Islam critiqued finds higher reason for suffering; is there any good in hardship?

In answer to the video "No Compulsion in Religion"

Allah tries all human beings indiscriminately through hardships or ampleness 7:168,13:26,30:37 with the aim of causing spiritual growth, purging the hearts. The preceding verses 4:75-77 speak of the divinely ordained law of fighting in the way of Allah those who unjustly oppress the weak for their religion, and some among the Muslims shallow in their faith held back. The Quran is full of admonition to such "believers" shallow in their faith who were averse at the thought of leaving their homes to defend the oppressed 8:5-6.

Those hypocritical "believers" viewed such hardship as "evil", a calamity brought upon them by the prophet "This is from you", while it is in fact a beneficial test coming from God with the aim of increasing them spiritually. In this sense, all things coming from Allah -hardships or ampleness- are good regardless of our perception "All is from Allah".

Hardship is, then, the preliminary to higher, more advanced states of being; it prepares man for the recompense that awaits him, and from his response to it, his true nature is brought to light.

This turns the concept of 'Evil' into a relative one that is subject to interpretation. In other words, what is considered 'evil' to us might actually be 'good' when it brings about a greater benefit such as successful completion of the test, ie spiritual betterment.

But perfection is not what God expects from us, He wants us to turn to Him in all situations whether in happiness or despair

12:87"despair not of Allah's mercy; surely none despairs of Allah's mercy except the unbelieving people".

God repeatedly encourages the transgressor to turn to Him, because it is expected from a faillible creature to fall short sometimes 3:133-5.

The fact that all things befalling man, whether what we interpret as "good" or what we perceive as "bad" are ultimately in Allah's Hands is reflected in 27:47 when the nation of prophet Salih accused him of being the cause of their confusion (ta'ir)

"He said: The cause of your confusion (ta'irukum) is with Allah".

His answer, as that of Moses quoted elsewhere 7:131 reflects a higher reality; all things are encompassed by Allah's knowledge and power because He is the One who created and controls the laws of causality. All consequences of man's actions are encompassed by Allah's knowledge 11:6,57:22 and power 11:56 because He could, against man's will, prevent sin 18:65-82,6:107,112,137 or force belief 2:253,5:48,6:35,149,11:118. But this does not in anyway dismiss the person's responsibility in the process of causality.

The deeds of a person are so closely tied to him that

17:13"We have made every man's ta'ir/actions (good or bad) to cling to his neck".

A person's deeds are as inseparable from him as the neck from the rest of the body. Whereas other body parts can be cut off without causing death, it isnt the case with the neck. The higher reality implied in the statement that the current state of affairs of a person is with Allah, thus means that the chain reaction that leads to a person's good or bad position, originates from Allah since He established that system, and is allowed by Him, since He encompasses it with His knowledge and power at each instant.

This means the consequences of any action, even those that result in sin or confusion as in the prophet Salih's reply, can be said to be indirectly caused by Allah

27:46-47"why do you seek to hasten on the evil before the good? Why do you not ask forgiveness of Allah so that you may be dealt with mercifully. They said: We are confused because of you..He said: The cause of your confusion is with Allah; nay, you are a people who are tried".

He created the system of causality, of which man's freewill is fully part of, and He could stop it at anytime. That is why there is no entity other than Allah to turn to to ward of a chain of causality leading to hardships or evil, just as there is none to beseech for benefits

6:17"And if Allah touch you with affliction, there is none to take it off but He. And if He visits you with good, then He has power over all things".

The universal system of causality is entirely in Allah's hands, encompassing and sustaining it at each instant. He could allow or prevent its intricate effects in such a way that could result in harm or benefit upon a person. All is done with perfect wisdom. When speaking of the hardship of the Israelites' life of slavery, the Quran attributes the direct action to the tyrants, all the while keeping the divine will in the background 

14:6"Remember the favor of Allah upon you when He saved you from the people of Pharaoh, who were afflicting you with the worst torment and were slaughtering your [newborn] sons and keeping your females alive. And in that was a great trial from your Lord".

The tyrants' evil deeds were permitted to unfold, but with ultimate wisdom, becoming a trial to those suffering the consequences.

As to sinning,  God does not want man to sin, hence the Quran's continuous pointing the means by which to do good. From turning the attention to man's ingrained cognition of higher moral truths, to the sending of revelation through special agents. Or the external guidance in the shape of recognizable signs of nature within and outside of man, signs so obvious that the Quran uses the powerful image of God's face being anywhere man looks.

 All these guiding mechanisms cannot be imposed upon anyone, in a system of causality that integrates freewill. One has to make use of these tools freely and choose whether he wants to live within the prescribed limits or not. God therefore does not want man to transgress but when he does, God allows it according to His established system of freewill in which man is fully responsible. There is therefore a fine line between moral accountability and God's supreme will and control over the current of causality 

7:28"And whenever they commit an indecency, they say: 'We found our fathers (acting) upon it, and Allah has enjoined it on us.' Say: 'Verily Allah does not enjoin indecencies. Do you say against Allah what you do not know?".

Disbelievers of all times objected that if Allah truly disapproved of their behaviour, then he would have prevented them from it

43:20,16:35-6"And they who give associates (to Allah) say: If Allah had pleased, we would not have served anything besides Allah, (neither) we nor our fathers, nor would we have prohibited anything without (order from) Him".

This corrupt and self-serving view of causality doesnt take into account that Allah has established that eachone is to choose his own path freely. The evidence for this is that we are capable of perceiving the spiritual significances of all things, as well as the fact we were sent guidance and messengers that clarify the right from the wrong

"Thus did those before them; is then aught incumbent upon the messengers except a plain delivery (of the message)? And certainly We raised in every nation a messenger saying: Serve Allah and shun the Shaitan. So there were some of them whom Allah guided and there were others against whom error was due".

This refutes determinism or predetermination, while also dismissing the excuse that one's spiritual condition, despite being exposed to guidance, could be due to worldly circumstances beyond one's control 4:97,16:41-2,23:106-8. Those who regard themselves as compelled to conform to society are strongly condemned and their excuse for failing to assume responsibility is rejected, except for the weak who have no possibility but to remain in an environment that conflicts with the right path 4:98-99.  But those who have the ability to bring about a change in their atmosphere or shift to a better environment, but that do not, then they will be fully accountable for having continued to live in the same corrupt environment. One should not hesitate to migrate in Allah's way to seek conditions more spiritually appropriate. Neither should one be disheartened by the material uncertainties of a new life to the point one remains in a place that could jeopardize one's situation in the Hereafter 29:56-60. In the NT, Jesus warned his followers to the same effect Matt6:24-34.

The word hijra in such verses denotes the spiritual as well as physical migration from an environement of sin to that of righteousness. This is a continuous demand from Allah that one should strive to purify his atmosphere from evil. Hijra is, according to God's messenger, the abandonning of what is condemned and disliked by Allah. Whenever the worship of God becomes impossible, the believer is obliged to forsake the domain of evil and to "migrate unto God", to a place where it is possible to live in accordance with one's faith

29:56,39:10"and Allah's earth is spacious".

The prophets themselves from Abraham to Moses, down to Muhammad, did not hesitate to leave their homeland and their loved ones behind, when the pressure to be expelled reached life-threatening proportions. Other prophets, the likes of Shuayb were rescued by God through the sending of calamities upon the rejecters before their forced exile came to fruition 7:88-91.