Is our experiment generating the output we are hoping for? Due to something we are not capable enough to differentiate between our measurement error.

Hypothesis Testing is a powerful statistical method to solve such experiments. We will consider a coin-tossing event with unknown parameter p.

The first step is to generate the separate hypothesis, Ho is called Null-Hypothesis and Ha is called Alternate Hypothesis.

Now, the question that arises is which hypothesis is more consistent with data? To choose between these, we need a statistical test, we will apply a function J, of the sample set Xn = {Xi }n into the real line, where Xi is the heads or tails outcome (Xi ∈ {0, 1}).

If you got the concept, in other words, we are calculating J(Xn) and check if it exceeds a threshold θ. If not, then we accept Ho (Null Hypothesis) otherwise accept Ha (Alternate Hypothesis).

Notationally, this is the following:

We have the observed data Xn and a function J that maps that data onto the real line. Then, using the constant θ as a threshold, the inequality effectively divides the real line into two parts, one corresponding to each of the hypotheses.

Whatever this test J is, it will make mistakes of two types — false negatives and false positives. The false positives come from the case where we accept Ho (Null Hypothesis) when the test says we should accept Ha (Alternate Hypothesis).

For this example, here are the false positives: