Run 20 tests on pure noise and what happens? You have about a 64 percent chance of finding at least one “significant” result. That single number shows why hypothesis testing needs clear logic, not just a formula. This guide walks through that logic with a courtroom, a coffee machine, and a website test, all with real numbers.
- Hypothesis testing assumes no effect (H0) and rejects it only when the data make that assumption hard to believe.
- A Type I error is a false alarm; a Type II error is a missed real effect.
- Lowering one error rate raises the other unless you collect a larger sample.
- Power is the chance of catching a real effect; 80 percent is a common planning target.
- Statistically significant does not mean large or important, especially with huge samples.
What Is Hypothesis Testing?
Hypothesis testing is a method for deciding whether sample data give strong enough evidence against a default claim, called the null hypothesis. It works much like a criminal trial.
In court, the defendant is presumed innocent. The null hypothesis (H0) plays that role: no effect, no difference, nothing unusual. The prosecution must bring evidence strong enough to convict “beyond reasonable doubt.” In statistics, that bar is the significance level, alpha, often set at 0.05.
The verdict wording matters. A jury says “not guilty,” never “innocent.” Weak evidence does not prove innocence; it only fails to prove guilt. A test works the same way. We either “reject H0” or “fail to reject H0,” and we never “accept H0.” A small sample can miss a real effect, just as a thin case can free a guilty person.
How Do You Write the Null and Alternative Hypotheses?
Turn your question into two claims about a population number. H0 states “no change” with an equals sign, and H1 states the effect you want to show.
Start with the plain question, then name the parameter. Three examples show the pattern:
- “Does this machine fill 250 ml cups correctly?” becomes H0: mean = 250 ml and H1: mean is not 250 ml.
- “Does the new checkout page sell more?” becomes H0: p(new) = p(old) and H1: p(new) > p(old).
- “Is the defect rate above 2 percent?” becomes H0: p = 0.02 and H1: p > 0.02.
Write both hypotheses before you see the data. Choosing the direction after peeking at results inflates false alarms. The claim you hope to support always goes in H1, because a test can only build evidence against H0.
How Does a Hypothesis Test Work From Start to Finish?
A test states the hypotheses, sets alpha, computes a test statistic from the sample, converts it to a p-value, and compares that p-value with alpha.
Worked Example: The 250 ml Coffee Machine
A cafe owner measures 40 cups. The sample mean is 247.8 ml and the standard deviation is 6.1 ml. Set H0: mean = 250 and H1: mean is not 250, with alpha = 0.05.
The standard error is 6.1 / sqrt(40) = 0.964 ml. The t statistic is (247.8 – 250) / 0.964 = -2.28, with 39 degrees of freedom. That gives a two-sided p-value of 0.028. Since 0.028 is below 0.05, we reject H0: the machine underfills. What the 0.028 actually means is covered in what a p-value really tells you.
The same data give a 95 percent interval of 245.8 to 249.8 ml, which excludes 250. That match is no coincidence, as confidence intervals explained shows. To run this test on your own numbers, use the hypothesis test calculator for z, t, and proportion tests.
What Are Type I and Type II Errors?
A Type I error rejects a true null hypothesis, a false alarm; a Type II error fails to reject a false one, a missed effect. Their rates are alpha and beta.
The two errors pull against each other. Demand stronger evidence and you convict fewer innocents but free more guilty people. Which error hurts more depends on the setting:
| Setting | Type I error means | Type II error means | Usual priority |
|---|---|---|---|
| Medical screening (H0: no disease) | A healthy person gets a follow-up test | A real disease goes undetected | Avoid Type II; accept more false alarms |
| Spam filter (H0: email is legitimate) | A real email lands in spam | A spam message reaches the inbox | Avoid Type I; accept some missed spam |
How Do Power and Sample Size Affect a Test?
Power is the probability that a test detects a real effect, equal to 1 – beta. Larger samples, larger effects, and a higher alpha all raise it.
The figure shows a one-sided z test where the true effect sits 2.5 standard errors from H0. The cutoff at 1.645 leaves alpha at 0.05 and power at 0.80.
Back to the coffee machine. Suppose the true mean is 248 ml, a 2 ml shortfall. With 40 cups and a spread of 6.1 ml, a simulation of 40,000 samples gives power of only about 53 percent. That is close to a coin flip. Measuring about 75 cups lifts power to 80 percent, and 100 cups reaches about 90 percent. Plan the sample size before collecting data, not after a disappointing result.
Why Can a Significant Result Still Mislead You?
Significance says an effect is unlikely to be zero, not that it is large. It also breaks down when you run many tests at once.
A Tiny Effect With a Huge Sample
Consider an A/B test of two signup pages. Page A converts 200 of 4,000 visitors (5.0 percent) and page B converts 250 of 4,000 (6.25 percent). A two-proportion z test gives z = 2.43 and p = 0.015, a real and useful 1.25-point lift.
Now scale up to 1,000,000 visitors per page, with rates of 5.00 and 5.10 percent. The test gives z = 3.23 and p = 0.0012, an even smaller p-value. Yet the lift is just 0.1 point, which rarely pays for a redesign. Always report the effect size next to the p-value.
The Multiple-Comparisons Trap
Each test at alpha 0.05 carries a 5 percent false-alarm risk. Across 20 independent tests with no real effects, the chance of at least one false positive is 1 – 0.95^20 = 0.64. The expected number of false positives is exactly one. A simple fix, the Bonferroni correction, divides alpha by the number of tests, giving 0.05 / 20 = 0.0025 per test.
The Hypothesis Test Calculator runs one-sample z, t, and one-proportion tests and returns the test statistic, p-value, and decision.
FAQs About Hypothesis Testing
What Is Hypothesis Testing in Simple Terms?
It is a way to check whether data give strong evidence against a default claim of no effect. You assume the null hypothesis is true, measure how surprising your sample is, and reject the null only when the evidence is strong.
What Is the Difference Between the Null and Alternative Hypothesis?
The null hypothesis states no effect or no difference and always contains an equals sign. The alternative hypothesis states the effect you want evidence for, such as a mean that is higher, lower, or simply different.
Does Failing to Reject the Null Hypothesis Prove It Is True?
No. Like a “not guilty” verdict, it means the evidence was not strong enough. A small sample or low power can hide a real effect. The correct wording is “fail to reject,” never “accept.”
Which Is Worse, a Type I or a Type II Error?
It depends on the cost of each mistake. In medical screening, missing a disease (Type II) is usually worse. In spam filtering, sending a real email to spam (Type I) is usually worse.
How Large a Sample Do I Need for a Hypothesis Test?
Enough to give about 80 percent power for the smallest effect you care about. In the coffee example, detecting a 2 ml shortfall with a 6.1 ml spread takes about 75 cups. With 40 cups, power is only about 53 percent.
Why Is 0.05 the Usual Significance Level?
It is a long-standing convention that balances false alarms against missed effects, not a law of nature. Fields with costly false alarms often use 0.01 or stricter, while early exploratory work sometimes uses 0.10.
Can a Result Be Statistically Significant but Not Important?
Yes. With 1,000,000 users per group, a lift from 5.00 to 5.10 percent gives p = 0.0012, yet the 0.1-point gain is tiny. Always judge the effect size alongside the p-value.
Sources
References Used in This Article
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.1.3 What Are Statistical Tests?
- OpenStax, Introductory Statistics 2e, Chapter 9: Hypothesis Testing With One Sample
- Penn State STAT 200, Lesson 6: Errors, Power, Multiple Testing, and Practical Significance
- Wasserstein and Lazar, The ASA Statement on p-Values, The American Statistician (2016)
This article explains the logic of hypothesis testing for general education. Worked numbers were computed and checked independently. Reviewed for accuracy by Prof. Dr. Khalil Mudassar, PhD. Last updated September 27, 2026.
Author
Shakeel Muzaffar is the Founder and Editor-in-Chief of MultiCalculators.com, bringing over 15 years of experience in digital publishing, product strategy, and online tool development. He leads the platform's editorial vision, ensuring every calculator meets strict standards for accuracy, usability, and real-world value. Shakeel personally oversees content quality, formula verification workflows, and the platform's commitment to publishing tools that are genuinely useful for students, professionals, and everyday users worldwide.
