What Is a P-Value?

Would you call a coin unfair after it lands heads 9 times in 10 flips? A fair coin does that, or something more lopsided, only about 2 times in 100 tries. That number, 0.0215, is a p-value. It is the most reported number in science and one of the most misread. This guide explains what a p-value really says, what it never says, and how to read one with confidence.

What Is a P-Value in Plain Terms?

A p-value is the probability of seeing data at least as extreme as yours when the null hypothesis is true. It measures surprise. The null hypothesis is the boring default, such as “this coin is fair” or “this drug does nothing.”

Take the coin. Flip it 10 times and get 9 heads. A fair coin gives 9 or 10 heads in only 11 of the 1,024 equally likely sequences. That makes the one-sided p-value 11/1024, or 0.0107.

Most tests are two-sided, because 9 tails would look just as suspicious. Adding the matching tail gives 22/1024, or 0.0215. In plain words, a fair coin produces a result this lopsided about 2.15 percent of the time.

Notice what the calculation assumed. Every step pretended the coin was fair. The p-value never asks whether the coin is fair. It asks how odd your flips would look if it were.

What Does P = 0.03 Actually Mean?

A p-value of 0.03 means that, with no real effect, data this extreme or more would appear in about 3 percent of repeated studies. That is the whole statement. Everything beyond it needs extra information.

What P = 0.03 Does Mean

Your data sit in a rare corner of what the null model predicts. Either a rare event happened, or the null model is wrong in some way. The data are fairly incompatible with “no effect.”

What P = 0.03 Does Not Mean

It does not mean a 3 percent chance that the null hypothesis is true. It does not mean a 97 percent chance your finding is real. It says nothing about how large or useful the effect is.

Where does a p-value fit in a full study? It is one step in a larger procedure with hypotheses, a significance level, and error types. Our guide to how hypothesis testing works step by step covers that procedure. To turn a z or t statistic into a p-value, use the P-Value Calculator for z and t statistics.

What Are the Most Common P-Value Misconceptions?

The four common myths read p as the odds H0 is true, a certainty score, proof of nothing, or a size gauge. All four are wrong. Each one flips a conditional probability or ignores sample size.

Myth 1: p is the probability that H0 is true. The p-value assumes H0 is true from the start. It cannot then report the chance of its own assumption.

Myth 2: 1 – p is the probability the result is real. A p of 0.01 does not give 99 percent confidence in the effect. That answer also depends on prior plausibility and study design.

Myth 3: p above 0.05 proves no effect. A study with 20 people can miss a real effect entirely. A large p often means “not enough data yet.”

Myth 4: a smaller p means a bigger effect. A tiny effect in a huge sample yields a tiny p. The table further down shows this with real numbers.

What Did the ASA Say About P-Values?

In 2016 the American Statistical Association issued six principles on p-values. It was the first formal statement of its kind from the society’s board. Here they are in plain words.

• Principle 1: A p-value shows how incompatible the data are with a stated statistical model.
• Principle 2: It does not give the probability that a hypothesis is true, or that chance alone produced the data.
• Principle 3: Scientific, business, and policy decisions should not rest only on crossing a threshold.
• Principle 4: Sound inference needs full reporting, including every analysis that was run.
• Principle 5: A p-value does not measure the size of an effect or its importance.
• Principle 6: On its own, a p-value is not a good measure of evidence for a model or hypothesis.

Principles 2 and 5 map directly onto the four myths above. Principle 4 targets a quieter problem, which the next section covers.

Why Is 0.05 the Standard Cutoff?

The 0.05 cutoff is a convention, not a mathematical law. Ronald Fisher suggested 1 in 20 as a handy benchmark in his 1925 textbook. Later researchers turned his rule of thumb into a hard line.

A hard line causes trouble. A p of 0.049 and a p of 0.051 carry nearly identical evidence, yet one gets published and the other gets filed away. In 2019, more than 800 scientists signed a Nature comment calling for an end to the “significant or not” split.

P-Hacking and Multiple Comparisons

P-hacking means trying many analyses and reporting only the one that crosses 0.05. Common moves include dropping outliers, adding subgroups, or stopping data collection once p dips low enough.

Multiple comparisons create the same trap by accident. Run 20 independent tests where nothing is going on, each at 0.05. The chance of at least one “significant” result is 1 – 0.95^20, or about 64 percent. That is why ASA Principle 4 demands full reporting.

Why Does Sample Size Change the P-Value?

The p-value depends on both the effect size and the sample size, so the same effect gives smaller p-values as n grows. Hold the effect fixed at 0.2 standard deviations and watch what happens in a one-sample t-test.

The effect never changed, yet the verdict swung from “not significant” to “highly significant.” Quadrupling the sample doubles the t statistic, because t equals the effect times the square root of n.

This is why effect sizes and intervals belong beside every p-value. The interval shows the plausible range, and our guide to how confidence intervals work explains how to read one. The n = 25 interval spans zero yet still allows an effect as large as 0.61.

FAQs About P-Values

Sources

This article explains what a p-value means and how it is misread. It is general statistics education, not advice for any specific study. Reviewed for accuracy by Prof. Dr. Khalil Mudassar, PhD. Last updated September 27, 2026.