What Does a P-Value Actually Say? 6 Misconceptions Still Taught in Many Classrooms

If there is one number that is most often worshipped and most often misunderstood in research, it is the p-value. Students memorize it: “if p is less than 0.05, the hypothesis is accepted; if more, it is rejected.” Lecturers ask about it at the defense. Journals use it as a yardstick. And in the middle of all that, almost nobody can truly explain what the number means.

I am not blaming anyone. The definition really is not intuitive. But understanding it correctly can save you from wrong conclusions, in a thesis and in the workplace. Let us start with the definition, then take apart the six most common misconceptions.

The correct definition, in human language

The p-value is the probability of getting a result as extreme as the one we observed (or more extreme), assuming the null hypothesis is true. The null hypothesis usually says “there is no effect” or “there is no difference”.

Imagine you suspect a coin is unfair. You flip it ten times and get nine heads. The question: if this coin were actually fair, how likely would we be to get a result this extreme? The probability of getting nine or ten heads out of ten flips of a fair coin is about 1.1%. That is the p-value (for a one-sided test): about 0.011. A small number means your result is surprising if the coin is fair, so you begin to doubt the assumption that the coin is fair.

Notice what it does not say: the p-value does not tell you the probability that the coin is fair. It only tells you how odd the data is if the coin is fair. Two very different things, and that is the root of the misconceptions that follow.

Misconception 1: “p = 0.03 means the null hypothesis has a 3% chance of being true”

Wrong. The p-value is the probability of the data (or more extreme) assuming the null hypothesis is true, not the probability that the null hypothesis is true given the data. Flipping that sentence is like equating “the probability that someone carries an umbrella if it is raining” with “the probability that it is raining if someone carries an umbrella”. The two are clearly different. To get the probability of the hypothesis itself, we need a different approach (for example Bayesian) and prior information about how plausible the hypothesis was before seeing the data.

Misconception 2: “p < 0.05 means the effect is important”

No. The p-value does not measure the size of an effect. With a very large sample, a trivial difference (for example a satisfaction score difference of 0.02 on a scale of 1 to 5) can produce a very small p. Conversely, with a small sample, a practically important effect can produce a large p because the data is not strong enough to establish it.

To judge the importance of an effect, report the effect size (for example a difference in means, a regression coefficient, Cohen’s d, or R²) along with a confidence interval. A confidence interval is far more informative: it shows the range of plausible effect values, not just yes or no.

Misconception 3: “p > 0.05 means there is no effect”

Also wrong. A non-significant result only means the data is not strong enough to reject the null hypothesis. Perhaps the effect truly does not exist. Perhaps the effect exists, but the sample is too small to detect it (low power). “No evidence of an effect” is not the same as “evidence of no effect”. In a report, write “insufficient evidence was found” and show the confidence interval to make clear whether a meaningful effect is still possible.

Misconception 4: “0.05 is a sacred number”

The 0.05 threshold is a convention, not a law of nature. It became popular in the early 20th century because it was easy to use with statistical tables, not because there is any scientific reason that 0.049 is fundamentally different from 0.051. Treating p = 0.049 as a “finding” and p = 0.051 as “not a finding” is a fragile way of thinking. In its 2016 statement, the American Statistical Association warned that scientific conclusions and policy decisions should not be based only on whether a p-value passes a specific threshold.

Misconception 5: “If I try many analyses and one is significant, that is my finding”

This is a subtle trap, and often unnoticed by those who fall into it. If you test 20 relationships that do not actually exist, with a 0.05 threshold, you should expect about one “significant” result by chance. The probability that at least one of 20 independent tests produces p < 0.05 when there is no effect at all is about 64%. If you report only the significant one, you are presenting coincidence as discovery.

The practice of trying various analyses, picking subgroups, or deleting data until a significant result appears is called p-hacking. How to guard against it: set your hypotheses and analysis plan before looking at the data, report all the analyses you ran (not just the ones that worked), and use corrections for multiple testing when testing many hypotheses at once.

Misconception 6: “A very small p-value means the result is certainly true”

A small p-value does not guarantee that the study design is good, the measurement valid, the sample representative, or that there is no bias. A perfect analysis of flawed data still produces flawed conclusions. The p-value answers only one narrow question about random variation, and does not touch questions about data quality or research logic.

So how should you use the p-value?

The view I share with students and researchers:

1. Understand what is being tested. Write the null hypothesis in plain words.
2. Report the actual p-value (for example p = 0.031), not just “p < 0.05”.
3. Always include the effect size and confidence interval.
4. Consider power. Is your sample sufficient to detect a meaningful effect?
5. Tell the whole story of your analysis. Do not hide analyses that did not work.
6. Combine it with theory and common sense. A significant result that contradicts all existing theory deserves suspicion and replication.

The p-value is not the enemy. It is a useful tool, as long as it is used to answer the right question. What is dangerous is turning it into a traffic light that decides “true or false” without thinking.

A numerical example: the same effect, different conclusions

Imagine a training program raises the average competence score by 2 points on a scale with a standard deviation of 10 (a small effect, Cohen’s d = 0.2). If we test it on 30 people, that effect will almost certainly not be detected (large p), and we might conclude ‘the training has no effect’. If we test it on 3,000 people, the same effect will produce a very small p, and we conclude ‘the training has a significant effect’. The effect is exactly the same: two points. Only the sample size changed. The question that really matters is: is two points meaningful in practice, and is the cost of the training proportionate to the benefit? The p-value does not answer that.

Alternatives and complements

Several increasingly recommended approaches: confidence intervals to show the range of plausible effects, effect sizes to state magnitude, power analysis before a study to make sure the sample is sufficient, pre-registration of analysis plans to prevent choosing analyses after seeing results, and replication to check whether findings hold up. Bayesian approaches also offer a way to state how strongly the evidence supports a hypothesis, though they require additional understanding. The point is not to throw away the p-value, but to place it as one piece of evidence among several.

---