SCIENCEYE BLOG

Contact

P-value

20 min read

Updated on November 15, 2026
Painterly bell shaped hill of blue mist on a dark background its outer tails glowing orange beside a single white lantern illustrating the p value as the tail area beyond an observed result

SHARE:

Introduction

Definition

A p-value is a probability calculated under a specified statistical model. It is the chance that a summary of the data would be equal to or more extreme than the value actually observed.1 The model usually includes a null hypothesis, most often the assumption of no effect or no difference. A small p-value signals that the data sit poorly with that model; it does not give the probability that the null hypothesis is true.12

Context

The p-value grew out of the significance tests of the statistician Ronald Fisher. Jerzy Neyman and Egon Pearson later recast testing as a decision rule with a fixed long-run error rate, and modern practice blends the two.3 Today the p-value dominates biomedical reporting. Among papers that report p-values, 96% report at least one at or below 0.05. In a sample of 796 abstracts with data, only 2.3% gave confidence intervals.4 It is also widely misread: 99% of 362 psychology students and researchers misinterpreted at least one p-value statement.5

Scope

This entry defines the p-value and its parts and shows how it is calculated. It then reviews its use and misuse, good reporting, and reform debates. For an accessible introduction, see the blog post P-Value Myths: Why p < 0.05 Is Not 95% Sure.

The p-value as the tail area under the null distribution Animation. A bell-shaped curve shows the test statistic expected if there is no effect. Shaded tails beyond the observed value grow smaller as the observed value moves out: z of 1.0 gives p of 0.32, 1.64 gives 0.10, 1.96 gives 0.05, and 2.58 gives 0.01. The loop restarts every 8 seconds. If there were no effect ObservedShaded tails: p = 0.32 ObservedShaded tails: p = 0.10 ObservedShaded tails: p = 0.05 ObservedShaded tails: p = 0.01 -3 -2 -1 0 1 2 3 Test statistic (z)
The p-value is the shaded tail area beyond the observed result, assuming no effect. Values computed from the standard normal distribution (two-sided).

Detailed Explanation

Components

Every p-value rests on the same small set of parts. First comes the null hypothesis, a precise statement such as “the new drug and the old drug lower blood pressure by the same amount.” Next, the analyst chooses a test statistic. This single number summarizes how far the data depart from that hypothesis, such as a difference in means divided by its standard error. Under the null hypothesis and the other model assumptions, the test statistic has a known sampling distribution. This is the spread of values it would take across many repeats of the study.

The observed test statistic is then located in that distribution. The p-value is the area of the distribution at or beyond the observed value, in one tail or both. Many studies also fix a significance level, called alpha, before collecting data, conventionally 0.05; a p-value below alpha is then labelled “statistically significant.” However, the p-value is computed under the whole statistical model, so it reflects every assumption in that model, not only the null hypothesis.1 If the data were analyzed in several ways and only the smallest p-value was reported, the calculated number no longer means what it claims.2

Alpha also sets the type I error rate, the long-run share of true null hypotheses that a test would wrongly reject.3 Its partner, the type II error rate, is the share of real effects a study would miss. One minus that rate is the study’s power.2 A p-value says nothing about power on its own. So a non-significant result from a small study is weak evidence either way.

A smoke alarm offers an analogy. Alpha is how often it sounds with no fire, and power is how often it sounds when there is one. The p-value is how loudly it rings this time.

ComponentDescriptionRole in the p-value
Null hypothesisA precise claim, usually of no effect or no differenceDefines the world in which the p-value is calculated
Statistical modelAll assumptions, including random sampling and the data’s distributionMust hold for the p-value to mean what it says
Test statisticOne number summarizing the departure from the null (for example z or t)Converts the data into a single comparison
Sampling distributionValues the test statistic would take over many repeats if the null were trueSupplies the reference against which the result is judged
Tail areaProbability at or beyond the observed statistic, one-sided or two-sidedIs the p-value itself
Significance level (alpha)A cut-off fixed in advance, conventionally 0.05Turns the p-value into a yes or no decision
Components of a p-value.12

Tests can be one-sided or two-sided. A two-sided test counts results that are extreme in either direction. For symmetric distributions, it doubles the single-tail area. A one-sided test counts only one direction.

The choice of sides can change the verdict, however. For the coin example under Statistical Basis, the one-sided p-value is 0.028, below 0.05, while the two-sided value is 0.057, above it. Picking the direction after seeing the data halves the p-value without adding any evidence.

Variations

Two schools shaped how p-values are used. In Fisher’s approach, the p-value is a continuous measure of evidence against the null hypothesis, read alongside everything else known about the question. In the Neyman-Pearson approach, the analyst sets alpha in advance and simply accepts or rejects the hypothesis. This controls the rate of false rejections over many uses of the procedure.3

Most textbooks now teach a hybrid called null-hypothesis significance testing. Steven Goodman calls the resulting belief the “P value fallacy.” It is the idea that one number can serve both as a long-run error rate and as the evidence from a single study.6

P-values also differ in how they are computed. Exact tests use the precise distribution of the test statistic, as the coin example does with the binomial distribution. Many common tests rely on approximations, such as the normal distribution, which work well for large samples but less well for small ones. Permutation tests build the reference distribution by reshuffling the observed data many times.

The distinction matters in practice. Under the decision view, p = 0.049 and p = 0.051 lead to opposite verdicts, although the two results carry almost the same evidence. Likewise, p = 0.001 and p = 0.04 are both simply “significant.” Under the evidential view, they mean nearly the same thing, and neither settles the question without effect sizes, prior knowledge, and study quality.6

Running many tests also creates a related problem. If 20 independent tests are run when every null hypothesis is true, the chance that at least one gives p < 0.05 is 64%. Adjusted p-values correct for this. The simplest, the Bonferroni method, divides alpha by the number of tests. Unreported multiple testing is one form of the analytic flexibility described under Evidence below.

What a p-value is not

Statisticians have catalogued the common misreadings in detail. Goodman lists 12 misconceptions, and a 2016 guide by Sander Greenland and colleagues lists 25 misinterpretations of tests, p-values, confidence intervals, and power.72 One common misreading treats p as the probability that the null hypothesis is true, or that the data arose by chance alone. The American Statistical Association states that a p-value measures neither.1

Another common error treats a non-significant result as proof of no effect. In fact, absence of evidence is not evidence of absence.8 An analysis of 791 articles found that about half made exactly this mistake.9

A third error equates p = 0.05 with a 5% chance that a “discovery” is false. David Colquhoun applied the logic of screening tests to this question. He showed that treating p = 0.05 as a discovery leads to being wrong at least 30% of the time under plausible assumptions.10 To keep that false discovery rate below 5%, he suggested a threshold near p = 0.001, a “three-sigma” rule.10

A fourth error reads 1 minus p as the chance that a result will replicate. Replication depends on the true effect and the study’s power, not on the original p-value alone. The simulation under Statistical Basis shows how widely p-values vary across identical studies. A fifth error assumes that a small p-value means a large or important effect; the sample-size example under Statistical Basis shows why it does not.

A p-value measures how well the data fit a model of no effect. It is not the probability that the null hypothesis is true, and it is not the chance that a finding is false.

Statistical Basis

How a p-value is calculated

A simple example shows the arithmetic. Suppose a coin lands heads 60 times in 100 tosses, and the null hypothesis says the coin is fair. Under that hypothesis, the number of heads follows a binomial distribution with a mean of 50. The exact probability of 60 or more heads is 0.0284. A two-sided test also counts 40 or fewer heads, giving a p-value of 0.057. That value sits just above the threshold, which is why the verdict flips with a one-sided test.

By the 0.05 convention, the result is “not significant,” yet 60 heads is still unusual for a fair coin.

Most real analyses follow the same steps with a different distribution. For a difference between two group means, the test statistic is the difference divided by its standard error. The p-value is the area beyond that value under a normal or t distribution. The figure above shows how the shaded area shrinks as the observed statistic moves away from zero. A z value of 1.96 leaves 5% in the two tails, while 2.58 leaves 1%.

Because the standard error shrinks as samples grow, the p-value depends heavily on sample size. Consider a difference of only 0.02 standard deviations between two groups of 40,000 people. It gives z = 2.83 and p = 0.005, a highly “significant” result for an effect too small to matter in practice. Conversely, a small study can miss a large effect. The p-value therefore cannot stand in for the size or importance of an effect.

How much p-values vary

P-values are also noisy. Because each study draws a fresh sample, identical studies give different p-values even when the true effect stays the same. One simulation ran 20 identical studies, each comparing two groups of 32 people with a true effect of half a standard deviation. The p-values ranged from 0.0001 to 0.68, and 13 of the 20 fell below 0.05. Each study had about a 52% chance of reaching significance. A single result, significant or not, therefore says little about whether the next study will agree.

Under a true null hypothesis the picture changes. Every p-value between 0 and 1 is then equally likely. So 5% of studies of an effect that does not exist will still reach p < 0.05.

Simple arithmetic shows what this means for a field. Suppose 10% of tested hypotheses are true and studies have 80% power. Of 1,000 tests, the 100 true effects yield 80 significant results, while the 900 null effects yield 45 false positives. So 36% of the significant findings are false, far more than the 5% that alpha might suggest.

Evidence

Large studies of the research literature show how p-values behave in practice. The Open Science Collaboration repeated 100 psychology studies. Of the originals, 97% had reported significant results, but only 36% of the replications did. Replication effects were also about half the original size.11 Only 47% of original effect sizes fell within the replication’s 95% confidence interval, and combining original and replication data left 68% significant.11

Text-mining of open-access papers found more p-values just below 0.05 than expected, in both results sections and abstracts. This pattern is a signature of p-hacking, the practice of trying analyses until p falls below 0.05. A ratio that should sit at 0.5 without p-hacking reached 0.546 in results sections across 14 disciplines and 0.537 in abstracts. Its effect on overall conclusions appeared weak, however.12

Joseph Simmons and colleagues showed how this happens. Undisclosed flexibility in data collection and analysis can push the false-positive rate far above the nominal 5%.13 John Ioannidis made a broader argument. Whether a significant finding is true, he showed, depends on four things: statistical power, bias, the number of relationships tested, and the prior odds that the effect is real.14

Misunderstanding is not limited to students. In one study, 120 researchers did no better than 442 students at spotting false statements about a confidence interval. Both groups endorsed more than three of six false statements on average, and only 3% of the researchers rejected all six.15

StudyYearKey findingDOI
Open Science Collaboration201536% of 100 replications were significant, against 97% of the originals10.1126/science.aac4716
Head et al.2015More p-values just below 0.05 than expected across disciplines10.1371/journal.pbio.1002106
Chavalarias et al.201696% of papers with p-values reported one at or below 0.0510.1001/jama.2016.1952
Lyu et al.201899% of 362 participants misread at least one p-value item10.3389/fpsyg.2018.00868
Hoekstra et al.2014Researchers and students both endorsed false interval statements10.3758/s13423-013-0572-3
Simmons et al.2011Flexible analysis inflates false positives above 5%10.1177/0956797611417632
Major studies of p-value use and misuse.

Current understanding

In 2016 the American Statistical Association issued a formal statement setting out six principles for using and reading p-values. Its second principle states that p-values do not measure the probability that the studied hypothesis is true. Nor do they measure the probability that the data were produced by chance alone.1

Bayesian analysis gives a way to translate a p-value into evidence. Thomas Sellke, Maria Bayarri, and James Berger calibrated p-values into a lower bound on the Bayes factor, using the expression −e·p·ln(p). A Bayes factor is the ratio of how well two hypotheses predict the data.16 At p = 0.05, the bound is 0.407, roughly 1 to 2.5 odds against the null. With even prior odds, the null hypothesis still has at least a 28.9% probability of being true.16

In a 1999 analysis, Steven Goodman concluded that p-values greatly overstate the evidence against the null hypothesis. He proposed the minimum Bayes factor as an objective alternative way to express the evidence from a study.17

The same calibration shows how the evidence grows as p falls. With even prior odds, the null hypothesis keeps at least an 11% chance of being true at p = 0.01, 7% at p = 0.005, and 2% at p = 0.001. Because these are lower bounds, the true probability for a given study can be higher.16 Read this way, the figure explains why p = 0.05 is weak evidence on its own. Statisticians broadly agree on what a p-value is. The disagreement concerns how much weight it should carry and whether fixed thresholds help.118

Lowest probability that the null hypothesis is true, by p-value Bar chart. With even prior odds, the lowest posterior probability of the null is about 38% at p of 0.10, 29% at 0.05, 11% at 0.01, 7% at 0.005, and 2% at 0.001. Chance the null is still true (lowest value, even prior odds) 38% 0.1 29% 0.05 11% 0.01 7% 0.005 2% 0.001 p-value
Lowest posterior probability of the null hypothesis for common p-values, with even prior odds, from the calibration −e·p·ln(p). Values computed from the formula.16

Practical Applications

Relevance

P-values shape which treatments reach patients, which findings make headlines, and which studies get published. A clinician reading a trial needs to know whether a non-significant result means “no benefit” or “not enough data,” and the p-value alone cannot say. Jonathan Sterne and George Davey Smith reviewed the history of significance tests and the case against relying on them, written for clinicians.19

Small studies, however, make this problem common. A trial with 20 participants per group has only about a 35% chance of detecting a true effect of half a standard deviation at p < 0.05. With 64 per group, the chance rises to about 81%. In the smaller trial, most real effects of that size would come back “not significant.”

In health news, exaggeration can begin with the press release. A third of 462 university press releases exaggerated causal claims. When they did, 81% of the news stories based on them repeated the exaggeration.20

A recent Meta-Analysis, a study that pools the results of many studies, illustrates the point. It found that a single bout of stretching produced a non-significant reduction in arterial stiffness (p = 0.258). Yet the effect size of −0.22 had a 95% confidence interval from −0.62 to 0.17, which spans a moderate benefit and a slight harm.21 Read correctly, this result means the question is still open; it does not show that acute stretching does nothing. The SciWiki entry on meta-analysis explains how such pooled estimates are built.

Reporting

Good reporting gives readers more than a verdict. Reform advocates recommend treating p as a continuous measure, weighed with prior evidence, plausibility, and study design, rather than as a pass or fail.18 They also suggest reading confidence intervals as “compatibility intervals,” the range of effects most compatible with the data.9

Confidence intervals are misread as well: 93% of the 362 surveyed psychology students and researchers misread at least one confidence interval item.5 Biomedical practice still falls short. In a manual sample of 796 abstracts with data, 15.7% reported p-values, 2.3% reported confidence intervals, and none reported Bayes factors. Only 1.8% gave both an effect size and a confidence interval.4

A complete report lets readers answer three questions. How large is the effect? How precisely is it estimated? And how surprising would the data be if there were no effect? The effect size answers the first, the confidence interval the second, and the p-value only the third. Giving the exact value, such as p = 0.03 rather than p < 0.05, also lets readers weigh the evidence on a continuous scale.18

QuantityTypical formWhat it tells the reader
Exact p-valueA number from 0 to 1, for example p = 0.03How well the data fit the null model
Effect sizeMean difference, ratio, or standardized differenceHow large the effect is
95% confidence intervalA range around the effect sizeWhich effect sizes are compatible with the data
Bayes factorA ratio comparing two hypothesesHow strongly the data favor one hypothesis over another
What to report alongside a p-value.1817

Good practice

Several methods address the p-value’s blind spots. Equivalence testing turns “no significant difference” into a testable claim. The researcher first sets equivalence bounds at the smallest effect that would matter in practice. Two one-sided tests then check whether the effect lies above the lower bound and below the upper bound. If both succeed, the data reject any effect large enough to matter.22

Researchers can justify their choice of alpha for each study rather than defaulting to 0.05. The authors of that proposal argued that every design choice, including the threshold, should be reported and justified.23

Bayes factors offer a direct comparison of how well two hypotheses predict the data. A Bayes factor of 3, for example, means the data are three times more likely under one hypothesis than under the other.17 Bayesian analysis already shapes major trials. The Pfizer COVID-19 vaccine trial reported a Bayesian credible interval, although its report labelled it a confidence interval.24 Planning sample size also matters. A trial that wants an 80% chance of detecting a true effect of half a standard deviation needs about 64 participants per group.

Transparency also matters as much as method. Simmons and colleagues proposed six requirements for authors and four guidelines for reviewers so that this analytic flexibility is disclosed rather than hidden.13

Readers can apply four quick checks to any reported p-value. Is the exact value given? Is there an effect size with a confidence interval? Was the analysis planned in advance or chosen after seeing the data? And how many tests did the study run?

A p-value that passes all four checks deserves far more weight than one that fails them. Each check takes a minute, but together they change how much a result should count.

Report the exact p-value with the effect size and its confidence interval. When the question is whether an effect is absent, use equivalence tests or Bayes factors.

Controversies and Limitations

Debates

Statisticians agree that p-values are misused but disagree about the cure. In 2018, 72 researchers led by Daniel Benjamin proposed lowering the default threshold from 0.05 to 0.005.25 A group of 88 authors replied that researchers should justify their alpha instead. Valentin Amrhein and Greenland argued for removing significance thresholds rather than redefining them.2326

Blakeley McShane and colleagues proposed abandoning statistical significance as a default, without banning p-values.18 In 2019, more than 800 scientists signed a call by Amrhein, Greenland, and McShane to retire statistical significance.9 One journal, Basic and Applied Social Psychology, went further in 2015 and banned null-hypothesis significance testing, confidence intervals included.27 These proposals share a goal, fewer false claims, but disagree on whether a threshold of any kind helps. All except the outright ban keep some role for the p-value.

ProposalSourceWhat it changes
Lower the threshold to 0.005Benjamin et al., 2018Default cut-off moves from 0.05 to 0.005
Justify alpha per studyLakens et al., 2018Each study chooses and justifies its own alpha
Remove thresholdsAmrhein and Greenland, 2018No cut-off; p read as a continuous measure
Abandon significance as defaultMcShane et al., 2019p weighed with prior evidence, plausibility, and design
Ban significance testingTrafimow and Marks, 2015Tests and confidence intervals removed in one journal
Main proposals for reforming p-value use.

Limitations

Even so, some limits are built into the p-value. It depends on sample size, on every model assumption, and on how many analyses were run. It also cannot measure the size or importance of an effect.2 Nor can it show that an effect is absent; that is the job of equivalence tests or Bayes factors.

Open questions remain about which reforms improve the reliability of published research in practice. It is also unclear how best to teach correct interpretation, given how persistent the misreadings have proved. Useful next steps include testing which teaching methods reduce misreadings and tracking how journal policies change what authors report.

References

  1. Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129-33. doi:10.1080/00031305.2016.1154108
  2. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337-50. doi:10.1007/s10654-016-0149-3
  3. Goodman SN. p values, hypothesis tests, and likelihood: implications for epidemiology of a neglected historical debate. Am J Epidemiol. 1993;137(5):485-96. doi:10.1093/oxfordjournals.aje.a116700
  4. Chavalarias D, Wallach JD, Li AHT, Ioannidis JPA. Evolution of reporting P values in the biomedical literature, 1990-2015. JAMA. 2016;315(11):1141-8. doi:10.1001/jama.2016.1952
  5. Lyu Z, Peng K, Hu CP. P-value, confidence intervals, and statistical inference: a new dataset of misinterpretation. Front Psychol. 2018;9:868. doi:10.3389/fpsyg.2018.00868
  6. Goodman SN. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med. 1999;130(12):995-1004. doi:10.7326/0003-4819-130-12-199906150-00008
  7. Goodman S. A dirty dozen: twelve P-value misconceptions. Semin Hematol. 2008;45(3):135-40. doi:10.1053/j.seminhematol.2008.04.003
  8. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485. doi:10.1136/bmj.311.7003.485
  9. Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305-7. doi:10.1038/d41586-019-00857-9
  10. Colquhoun D. An investigation of the false discovery rate and the misinterpretation of p-values. R Soc Open Sci. 2014;1(3):140216. doi:10.1098/rsos.140216
  11. Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349(6251):aac4716. doi:10.1126/science.aac4716
  12. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and consequences of p-hacking in science. PLoS Biol. 2015;13(3):e1002106. doi:10.1371/journal.pbio.1002106
  13. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22(11):1359-66. doi:10.1177/0956797611417632
  14. Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124
  15. Hoekstra R, Morey RD, Rouder JN, Wagenmakers EJ. Robust misinterpretation of confidence intervals. Psychon Bull Rev. 2014;21(5):1157-64. doi:10.3758/s13423-013-0572-3
  16. Sellke T, Bayarri MJ, Berger JO. Calibration of p values for testing precise null hypotheses. Am Stat. 2001;55(1):62-71. doi:10.1198/000313001300339950
  17. Goodman SN. Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med. 1999;130(12):1005-13. doi:10.7326/0003-4819-130-12-199906150-00019
  18. McShane BB, Gal D, Gelman A, Robert C, Tackett JL. Abandon statistical significance. Am Stat. 2019;73(Suppl 1):235-45. doi:10.1080/00031305.2018.1527253
  19. Sterne JA, Davey Smith G. Sifting the evidence: what’s wrong with significance tests? BMJ. 2001;322(7280):226-31. doi:10.1136/bmj.322.7280.226
  20. Sumner P, Vivian-Griffiths S, Boivin J, Williams A, Venetis CA, Davies A, et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ. 2014;349:g7015. doi:10.1136/bmj.g7015
  21. Logan JG, Chauntry AJ, Zhao H, Park S, Stoner L. Musculoskeletal stretching and arterial stiffness: systematic review and Meta-Analysis of acute and longitudinal interventions. Sports Med. 2026. Epub 2026 Jul 26. doi:10.1007/s40279-026-02483-8
  22. Lakens D. Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Soc Psychol Personal Sci. 2017;8(4):355-62. doi:10.1177/1948550617697177
  23. Lakens D, Adolfi FG, Albers CJ, Anvari F, Apps MAJ, Argamon SE, et al. Justify your alpha. Nat Hum Behav. 2018;2(3):168-71. doi:10.1038/s41562-018-0311-x
  24. Ji Y, Yuan S. Lessons learned from the Bayesian design and analysis for the BNT162b2 COVID-19 vaccine phase 3 trial. N Engl J Stat Data Sci. 2025;3(2):159-63. doi:10.51387/26-NEJSDS93
  25. Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers EJ, Berk R, et al. Redefine statistical significance. Nat Hum Behav. 2018;2(1):6-10. doi:10.1038/s41562-017-0189-z
  26. Amrhein V, Greenland S. Remove, rather than redefine, statistical significance. Nat Hum Behav. 2018;2(1):4. doi:10.1038/s41562-017-0224-0
  27. Trafimow D, Marks M. Editorial. Basic Appl Soc Psych. 2015;37(1):1-2. doi:10.1080/01973533.2015.1012991

External Resources

Leave a Reply

Your email address will not be published. Required fields are marked *