The clinician’s guide to p values, confidence intervals, and magnitude of effects (2024)

Perhaps the most universally recognized statistic is the p-value. Most individuals understand the notion that (usually) a p-value <0.05 signifies a statistically significant difference between the two groups being compared. While this understanding is shared amongst most, it is far more important to understand what a p-value does not tell us. Attempting to inform clinical practice patterns through interpretation of p-values is overly simplistic, and is fraught with potential for misleading conclusions. A p-value represents the probability that the observed result (difference between the groups being compared)—or one that is more extreme—would occur by random chance, assuming that the null hypothesis (the alternative scenario to the study’s hypothesis) is that there are no differences between the groups being compared. For example, a p-value of 0.04 would indicate that the difference between the groups compared would have a 4% chance of occurring by random chance. When this probability is small, it becomes less likely that the null hypothesis is accurate—or, alternatively, that the probability of a difference between groups is high [1]. Studies use a predefined threshold to determine when a p-value is sufficiently small enough to support the study hypothesis. This threshold is conventionally a p-value of 0.05; however, there are reasons and justifications for studies to use a different threshold if appropriate.

What a p-value cannot tell us, is the clinical relevance or importance of the observed treatment effects. [1]. Specifically, a p-value does not provide details about the magnitude of effect [2,3,4]. Despite a significant p-value, it is quite possible for the difference between the groups to be small. This phenomenon is especially common with larger sample sizes in which comparisons may result in statistically significant differences that are actually not clinically meaningful. For example, a study may find a statistically significant difference (p < 0.05) between the visual acuity outcomes between two groups, while the difference between the groups may only amount to a 1 or less letter difference. While this may be in fact a statistically significant difference, the difference is likely not large enough to make a meaningful difference for patients. Thus, p-values lack vital information on the magnitude of effects for the assessed outcomes [2,3,4].

To overcome this limitation, it is important to consider both (1) whether or not the p-value of a comparison is significant according to the pre-defined statistical plan, and (2) the magnitude of the treatment effects (commonly reported as an effect estimate with 95% confidence intervals) [5]. The magnitude of effect is most often represented as the mean difference between groups for continuous outcomes, such as visual acuity on the logMAR scale, and the risk or odds ratio for dichotomous/binary outcomes, such as occurrence of adverse events. These measures indicate the observed effect that was quantified by the study comparison. As suggested in the previous section, understanding the actual magnitude of the difference in the study comparison provides an understanding of the results that an isolated p-value does not provide [4, 5]. Understanding the results of a study should shift from a binary interpretation of significant vs not significant, and instead, focus on a more critical judgement of the clinical relevance of the observed effect [1].

There are a number of important metrics, such as the Minimally Important Difference (MID), which helps to determine if a difference between groups is large enough to be clinically meaningful [6, 7]. When a clinician is able to identify (1) the magnitude of effect within a study, and (2) the MID (smallest change in the outcome that a patient would deem meaningful), they are far more capable of understanding the effects of a treatment, and further articulate the pros and cons of a treatment option to patients with reference to treatment effects that can be considered clinically valuable.

Authors and Affiliations

1. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada

   Mark R. Phillips,Lehana Thabane,Mohit Bhandari&Varun Chaudhary

2. Retina Consultants of Texas (Retina Consultants of America), Houston, TX, USA

   Charles C. Wykoff

3. Blanton Eye Institute, Houston Methodist Hospital, Houston, TX, USA

   Charles C. Wykoff

4. Biostatistics Unit, St. Joseph’s Healthcare-Hamilton, Hamilton, ON, Canada

   Lehana Thabane

5. Department of Surgery, McMaster University, Hamilton, ON, Canada

   Mohit Bhandari&Varun Chaudhary

6. NIHR Moorfields Biomedical Research Centre, Moorfields Eye Hospital, London, UK

   Sobha Sivaprasad

7. Cole Eye Institute, Cleveland Clinic, Cleveland, OH, USA

   Peter Kaiser

8. Retinal Disorders and Ophthalmic Genetics, Stein Eye Institute, University of California, Los Angeles, CA, USA

   David Sarraf

9. Department of Ophthalmology, Mayo Clinic, Rochester, MN, USA

   Sophie J. Bakri

10. The Retina Service at Wills Eye Hospital, Philadelphia, PA, USA

    Sunir J. Garg

11. Center for Ophthalmic Bioinformatics, Cole Eye Institute, Cleveland Clinic, Cleveland, OH, USA

    Rishi P. Singh

12. Cleveland Clinic Lerner College of Medicine, Cleveland, OH, USA

    Rishi P. Singh

13. Department of Ophthalmology, University of Bonn, Boon, Germany

    See Also

    On p-Values and Statistical Significance

    Frank G. Holz

14. Singapore Eye Research Institute, Singapore, Singapore

    Tien Y. Wong

15. Singapore National Eye Centre, Duke-NUD Medical School, Singapore, Singapore

    Tien Y. Wong

16. Centre for Eye Research Australia, Royal Victorian Eye and Ear Hospital, East Melbourne, VIC, Australia

    Robyn H. Guymer

17. Department of Surgery (Ophthalmology), The University of Melbourne, Melbourne, VIC, Australia

    Robyn H. Guymer

Consortia

Contributions

MRP was responsible for conception of idea, writing of manuscript and review of manuscript. VC was responsible for conception of idea, writing of manuscript and review of manuscript. MB was responsible for conception of idea, writing of manuscript and review of manuscript. CCW was responsible for critical review and feedback on manuscript. LT was responsible for critical review and feedback on manuscript.

Corresponding author

Correspondence to Varun Chaudhary.

Competing interests

MRP: Nothing to disclose. CCW: Consultant: Acuela, Adverum Biotechnologies, Inc, Aerpio, Alimera Sciences, Allegro Ophthalmics, LLC, Allergan, Apellis Pharmaceuticals, Bayer AG, Chengdu Kanghong Pharmaceuticals Group Co, Ltd, Clearside Biomedical, DORC (Dutch Ophthalmic Research Center), EyePoint Pharmaceuticals, Gentech/Roche, GyroscopeTx, IVERIC bio, Kodiak Sciences Inc, Novartis AG, ONL Therapeutics, Oxurion NV, PolyPhotonix, Recens Medical, Regeron Pharmaceuticals, Inc, REGENXBIO Inc, Santen Pharmaceutical Co, Ltd, and Takeda Pharmaceutical Company Limited; Research funds: Adverum Biotechnologies, Inc, Aerie Pharmaceuticals, Inc, Aerpio, Alimera Sciences, Allergan, Apellis Pharmaceuticals, Chengdu Kanghong Pharmaceutical Group Co, Ltd, Clearside Biomedical, Gemini Therapeutics, Genentech/Roche, Graybug Vision, Inc, GyroscopeTx, Ionis Pharmaceuticals, IVERIC bio, Kodiak Sciences Inc, Neurotech LLC, Novartis AG, Opthea, Outlook Therapeutics, Inc, Recens Medical, Regeneron Pharmaceuticals, Inc, REGENXBIO Inc, Samsung Pharm Co, Ltd, Santen Pharmaceutical Co, Ltd, and Xbrane Biopharma AB—unrelated to this study. LT: Nothing to disclose. MB: Research funds: Pendopharm, Bioventus, Acumed – unrelated to this study. VC: Advisory Board Member: Alcon, Roche, Bayer, Novartis; Grants: Bayer, Novartis – unrelated to this study.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original version of this article was revised: In this article the middle initial in author name Sophie J. Bakri was missing.

Reprints and permissions