Jump to content

Post hoc analysis: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous edit
Content deleted Content added
VisualWikitext
Latin terms common in English are not italicized.
Argued that some tests conventionally called “post-hoc” should be decided on before looking at the data.
Tags: Mobile edit Mobile app edit iOS app edit App section source
 
(18 intermediate revisions by 14 users not shown)
Line 1: Line 1:
{{Short description|Statistical analyses that were not specified before the data were seen}}
{{Short description|Statistical analyses that were not specified before the data were seen}}
{{Distinguish|Post hoc theorizing}}
{{Distinguish|Post hoc theorizing}}
In a scientific study, '''post hoc analysis''' (from [[Latin language|Latin]] ''[[post hoc (disambiguation)|post hoc]]'', "after this") consists of [[statistics|statistical analyses]] that were not specified before the data was seen. This typically creates a [[multiple testing]] problem because each potential analysis is effectively a [[Statistical hypothesis testing|statistical test]]. Multiple testing procedures are sometimes used to compensate, but that is often difficult or impossible to do precisely. Post hoc analysis that is conducted and interpreted without adequate consideration of this problem is sometimes called ''[[data dredging]]'' by critics because the statistical associations that it finds are often spurious.
In a scientific study, '''post hoc analysis''' (from [[Latin language|Latin]] ''[[post hoc (disambiguation)|post hoc]]'', "after this") consists of [[statistics|statistical analyses]] that were specified after the data were seen.<ref>{{Cite web |title=What is the significance and use of post-hoc analysis studies? |url=https://www.cwauthors.com/article/significance-and-use-of-post-hoc-analysis-studies |access-date=2022-12-09 |website=www.cwauthors.com}}</ref><ref>{{Cite web |date=2019-11-12 |title=11.8: Post Hoc Tests |url=https://stats.libretexts.org/Bookshelves/Applied_Statistics/Book%3A_An_Introduction_to_Psychological_Statistics_(Foster_et_al.)/11%3A_Analysis_of_Variance/11.08%3A_Post_Hoc_Tests |access-date=2022-12-09 |website=Statistics LibreTexts |language=en}}</ref> They are usually used to uncover specific differences between three or more group means when an [[analysis of variance]] (ANOVA) test is significant.<ref>{{Cite web |title=SAGE Research Methods - The SAGE Encyclopedia of Communication Research Methods |url=https://methods.sagepub.com/reference/the-sage-encyclopedia-of-communication-research-methods/i11008.xml |access-date=2022-12-09 |website=methods.sagepub.com |language=en}}</ref> This typically creates a [[multiple testing]] problem because each potential analysis is effectively a [[Statistical hypothesis testing|statistical test]]. Multiple testing procedures are sometimes used to compensate, but that is often difficult or impossible to do precisely. Post hoc analysis that is conducted and interpreted without adequate consideration of this problem is sometimes called ''[[data dredging]]'' by critics because the statistical associations that it finds are often spurious.<ref>{{Cite journal |last=Zhang |first=Yiran |last2=Hedo |first2=Rita |last3=Rivera |first3=Anna |last4=Rull |first4=Rudolph |last5=Richardson |first5=Sabrina |last6=Tu |first6=Xin M. |date=2019-08-01 |title=Post hoc power analysis: is it an informative and meaningful analysis? |url=https://gpsych.bmj.com/content/32/4/e100069 |journal=General Psychiatry |language=en |volume=32 |issue=4 |pages=e100069 |doi=10.1136/gpsych-2019-100069 |issn=2517-729X|pmc=6738696 }}</ref>


Post hoc analyses are not inherently bad or good;<ref name="APA-Manual-7e">{{cite book |year=2020 |author=American Psychological Association |title=Publication Manual of the American Psychological Association: the Official Guide to APA Style |publisher=American Psychological Association |location=Washington, DC |isbn=978-1433832178 |edition=7th}}</ref>{{rp|12–13}} rather, the main requirement for their [[research ethics|ethical use]] is simply that their results not be [[scientific misconduct|mispresented]] as the original hypothesis.<ref name="APA-Manual-7e"/>{{rp|12–13}} Modern editions of scientific manuals have clarified this point; for example, [[APA style]] now specifies that "hypotheses should now be stated in three groupings: preplanned–primary, preplanned–secondary, and exploratory (post hoc). Exploratory hypotheses are allowable, and there should be no pressure to disguise them as if they were preplanned."<ref name="APA-Manual-7e"/>{{rp|12–13}}
⚫
== Causes ==
⚫
Sometimes the temptation to engage in post hoc analysis is motivated by a desire to produce positive results or see a project as successful. In the case of pharmaceutical research, there may be significant financial consequences to a failed trial, although the US [[Food and Drug Administration]] does not accept post hoc analysis.<ref name=":0">{{Cite news|url=https://www.nytimes.com/2017/11/28/magazine/a-failure-to-heal.html|title=A Failure to Heal|last=Mukherjee|first=Siddhartha|date=2017-11-28|work=The New York Times|access-date=2017-12-02|language=en-US|issn=0362-4331}}</ref>


== Common post hoc tests ==
In some cases, additional [[subgroup analysis]] may be requested by scientific peers or the editors of academic journals. In one such incident, journal editors demanded that the statistician [[Richard Peto]] provide a post hoc analysis of subgroups for the use of aspirin as [[secondary prevention]] for people who had experienced heart attacks. He refused the request as being statistically unsound and likely to lead to nonsensical results. When they persisted, he provided the editors with a subgroup analysis that evaluated the supposed response based upon the patients' [[Astrological sign|astrological signs]].<ref name=":0" /><ref>[[Richard Peto]], "Current misconception 3: that subgroup-specific trial mortality results often provide a good basis for individualising patient care", ''[[Br J Cancer]]'', 104(7), pages 1057-1058 (2011). {{doi|10.1038/bjc.2011.79}}</ref>
Some common post hoc tests include:<ref>{{Cite web |title=Post Hoc Definition and Types of Tests |url=https://www.statisticshowto.com/probability-and-statistics/statistics-definitions/post-hoc/ |access-date=2022-12-09 |website=Statistics How To |language=en-US}}</ref><ref>{{Cite web |last=Pamplona |first=Fabricio |date=2022-07-28 |title=Post Hoc Analysis: Process and types of tests |url=https://mindthegraph.com/blog/post-hoc-analysis/ |access-date=2022-12-09 |website=Mind the Graph Blog |language=en-US}}</ref>
* [[Holm–Bonferroni method|Holm-Bonferroni Procedure]]
* [[Newman–Keuls method|Newman-Keuls]]
* [[Rodger's method|Rodger’s Method]]
* [[Scheffé's method|Scheffé’s Method]]
* [[Tukey's range test|Tukey’s Test]] (see also: Studentized Range Distribution)

However, with the exception of Scheffès Method, these tests should be specified "a priori" despite being called "post-hoc" in conventional usage. For example, a difference between means could be significant with the Holm-Bonferroni method but not with the Turkey Test and vice versa. It would be poor practice for a data analyst to choose which of these tests to report based on which gave the desired result.

⚫
== Causes ==
⚫
Sometimes the temptation to engage in post hoc analysis is motivated by a desire to produce positive results or see a project as successful. In the case of pharmaceutical research, there may be significant financial consequences to a failed trial.{{citation needed|date=November 2023}}


== See also ==
== See also ==
*[[Testing hypotheses suggested by the data]]
* [[HARKing]]
* [[Testing hypotheses suggested by the data]]
* [[Nemenyi test]]
* [[Outcome switching]]


==References==
==References==

Latest revision as of 16:35, 29 December 2024

Not to be confused with Post hoc theorizing.

In a scientific study, post hoc analysis (from Latin post hoc, "after this") consists of statistical analyses that were specified after the data were seen.[1][2] They are usually used to uncover specific differences between three or more group means when an analysis of variance (ANOVA) test is significant.[3] This typically creates a multiple testing problem because each potential analysis is effectively a statistical test. Multiple testing procedures are sometimes used to compensate, but that is often difficult or impossible to do precisely. Post hoc analysis that is conducted and interpreted without adequate consideration of this problem is sometimes called data dredging by critics because the statistical associations that it finds are often spurious.[4]

Post hoc analyses are not inherently bad or good;[5]: 12–13  rather, the main requirement for their ethical use is simply that their results not be mispresented as the original hypothesis.[5]: 12–13  Modern editions of scientific manuals have clarified this point; for example, APA style now specifies that "hypotheses should now be stated in three groupings: preplanned–primary, preplanned–secondary, and exploratory (post hoc). Exploratory hypotheses are allowable, and there should be no pressure to disguise them as if they were preplanned."[5]: 12–13 

Common post hoc tests

[edit]

Some common post hoc tests include:[6][7]

However, with the exception of Scheffès Method, these tests should be specified "a priori" despite being called "post-hoc" in conventional usage. For example, a difference between means could be significant with the Holm-Bonferroni method but not with the Turkey Test and vice versa. It would be poor practice for a data analyst to choose which of these tests to report based on which gave the desired result.

Causes

[edit]

Sometimes the temptation to engage in post hoc analysis is motivated by a desire to produce positive results or see a project as successful. In the case of pharmaceutical research, there may be significant financial consequences to a failed trial.[citation needed]

See also

[edit]

References

[edit]
  1. ^ "What is the significance and use of post-hoc analysis studies?". www.cwauthors.com. Retrieved 2022-12-09.
  2. ^ "11.8: Post Hoc Tests". Statistics LibreTexts. 2019-11-12. Retrieved 2022-12-09.
  3. ^ "SAGE Research Methods - The SAGE Encyclopedia of Communication Research Methods". methods.sagepub.com. Retrieved 2022-12-09.
  4. ^ Zhang, Yiran; Hedo, Rita; Rivera, Anna; Rull, Rudolph; Richardson, Sabrina; Tu, Xin M. (2019-08-01). "Post hoc power analysis: is it an informative and meaningful analysis?". General Psychiatry. 32 (4): e100069. doi:10.1136/gpsych-2019-100069. ISSN 2517-729X. PMC 6738696.
  5. ^ a b c American Psychological Association (2020). Publication Manual of the American Psychological Association: the Official Guide to APA Style (7th ed.). Washington, DC: American Psychological Association. ISBN 978-1433832178.
  6. ^ "Post Hoc Definition and Types of Tests". Statistics How To. Retrieved 2022-12-09.
  7. ^ Pamplona, Fabricio (2022-07-28). "Post Hoc Analysis: Process and types of tests". Mind the Graph Blog. Retrieved 2022-12-09.
Retrieved from "https://en.wikipedia.org/enwiki/w/index.php?title=Post_hoc_analysis&oldid=1265996950"