How should one handle post hoc analyses or HARKing? What are the problems with considering results that emerge rather than those that were hypothesized? Should researchers simply not do this, or are there good reasons for conducting such analyses?

Question

What will be an ideal response?

Answer 1

Ans: While Bem suggested that we should write our research papers based on the results that emerged, Kerr and others argue that there are problems with presenting post hoc hypotheses as if they had been hypothesized all along. Some of the problems with the latter include Type I errors are translated into theory, such hypotheses fail Popper’s criterion of falsifiability, and an explanation designed to fit post hoc results are less likely to be correct. Other problems include losing information about the original hypothesis, statistical abuses are promoted, and communicating an incorrect model of science where there is always a result to be found. One of the solutions to this problem of HARKing is to not do it. That is, researchers should not report analyses that were not originally planned. Instead, when something interesting is observed that was not expected, this observation should lead to further research on the part of the researcher. This assumes that the researcher wants to go in that direction and has the time or even plans on conducting more research in the future. On the other hand, not reporting information is almost as big of a problem as the problems listed above. This problem is related to the file drawer phenomenon—ignoring findings of any kind can represent a certain kind of bias as well. In my opinion, these analyses should be conducted and reported with complete transparency. They should be labeled as post hoc analyses, and any conclusion should be tentative at best. Then, as with any scientific endeavor, they should be explored fully in a follow-up paper if possible.