Describe the phenomenon of fishing and explain how it is related to error

Question

What will be an ideal response?

Answer 1

In anything but the most trivial research study, the researcher spends a considerable amount of time analyzing the data for relationships. Of course, it's important to conduct a thorough analysis, but most people are well aware of the fact that if you torture data long enough, you can usually turn up results that support or corroborate your hypotheses. In more everyday terms, you are "fishing" for a specific result by analyzing the data repeatedly under slightly differing conditions or assumptions.

Let's think about why this is a problem. A major assumption that underlies most statistical analyses is that each analysis is independent of the other. However, that is usually not true when you conduct multiple analyses of the same data in the same study. For instance, let's say you conduct 20 statistical tests and for each one you use the .05 level criterion for deciding whether you are observing a relationship. For each test, the odds are 5 out of 100 that you will see a relationship even if there is not one there. Odds of 5 out of 100 are equal to the fraction 5/100 which is also equal to 1 out of 20 . Now, in this study, you conduct 20 separate analyses. Let's say that you find that of the twenty results, only one is statistically significant at the .05 level. Does that mean you have found a real relationship? If you had only done the one analysis, you might conclude that you found a relationship in that result. However, if you did 20 analyses, you would expect to find one of them significant by chance alone, even if no real relationship exists in the data. This threat to conclusion validity is called the fishing and the error rate problem.

The basic problem is that, like a fisherman, you repeatedly try to catch something—each analysis is like another cast into the water, and you keep going until to "catch" one. In fishing, it doesn't matter how many times you try to catch one (depending on your patience). In statistical analysis, it does matter because if you report just the one you caught, we would not know how many other analyses were run in order to catch that one.

Instead, when you conduct multiple analyses, you should adjust the error rate (the significance level or alpha level) to reflect the number of analyses you are doing, and this should be planned from the time you develop your hypothesis. The bottom line is that you are more likely to see a relationship when there isn't one if you keep reanalyzing your data and don't take your fishing into account when drawing your conclusions.