**1. Opportunistic** – (a) poring over data not collected *specifically* for the current purpose until an alleged statistically significant association is found between
variables and then (b) devising a plausible hypothesis to fit the association.

One can easily find significant results where none exist simply by making multiple comparisons. Using the widely accepted p-value of 0.05 (i.e., willingness to take a 5 percent risk of declaring something is significant when it isn’t), more comparisons means more opportunities for random events to be declared significant due just to chance. For two tests the probability that
*at least one* "significant" difference could possibly be declared randomly is 10 percent (1 – (0.95 x 0.95)). For 20 tests, it is 64 percent (1 - 0.95**20)).

If one is on a “fishing expedition” with such a data set – once again, I emphasize that this term applies only to data (usually a tabulation) that wasn’t collected *specifically* for the current purpose – one should at least adjust decision criteria to make the
**overall** risk 5 percent. This significance value is dependent on the number of possible comparisons. There are several ways to do this, but one example would say that the threshold to declare significance for two potential individual comparisons should each be p < 0.025. Similarly, for 20 comparisons, this would be p < 0.0036.

Further, if the fishing expedition catches a boot, the fishermen should throw it back and not
claim that they were fishing for boots. The honest investigator will limit the study to focused questions, all of which make sense in the given context -- which can then be subsequently tested with an appropriately **designed** study. The data torturer will act as if every positive result confirmed a major hypothesis.

Unfortunately, when this type of data torturing is done well, it may be impossible for readers to tell that the positive
association did not spring from an *a priori* hypothesis.

**2. Procrustean** - deciding on the hypothesis to be proved, then making the data fit the hypothesis.

This requires selective reporting, one of the most common being the intentional suppression
of contradictory data. It is more difficult to carry out than opportunistic data torturing, but its results are often more believable if one starts with a popular hypothesis that appears to have been “proven.”

One should suspect data torturing whenever subjects are dropped without a clear reason, or when a large proportion of subjects are excluded for any reason. Ask: Is the rationale for the subgroup analyses convincing?

In the case
of medicine, remember that two sexes, multiple age groups, and different clinical features such as stages of disease make it possible for the investigators to examine the data in many different ways.

If a drug is reported as working only in women over 60 years of age, the savvy reader should at least suspect a chance finding.