Strategic Sample Selection

Résumé

An evaluator tests an empirical hypothesis based on evidence provided by a researcher who is biased toward acceptance. The researcher constructs the sample by first collecting a pre-sample and then strategically selecting data for the sample. Selected data are not Blackwell comparable to random data. The paper develops tools to characterize the impact of strategic sample selection on the payoff of evaluator and researcher. If the experiment generating the data is symmetric and satisfies equipoise (indifference between acceptance and rejection at the prior), a passive evaluator who is unaware of strategic sample selection is unaffected by data manipulation because the increase in false positives is exactly offset by the payoff gain due to the increase in false negatives. Under symmetry and equipoise, a wary evaluator optimally raises the acceptance standard; whether the researcher benefits or not from manipulation in equilibrium depends on the thickness of the tails of the data distribution. Retaining symmetry but relaxing equipoise, a wary evaluator benefits from strategic sample selection unless either the data distribution has sufficiently thick tails and the hypothesis would strongly be rejected at the prior, or the data distribution has sufficiently thin tails and the hypothesis would strongly be accepted at the prior. When the data follows Gumbel’s extreme value distribution a wary evaluator is always unaffected by strategic sample selection; in that case data manipulation is a pure rat race and results in a loss for the researcher equal to the cost of all the extra data collected. joint with Alfredo Di Tillio and Peter N. Sorensen