Diagnosing a Correlation Paradox: Jon’s ASC 2026 Talk

A recorded version of Jon’s ASC talk on how loose derivation chains let “successful” correlations hide theoretical anomalies, a link to the paper, and an invitation to come talk with me in person.

Hirschi’s (2004) Arkansas data: r = −.31, with 297 of 754 students mispredicted

What does a “successful” correlation actually tell us about a theory? In criminology, a statistically significant correlation in the predicted direction usually gets read as support, and the conversation moves on. In my (Jon’s) talk at this year’s American Society of Criminology meeting, I try to slow that conversation down. Using data from Hirschi’s (2004) chapter, where he redefined self-control as inhibitions, I show how a correlation of −.31 between self-control and delinquency (the kind of result most of us would happily report as evidence for the theory) coexists with the theory mispredicting 297 of 754 Arkansas students, or nearly 40% of the sample.

I recorded a version of the talk in advance, so you can watch it here (about 17 minutes):

The short version. Drawing on Paul Meehl’s idea of logical derivation chains, I argue that every empirical test of a theory is really a test of the theory plus a long list of auxiliary assumptions about measurement and statistics; for instance, that survey items mean the same thing to respondents as they do to researchers, that summing those items preserves what the theory says, and that a significant correlation indexes the process the theory describes. When those links are loose, a failed test cannot tell us which link broke, and a successful one cannot tell us which link held. Loose links can also manufacture success on their own. Overlapping item content, reverse causation, and selective publication can all push correlations in the predicted direction whether or not the theory has any truth to it, which may help explain why our journals report support far more often than Meehl’s arithmetic suggests even a true theory, tested well, should earn it.¹

How many people do the correlations we cherish get wrong?

The good news, if you can call it that, is that none of this requires new data. It requires mapping our auxiliaries (including the rival chains that could produce the same result), looking at items before scales, counting hits and misses beside every correlation, and treating the cases our theories get wrong as phenomena to explain rather than error variance to minimize.

The paper. The talk is based on my article, “Diagnosing a Correlation Paradox: How Loose Derivation Chains Conceal Theoretical Anomalies in Criminological Research,” recently published in the European Journal of Criminology. An earlier preprint version is available on SocArXiv, and the data, code, and supplement are on OSF.

Come talk with us. If you will be at ASC, I would love to see you at our session, “Innovations in Survey Design and Analysis,” on Saturday, November 21, from 8:00 to 9:20am in LaSalle 3 (7th floor) (program entry). Yes, that is an early Saturday slot at the end of a long week, so think of it as a reward for the truly committed. In the same session, Jake Day will offer a sneak peek of our work on detecting fraud and near-duplicate responses in survey data (“Match, but Verify: An Information-Theoretic Approach to Survey Fraud Detection”), which pairs percent-match screening with surprisal and entropy to cut false positives substantially. Come watch, ask hard questions, and tell us where you think we are wrong. I am presenting this work because I think our field’s evidentiary habits deserve more scrutiny than they usually get (mine included), and that kind of scrutiny benefits from open, frank discussion with smart people who see things differently. If you cannot make it, feedback in the comments below or by email is just as welcome.

Footnotes

1. Meehl (1990b, p. 220) prints .59 as the probability that a true theory passes a well-designed test, but the product of his own stated inputs is about .50, which is the figure I use in the talk.