Jeffrey N. Rouder (UCI)
Conceptualizing and Measuring Individual Variation in Simple Cognitive Tasks: The Case of Inhibition.

In modern individual-difference studies, researchers often correlate performance on various experimental tasks to uncover common latent processes. Yet, the results have been disappointing as correlations among tasks that surely have processes in common are often quite low. Whether this low correlation implies separate processes or is a statistical consequence of low reliability remains unknown. Here, I take a critical look at the problem: Experimental tasks are defined as having an embedded contrast, say the difference in performance between primed and unprimed stimuli or between congruent and incongruent conditions. This contrast is signed, and one expects true performance to be in a certain direction, say responses are faster for unprimed than primed stimuli. The question then is do all individuals truly have effects in this certain direction. Or, alternatively, are there some individuals that have no true effect, or even some that have a true reversal? I discuss these alternatives and provide a statistical approach to address the "does everybody" question. My colleagues and I observe that for inhibition tasks, the answer is "yes." It is plausible that each individual has true Simon, Stroop, Flanker, and priming effect in the expected direction. This observation, in turn, leads almost immediately to a deep problem in the current literature: If each individual has the same sign, and the overall effect is relatively small, say 50 ms, then the variability among individuals must be relatively small as well. I show that given the noise from finite trials, common individual differences studies were doomed from the start to lack the resolution to find correlations should they exist. One possible solution is to use hierarchical models to account for trial variation as well as covariation across individuals. Yet, the results with this approach are not that promising, and the enterprise of studying covariation across individuals with experimental tasks is far harder than may be realized.