In a previous post I was very critical of a study claiming to show gender bias in journal publications in political science. Like too many studies of this kind the data only supported the judgement of gender bias to the extent one was already inclined to believe gender bias was the appropriate explanation for gender disparities in the field. However, not all studies suffer from these flaws so when I heard about a recent study in PNAS examining how an individuals race affects how police treat them at traffic stops and saw that it was well done I thought I should post an example of the right way to engage in this kind of study (and the important/unexpected information one gets when one studies bias rigorously).
What the authors of this paper did was take body camera footage from Oakland police officers in April 2014 and examine vehicle stops they made. They had human raters (I presume college students) examine transcripts of the interactions (without knowledge of officer or civilian’s race) and rate them based on respectfulness, formality, friendliness, politeness and impartiality. After determining that such ratings were repeatable (different raters tended to agree on scoring) they then trained a computational model to predict both respect and formality which they verified against human ratings. I’ll let the paper’s authors speak for themselves about the results.
Controlling for these contextual factors, utterances spoken by officers to white community members score higher in Respect [β = 0.05 (0.03, 0.08)]. Officer utterances were also higher in Respect when spoken to older [β = 0.07 (0.05, 0.09)] community members and when a citation was issued [β = 0.04 (0.02, 0.06)]; Respect was lower in stops where a search was conducted [β = −0.08 (−0.11, −0.05)]. Officer race did not contribute a significant effect. Furthermore, in an additional model on 965 stops for which geographic information was available, neither the crime rate nor density of businesses in the area of the stop were significant, although a higher crime rate was indicative of increased Formality [β = 0.03 (0.01, 0.05)].
Note that the authors themselves raised the possibility that geographic region might play a confounding role, e.g., people in high crime areas might be treated more suspiciously, and rejected it. However, one still might worry that any effect we are seeing is a result of minorities being more inclined toward criminal behavior and thus more frequently pulled over on suspicion of serious infractions but that too is considered and rejected.
One might consider the hypothesis that officers were less respectful when pulling over community members for more severe offenses. We tested this by running another model on a subset of 869 interactions for which we obtained ratings of offense severity on a four-point Likert scale from Oakland Police Department officers, including these ratings as a covariate in addition to those mentioned above. We found that the offense severity was not predictive of officer respect levels, and did not substantially change the results described above. To consider whether this disparity persists in the most “everyday” interactions, we also reran our analyses on the subset of interactions that did not involve arrests or searches (N = 781), and found the results from our earlier models were fundamentally unchanged.
Finally, the paper authors are careful to acknowledge limitations of their analysis. In particular, they acknowledge the limitations of their study in identifying the cause of these disparities in treatment/language and with respect to the possibility that it is differences in minority behavior which itself causes officers to respond differently they say:
The racial disparities in officer respect are clear and consistent, yet the causes of these disparities are less clear. It is certainly possible that some of these disparities are prompted by the language and behavior of the community members themselves, particularly as historical tensions in Oakland and preexisting beliefs about the legitimacy of the police may induce fear, anger, or stereotype threat. However, community member speech cannot be the sole cause of these disparities. Study 1 found racial disparities in police language even when annotators judged that language in the context of the community member’s utterances. We observe racial disparities in officer respect even in police utterances from the initial 5% of an interaction, suggesting that officers speak differently to community members of different races even before the driver has had the opportunity to say much at all.
I feel that this analysis considered and fairly convincingly rejected all the plausible confounders. Of course others might disagree and suggest some other factor, e.g., expensiveness of car, is responsible but even if you are inclined to take such a line you have to admit that this study provides some pretty damn good evidence by ruling out many other plausible confounding variables.
Having said this one should still be careful (as the authors of this paper are) in interpreting the results. In particular, we don’t have a good sense of what the psychological reason for officers different behavior with minorities. Is it because they judge them to be less deserving of respect? Or maybe officers judge minorities to be less respectful to them so begin the interaction less respectfully? Or some other explanation? If the goal is making the world a better place and not merely assigning blame those answers matter and hopefully more good scientific studies will reveal them.
I’d like to close with what I take to be one of the most important reasons to do this research rigorously. While most people could have probably guessed that officers would be less respectful to minority drivers it wasn’t at all obvious that officer race wouldn’t play a factor in respectfulness to minorities. Nor was it obvious that we would see a difference in treatment from a broad swath of police officers not merely a few particularly biased officers. The reason this kind of research is important (in addition to validating minority claims of police treatment) is that we need to learn how and why minorities are treated differently if we are going to fix the problem. Without studies like this I think many people’s natural assumption is that hiring minority officers would address these problems. It doesn’t.