Image by wlef70 on Flickr, CC BY-NC-SA 2.0

How Generalizable are Findings from Police Surveys? A Review of Multi-Agency Studies

How Generalizable are Findings from Police Surveys? A Review of Multi-Agency Studies


Policing scholars frequently use surveys to understand officer attitudes and behavioral intentions. Yet, it is difficult to gain access to one - let alone multiple - agencies. Thus, officer surveys often reflect views in a single department, making it unclear how generalizable the findings are. For the present study, we conducted an exploratory review of articles published in 16 criminology and policing journals from 2000 to 2017. We identified 600 studies that involved surveying one or more samples of police officers. From this list, we set out to determine: (1) how often authors administered their surveys to more than one sample, and (2) when surveys were administered to more than one sample, how often were results consistent across samples? We found eighty-seven (14.5%) articles that involved collecting survey data from multiple agencies, though only 29 (4.8% overall, 33.3% of multi-agencies studies) met our inclusion criteria. Importantly, only 15 studies could be analyzed as some authors no longer had data, could not share data, or did not respond to our emails. Results were fully consistent across samples in just one published study. In the other studies, findings partially replicated - though sometimes results were in the opposite direction across departments. Thus, replication is critical before policy is created from single-agency surveys.

Police Practice and Research, 24(5), 597-611


Like 5 years ago, Erin Kearns and I set out to answer two relatively simple questions:

  1. How often do police researchers administer their surveys to officers in more than one agency?

  2. When researchers do survey officers in more than one agency, how often are results consistent across agencies?

Erin had recently published this paper in Justice Quarterly, where she administered surveys about community policing to officers in three departments. Afterward, she noted her conclusions would have been quite different if she’d only administered her survey to one department.


To answer our questions, we looked at studies published between 2000 and 2017 in 16 criminology/policing journals (the same sampling frame my colleagues and I used for this paper on response rates). One limitation of our approach is that we probably missed some studies where authors surveyed just one agency in an effort to replicate an existing study.1 However, we believe such studies are likely rare because they are disincentivized. Over in economics, for example, Galiani et al. (2017, p. 2) recently reported that “In a survey of editors, almost all responded they would in principle publish a replication study that overturned the results of the original study, but only 29% responded that they would consider publishing a replication study that confirmed the original study results.” And in criminology, it has been estimated that replication studies make up somewhere between 0.45% and 2.34% of the published literature.


Ultimately, we identified 600 studies that involved surveying one or more police departments. Just 87 of them surveyed more than a single department. After excluding studies that didn’t meet our inclusion criteria (see below), we were left with 29 studies. Finally, for 15 of those 29 studies, we were able to scrape the information needed (or contact the authors to obtain it) for our analysis.


The answer to our first question, then, was that just 14.5% of the 600 police survey studies published in these 16 journals from 2000 to 2017 involved more than one department. Here’s what it looks like when we plot the years:


The answer to our second question wasn’t quite as straightforward. One study (this one) fully replicated. Another (this one) mostly replicated. Most of the others were decidedly mixed - see Table 2 of our post-print for detailed discussion.

Bottom Line

Researchers (ourselves included) are often quick to point out the heterogeneity in American policing, which is comprised of some 18,000 agencies. Thus, we have to be careful when we interpret the results of studies involving just one of those agencies - we can’t be certain the results would hold up elsewhere. Maybe they would! But maybe not. Simply put: we need more replication research.

Our field - authors, editors, and reviewers alike - must take the need for replication studies more seriously.

  1. Another limitation is that we restricted our search to criminology and policing journals. Thus our study can’t speak to the state of affairs in, e.g., economics and political science journals. ^