Good Cop, Bad Cop

Does proactive policing—often referred to as “broken windows” policing—improve or harm public safety? Research published in 2017, based on data from a period when the New York Police Department slowed these practices, showed that reducing proactive policing can benefit public safety. But when Associate Professor of Criminology Aaron Chalfin and colleagues from Barnard College and the University of Connecticut tried to replicate the findings, they determined that conclusion was based on a faulty research design. They outline why and reevaluate the data in an article in the Journal of Comments and Replications in Economics.

“The 2017 article is an example of a bad research design that led to a conclusion that the data does not actually support,” says Chalfin, whose co-authors include Barnard Assistant Professor Morgan C. Williams, Jr., and David Mitre Becerril, a UConn assistant professor who was a Penn doctoral student when the work took place.

The research focused on the aftermath of a December 2014 event. Two New York City police officers were targeted, shot, and killed in their squad car. Shortly after, then-mayor Bill de Blasio commented about concerns for his biracial son when interacting with police. Many officers interpreted this as inflammatory, and an unofficial protest ensued. Officers slowed all proactive policing like traffic stops but continued to arrest for more serious crimes. This unusual confluence of events was prime for study, says Chalfin.

The researchers who undertook the original research had “good data, but didn’t analyze it properly,” comparing the period in 2014–2015 to the same period a year before, Chalfin says. “The authors used a pre-post comparison, and no one should be drawing causal inferences from a pre-post comparison. You always need a counterfactual, a stand in for what would’ve happened in the absence of this intervention, which these researchers did not have.”

Additionally, Chalfin and his co-authors found considerable variation in the intensity of the policing slowdown across New York City communities, and no evidence that highly exposed communities and less-exposed communities experienced different levels of major crimes.

“Therefore, I would say to those who read the research—or a headline based on it—that the initial results are not credible,” Chalfin says. “When the data was applied properly, we actually found that crime didn’t change one way or another, which is, in and of itself, an interesting result.”