{"id":263195,"date":"2024-06-03T12:00:37","date_gmt":"2024-06-03T11:00:37","guid":{"rendered":"https:\/\/www.transcend.org\/tms\/?p=263195"},"modified":"2024-05-31T08:54:30","modified_gmt":"2024-05-31T07:54:30","slug":"race-policing-and-the-limits-of-social-science","status":"publish","type":"post","link":"https:\/\/www.transcend.org\/tms\/2024\/06\/race-policing-and-the-limits-of-social-science\/","title":{"rendered":"Race, Policing, and the Limits of Social Science"},"content":{"rendered":"
\"\"<\/a>

Image: Derek Simeone \/ Flickr<\/p><\/div>\n

Studying the social world requires more than deference to data\u2014no matter the prestige or sophistication of the tools with which they are parsed.<\/em><\/p><\/blockquote>\n

6 May 2024 <\/em>– In 2016, economist Roland G. Fryer, Jr., the youngest African American ever to be awarded tenure at Harvard, came upon what he would call the \u201cmost surprising result of my career.\u201d In a study<\/a> of racial differences in the use of force by police officers, Fryer found that Black and Hispanic civilians were no more likely than white civilians to be shot to death by police. \u201cYou know, protesting is not my thing,\u201d Fryer told<\/a> the New York Times.<\/em> \u201cBut data is my thing. So I decided that I was going to collect a bunch of data and try to understand what really is going on when it comes to racial differences in police use of force.\u201d<\/p>\n

\n
The entwining of data and theory is especially fraught in the study of race.<\/strong><\/em><\/div>\n<\/blockquote>\n

Three thousand hours later, after meticulous records collection and analysis, the data had spoken. Although Black people are significantly more likely to experience non-lethal<\/em> force at the hands of police than white people in similar situations, Fryer concluded, there is no racial bias in fatal police shootings. The findings appeared to have direct implications for the growing protest movement that had swept across the United States following the police killings of Michael Brown, Eric Garner, Tamir Rice, Freddie Gray, Philando Castile, among so many others. \u201cIt is plausible that racial differences in lower-level uses of force,\u201d Fryer wrote at the end of the paper, \u201care simply a distraction and movements such as Black Lives Matter should seek solutions within their own communities rather than changing the behaviors of police and other external forces.\u201d<\/p>\n

The study quickly came under fire. For the most part, critics took one of two tacks. One was to argue that the research failed on its own technical terms: the data were erroneous or misleading; there was a mathematical error in the analysis; the statistical protocol was inappropriate. The other tack was to undermine the legitimacy of the effort on auxiliary grounds, pointing out that economists are not experts in the study of police shootings and that the profession of economics suffers from a \u00ad\u00ad\u00adconservative bias. These two types of reply exemplify a common pattern of response to the results of quantitative social science, and they are also once again on wide display in national conversations about policing. But they illustrate a deep problem with the way we think about the nature of social scientific inquiry\u2014and, consequently, its capacity to inform our thinking about politics and policy.<\/p>\n

On these two views, scientific method is either so airtight that only errors from within can undermine it or so porous that its validity turns entirely on outside interests. There are certainly cases of each kind of blunder: recall the Reinhart-Rogoff Excel spreadsheet snafu<\/a> on the one hand, tens of millions of dollars funneled by ExxonMobil to fund climate change denialism<\/a> studies on the other. We misunderstand run-of-the-mill scientific practice, however, if we view it as either everywhere or nowhere settled by data. As historians and philosophers of science have long emphasized, \u201cthe data\u201d can never take us all the way from observation to conclusion; we can interpret them only against some background theory that settles what the data are evidence of<\/em>. Far from playing no role in quantitative social science, a shared set of theoretical and normative commitments is what allows data-first methods to work at all.<\/p>\n

\n
What we should believe\u2014and what we should give up believing\u2014can never be decided simply by brute appeals to data.<\/strong><\/em><\/div>\n<\/blockquote>\n

This entwining of data and theory runs through any application of quantitative methods, but it is especially fraught today in the study of race. Since the 1970s, the development of causal inference methodology<\/a> and the rise of large-scale data collection efforts have generated a vast quantitative literature on the effects of race in society. But for all its ever-growing technical sophistication, scholars have yet to come to consensus on basic matters regarding the proper conceptualization and measurement of these effects. What exactly does it mean for race to act as a cause? When do inferences about race make the leap from mere correlation to causation? Where do we draw the line between assumptions about the social world that are needed to get the statistical machinery up and running and assumptions that massively distort how the social world in fact is and works? And what is it that makes quantitative analysis a reliable resource for law and policy making?<\/p>\n

In both academic and policy discourse, these questions tend to be crowded out by increasingly esoteric technical work. But they raise deep concerns that no amount of sophisticated statistical practice can resolve, and that will indeed only grow more significant as \u201cevidence-based\u201d debates about race and policing reach new levels of controversy in the United States. We need a more refined appreciation of what social science can offer as a well of inquiry, evidence, and knowledge, and what it can\u2019t. In the tides of latest findings, what we should believe\u2014and what we should give up believing\u2014can never be decided simply by brute appeals to data, cordoned off from judgments of reliability and significance. A commitment to getting the social world right does not require deference to results simply because the approved statistical machinery has been cranked. Indeed in some cases, it may even require that we reject findings, no matter the prestige or sophistication of the social scientific apparatus on which they are built.<\/p>\n

An instructive object lesson in these issues can be found in a controversy ignited last summer, when a paper<\/a> published in the American Political Science Review<\/em> (APSR) in late May questioned the validity of many recent approaches to studying racial bias in police behavior, including Fryer\u2019s. A very public skirmish among social scientists ensued, all against the backdrop of worldwide protests over the murder of George Floyd by Minneapolis police officer Derek Chauvin.<\/p>\n

The APSR paper focused, in particular, on the difficulties of \u201cstudying racial discrimination using records that are themselves the product of racial discrimination.\u201d The authors\u2014Dean Knox, Will Lowe, and Jonathan Mummolo\u2014argued that<\/p>\n

when there is any racial discrimination in the decision to detain civilians\u2014a decision that determines which encounters appear in police administrative data at all\u2014then estimates of the effect of civilian race on subsequent police behavior are biased absent additional data and\/or strong and untestable assumptions.<\/em><\/p><\/blockquote>\n

The trouble, in short, is that \u201cpolice records do not contain a representative sample\u201d of people observed by the police. If there is racial bias reflected in who gets stopped and why\u2014and we have independent reason to believe that it does\u2014then police data for white and non-white arrestees are not straightforwardly comparable without making additional implausible or untestable assumptions. Such \u201cpost-treatment bias\u201d in the data would thus severely compromise any effort to estimate the \u201ctrue\u201d causal effects of race on law enforcement behavior, even if we are only interested in what happens after the stop takes place. \u201cExisting empirical work in this area is producing a misleading portrait of evidence as to the severity of racial bias in police behavior,\u201d the authors conclude. Such techniques \u201cdramatically underestimate or conceal entirely the differential police violence faced by civilians of color.\u201d The authors therefore call for \u201cfuture research to be designed with this issue in mind,\u201d and they outline an alternative approach.<\/p>\n

\n
Where do we draw the line between assumptions needed to do the statistics and assumptions that distort how the social world really works?<\/em><\/strong><\/div>\n<\/blockquote>\n

A critical response<\/a> by several other scholars\u2014Johann Gaebler, William Cai, Guillaume Basse, Ravi Shroff, Sharad Goel, and Jennifer Hill\u2014appeared a month later, in June; for simplicity, call this group of scholars the second camp<\/em>. Disputing the APSR authors\u2019 pessimistic assessment of research on racial bias in policing, they countered that the APSR paper rested on a \u201cmathematical error.\u201d The usual methods could<\/em> still recover reliable estimates of the causal effect of race on law enforcement behavior after a stop has been made, even if police stops are themselves racially biased. The error, they assert, lay in assuming that certain conditions had<\/em> to be assumed in order to make reliable estimates using data like Fryer\u2019s. In fact, these scholars wrote, a weaker statistical condition\u2014what they term \u201csubset ignorability\u201d\u2014would also suffice, and it was more likely to hold, \u201cexactly or approximately,\u201d in practice. They then attempt to show how the standard causal estimation techniques can be saved by putting forth their own analysis of racial bias in prosecutors\u2019 decisions to pursue charges (again relying on the sort of data from police records that the APSR authors find problematic).<\/p>\n

In the days following this exchange, what ensued can only be described as a high-profile statistical showdown on Twitter, punctuated by takes from interested onlookers. The second camp mounted a defense of the mathematics, arguing that progress in statistical methods should not be foreclosed for fear of unobservable bias. In a policy environment that increasingly looks to quantitative analyses for guidance, Goel wrote<\/a>, \u201ccategorically barring a methodology . . . can have serious consequences on the road to reform.\u201d The APSR authors, by contrast, emphasized what they took to be the purpose of applied social scientific research: to provide analysis at the service of real-world policy and practical political projects. Knox, for example, wrote<\/a> that their critics\u2019 argument \u201ctreats racial bias like a game with numbers.\u201d Instead, he went on, he and his co-authors \u201cuse statistics to seek the best answers to serious questions\u2014not to construct silly logic puzzles about knife-edge scenarios.\u201d This is no time, the APSR authors argued, to fetishize mathematical assumptions for the sake of cranking the statistical machinery.<\/p>\n

What are we to make of this debate? Despite the references to mathematics and the sparring of proof-counterexample-disproof, which suggest a resolution is to be found only in the realm of pure logic, the dispute ultimately comes down to a banal, congenital feature of statistical practice: the plausibility of the assumptions one must make at the start of every such exercise. For the APSR authors, even the second camp\u2019s weaker assumption of subset ignorability fails the test of empirical scrutiny: to them it is clearly implausible as a matter of how the social world in fact is and works. Ironically though, given their forceful criticism of the APSR paper, the second camp comes to the same conclusion in their own analysis of prosecutors\u2019 charging decisions, conceding that \u201csubset ignorability is likely violated\u201d\u2014thus rendering their own results empirically suspect.<\/p>\n

\n
We cannot justify our use of implausible assumptions solely on the basis of mathematical convenience.<\/strong><\/em><\/div>\n<\/blockquote>\n

This curious episode demonstrates how the social scientist is so often trapped in a double bind in her quest to cleave to her empirical commitments, especially when it comes to the observational studies\u2014as opposed to randomized experiments\u2014that are the bread and butter of almost all quantitative social science today. Either she buys herself the ability to work with troves of data, at the cost of implausibility in her models and assumptions, or she starts with assumptions that are empirically plausible but is left with little data to do inference on. By and large, quantitative social science in the last two decades has taken the former route, thanks in significant part to pressure from funding incentives. If implausible assumptions are the price of entry, the Big Data revolution promises the payment is worth it\u2014be it in profit or professional prestige. As the mathematical statistician David A. Freedman wrote<\/a>, \u201cIn the social and behavioral sciences, far-reaching claims are often made for the superiority of advanced quantitative methods\u2014by those who manage to ignore the far-reaching assumptions behind the models.\u201d<\/p>\n

But if the social scientist is genuinely committed to being empirical, this choice she must make between plausible assumptions and readily available data must itself be justified on the basis of empirical evidence. The move she winds up making thus tacitly reveals the credence she has toward the theories of the social world presently available to her, or at least the kind of commitments she is willing to be wrong about. Precisely to the extent that social science is something more than mathematics\u2014in the business of figuring out how the world is, or approximately is\u2014statistical assumptions can never shake off their substantive implications. The requirement that social science be truly \u201cevidence-based\u201d is thus extremely demanding: it means that we cannot justify our use of implausible assumptions solely on the basis of mathematical convenience, or out of sheer desire to crank the statistical machinery. It is only in the belief that our assumptions are true, or true enough, of the actually existing world that social science can meet this exacting demand.<\/p>\n

Notice the role that normativity plays in this analysis. If, as the first step to embarking on any statistical analysis, the quantitative social scientist must adopt a set of assumptions about how the social world works, she introduces substantive theoretical commitments as inputs<\/em> into her inquiry. This initial dose of normativity thus runs through the entire analysis: there is simply no escaping it. Whether any subsequent statistical move is apt will depend, in however complex ways, on one\u2019s initial substantive views about the social world.<\/p>\n

\n
A shared set of theoretical and normative commitments is what allows data-first methods to work at all.<\/em><\/strong><\/div>\n<\/blockquote>\n

What do these reflections mean in the specific case of research on race and policing? Whether one has in fact distilled the causal effect of race on police behavior in any particular study will depend on what one believes to be true about the racial features of policing more broadly. And since what positions you take on these matters depend on your background views regarding the prevalence and severity of racial injustice as an empirical phenomenon, whether a finding ends up passing statistical muster and therefore counts as an instance of racially discriminatory police action will depend on your broader orientation to the social world.<\/p>\n

The upshot of these considerations is that statistical analysis is inescapably norm-laden; \u201cfollowing the data\u201d is never a mechanical or purely mathematical exercise. But this fact should not lead us to discard any commitment to empirical validity as such. On the contrary, it should simply serve to remind us that standards of empirical scrutiny apply throughout the whole of any methodology. As Freedman put it, \u201cThe goal of empirical research is\u2014or should be\u2014to increase our understanding of the phenomena, rather than displaying our mastery of technique.\u201d<\/p>\n

One important consequence of this orientation, I think, is that we ought to subject not just assumptions but also conclusions to empirical scrutiny. To some observers of our social world, the conclusion that there is no causal effect of race in police shootings is not only implausible: it is simply and patently false. For even a cursory glance at descriptive summary statistics reveals wide gulfs in the risk of being killed by police for Blacks compared to whites. According to one study<\/a>, Black men are about 2.5 times more likely to be killed by police than white men, and data<\/a> from the 100 largest city police departments show that police officers killed unarmed Black persons at four times the rate of unarmed white persons\u2014statistical facts that speak nothing of the immense historical record of overtly racist policing, which does not lend itself so easily to quantification. If certain methods erase these stark (and undisputed) disparities, painting a picture of a social landscape in which race does not causally influence police shooting behaviors, then so much worse for those methods. From this vantage, failing to take account of the many different forms of evidence of decades of racialized policing and policymaking is not only normatively wrong. It is also empirically absurd, especially as a self-styled \u201cevidence-based\u201d program that seeks to illuminate the truths of our social world.<\/p>\n

\n
Rejecting a study\u2019s methods on the basis of disagreement with its results is a completely legitimate inferential move.<\/strong><\/em><\/div>\n<\/blockquote>\n

This suggestion\u2014that we sometimes ought to reject a finding on the grounds that it does not accord with our prior beliefs\u2014might seem downright heretical to the project of empirical science. And indeed, there is some danger here; at the extreme, indiscriminate refusal to change our minds in the light of evidence reeks of a sham commitment to empirical study of the world. But the truth is that scientists reject findings on these sorts of grounds all the time<\/em> in the course of utterly routine scientific practice. (For just one recent newsworthy example<\/a>, consider a 2011 study that found evidence for extrasensory perception.) The move need not signal a failure of rationality; indeed it can often be a demand<\/em> of it. Determining which it is, in any particular case, cannot be settled by asking whether one has been faithful to \u201cfacts and logic,\u201d as so many like to say, or to the pure rigors of mathematical deduction.<\/p>\n

Instead, when a scientific finding conflicts with one of our convictions, each of us must comb over what philosopher W. V. O. Quine so charmingly called<\/a> our \u201cweb of belief,\u201d considering what must be sacrificed so that other beliefs might be saved. And since our webs are not all identical, what rational belief revision demands of us will also vary. One man\u2019s happily drawn conclusion (p, <\/em>therefore q<\/em>!) is another\u2019s proof by contradiction (surely not q<\/em>, therefore not p<\/em>!). Or as the saying goes, one man\u2019s modus ponens <\/em>is another man\u2019s modus tollens.<\/em> Rejecting a study\u2019s methods or its starting assumptions on the basis of disagreement with its results is a completely legitimate inferential move. We tend to overlook this feature of science only because for most of us, so much of the nitty-gritties of scientific inquiry have little direct bearing on our daily lives. Our webs of belief usually are not touched by the latest developments in science. But once scientific findings make contact with\u2014and perhaps even run up against\u2014our convictions, we become much more sensitive to the way the chain of reasoning is made to run.<\/p>\n

The fact that good faith efforts at rationality might lead different people to different or even opposite conclusions is a basic, if unsettling, limitation of science. We cannot hope for pure knowledge of the world, deductively chased from data to conclusion without mediating theory. In the end, the Fryer study controversy has been one long object lesson in how our empirical commitments are invariably entangled with normative ones, including commitments more typically thought of as ethical or political. The choice to sacrifice empirical plausibility in one\u2019s assumptions, in particular, is not just a \u201cscientific\u201d matter, in the oversimplified sense of \u201cjust the facts\u201d: it is inevitably interwoven with our ethical and political commitments. In bringing one\u2019s web of beliefs to bear on the debate over what constitutes proper study of effects of race in policing, one puts forth not just prior empirical beliefs about, say, the prevalence of racial targeting or the fidelity of police reporting practices, but also one\u2019s orientation toward matters of racial justice and self-conceptualization as a researcher of race and the broader system of policing and criminal justice.<\/p>\n

\n
Good faith efforts at rationality might lead different people to different or even opposite conclusions.<\/strong><\/em><\/div>\n<\/blockquote>\n

For the APSR authors, bias in policing presents both enough of a normative concern and an empirical possibility to license, as a matter of good scientific practice, the sacrifice of certain business-as-usual approaches. The second camp, by contrast, is loath to make the leap to discard approaches held in such high esteem. Their commitment to the usefulness of the standard approaches runs so deep that they do not yet see sufficient cause for retreat. In a revision of their paper<\/a> released in October, the authors remove the explicit assertion of a \u201cmathematical error\u201d but find \u201creason to be optimistic\u201d that many cases of potential discrimination do <\/em>meet the empirical conditions prescribed by the statistical assumptions proposed to salvage the usual approaches.<\/p>\n

What exactly these reasons for optimism are remains unclear. By the second camp\u2019s own admission, because \u201cone cannot know the exact nature and impact of unmeasured confounding . . . we must rely in large part on domain expertise and intuition to form reasonable conclusions.\u201d And yet without reference to any such further evidence or support, they nevertheless conclude: \u201cIn this case, we interpret our results as providing moderately robust evidence that perceived gender and race have limited effects on prosecutorial charging decisions in the jurisdiction we consider.\u201d Such a claim ultimately says much more about their web of belief than about the actually existing social world.<\/p>\n

For those whose beliefs, empirical and ethical, are forged in participation in radical sociopolitical movements from below, to be ill-inclined to accept certain findings about race and policing is to remain steadfast in a commitment to a certain thick set of empirical and ethical propositions in their <\/em>webs of beliefs: that systems of policing and prisons are instruments of racial terror and that any theory of causation, theory of race, and statistical methods worth their salt will see race to be a significant causal factor affecting disparate policing and prison outcomes. This <\/em>just is the first test of \u201cfitting the data.\u201d It is not a flight from rationality but an exercise of it.<\/p>\n

\n
We need a conception of social theory that is at once \u201cempirical, interpretative, and critical.\u201d<\/strong><\/em><\/div>\n<\/blockquote>\n

Does this view of social science transform an epistemic enterprise into a crudely political one? Does a readiness to sacrifice some scientific findings to save ethical or political commitments endanger the status of science as a distinctive project that seeks to produce new knowledge about the world? I think it doesn\u2019t have to. Even the hardest-nosed empiricist starts from somewhere. She must interpret her data against some background theory that she takes to be the most natural, most plausible, and most fruitful. Deviations from this position that are self-consciously animated by politics need not be less genuinely truth-seeking than self-styled neutral deference to the status quo.<\/p>\n

This fact tends to get lost in debates about where science sits along a continuum that runs from \u201cobjective\u201d (protected from bias and outside interference) to \u201cpolitical\u201d (a no-holds-barred struggle for power, the label of \u201cscience\u201d slapped onto whatever the winner wishes). What that picture elides is how science unfolds in the trenches of knowledge production: in the methodological minutiae that determine which assumptions must be sacrificed and which can be saved, when abstraction leads to silly logic puzzles and when it is a necessary evil, which conclusions trigger double-takes and which signal paradigm shifts, and so on. To acknowledge that these struggles cannot take us beyond the never-ending tides of the \u201clatest findings\u201d is not to give up on quantitative social science as a venture for better understanding the world. It is simply to embrace a conception of social inquiry that is always, as philosopher Richard J. Bernstein put<\/a> it, at once \u201cempirical, interpretative, and critical.\u201d<\/p>\n

________________________________________<\/p>\n

Lily Hu\u00a0is Assistant Professor of Philosophy at Yale and a contributing editor at <\/em>Boston Review. She works in philosophy of science and social science, and political and social philosophy.<\/em><\/p>\n

Go to Original – bostonreview.net<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

6 May 2024 – Studying the social world requires more than deference to data\u2014no matter the prestige or sophistication of the tools with which they are parsed.<\/p>\n","protected":false},"author":4,"featured_media":263196,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48],"tags":[2449,1778,2579,1755,354,308,676,1281,552,103,108,70],"class_list":["post-263195","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-in-focus","tag-big-data","tag-conflict-analysis","tag-critical-race-theory","tag-data-protection","tag-economics","tag-philosophy","tag-police","tag-police-brutality","tag-race","tag-racism","tag-social-sciences","tag-usa"],"_links":{"self":[{"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/posts\/263195","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/comments?post=263195"}],"version-history":[{"count":2,"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/posts\/263195\/revisions"}],"predecessor-version":[{"id":263198,"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/posts\/263195\/revisions\/263198"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/media\/263196"}],"wp:attachment":[{"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/media?parent=263195"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/categories?post=263195"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.transcend.org\/tms\/wp-json\/wp\/v2\/tags?post=263195"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}