## Notes Lily Hu argues that, in social and political spheres such as economics, law and policing, questions of statistical methodology cannot be disentangled from normative judgements about the society and body politic for which those methods are used to draw conclusions. Referencing a debate among quantitative social scientists about how to study racial discrimination in the criminal justice system, she points out: > Shrouded in tables, figures, “preferred model” specifications, double asterisks indicating statistical significance, the whole nine-yards of econometric analysis, statistical disagreement about method just _is_, in many such cases, substantive disagreement about race. It is hard to see how expert analyses so thoroughly imbued with normative thinking about race can be seen to present cold hard expert social scientific evidence on some "true" effect of race. She argues that statistical methods can still be used to fill in our understanding of the world, but should never be sources of social innovation, and should always be subject to scrutiny. Further, the debate over statistical methods should be as much a part of the normative debate as any other questions on race, politics, etc. > ...getting the substantive questions about methodology, our social concepts, and political epistemology right—_should_ be a part of any political struggle rooted in a critical assessment of the world as it stands. *Lily Hu is a PhD candidate in Applied Mathematics and Philosophy at Harvard University.* ## Highlights Frameworks that can side-step moral and political dispute with an agree-to-disagree convergence on a simplified empirical analysis gain much in practical directedness and operationalizability. The law must, at the end of the day, *do* something. ([View Highlight](https://read.readwise.io/read/01j5tnqhqy62fkbvegvpz72x3x)) --- A [recent](https://www.cambridge.org/core/journals/american-political-science-review/article/administrative-records-mask-racially-biased-policing/66BC0F9998543868BB20F241796B79B8) [methodological](https://5harad.com/papers/post-treatment-bias.pdf) [dispute](https://dcknox.github.io/files/KnoxLoweMummolo_PostTreatmentSelectionPolicing.pdf) among quantitative social scientists about how to properly study racial discrimination in the criminal justice system well illustrates how the nuts and bolts of causal inference—in this case about the quantitative ventures to compute “effects of race”—feature a slurry of theoretical, empirical, and normative reasoning that is often displaced into debates about purely technical matters in methodology. The crux of the debate concerned the possible distortionary effect of using arrest data from administrative police records to measure causal effects of race. If the composition of police encounters recorded is biased by upstream racially biased policing procedures, one camp argued, standard inference procedure produces erroneous estimates of causal effects of race, and correspondingly, of racial discrimination. The opposing side disagreed. According to these researchers, biased policing practices do not necessarily invalidate the usual methods, since other statistical facts of the data might shake out in a way that preserves our standard inferential capacities. For example, even if policing is biased such that Black people are stopped for much more minor “violations” than are whites, it is possible that the nature of these Black police stops is such that potential police violence in these encounters compared to white ones exactly counterbalances those differences in potential use of force due to the different composition of stops. That is, the causal effect of race on police of force could be biased by, on the one hand, the kinds of stops that Black vs. white people are subject to—i.e., jaywalk stops or assault stops—and on the other, the natureof the police encounters themselves—i.e., police officer’s sense of “suspicion” or “level of threat” in the encounter. There’s no saying that these two sources of bias might not happily cancel each other out so to allow statistical inference to carry on business as usual. The dispute, then, came down to the oldest problem in the book of statistical inference: whether *any* estimand in an inference exercise can be identified depends crucially on whether some set of assumptions about the data can be made. ([View Highlight](https://read.readwise.io/read/01j5tnrm00mg35p4jx53kq0d6n)) --- The statistical skirmish among the social scientists this past summer thrusts into open view the strange place of the theoretical and the normative *within* empirical methods. Whether the causal effect of race on police use of force can be identified statistically using administrative police records hangs on assumptions that the analyst is willing to put forth about how policing works and how race works in our society—questions, as I have just argued, that must be decided priorto the use of inference methods to “detect discrimination.” ([View Highlight](https://read.readwise.io/read/01j5tntf79vqahsassmfs1b796)) --- To get at the causal effect of race on sentencing, says one expert, we need to stratify the data based on past parole violation because those who do and do not violate parole constitute distinct classes of individuals—classes independent of race for which causal effects of race must be measured separately. No, that’s not right, says another, past parole violation is a variable downstream of race, and so conditioning on it induces post-treatment bias (no need to sweat the technical details here). ([View Highlight](https://read.readwise.io/read/01j5tnvc37dcrdy5c0pzg7g9bd)) --- Shrouded in tables, figures, “preferred model” specifications, double asterisks indicating statistical significance, the whole nine-yards of econometric analysis, statistical disagreement about method just *is*, in many such cases, substantive disagreement about race. It is hard to see how expert analyses so thoroughly imbued with normative thinking about race can be seen to present cold hard expert social scientific evidence on some “true” effect of race. ([View Highlight](https://read.readwise.io/read/01j5tnw36tsvvdbaknpk5gqb67)) > Note: Key point here: the debate over statistical methods is really a debate over race shrouded in technical jargon. --- Statistical methods can never be *sources* of normative innovation. Instead, we should think their role to be to fill in a more detailed portrait of the social world, given some substantive (qualitative, interpretive) starting sketch. Both the starting sketch and the final portrait are subject to standards of empirical and political scrutiny. They can give more or less accurate accounts of how the world in fact is, and they can be more or less useful for our political projects. ([View Highlight](https://read.readwise.io/read/01j5tnxh4xpnctdfg62hppxz2k)) --- Investment in getting statistical analyses right—getting the substantive questions about methodology, our social concepts, and political epistemology right—*should* be a part of any political struggle rooted in a critical assessment of the world as it stands. Progressive interpretations of anti-discrimination and equal protection are exemplary of legal ideals that are future-looking towards a horizon of egalitarianism but are historically borne out of and presently grounded in a critical orientation toward a set of empirical facts about social injustice and legally enshrined inequality. Empirical analyses in these areas should be seen as potential allies—though, of course, not automatic ones—to a progressive legal agenda. ([View Highlight](https://read.readwise.io/read/01j5tnxaa5ndqvsdctx25hggbp)) ---