THE ALGORITHM'S SHADOW

Predictive Policing and Racial Bias in Artificial Intelligence

A Feature From The Sankofa Question Magazine Desk

"Technology measures patterns of policing, not criminal behaviour."

Across the United States and Europe, law enforcement agencies increasingly deploy artificial intelligence systems to predict where crimes will occur and who will commit them. Police departments claim these tools allocate resources more efficiently, reduce human bias, and improve public safety outcomes. But researchers examining the underlying mathematics and real-world consequences tell a different story.

Predictive policing algorithms ingest historical arrest data, crime reports, and demographic information. They identify geographic hotspots and flag individuals deemed "high risk." But these systems do not measure actual criminal activity—they measure patterns of policing. Arrest records reflect where police chose to patrol, whom they detained, and which charges officers filed. Systematic over-policing of Black and Brown neighbourhoods produces disproportionate arrest data from those areas. Algorithms then interpret this skewed data as evidence that those communities pose higher criminal risk. This creates feedback loops: increased patrols generate more arrests, generating more data reinforcing the algorithm's predictions, justifying further police deployment (O'Neil, 2016).

Amnesty International UK documented 33 police forces using predictive profiling or risk prediction systems built on discriminatory data in a 2025 report—the first report to demonstrate that these systems breach the UK's national and international human rights obligations (Amnesty International UK, 2025). ProPublica's investigation of COMPAS, one of the most widely used risk assessment algorithms in the United States, revealed that Black defendants who did not reoffend were flagged as high risk at nearly double the rate of White defendants (Angwin et al., 2016). Commercial gender classification systems misclassified darker-skinned females at rates up to 34.7 percent, while error rates for lighter-skinned males never exceeded 0.8 percent (Buolamwini & Gebru, 2018). These technologies embed racial bias within mathematical formulas presented as objective neutrality.

SIDEBAR: BY THE NUMBERS

How Algorithms Learn Discrimination

Machine learning algorithms operate on principles that appear mathematically neutral but encode social reality. Developers feed training data drawn from historical records. The algorithm detects patterns correlating variables with outcomes. It identifies features statistically associated with higher risk. Then it applies those correlations to future cases.

The problem lies in data selection. Historical crime data does not represent actual crime rates—it represents policing practices. Communities experiencing aggressive patrol generate more arrest data regardless of actual offending. Low-income neighbourhoods with concentrated surveillance accumulate arrest records disproportionate to population size (Eubanks, 2018). Wealthier areas with minimal policing generate fewer arrests even when similar violations occur.

Consider drug offences. Consumption and trafficking occur across income classes, racial groups, and neighbourhoods. But police concentrate enforcement in Black urban areas. White suburban users face fewer stops. Resulting arrest data shows Black communities as drug crime hotspots. Algorithms trained on this data predict higher drug crime risk in Black neighbourhoods. This justification triggers more policing, creating self-fulfilling prophecy.

Researchers call this phenomenon "dirty data": algorithms ingest biased inputs produced by past civil rights violations and reproduce that bias in outputs (Richardson, 2019). The mathematics appears objective, but garbage in equals garbage out. Developers sometimes acknowledge this limitation. Few disclose it to public users.

COMPAS and Risk Assessment in Courts

Courts increasingly rely on algorithmic risk scores determining bail, sentencing, and parole decisions. The COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) system calculates probability that defendants will commit future crimes. Judges consult scores during judicial proceedings.

ProPublica's 2016 investigation revealed stark disparities. Analysing more than 7,000 individuals arrested in Broward County, Florida, investigators found that Black defendants who did not recidivate were incorrectly flagged as high risk at 44.9 percent, compared with 23.5 percent for White defendants. Conversely, White defendants who did reoffend were incorrectly labelled low risk at 47.7 percent, compared with 28 percent for Black defendants (Angwin et al., 2016). These miscalibrations meant Black defendants faced harsher sentences, longer detention, and greater supervision despite identical offence histories.

Algorithm designers defend these outcomes. They argue that COMPAS predicts recidivism with comparable accuracy across racial groups when measuring overall accuracy rates. They contend that base rate differences reflect genuine variation rather than bias. But fairness scholars dispute these claims. Different types of errors produce different harms. False positives trap innocent people in correctional systems. False negatives release offenders who reoffend. Equating them obscures disparate consequences.

Wisconsin's Supreme Court upheld the admissibility of COMPAS scores in sentencing in State v. Loomis (2016) while acknowledging the due process concerns raised by the tool's opacity (Columbia Human Rights Law Review, 2017). Defense attorneys rarely access proprietary algorithms limiting meaningful challenge. Vendor secrecy protects commercial interests while preventing judicial review. Defendants facing enhanced sentences cannot examine the mathematics underlying those determinations.

Geographic Prediction and Hotspot Policing

Companies like PredPol—rebranded as Geolitica in 2021—and HunchLab, sold by Azavea to SoundThinking (then ShotSpotter) in 2018, market software predicting where crimes will occur geographically (The Markup, 2023). Police departments deploy patrol cars to flagged locations during designated time windows. PredPol grew out of a Los Angeles Police Department–UCLA collaboration around 2010 and became the most widely used predictive policing software in the United States (Brennan Center for Justice, n.d.; FBI Law Enforcement Bulletin, 2012). Los Angeles and Santa Cruz adopted these systems beginning in 2011; Atlanta followed in 2013 (FBI Law Enforcement Bulletin, 2012; R Street Institute, 2024).

Analysis reveals problematic outcomes. Hotspot predictions cluster overwhelmingly in low-income communities of colour. Predictions reflect historical patrol patterns more than actual crime distributions. Police respond to algorithmic alerts generating increased arrests in already overpoliced areas. Data feeds back into the system strengthening future predictions.

Kristian Lum and William Isaac tested a publicly available version of PredPol's algorithm against Oakland, California's recorded drug crimes in a study published in Significance. The researchers found that the algorithm would have dispatched targeted policing almost exclusively to lower-income, minority neighbourhoods—regardless of where drug use actually occurred, which public health data showed was spread across the city. The algorithm, they wrote, would be "predicting future policing, not future crime" (Lum & Isaac, 2016). Their finding: the system learned where police had gone, not where crimes had occurred. Implementation reinforced existing patterns rather than discovering hidden risks.

A 2023 analysis by The Markup and Gizmodo examined 23,631 predictions Geolitica generated for the Plainfield, New Jersey, police department over ten months in 2018. Fewer than 100 predictions—less than half of one percent—matched a reported crime of the predicted type (The Markup, 2023). Separately, the RAND Corporation's foundational reference guide on predictive policing cautioned that such tools inherit the assumptions and limitations of the data and patrol strategies they automate (Perry et al., 2013).

Critics argue that geographic prediction legitimises concentration of police presence in already burdened communities. Residents experience intensified surveillance as technological necessity rather than policy choice. The algorithm appears neutral, absolving administrators from accountability for enforcement decisions.

Facial Recognition and Biometric Surveillance

Facial recognition technology compares photographs from surveillance cameras, social media profiles, or smartphone images against databases containing mugshots, driver licenses, and passport photos. Law enforcement uses these systems to identify suspects in criminal investigations. Civil liberty organisations warn that widespread deployment creates pervasive monitoring networks (Zuboff, 2019).

MIT researcher Joy Buolamwini, with Microsoft Research's Timnit Gebru, tested commercial gender classification systems from IBM, Microsoft, and Face++ using a dataset balanced by gender and skin type (Buolamwini & Gebru, 2018). Error rates varied dramatically across demographic groups. Lighter-skinned males experienced misclassification rates no higher than 0.8 percent. Darker-skinned females faced error rates of up to 34.7 percent. This discrepancy means darker-skinned women are far more likely to be wrongly classified by commercial facial analysis systems.

Wrongful arrests have occurred. Robert Williams, a Black Michigan man, was detained in January 2020 based on an erroneous facial recognition match. He spent 30 hours in jail before prosecutors dismissed charges (American Civil Liberties Union, n.d.). Similar cases involved Michael Oliver of Detroit and Nijeer Parks of Paterson, New Jersey, who spent ten days in jail before his case was dismissed for lack of evidence (CNN Business, 2021). All shared demographics: Black men.

More than 20 cities and jurisdictions have banned police use of facial recognition, beginning with San Francisco in May 2019 and including Boston and Portland, Oregon, in 2020 (Electronic Frontier Foundation, 2019; EPIC, 2020). Most jurisdictions permit unrestricted deployment. Police departments purchase systems from vendors including Clearview AI, whose database—built on images scraped from social media and other websites without consent—grew from roughly 3 billion faceprints at its 2020 exposure to 30 billion images by 2023, according to the company's chief executive (CiTiP KU Leuven, 2023; Business Insider, 2023). Privacy advocates argue that mass surveillance infrastructure enables authoritarian control disguised as public safety technology.

Algorithmic Auditing and Accountability Proposals

Growing evidence of bias has triggered calls for algorithmic auditing and regulation. Researchers propose standardized testing protocols evaluating discriminatory impacts across demographic groups. New York City enacted Local Law 144 in 2021, requiring independent bias audits of automated hiring tools before deployment—though no comparable federal mandate governs law enforcement technology, and federal legislation has stalled in Congress (New York City Department of Consumer and Worker Protection, 2021).

Academic researchers developed fairness metrics assessing algorithmic equity. Selbst et al. (2019) outlined five abstraction traps where technical implementations fail sociotechnical contexts: the Framing, Portability, Formalism, Ripple Effect, and Solutionism Traps. Raji and Buolamwini (2019) proposed actionable auditing methodologies enabling external verification through structured public disclosure. But vendor resistance remains significant. Commercial confidentiality claims protect proprietary code from independent examination.

Government oversight lags behind deployment speed. The Government Accountability Office issued an AI accountability framework in 2021 identifying key practices for responsible AI use, and a 2024 GAO review found that 15 of 20 federal agencies reporting current or planned AI uses maintained inventories containing inaccurate and incomplete information, prompting 35 recommendations to 19 agencies (U.S. Government Accountability Office, 2021, 2024). Without mandatory transparency requirements, developers face few incentives correcting discriminatory outputs.

Civil Rights Organisations Respond

Organisations including the ACLU, Electronic Frontier Foundation, Data & Society, and the Algorithmic Justice League litigate, publish reports, and advocate regulatory reform. They document wrongful convictions resulting from flawed algorithms. They test commercial systems identifying demographic disparities. They propose legislative frameworks protecting citizens from unregulated surveillance technology.

Amnesty International's 2025 report documented that the 33 UK police forces using predictive tools were operating systems built on discriminatory data, in breach of national and international human rights obligations (Amnesty International UK, 2025). American civil rights groups pursued litigation. Robert Williams, joined by the ACLU, filed a federal lawsuit against the City of Detroit that produced a landmark settlement barring arrests based solely on facial recognition leads (American Civil Liberties Union, n.d.).

Corporate Responsibility and Developer Ethics

Technology companies developing AI systems face growing pressure regarding social impacts. Employees organise demanding ethical review boards and public disclosure policies. Roughly 4,000 Google employees signed a petition against the Pentagon's Project Maven drone-imagery contract in April 2018, and by June the company announced it would not renew the contract (BBC News, 2018). Microsoft's president stated in June 2020 that the company would not sell facial recognition technology to U.S. police departments until a national law grounded in human rights governed its use (CNBC, 2020).

Industry initiatives have published responsible-deployment principles, but the commitments remain voluntary and carry no enforcement mechanism. Some developers adopt ethics-first approaches. Researchers at Stanford, MIT, and Carnegie Mellon publish bias detection toolkits enabling external verification. Open-source initiatives democratise access to algorithmic auditing methods. Yet these efforts remain marginal compared to commercial deployment scale.

THE SANKOFA QUESTION

If predictive policing, facial recognition, and risk assessment algorithms systematically discriminate against Black communities while operating as unaccountable black boxes shielding their mathematics from public scrutiny—how can affected populations demand accountability without access to the very systems harming them?

Algorithmic opacity prevents democratic oversight. When proprietary code remains secret, communities cannot evaluate whether mathematical claims justify deployed outcomes (Selbst et al., 2019). Three actions become necessary: First, mandate algorithmic disclosure requirements giving defendants, plaintiffs, and civil society access to source code, training data, and validation testing results (Raji & Buolamwini, 2019). Second, establish independent oversight bodies empowered to conduct audits, impose sanctions, and block deployments demonstrating discriminatory impacts. Third, ban specific applications causing demonstrable harm—including facial recognition for identification purposes, geographic prediction tools targeting low-income communities, and automated risk scores determining sentencing or parole eligibility (Richardson, 2019).

Sankofa teaches that understanding the present requires examining the past. But prediction technology looks toward the future—forecasting who will offend, where crimes will happen, which individuals pose danger. These projections inherit historical discrimination while projecting it forward as inevitable (Richardson, 2019). We must interrupt this trajectory. Technology should serve public safety without reproducing structural injustice. The question is not whether algorithms can be improved—they demonstrably can. The question is whether we possess political will mandating improvements before widespread deployment entrenches discrimination deeper into institutional practice. Protecting Black communities requires resisting algorithmic determinism masquerading as mathematical objectivity.

References

Amnesty International UK. (2025). Automated racism: How police data and algorithms code discrimination into policing. https://www.amnesty.org.uk/latest/uk-police-forces-supercharging-racism-crime-predicting-tech-new-report

Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). Machine bias: There's software used across the country to predict future criminals, and it's biased against Blacks. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

American Civil Liberties Union. (n.d.). Williams v. City of Detroit. https://www.aclu.org/cases/williams-v-city-of-detroit-face-recognition-false-arrest

BBC News. (2018, June 1). Google "to end" Pentagon artificial intelligence project. https://www.bbc.com/news/business-44341490

Brennan Center for Justice. (n.d.). Predictive policing explained. https://www.brennancenter.org/our-work/research-reports/predictive-policing-explained

Broussard, M. (2018). Artificial unintelligence: How computers misunderstand the world. MIT Press.

Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1–15. https://proceedings.mlr.press/v81/buolamwini18a.html

Business Insider. (2023, April 30). Clearview AI scraped 30 billion images from Facebook to share with police. https://www.businessinsider.com/clearview-scraped-30-billion-images-facebook-police-facial-recogntion-database-2023-4

CiTiP KU Leuven. (2023). Clearview AI: Illegally collecting and selling our faces in total impunity? (Part I). https://www.law.kuleuven.be/citip/blog/clearview-ai-illegally-collecting-and-selling-our-faces-in-total-impunity-part-i

CNBC. (2020, June 11). Microsoft says it will not sell facial recognition software to police. https://www.cnbc.com/2020/06/11/microsoft-says-will-not-sell-facial-recognition-software-to-police.html

CNN Business. (2021, April 29). Nijeer Parks was arrested due to a false facial recognition match. https://www.cnn.com/2021/04/29/tech/nijeer-parks-facial-recognition-police-arrest

Electronic Frontier Foundation. (2019, December). The fight against government face surveillance: 2019 year in review. https://www.eff.org/deeplinks/2019/12/year-fight-against-government-face-surveillance

EPIC. (2020, September). Portland City Council votes to ban facial recognition. https://epic.org/portland-city-council-votes-to-ban-facial-recognition/

Eubanks, V. (2018). Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press.

FBI Law Enforcement Bulletin. (2012). Predictive policing: Using technology to reduce crime. https://leb.fbi.gov/articles/featured-articles/predictive-policing-using-technology-to-reduce-crime

Lum, K., & Isaac, W. (2016). To predict and serve? Significance, 13(5), 14–19. https://doi.org/10.1111/j.1740-9713.2016.00960.x

New York City Department of Consumer and Worker Protection. (2021). Automated employment decision tools (AEDT) — Local Law 144. https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page

O'Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. Crown.

Perry, W. L., McInnis, B., Carter Price, C., Smith, S., & Hollywood, J. S. (2013). Predictive policing: The role of crime forecasting in law enforcement operations (RR-233). RAND Corporation. https://www.rand.org/pubs/research_reports/RR233.html

Raji, I. D., & Buolamwini, J. (2019). Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 429–435. https://doi.org/10.1145/3306618.3314244

Richardson, R. (2019). Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice [Report]. https://www.rashidarichardson.com/algorithmic-bias

R Street Institute. (2024). What works in American policing, part 6: Predictive policing. https://www.rstreet.org/commentary/what-works-in-american-policing-part-6-predictive-policing-promise-pitfalls-and-the-limits-of-algorithmic-forecasting/

Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and abstraction in sociotechnical systems. Proceedings of the Conference on Fairness, Accountability, and Transparency, 59–68. https://doi.org/10.1145/3287560.3287598

The Markup. (2023, October 2). Predictive policing software terrible at predicting crimes. https://themarkup.org/prediction-bias/2023/10/02/predictive-policing-software-terrible-at-predicting-crimes

U.S. Government Accountability Office. (2021). Artificial intelligence: An accountability framework for federal agencies and other entities (GAO-21-519SP). https://www.gao.gov/products/gao-21-519sp

U.S. Government Accountability Office. (2024). Artificial intelligence: Agencies have begun implementation but need to complete key requirements (GAO-24-105980). https://www.gao.gov/products/gao-24-105980

Whittaker, M., Crawford, K., Dobbe, R., Fried, G., Kaziunas, E., Kak, A., Mathur, V., Richardson, R., Schultz, J., Schwartz, O., & West, S. M. (2018). AI now 2018 report. AI Now Institute. https://ainowinstitute.org/wp-content/uploads/2023/04/AI_Now_2018_Report.pdf

Zuboff, S. (2019). The age of surveillance capitalism: The fight for a human future at the new frontier of power. PublicAffairs.