How AI Can Lead to False Arrests and Wrongful Convictions

Why It Matters

When AI predictions are treated as proof, civil liberties erode and public trust in law enforcement wanes, prompting urgent calls for oversight and standards.

Key Takeaways

• AI misidentified a chip bag as a gun in Baltimore, 2025
• Facial recognition falsely tied Tennessee grandmother to out‑of‑state fraud
• Police treat AI probability scores as certain evidence, prompting arrests
• Alert thresholds are set by vendors, invisible to the public
• No national registry tracks AI policing tools across U.S. cities

Pulse Analysis

The adoption of AI‑driven surveillance and predictive‑policing platforms has accelerated in the past decade, promising faster threat detection and data‑driven resource allocation. Yet the technology’s core is statistical inference, not factual verification. When a camera in Baltimore mistook a Doritos bag for a firearm, or a facial‑recognition algorithm linked an innocent grandmother to fraud in a distant state, the resulting police actions highlighted a systemic flaw: officers are interpreting probability scores as incontrovertible proof. This conflation turns a tool designed to flag possibilities into a de‑facto decision‑maker, raising the specter of routine civil‑rights violations.

Technical design choices further compound the problem. AI models operate behind configurable confidence thresholds that dictate when an alert triggers. A low threshold captures more potential threats but inflates false positives; a high threshold reduces errors but may miss genuine risks. These settings are typically determined by vendors or internal police units and are rarely disclosed to the public. Without a transparent registry or standardized policy, municipalities adopt divergent practices, creating a patchwork of accountability that hampers oversight and public scrutiny.

Legal scholars argue that existing standards of proof—probable cause, preponderance of evidence, beyond reasonable doubt—are ill‑suited to algorithmic outputs that lack explicit uncertainty communication. To safeguard due process, policymakers must mandate clear disclosure of confidence levels, enforce independent audits, and establish a national inventory of AI policing tools. Embedding human judgment checkpoints, akin to medical diagnostics, can ensure that AI serves as a supplement rather than a substitute for investigative rigor. As AI permeates courts, schools, and hospitals, the imperative to design systems that admit uncertainty becomes a cornerstone of democratic governance.