We Need to Get Better at Asking Questions About Government AI Systems

Why It Matters

Without clear, public evaluation of government AI, citizens face opaque decision‑making that can amplify bias and waste public funds, undermining trust in digital public services.

Key Takeaways

• Lord Clement‑Jones’ AI queries went unanswered, highlighting transparency gaps
• LLMs are being piloted in fraud, housing, planning, and welfare services
• Power imbalance exists between public bodies, tech vendors, and citizens
• ATRS Hub offers a standardized way to publish algorithmic use details
• Five priorities: public evaluation, shared tools, dynamic processes, supplier data, standards

Pulse Analysis

The UK government’s rapid adoption of large language models (LLMs) reflects a broader global push to modernise public services. From automating fraud‑risk assessments in social housing to screening vulnerable claimants at the Department for Work and Pensions, AI promises efficiency gains and cost savings. Yet early missteps—such as the West Midlands police’s reliance on Microsoft Copilot hallucinations—underscore the real‑world consequences of opaque algorithms, from wrongful decisions to public backlash. These pilots expose a critical tension: while AI can streamline interactions with the state, it also magnifies existing data biases and raises environmental concerns tied to high‑energy models like Claude and ChatGPT.

Transparency is the linchpin of democratic oversight, yet the current landscape is fragmented. The Algorithmic Transparency Recording Standard (ATRS) and its public hub provide a nascent framework for documenting why and how AI tools are deployed, but they capture only formal projects, missing ad‑hoc uses of commercial services. Moreover, suppliers such as Anthropic, OpenAI, and Palantir often withhold training data and model specifications, making it difficult for legislators to assess legal risk or carbon footprints. Data Protection Impact Assessments (DPIAs) and emerging AI assurance initiatives aim to fill this gap, but they remain internal and rarely published, limiting external scrutiny from civil‑society watchdogs, technical journalists, and affected communities.

Crow’s five‑point roadmap offers a pragmatic path forward. Making evaluations publicly meaningful invites independent auditors to spot discriminatory outcomes, while shared open‑source toolkits—like Singapore’s AI Verify and the UK’s Public Sector AI Testing Framework—can lower the barrier for consistent testing across departments. Dynamic, integrated reporting mechanisms are needed to keep pace with the fast‑evolving LLM market, and procurement contracts must mandate comprehensive handover packages that include model documentation and performance metrics. By investing in a collaborative, transparent AI evaluation ecosystem, the public sector can reap the promised efficiencies without sacrificing accountability, ultimately delivering genuine public value and safeguarding democratic trust.