When I Selected, 'Rather Not Say', Gemini Said, 'I'll Decide for You'. In Case It's Not Obvious, Here's Why that Just Won't Do!

Why It Matters

Mis‑inferred identity data can erode trust, create legal exposure, and amplify inequities, making accurate, auditable AI essential for enterprise risk management.

Key Takeaways

• AI infers gender despite user “rather not say” setting
• Misclassifications shift correction burden onto users
• Bias persists across voice, face, and text models
• Governance frameworks lag behind AI identity inference
• Enterprises need transparent mechanisms for attribute correction

Pulse Analysis

Enterprise AI is no longer a peripheral tool; it now underpins daily workflows, from meeting transcription to biometric access control. By extracting signals—names, voices, faces—these systems generate identity attributes that feed into records, security decisions, and analytics. When a platform like Google Gemini overrides a user’s explicit gender preference, it demonstrates a design bias toward inference, embedding potentially inaccurate data into corporate knowledge bases without a clear path for amendment.

Research consistently shows that such inference mechanisms amplify existing societal biases. Studies from 2022 to 2025 reveal higher error rates for Black, Asian, and non‑binary individuals across voice‑biometric and facial‑recognition models. The resulting false rejections or mis‑gendered summaries impose an "administrative burden" on affected users, who must spend time correcting records or re‑verifying identity. This hidden cost is rarely captured in ROI calculations, yet it undermines inclusion goals and can damage brand reputation when errors become public.

Regulators are beginning to catch up. GDPR’s data‑accuracy rights and the EU AI Act’s high‑risk classification for biometric categorisation demand transparent risk management and human oversight. Enterprises should therefore audit which attributes their AI infers, publish demographic error metrics, and implement user‑driven correction workflows. By treating identity data as a shared responsibility rather than a black‑box output, organizations can mitigate legal exposure, improve system fairness, and preserve the trust essential for AI‑driven productivity.

When I selected, 'Rather not say', Gemini said, 'I'll decide for you'. In case it's not obvious, here's why that just won't do!

Author and publication details not provided.

Enterprise AI is increasingly positioned as infrastructure. Systems that summarize meetings, authenticate users, and moderate content are becoming embedded in organizational workflows. As these tools take on more authority, they also make decisions about identity – who someone is, how they should be described, and whether they are trusted.

A common response is to question whether AI systems should store identity attributes at all – there are risks associated with personal information being shared with our AI overlords and it is an emotive subject. Declared identity creates new data stores, new attack surfaces, and additional privacy risks. That concern is legitimate.

However, many systems already infer identity. They estimate gender from names, authenticity from voices, and demographic characteristics from faces. The operational question is not whether identity is processed, but what happens when systems infer incorrectly and whether users can correct the result.

When the record gets your identity wrong

I was recently added to a meeting where the host had enabled Google's Gemini Meet Notes. The assistant produces summaries, extracts action items, and generates a record of discussion. I am non‑binary and use they/them pronouns. This information is visible in my email signature, enterprise profile, and collaboration tools.

Gemini ignores it. In Google account settings, a gender field states that gender may be used for personalization across services. I had selected “Rather not say”. The system generated meeting notes referring to me as “she”. People who have known me for a long time occasionally make the same mistake – that’s part of human error and easily done. A system, however, encodes the inference into institutional records and circulates it without a mechanism for correction. The design choice is to infer identity rather than accept user‑provided information.

Research on gender bias in natural language processing shows that models trained on binary gender assumptions mis‑handle non‑binary identities and often infer gender from names rather than respecting stated identity. The MIS‑GENDERED+ benchmark, published in 2025 (https://arxiv.org/abs/2508.00788) found that many models remain vulnerable to this form of bias despite improvements in overall accuracy.

A friend of mine – a cisgender man – recently found that his bank's automated telephone system no longer recognized him. His voice biometric profile no longer matched his speech, resulting in account lockout and manual verification. His voice pitch is higher than average. The system treated this as an authentication failure. From the bank’s perspective, the system reduces fraud risk. From the user’s perspective, it produced a false rejection with no clear way to correct the underlying profile.

This pattern is documented. A 2022 study in Scientific Reports found measurable racial and gender disparities in voice biometric systems (https://www.nature.com/articles/s41598-022-06673-y) across both commercial and research implementations. Speaker identification accuracy varied across demographic groups, with some populations experiencing higher error rates.

In both cases, the systems inferred identity from signals but provided limited mechanisms for correction.

Not edge cases – design choices

These incidents are often framed as temporary limitations that will improve with better models. Evidence across machine learning suggests the issue is structural.

The US National Institute of Standards and Technology evaluated 189 face‑recognition algorithms from 99 developers. Many systems were significantly more likely to produce false matches for Black and East Asian faces than for white faces, with the highest false‑positive rates observed among African‑American women. Performance differences were linked to training‑data composition rather than inherent technical constraints. The same pattern appears across enterprise automation and algorithmic content moderation (https://diginomica.com/when-i-selected-rather-not-say-gemini-said-ill-decide-you-case-its-not-obvious-heres-why-just-wont#): systems infer identity from observable signals, resolve uncertainty through classification, provide limited correction mechanisms, and distribute error unevenly.

These dynamics predate current AI deployment. In a 2019 talk at Monkigras on diversity in design, researcher Eriol Fox (https://www.youtube.com/watch?v=vSU6JMjoX40&list=PLvsKqlNNP3R-Y274Vw7435FOSXLbFMXHQ&index=8) argued that software often assumes a “default user” and treats others as exceptions. Identity exists independently of system perception, yet automated systems frequently privilege inference over self‑declared information.

When systems mis‑classify users, the work of correction shifts onto those affected. I can manually edit meeting notes, repeat pronouns, and submit feedback. My friend must repeatedly verify his identity and explain authentication failures.

Public‑administration research describes this as administrative burden – the learning, compliance, and psychological costs required to access services. These costs are often produced by system design and distributed unevenly across populations. That overhead rarely appears in enterprise ROI calculations for AI assistants or biometric systems. Efficiency gains are measured in reduced manual effort and improved security, not in the work shifted onto misclassified users.

Automation does not eliminate work. It redistributes it.

The governance gap

Automated systems now produce records of participation, control access to services, moderate communication, and shape HR and customer identity data. In practice, they function as identity providers without corresponding governance frameworks.

Regulatory frameworks are beginning to address this. Under GDPR, individuals have rights related to automated decision‑making and data accuracy, though the status of AI‑inferred attributes remains uncertain. The EU AI Act classifies certain biometric categorization systems as high‑risk and imposes requirements around risk management, data governance, and human oversight.

Meeting transcription systems that infer gender or voice‑biometric systems with uneven failure rates may fall within these governance concerns. These outcomes represent not only inclusion challenges but operational and security trade‑offs. Before deploying identity‑inferring systems, organizations should ask:

• What identity attributes does the system infer, and from which signals?

• What are the documented error rates across demographic groups?

• What mechanisms allow users to correct inferred attributes?

If these questions cannot be answered clearly, the system’s behavior may not be fully understood.

My take

Enterprise AI is no longer just automating tasks — it's classifying people as data. Summarizing what they contributed, verifying whether they're allowed in, generating records that shape how organizations understand them. In the words of The Pretenders: don’t get me wrong. Verification matters given the ever‑growing deep fakes and credential fraud, and systems that confirm identity serve a real purpose. But verification only works when it’s accurate across the population it serves. My friend's bank thought it was verifying him. It was actually deciding his voice didn’t sound right. That’s inference with consequences, not security.

The same applies when a meeting assistant guesses my gender rather than checking my preferences. One claims to be verification, the other convenience, but both make assumptions about who I am without asking. The fixes are known: where users have declared a preference, respect it; where identity is uncertain, default to neutral; where the system gets it wrong, provide a way to correct the record. And underneath the technical fixes, a simpler expectation which shouldn’t be that hard to ask for – when a system decides something about who you are, you should be able to ask where that came from and on what basis. Systems worth trusting can answer that.

Most can’t.