ChatGPT Is Not Your Friend (Guest Post by Grace Helton)

Key Takeaways

• 20% of US high schoolers used AI romantically (2025 data)
• LLMs lack non‑derivative interests, preventing true caring
• Friendship requires mutual care for each other's sake
• Human‑LLM bonds risk instrumental harms and ethical concerns
• Industry pushes companion AI despite philosophical objections

Pulse Analysis

The market for AI companionship has exploded in recent years, with startups like Replika branding chatbots as "best friends" and media reports noting that one‑fifth of American high‑school students have experimented with AI in romantic contexts. These products tap into loneliness, social anxiety, and the desire for constant interaction, promising personalized conversation without judgment. Yet the rapid commercial rollout outpaces scholarly debate, leaving many users unaware of the deeper philosophical and ethical dimensions of treating a language model as a confidant.

Philosophers define friendship through a "caring constraint": both parties must care about each other for the other's sake and be disposed to act on that care. Grace Helton points out that current large language models lack consciousness, emotions, and non‑derivative interests, meaning they cannot experience or reciprocate genuine concern. While a human may feel attachment to an LLM, the AI cannot return care in a way that benefits the LLM itself, breaking the reciprocity essential to true friendship. This gap raises questions about the authenticity of AI‑human bonds and whether they merely simulate companionship without fulfilling its moral core.

The implications extend to policy and product design. Regulators may need to differentiate between tools that assist and those that masquerade as relational partners, potentially requiring disclosures about the limits of AI empathy. Designers could incorporate safeguards that prevent users from over‑relying on chatbots for emotional support, encouraging human‑to‑human interaction instead. As the debate evolves, integrating philosophical insights with technical development will be crucial to ensure AI serves as a complement—not a substitute—for genuine human connection.