AI systems are no longer just tools. They talk like us, respond to our emotions, and increasingly fill the roles of therapists, companions, and romantic partners. In 2025, personal and emotional support overtook content creation as the number one use case for generative AI. That shift raises urgent questions about what happens when machines become too human and when we forget they aren’t. In this article, you will learn why anthropomorphic AI systems need limits and why transparency rules alone are not enough.
An article by Valerie Bohn
We Are Wired to Anthropomorphize
Have you ever found yourself talking to a non-human object? Have you ever yelled at a machine because it wasn’t working, or begged it to do what it should? Or perhaps you recently thanked the AI chatbot of your choice for its responses, even though you know that an LLM doesn’t care whether you say thank you or not?
This is what is called anthropomorphism: The human tendency to attribute human characteristics to non-human objects. More than 80 years ago, Heider and Simmel showed that people interpret social motivations from geometric shapes just by observing how they move. And ever since Joseph Weizenbaum introduced Eliza, it became very clear – much to his surprise – that it doesn’t take much for people to reveal emotions toward a machine. We cannot help it: when something moves, speaks, or responds to us, our social brain engages.
Companion apps like Replika, which today counts over 40 million registered users, are explicitly designed around this dynamic. But even AI tools not built for companionship, such as ChatGPT, Claude or Gemini, are increasingly used for emotional support and intimate conversation, as thriving online communities like Reddit’s “r/MyBoyfriendIsAI” illustrate. According to a large case study by Marc Zao Sanders from 2025, personal and emotional support overtook content creation as the number one use case for generative AI. The result is something more than mere user engagement. As AI becomes more human-like in appearance and behavior, users are more likely to attribute mental states, intentions, and feelings to it that it simply does not possess.
When we interact with modern AI systems, which have become increasingly human-like in their manner of expression, we are highly exposed to the phenomenon of anthropomorphizing. Large language models can engage in nuanced conversations, display apparent empathy and can give us the feeling that someone is listening to us. They create an illusion of sentience: The misleading perception of interacting with a human being.
An Illusion That Survives the Truth
What makes this phenomenon particularly challenging is that the illusion of sentience may be enlightenment-resistant. Even users who fully understand the technical background, may still find themselves reacting emotionally as if they were interacting with a conscious being. Much like the well-known Müller-Lyer illusion persists after we’ve been told the lines are equal, the emotional impression of interaction with a conscious entity can survive rational knowledge to the contrary – knowledge does not dissolve the perception. In the same way, knowing that an AI cannot feel does not prevent users from feeling for it.
Philosopher Catrin Misselhorn describes the phenomenon of reacting emotional towards non-human objects through the concept of imaginative perception: a blending of perception and imagination that allows us to experience an object as if it were sentient or alive. This mechanism is not a cognitive failure. It is a feature of human social cognition.
The Risks of Emotional Dependency
Anthropomorphic features in AI-systems significantly increase user engagement and trust, which is precisely why they can be very attractive for AI-development. But the same trust that makes AI accessible also creates dependencies that can be leveraged against users’ interests.
Users of companion apps share deeply personal information, forming one-sided emotional bonds, and in some cases substitute AI relationships for human ones. Moreover, research suggests that heavy reliance on AI for social interaction may reinforce negative social habits, reduce face-to-face communication skills, and diminish the capacity to navigate conflict and disagreement. AI companions do not challenge users, do not disagree with them, or hold them accountable. They keep the user in a warm blanket of confirmation. Over time, that comfort may come at the cost of social resilience.
In extreme cases, the consequences have been fatal. There are already several documented cases of suicides and so-called AI psychoses resulting from interactions with human-like large language models. This already resulted in multiple court cases against larger companies such as Google and OpenAI, as well as smaller providers like Character.AI.
The rise of anthropomorphic AI systems, which are used to address the social needs we all share, gives rise to a structural problem: Companies have strong financial incentives to design systems that maximize emotional attachment. The more a user bonds with an AI, the more they return, the more data they share, and the more they are ready to pay for such a system. Left to market forces alone, incentives point toward deeper anthropomorphism, not less.
The Regulatory Landscape of Anthropomorphic AI
While the EU AI Act suggests transparency rules and a broad ban on manipulative and exploitative AI, it does not explicitly address the risks posed by anthropomorphic AI systems, such as emotional dependency. This leaves manufacturers largely guessing which standards will apply under EU product liability, which has recently been extended to include psychological harm and AI as a source of harm.
China on the other hand already moved to cover these risks explicitly in its product safety regulation: In December 2025, the Chinese Cyberspace Administration released a draft law specifically targeting anthropomorphic AI interaction services, prohibiting systems from making false promises that impair user behavior, fostering emotional dependency, damaging interpersonal relationships, or facilitating self-harm. The law came into effect a few weeks ago and is the first regulation to address anthropomorphism in AI directly.
Given that the illusion of sentience is potentially enlightenment-resistant and our human tendency to anthropomorphize and emotionalize human-like machines persists even when we understand the technical underpinnings, mere transparency rules are insufficient. The Chinese approach that explicitly regulates the anthropomorphic characteristics of AI systems seems to be a reasonable way to mitigate the risks.
Disrupting the Illusion by Design
From a developer’s perspective, the most obvious way to mitigate the risks posed by anthropomorphic AI is to avoid incorporating unnecessary human traits into AI systems. AI voice outputs, for example, do not need to laugh, sigh, or yawn to be a helpful tool for us. Such traits only lead to unnecessary emotional responses from the user.
In my PhD research, I am exploring an approach that traces back to the author and theatre scholar Bertolt Brecht. Brecht influenced dramaturgy through his famous “alienation effect”. This effect deliberately and regularly breaks the illusion of a play, reminding the audience that they are not part of the action. This emotional distancing enables rational reflection on the events. Implementing a kind of alienation effect in humanoid AI systems could be helpful in constantly reminding users that they are interacting with a machine.
A more comprehensive approach suggests a general social compatibility test for AI systems. Since the Turing Test was developed, AI development has primarily sought to make AI as human-like as possible. But instead of asking “Can machines think?”, it would probably be better to ask “Can machines contribute to the social common good?” This idea is pursued under the Star-Durkheim Test, conceived by Susanne Draheim, Ulrich Furbach, and Ralf Möller.
Regulation Must Go Further Than Disclosure
Because humans anthropomorphize automatically and because companies profit from emotional bonds, the burden should not only rest with individual users to resist the illusion. Whilst transparency and informed consent are important, they will not be sufficient to protect users from emotional attachment and potential addiction or dependency.
Of course, we should do our best to ensure that we don’t lose the major advantage of having a natural conversation with a helpful tool. However, the human-like qualities that we can already observe in LLMs today, and which we will eventually see in social robots as well, are not necessary and, in fact, rather dangerous.
What is needed is regulation that targets the design directly: standards that limit unnecessary anthropomorphic features, require illusion-disrupting interaction elements for high-risk use cases, and hold developers accountable for the emotional dependencies their systems generate. China’s 2025 draft legislation points in this direction. The EU should establish explicit rules for anthropomorphic AI systems. The approach should include a general distinction between entertainment contexts involving limited risk and companion systems specifically designed to foster a lasting emotional bond.
The Turing Test asked whether a machine could be indistinguishable from a human. That was never the right question. The right question should be whether a machine is good for the humans who use it. Answering that question well is the work ahead.
