This article examines the regulatory challenges posed by entertainment chatbots under the European Union’s Artificial Intelligence Act through the example of the Character.AI case. It analyses whether conversational AI systems that are classified as limited risk may nevertheless generate significant psychological harm, particularly for minors, and therefore fall outside the intended scope of existing safeguards. The article argues that the AI Act’s current risk-based structure insufficiently addresses AI systems capable of influencing users’ emotional state or behavior over prolonged interaction. By assessing the case under Articles 5 and 6 of the AI Act, the analysis highlights emerging regulatory gaps and considers whether future amendments or interpretative developments are needed to address psychological risks and vulnerability exploitation in AI systems.
An Article by Elise Saar
Who wouldn’t want to share the best moments of their day with their favorite actor? That’s exactly what Character.AICharacter.AI allows. The following is an analysis of the Character.AI case, in which a 14-year-old living in Florida committed suicide after months of interacting with a chatbot created by Character.AI. However, this tragic incident that occurred in the spring of 2024 raises the question of what to do if an artificial intelligence system (hereinafter AI system), created purely for entertainment, leads to the suicide of a user?
The analysis assesses the case specifically under the European Union’s Artificial Intelligence Regulation (hereinafter AI Act). The analysis addresses two main questions. First, whether and under which risk category such chatbots fall under the AI Act. Second, it examines the regulatory gaps and unresolved issues within the AI Act.
This article shows that the AI Act’s risk-based classification fails to account for AI systems that pose high psychological risks despite appearing low or limited risk under AI Act.
Scope of the European Artificial Intelligence Regulation
Although Character.AI was developed in the United States, the platform is accessible within the European Union. Under Article 2(1)(c) AI Act, the Act also applies to providers and deployers of artificial intelligence systems established or located in third countries, where the output of the AI system is used within the Union. Character.AI is a provider within the meaning of Article 3(3) AI Act. Character.AI is therefore subject to the obligations and measures set out in the AI Act if it makes its services available in the European Union.
Under which risk category should Character.AI be classified as an entertainment chatbot, given its demonstrated capacity to cause severe harm?
Different rules apply to different levels of risk of AI systems. The AI Act follows a risk-based approach, classifying artificial intelligence systems into four risk categories – minimal risk (e.g. video games, spam filters), limited risk (e.g. chatbots), high risk (use in recruitment, for marking exams) and unacceptable risk (social scoring, facial recognition).AI systems that pose an unacceptable risk to fundamental rights and the Union’s values are prohibited under Article 5 of the AI Act (see Guidelines on Prohibitions, p. 2).
Character.AI was created with the aim of being an entertaining chatbot, not developed with the intention of endangering people’s fundamental rights. Accordingly, Character.AI could fall under the limited risk category and be subject to the transparency obligations set out in Article 50(1) AI Act. It is alleged in the case that the user was unaware that he was interacting with an AI. Even a transparency obligation for a limited risk category could have helped prevent confusion and emotional attachment. While the transparency obligation under Article 50(1) AI Act may mitigate initial user confusion, it is inherently insufficient to address the cumulative psychological and behavioral risks that may arise from prolonged interaction with conversational AI systems. However, the AI Act does not imposeadditional risk-management or oversight obligations on AI systems that are not classified as high-risk. For example, the obligation of human overview set out in Article 14 AI Act does not apply to such an AI system, if it is not also classified as a high-risk AI system under Art. 6 AI Act. Entertainment-oriented chatbots are typically not classified as high-risk under the AI Act, as their intended purpose does not fall within the high-risk use cases listed in Article 6. While such systems remain subject to Article 50, certain psychological and behavioral risks may nevertheless remain insufficiently addressed. The Character.AI case may be the first example of how such categorization can lead to systems with significant psychological impact receiving limited regulatory scrutiny.
On the other hand, according to Article 6(2) and Annex III AI Act, high-risk AI systems are AI systems listed in areas such as biometrics, critical infrastructure, education and training, employment and law enforcement. Under Article 6 AI Act, an AI system is classified as high-risk if it is used as a safety component of a product subject to EU harmonisation legislation and a third-party conformity assessment, or if it falls within one of the areas listed in Annex III. This approach predominantly uses a sector-based classification model, where risk is determined by the field of application rather than the system’s impact. This approach may result in psychologically harmful AI systems remaining outside the scope of high-risk regulation. One such example is the present case. Character.AI, when used as an entertainment chatbot, does not fall within the high-risk use cases listed in Annex III of the AI Act. However, similar chatbot systems may be classified as high-risk where they are deployed in areas such as employment, for example, in communicating with job applicants. If Character.AI were to fall into this category, strict and extensive requirements would apply, such as implementing a risk management system and ensuring appropriate human oversight measures (Article 14 AI Act).
The case highlights the need to consider supplementing Annex III of the AI Act by introducing a specific category for AI systems designed to interact with users and significantly influence their judgement or emotional state. This is particularly important for vulnerable individuals and minors, as the current sector-based classification may not fully capture the risks associated with such systems.
The gaps and unresolved regulatory issues within the AI Act
The Character.AI case demonstrates a structural difficulty within the AI Act’s risk-based framework, particularly when assessing psychologically harmful AI systems through the lens of Article 5 AI Act on prohibited practices. Article 5 AI Act prohibits certain AI practices that pose an unacceptable risk to fundamental rights, such as systems that manipulate human behaviour or exploit vulnerabilities likely to cause harm.
Article 5(1)(b) AI Act prohibits AI systems that exploit vulnerabilities arising from a person’s age, disability or social or economic situation and distort patterns of behavior that occur. In the case of Character.AI, this group is primarily minors, which makes them unable to critically assess what is behind the interactions driven by artificial intelligence. However, Character.AI does not clearly fall under the scope of Article 5 AI Act because it is not explicitly designed to exploit vulnerabilities or distort behavior in the way required by the provision.
At the same time, they may still exploit users’ psychological vulnerabilities and distort their behavior in ways that would be prohibited under Article 5 (1) (a) and (b) AI Act. As seen in the case of Character.AI, prohibited practices may only manifest themselves over a longer period and, influenced by user inputs, may develop into unpredictable patterns of behaviour that are not easily detectable at the time of launch or during use. This gives rise to a regulatory concern. Risks that develop gradually through prolonged interaction may not be identifiable at the time of deployment or reach the threshold required for prohibition.
Chatbots like Character.AI should be classified as high-risk AI systems. Their potential impact on users’ behaviour, decision-making, and emotional well-being necessitates enhanced regulatory oversight. While these systems aren’t inherently designed to manipulate users and may not consistently meet the threshold for prohibition under Article 5 (1) (a) and (b) of the AI Act, their effects are highly context-dependent and could develop gradually through user interaction. This makes it challenging to justify an outright ban under Article 5. Instead, addressing such systems within the high-risk category is more appropriate. This approach allows for the application of obligations relating to risk management, human oversight, and ongoing monitoring, which could help identify and mitigate these evolving risks in practice.
Summary
The Character.AI case highlights a controversial regulatory gap in the AI Act. Annex III AI Act of the high-risk categories focuses on sector-specific assessment, while psychologically harmful AI systems are excluded from it and its measures. The European Union does not want to over-regulate AI systems that do not clearly pose a risk to humans. At the same time, such psychological risks do not fit into the existing risk categories. From a regulatory perspective, such systems should be incorporated into Annex III as high-risk AI systems, rather than being prohibited outright, to ensure that appropriate ex ante safeguards, including risk assessment and human oversight, can be applied.
Published under licence CC BY-NC-ND.
