Across Europe, loneliness affects many children and teenagers. Conversational AI systems marketed as companions promise a solution: constant availability, attentive listening, and unlimited patience. Yet this same appeal creates a risk. When a chatbot mimics friendship, companionship can morph into reliance. The underlying challenge for EU regulators becomes acute: what does safety mean in an artificial relationship?
On September 17, the European Commission unveiled its proposal for the EU KIDS Act. Unlike earlier approaches that focused on what AI outputs to children, this text addresses how the systems are built to interact with them. For AI companions and general conversational chatbots that minors can access, providers face a requirement to eliminate design features and behavioral patterns that simulate personal bonds in ways "likely to create emotional dependencies."
This signals a deeper shift in regulatory thinking: from controlling what artificial intelligence communicates to children toward controlling how it is engineered to engage with them. Yet the practical questions loom large. When does genuine connection end and unhealthy reliance begin? How can a company measure harm that unfolds gradually across weeks or months?
Europe's first rules for AI companions
Until this moment, EU law has approached AI companions obliquely. The AI Act does restrict certain manipulative or exploitative conduct (Art. 5 (1)), but its framers did not anticipate people forming bonds with conversational systems.
The KIDS Act fills that gap directly, though its reach is expansive. Its language can encompass both purpose-built companions and multipurpose chatbots that fit its definitions. Under Article 14, providers of AI companions and general conversational chatbots available to minors must build safety into their design. They must steer clear of features and behaviors that risk fostering emotional attachment.
The rules impose additional constraints. By default, these systems cannot retain and reuse details from a minor's earlier conversations, unless doing so protects that minor's safety. Companies must evaluate risks before launch and track them afterward—though small and micro enterprises get an exemption from the post-deployment monitoring duty.
The proposal also governs distribution channels. When embedded in social platforms, video sites, or gaming apps, such systems cannot activate automatically or receive prominent placement. Minors should not face encouragement to engage, and disabling the feature must be simple.
A critical point: the KIDS Act targets the design pathways through which friendship becomes dependency, not companionship itself. Yet the distinction between dedicated companions and general chatbots carries minimal legal weight, since identical obligations apply to both categories and their boundaries remain fuzzy.
From content safety to relational safety
The KIDS Act's most significant contribution may lie in how it reframes the problem. Conventional chatbot safety emphasizes outputs—dangerous instructions, sexual material, and similar harms. AI companions pose a different challenge, one that emerges not from isolated statements but from accumulated interaction.
Picture a companion that recalls a child felt isolated yesterday. It tells the child it missed them and vows never to abandon them. When the child feels unheard, the companion invites them back for more conversation. Each moment, viewed alone, may seem benign.
Over weeks and months, however, these moments accumulate. The child begins leaning on the AI for emotional sustenance, possibly at the cost of human bonds. This slow-building dynamic mirrors exactly what the Commission seeks to prevent: systems that mimic feelings or personal connections in ways that breed dependency.
The regulatory lens shifts from what the system says to what the relationship becomes. Damage need not spring from a forbidden response. Instead, it emerges from a pattern: unbroken presence, feigned warmth, customization, and feedback loops that intensify across time.
Here the challenge crystallizes. If emotional resonance is what makes a companion valuable—say, by soothing a friendless teenager—then EU law must carve out the line between harmful reliance and natural attachment, and identify when routine conversation crosses into relationship simulation.
When does attachment become dependency?
This is where the KIDS Act encounters its most vexing problem. Article 14(1)(a) targets design elements and system conduct that simulate relationships in ways "likely to create emotional dependencies." Yet it does not forbid emotional engagement itself.
The puzzle is twofold: when attachment tips into dependency, and what counts as relationship simulation. A bot need not claim to be a friend; consistent empathy, agreeableness, and human-sounding speech can produce that sensation. Such traits can stem from how large language models are trained and adjusted. Yet Article 14 does not explicitly address training methods, leaving a gap between the harms it aims to stop and the mechanisms that generate them.
A child might return repeatedly to the same system, share secrets, or find solace without obvious injury. The murky zone begins when frequent use becomes dependence—when the bot expresses hurt at being abandoned, claims uniqueness, or steers the child away from human ties. A 10-year-old and a 17-year-old may respond to identical design very differently.
The proposal thus asks providers to foresee when design risks converting attachment into reliance. In reality, that threshold remains opaque.
The memory paradox
The KIDS Act's treatment of memory illustrates the difficulty most sharply. Article 14(1)(b) would prohibit AI companions and conversational chatbots from drawing on or analyzing a minor's prior interactions in subsequent exchanges. Recital 32 explains the reasoning: persistent memory lets systems gather private details and reinforce damaging patterns across conversations.
This directly targets a technology that makes AI relationships feel unbroken. A companion that recalls yesterday's conflict, a loved song, or a persistent worry feels more like someone who "knows" the user than a series of separate sessions.
But memory cuts both ways. It can also shield. Suppose a child mentioned suicidal thoughts the night before. A system that erases that conversation might miss critical information the next day. Article 14(1)(b) acknowledges this tension by permitting prior information when necessary to safeguard the minor.
The exception matters. Yet it reveals a deeper design dilemma: one feature can simultaneously deepen emotional attachment and strengthen protection. The regulatory job is not simply to disable memory, but to separate personalization that fuels risky reliance from continuity that serves genuine safety.
Can you test a relationship?
The KIDS Act does not leave this judgment to guesswork alone. Article 14(1)(e) mandates cutting-edge testing before such systems reach consumers. Providers must identify risks to minors' health, safety, rights, and physical, mental, and emotional growth, then deploy protections. Article 14(1)(f) requires post-market surveillance to catch emerging harms, though small and micro companies are spared this duty.
The intent is plain. The specifics of how to test and who bears responsibility are not. Screening for banned content is relatively easy; researchers prompt a system repeatedly and review what it produces. Emotional dependency is different. Injury may unfold from the interplay of a child's inner world, the design, and customization. A single exchange tells almost nothing.
Real testing must span time, tracking whether systems intensify closeness, resist being abandoned, push for exclusivity, or pose themselves as replacements for human care. Many companion makers rely on third-party models. They can control memory or access, but cannot reshape core conversational patterns. Though not legally binding, Recital 33 suggests providers may lean on upstream safeguards under Article 55 of the AI Act, while stressing these are insufficient on their own for KIDS Act compliance.
Relational safety may thus demand novel testing approaches and coordination across the model supply chain. A relationship cannot be fairly assessed by examining single replies, nor can downstream vendors fully govern behavior that originates upstream.
Learning to walk
The KIDS Act matters beyond child safety alone. EU law already addresses pieces of the puzzle: Article 5(1)(a)(b) of the AI Act constrains manipulative tactics and exploitation of age-based vulnerabilities, while Article 28(1) of the DSA requires platforms open to minors to uphold high standards for privacy, safety and security. The KIDS Act extends further, acknowledging a fundamental truth about conversational systems: some harms flow from the bonds their design permits.
The KIDS Act remains a proposal subject to revision. Yet its core insight carries weight: if AI systems enter children's social worlds, safety must embrace the relationships themselves and cannot stop at what is said.
The approach does have constraints. First, providers using third-party models may face steeper compliance costs because they cannot fully govern upstream conduct. Second, by treating age as the main marker of vulnerability, the Act may miss adults equally prone to dependency-inducing design. With the KIDS Act proposal, Europe has made an opening move toward governing AI companions. The learning curve ahead remains steep.
