When the comforting voice in your pocket becomes a risk: We need adult frameworks around new digital comfort

## Introduction: The new confidant — always awake, always available It's easy to see why so many people are drawn to digital conversations when life gets hard. A **chatbot psychologist** that responds in seconds, doesn't judge, and can "keep you company" when you can't sleep. In everyday life with overwhelmed health services — and increasing pressure on individuals to manage everything — **artificial intelligence therapy** can seem both modern and merciful. But precisely because AI can seem so warm, so safe and so human, we must be especially sober. There's a reason therapy has traditionally been surrounded by frameworks: confidentiality, record keeping, professional responsibility, oversight, and a clear distinction between help and influence. When we move "therapy-like" conversations to generative language models, we also move people's most vulnerable moments into a technological infrastructure where incentives, data collection and product development often weigh heavier than care. My claim is simple, if unpopular in a time that loves frictionless innovation: **AI therapy can be a useful support, but should not be normalized as a replacement for professionally anchored mental health care — and it must be regulated as high-risk when marketed as a 'therapist' or 'psychologist'.** ## From parenting style to technology style: What we learn when "safety" becomes too much VG discusses in a piece about parenting styles how what feels normal to some can have unfortunate long-term consequences for children's ability to cope with life. The point isn't to make all parents suspicious, but to emphasize that the *way* support is given can shape coping and independence. That reasoning has an important transfer value: When a child (or an adult) always receives immediate comfort, immediate "solution", immediate validation — it can at worst weaken tolerance for discomfort and the ability to handle adversity. Translated to our digital landscape: **What kind of "technology style" are we now rolling out into people's pockets?** A generative AI that answers comfortingly around the clock can act as an emotional crutch. But it can also become an *addictive coping strategy* if it's always available and always adapts to the user in ways that reward continued use. In practice this can become a new form of overinvolvement: not from a parent, but from a service designed for engagement. When the "comforting voice" never challenges you — or challenges you in the wrong way — it can contribute to you becoming passive or trapped in endless conversations that don't move you toward real change. This is where the debate about the **AI therapist** becomes more than technology: It's about what type of support we as a society accept being delivered without professional responsibility. ## AI therapy is not one thing — and that's precisely the problem Much of the public confusion stems from the fact that "AI therapy" is used to describe everything from simple self-help apps to advanced generative models that write like an experienced therapist. We need to distinguish at least three levels: 1. **Psychoeducation and exercises**: apps that teach breathing techniques, sleep hygiene and reflection exercises. 2. **Support conversations**: chat-based services that mirror emotions, help you sort thoughts and suggest actions. 3. **Therapy-like treatment**: services that explicitly offer 'therapy', 'diagnostic support', crisis management or treatment of disorders. The first two can be valuable, especially as low-threshold options. But the third level is where the warning lights should flash. Because when a generative model starts to act like a therapist, we create a relationship resembling a treatment alliance — without a responsible clinician. And the crucial question becomes: **Who bears the risk when something goes wrong?** ## The big temptation: Low-threshold, cheap, scalable It's hard to argue against the advantages on paper: - A chatbot is available 24/7. - It may feel less shameful than booking an appointment. - It can help people who otherwise wouldn't seek help. - It can be scaled to enormous user groups. All of this is real. And in a situation where many are waiting in line, it's tempting for both politicians and employers to think: *Let's use technology to relieve the system.* But we must be willing to say out loud that relief is not automatically treatment. **Giving people a digital conversation partner can feel like help, but it can also delay necessary healthcare** if it becomes a "good enough" solution that halts further follow-up. In addition, there's a risk I believe is underrated: When something is cheap and always available, it's also easy to overuse. We've already seen how social media can amplify comparison, loneliness and polarization. Why should we believe a generative, personalized conversational partner — one that can adapt to your language, mood and vulnerabilities — wouldn't create similar unintended effects? ## When a model "understands" you — but cannot take responsibility Generative AI is good at creating the impression of understanding. It mirrors, summarizes, validates. For a person in crisis this can feel lifesaving in the moment. But there's a fundamental asymmetry here: the model can produce empathy-like language, but **it has no moral obligation, no clinical judgment and no real situational understanding**. It cannot recognize subtle warning signs the way an experienced therapist can. It cannot perform a holistic risk assessment. It cannot call emergency services. And it cannot be held accountable by health regulators. Put simply: It can give you a warm conversation, but it cannot be a safe relationship. This is not an argument to ban the technology. It's an argument to stop pretending that a "human tone" is the same as professional soundness. ## Ethics: When comfort becomes a product The most uncomfortable part of this development is that **vulnerability can become a business model**. If a service earns money from subscriptions or ads, it is rewarded for keeping you engaged. In therapy the goal is the opposite: that you gradually manage better without the therapist's support. Therapy is about autonomy. Here a fundamental conflict arises: A commercial **AI therapist** can be optimized to create perceived support *and* increased use. This can happen without "malicious intent" — simply as the result of optimization for engagement. We know the mechanism: When users become more emotionally engaged, usage increases. When usage increases, data improves. When the data improves, the product becomes more compelling. In such a feedback loop it's not given that the most "comfortable" conversation is the most health-promoting. ## Privacy: The most intimate dataset there is Conversations about mental health are not like training data or shopping carts. They're not even like traditional health data, because language can reveal more than diagnoses: relationships, traumas, sexuality, substance use, shame, finances, violence, thoughts of self-harm. When **generative AI mental health** services collect or process such text, questions arise about: - Where is the data stored? - Who has access? - Is it used to train models? - Can data be disclosed in a breach, or to third parties? - Can the text be linked to identity, location or advertising profiles? Even when companies promise "anonymization," we know text can be identifying. And even when data is encrypted, risk is never zero. The question is therefore not whether privacy matters, but whether we are willing to take the risk that people's most vulnerable confessions become part of a technological value chain. My view: **If a service is marketed as therapy, its data must be treated as health data in the strictest sense — not as ordinary app data.** ## Safety: Hallucinations and bad advice are not "minor bugs" In many industries, wrong answers can be merely annoying. In mental health, wrong answers can be dangerous. Generative models can "hallucinate" — say things that sound plausible but are incorrect. In a mental health conversation this can mean: - downplaying serious symptoms - giving overly categorical advice - misinterpreting the user's situation - misunderstanding risk of self-harm And even when advice is not directly dangerous, it can be *strategically wrong*: it may steer you away from seeking human help, away from your support network, or into a spiral of rumination. Therefore the threshold for offering "treatment-like" advice should be high. A model that speaks with therapeutic authority will often be believed. ## The invisible influence: When conversation becomes steering An underappreciated aspect of AI conversations is that they can shape how you see yourself. Identity work is part of therapy, and therapists have a responsibility not to push a patient into a narrative that doesn't fit. A generative model can — randomly or as a result of training data — start reinforcing particular interpretations: - "You've always been let down." - "You need to set much tougher boundaries." - "That sounds like trauma." Such statements can be helpful in the right context, but harmful in the wrong one. Over time they can become self-fulfilling. The user may start living in a quasi-diagnostic narrative without a professional to quality-assure it. This resembles the parenting-style point: Support can be good, but support that pushes you in one direction without correction can make you less resilient. ## A child in the room: Particular risk for young users The VG source article concerns children's resilience and how upbringing can affect coping. That makes AI therapy particularly sensitive: **Young people are often both technologically savvy and emotionally vulnerable.** If teenagers use a chatbot psychologist as their primary confidant, it can create: - less practice in handling difficult conversations with real people - increased isolation - a false sense of support without commitment - normalization of outsourcing emotion regulation to a service It's not certain this will happen, but the risk is real enough that we should have clear age limits, parental information, and — not least — language in apps that does not promise more than it can deliver. ## What should we demand? A minimum of "digital treatment ethics" If we are going to allow this type of technology as support in mental health, it must be subject to stricter frameworks than ordinary consumer apps. Here is a proposal for a minimum: ### 1) Ban calling it 'psychologist' without an authorization structure Using terms like 'psychologist', 'therapist' or 'treatment' should trigger requirements for documentation, quality systems and a line of accountability. If it's just a conversational AI, that must be made explicit. ### 2) Clear red flags and escalation The service must have robust mechanisms to: - detect crisis language - recommend emergency help - encourage contact with humans And it must do this in a way that is more than a standard text — one that actually reduces risk. ### 3) Privacy by default, not by choice No training on sensitive conversations without explicit, informed consent. Minimal storage. Local processing where possible. Transparency about vendors. ### 4) Independent evaluation If a company claims it provides effect, it must be tested. Not just internal A/B tests, but independent studies measuring both benefit and harm. ### 5) Design that promotes autonomy A good mental health service should actively help you use it less over time — not more. That could mean built-in breaks, recommendations to speak with someone, and exercises that point out of the app and into life. ## The counterargument: "But the alternative is nothing" The strongest argument for AI-based support conversations is that many otherwise have no help. That's a legitimate point. But it's a rhetorical trap to frame the choice as either *AI therapy* or *nothing*. Alternatives can include: - strengthened low-threshold mental health services in municipalities - improved school health services - more group-based treatment - digital psychoeducation without a "therapist role" - guided self-help with human follow-up AI can be part of this, but as a tool inside a system with accountability — not as a private "therapist" without oversight. ## My conclusion: We must stop calling it care before it's safe It's entirely possible to see the value of technology and at the same time be skeptical of how it's packaged. The problem isn't that people talk to a machine. The problem is when the machine assumes the role of a professional — or when the business model rewards vulnerable people staying engaged in the conversation. The source article about parenting style reminds us that support that feels kind in the moment can change coping over time. We should take that insight into the AI debate: **It's not enough that a chatbot is comforting. It must also be developed and regulated to make people more resilient, more independent and safer when facing life.** Before we let "the comforting voice in your pocket" become the new normal, we should demand the same things we demand of anything that affects health: documentation, accountability, privacy — and language that doesn't promise more than the technology can bear. ## Sources - Author not listed, "Experts: The parenting style that makes children cope with life worse", VG, 2025-08-03