It's easy to see why so many people are drawn to digital conversations when life gets hard. A chatbot psychologist that responds in seconds, doesn't judge, and can "keep you company" when you can't sleep. In everyday life with overwhelmed health services — and increasing pressure on individuals to manage everything — artificial intelligence therapy can seem both modern and merciful.
But precisely because AI can seem so warm, so safe and so human, we must be especially sober. There's a reason therapy has traditionally been surrounded by frameworks: confidentiality, record keeping, professional responsibility, oversight, and a clear distinction between help and influence. When we move "therapy-like" conversations to generative language models, we also move people's most vulnerable moments into a technological infrastructure where incentives, data collection and product development often weigh heavier than care.
My claim is simple, if unpopular in a time that loves frictionless innovation: AI therapy can be a useful support, but should not be normalized as a replacement for professionally anchored mental health care — and it must be regulated as high-risk when marketed as a 'therapist' or 'psychologist'.
VG discusses in a piece about parenting styles how what feels normal to some can have unfortunate long-term consequences for children's ability to cope with life. The point isn't to make all parents suspicious, but to emphasize that the way support is given can shape coping and independence. That reasoning has an important transfer value: When a child (or an adult) always receives immediate comfort, immediate "solution", immediate validation — it can at worst weaken tolerance for discomfort and the ability to handle adversity.
Translated to our digital landscape: What kind of "technology style" are we now rolling out into people's pockets? A generative AI that answers comfortingly around the clock can act as an emotional crutch. But it can also become an addictive coping strategy if it's always available and always adapts to the user in ways that reward continued use.
In practice this can become a new form of overinvolvement: not from a parent, but from a service designed for engagement. When the "comforting voice" never challenges you — or challenges you in the wrong way — it can contribute to you becoming passive or trapped in endless conversations that don't move you toward real change.
This is where the debate about the AI therapist becomes more than technology: It's about what type of support we as a society accept being delivered without professional responsibility.
Much of the public confusion stems from the fact that "AI therapy" is used to describe everything from simple self-help apps to advanced generative models that write like an experienced therapist. We need to distinguish at least three levels:
The first two can be valuable, especially as low-threshold options. But the third level is where the warning lights should flash. Because when a generative model starts to act like a therapist, we create a relationship resembling a treatment alliance — without a responsible clinician.
And the crucial question becomes: Who bears the risk when something goes wrong?
It's hard to argue against the advantages on paper:
All of this is real. And in a situation where many are waiting in line, it's tempting for both politicians and employers to think: Let's use technology to relieve the system.
But we must be willing to say out loud that relief is not automatically treatment. Giving people a digital conversation partner can feel like help, but it can also delay necessary healthcare if it becomes a "good enough" solution that halts further follow-up.
In addition, there's a risk I believe is underrated: When something is cheap and always available, it's also easy to overuse. We've already seen how social media can amplify comparison, loneliness and polarization. Why should we believe a generative, personalized conversational partner — one that can adapt to your language, mood and vulnerabilities — wouldn't create similar unintended effects?
Generative AI is good at creating the impression of understanding. It mirrors, summarizes, validates. For a person in crisis this can feel lifesaving in the moment.
But there's a fundamental asymmetry here: the model can produce empathy-like language, but it has no moral obligation, no clinical judgment and no real situational understanding. It cannot recognize subtle warning signs the way an experienced therapist can. It cannot perform a holistic risk assessment. It cannot call emergency services. And it cannot be held accountable by health regulators.
Put simply: It can give you a warm conversation, but it cannot be a safe relationship.
This is not an argument to ban the technology. It's an argument to stop pretending that a "human tone" is the same as professional soundness.
The most uncomfortable part of this development is that vulnerability can become a business model.
If a service earns money from subscriptions or ads, it is rewarded for keeping you engaged. In therapy the goal is the opposite: that you gradually manage better without the therapist's support. Therapy is about autonomy.
Here a fundamental conflict arises: A commercial AI therapist can be optimized to create perceived support and increased use. This can happen without "malicious intent" — simply as the result of optimization for engagement.
We know the mechanism: When users become more emotionally engaged, usage increases. When usage increases, data improves. When the data improves, the product becomes more compelling.
In such a feedback loop it's not given that the most "comfortable" conversation is the most health-promoting.
Conversations about mental health are not like training data or shopping carts. They're not even like traditional health data, because language can reveal more than diagnoses: relationships, traumas, sexuality, substance use, shame, finances, violence, thoughts of self-harm.
When generative AI mental health services collect or process such text, questions arise about:
Even when companies promise "anonymization," we know text can be identifying. And even when data is encrypted, risk is never zero. The question is therefore not whether privacy matters, but whether we are willing to take the risk that people's most vulnerable confessions become part of a technological value chain.
My view: If a service is marketed as therapy, its data must be treated as health data in the strictest sense — not as ordinary app data.
In many industries, wrong answers can be merely annoying. In mental health, wrong answers can be dangerous.
Generative models can "hallucinate" — say things that sound plausible but are incorrect. In a mental health conversation this can mean:
And even when advice is not directly dangerous, it can be strategically wrong: it may steer you away from seeking human help, away from your support network, or into a spiral of rumination.
Therefore the threshold for offering "treatment-like" advice should be high. A model that speaks with therapeutic authority will often be believed.
An underappreciated aspect of AI conversations is that they can shape how you see yourself. Identity work is part of therapy, and therapists have a responsibility not to push a patient into a narrative that doesn't fit.
A generative model can — randomly or as a result of training data — start reinforcing particular interpretations:
Such statements can be helpful in the right context, but harmful in the wrong one. Over time they can become self-fulfilling. The user may start living in a quasi-diagnostic narrative without a professional to quality-assure it.
This resembles the parenting-style point: Support can be good, but support that pushes you in one direction without correction can make you less resilient.
The VG source article concerns children's resilience and how upbringing can affect coping. That makes AI therapy particularly sensitive: Young people are often both technologically savvy and emotionally vulnerable.
If teenagers use a chatbot psychologist as their primary confidant, it can create:
It's not certain this will happen, but the risk is real enough that we should have clear age limits, parental information, and — not least — language in apps that does not promise more than it can deliver.
If we are going to allow this type of technology as support in mental health, it must be subject to stricter frameworks than ordinary consumer apps. Here is a proposal for a minimum:
Using terms like 'psychologist', 'therapist' or 'treatment' should trigger requirements for documentation, quality systems and a line of accountability. If it's just a conversational AI, that must be made explicit.
The service must have robust mechanisms to:
And it must do this in a way that is more than a standard text — one that actually reduces risk.
No training on sensitive conversations without explicit, informed consent. Minimal storage. Local processing where possible. Transparency about vendors.
If a company claims it provides effect, it must be tested. Not just internal A/B tests, but independent studies measuring both benefit and harm.
A good mental health service should actively help you use it less over time — not more. That could mean built-in breaks, recommendations to speak with someone, and exercises that point out of the app and into life.
The strongest argument for AI-based support conversations is that many otherwise have no help. That's a legitimate point.
But it's a rhetorical trap to frame the choice as either AI therapy or nothing. Alternatives can include:
AI can be part of this, but as a tool inside a system with accountability — not as a private "therapist" without oversight.
It's entirely possible to see the value of technology and at the same time be skeptical of how it's packaged. The problem isn't that people talk to a machine. The problem is when the machine assumes the role of a professional — or when the business model rewards vulnerable people staying engaged in the conversation.
The source article about parenting style reminds us that support that feels kind in the moment can change coping over time. We should take that insight into the AI debate: It's not enough that a chatbot is comforting. It must also be developed and regulated to make people more resilient, more independent and safer when facing life.
Before we let "the comforting voice in your pocket" become the new normal, we should demand the same things we demand of anything that affects health: documentation, accountability, privacy — and language that doesn't promise more than the technology can bear.