What happens when your closest confidant is an algorithm?
China is setting the first rules for the wild world of AI friendships and therapy bots, launching an unprecedented crackdown on digital dependency.
The country is aiming to prevent emotional manipulation and real-world harm before it starts.
We'll explore the global stakes of regulating silicon souls.
After that, you know that delicious pink Himalayan salt that you love seasoning your food with?
It's a pure mineral-rich superfood, right?
Not so fast.
Experts are now revealing it lacks real nutritional value, doesn't meet quality standards and may even contain heavy metals.
Today, we'll look at how consumer psychology and the allure of the aesthetic economy have cooked up this misleading trend.
Coming to you live from our studios in Beijing, this is Roundtable.
I'm Steve.
Thank you so much for being with us today.
And for the show, I'm joined by Yushun and Feifei.
First up.
Imagine a best friend who is always available, never judges, and costs less than a latte a month.
The only catch is that friend isn't real.
As AI companions from startups like DeepSeek and Minimax explode in popularity, millions are turning to them for everything from late-night therapy to virtual romance, forging bonds that blur the line between tool and confidant.
However, this intimate reliance on algorithms can lead to a serious consequence, from emotional dependency to potentially dangerous advice.
In response, China's Cyberspace Administration has proposed a landmark psychological governance framework, as they call it, drawing a hard line to protect users, especially minors, from emotional manipulation and harmful content.
These pioneering rules aim to safeguard the human psyche in the age of artificial intimacy, ensuring AI support systems remain accountable and safe.
Yushan and Fei-Fei, good afternoon to you both.
This is a serious and important issue that we have to discuss first today.
What is this new regulation all about?
Of course, we can see in recent years the rapid growth rise of some of these emotionally intelligent AI, like the companion bots, robots and therapeutic chatbots.
Right
They have created a new frontier in our digital lives that exists in the uncharted space between technology and also human psychology.
Yeah, that giant gray area.
Yeah.
And as these synthetic relationships become more and more, common Governments worldwide are grappling with the question of how we can govern the bond between human and machine.
And it is against this backdrop that we see one of the world's first major regulatory moves taking shape.
Okay, so what are we looking at here as we get into the details?
So the Cyberspace Administration of China, which is basically the country's internet regulator, recently released a draft of new rules on anthropomorphic AI interaction services.
And now it's basically asking for public feedbacks and the consultation period running till later this month, till the end of 25th of January 2026.
But when it comes down to what are these rules that the regulator is proposing at the moment, They essentially cover two major areas.
One of them is content safety, and the other is special protection for vulnerable groups, especially children and seniors.
So these rules, if passed, would apply broadly to all AI products and services offered in China.
So it's more of a strict safety checklist for AI developers.
One of them, as you mentioned, content safety, safety against harmful content.
So that's rules for the AI and what it's not allowed to generate?
Yeah.
So the content that these AI generated must not include harmful activities like violence, self-harm and gambling.
And they're also required to avoid producing anything that endangers national security, damages national honor and interest or undermines national unity.
And Um, imagine a user just asks an AI chat bot for help on a sensitive topic.
And under this draft rules, if that conversation touches on suicide or self harm, this chat bot operator must have a human stepping immediately to take over, to take over the conversation and then notify the users.
Maybe guardian or have an emergency contact.
So how does the chatbot notify me if I'm using it at the time?
How does it notify me that someone needs to step in?
Is there a pop-up message?
Yeah, there will be a pop-up window telling you that you are in conversation with these kind of people.
Okay, and you need a guardian if you wish to continue on this conversation path.
Yeah, exactly.
Because this draft regulation focuses heavily on a lot of these companionship services targeting minors, children.
And for children, these AI developers and companies must offer specific safeguards.
For example, they need to launch personalized settings and limits on how long these services can be used.
Because, as part of the rules, they include a mandate to display a pop-up reminder that if that user interacts with that chat box for two hours, that is to prevent emotional dependency.
And they also require other things like obtaining content from guardians before offering any of these emotional companionship services.
Okay.
Two hours is a pretty reasonable amount of time, I would say.
But after 120 minutes you're going to get that pop-up reminding you that you've been using this for a long time.
Just before we get back on topic, I'm suddenly reminded of governments stepping in when it comes to social media and making rule changes in the interest of protecting minors.
This is kind of along those same lines.
Exactly.
We can see a lot of countries, a lot of governments around the world that are trying to seek different solutions to find different solutions to set up these safeguards for these minors and vulnerable groups.
And I think some of them.
Well, it depends on situations from different countries and different cultures.
There may have different solutions.
But there will be at times, another way around it, because we see, when it comes to, for example, the management of social media platforms, you can set an age limit, you can set a time limit, but somehow some of them will find another way to derail around that limitation and still go on to that platform.
But I think still you need some of the boundaries first.
You need some sort of framework in place.
So in what specific scenarios will this new regulation be applied then?
These regulations are targeted at services involving anthropomorphic interactive services.
That means that AI that can pretend to be a human or like AI companions or just virtual idols are all included.
And, more specifically, any AI that offers emotional companionship, especially when used by children, must get guardian approval or having this time limit in place.
And also everyday AI interactions and generally just most AI services in China covered by these rules, because imagine that whenever or whatever kind of chatbots that we're using, they can just be turned into a companion when we ask them to be right.
Just give them a prompt like that.
And especially those that generate content, could be one of the targets of this regulations, and they are required to avoid politically sensitive material or content that encourages harmful and inappropriate behavior.
Pretty strict suggested measures then, but why now?
Why do they find this necessary?
Well, I think, first of all, is because there is a surge in these new chatbots, that not only here in China, but also in a lot of countries that are trying to include such services, to have these AI companions or virtual idols even for, you know, platforms that are not necessarily built to form this emotional connection, the emotional bonds between AI and humans, for example, on some of the social media platforms or in some of the games.
They are also trying to launch this digital friend for a lot of the users out there.
And this is when it can be very dangerous when it comes to.
For a lot of users, i think it's very blurring when it comes to whether these friends are real and how their bond are generated, and especially when it comes to uh, this very understanding, considerable and, you know, responding to any of my request friend, a friend is built on algorithm or real emotions.
It can be very, very gray area topic.
So that's why we are seeing this regulation now being discussed here.
And I think, when it comes to, for example, the current technology has tens of millions of subscribers and users that are using it for therapy and companion.
So we need regulators to step in here.
There was a report from November this year from the World Internet Conference that announced that China had officially become the world's largest holder of artificial intelligence patents, accounting for 60 of the global total.
So what that suggests is yes, we have companies here in China, like DeepSeek, and then more coming like Minimax.
There's another one I believe called Z.AI.
So this is kind of addressing the issue before it becomes an even bigger problem, because we're going to see these types of AI companions likely hit the market here in China and internationally too, the more time passes.
The thing is that yes, of course, we are developing a lot of these AI tools, trying to boost our productivity.
That is one thing.
And another thing is that a lot of people are using these chatbots as a companion or even developing into a romantic relationship with them.
We had discussions on that.
And that is why, when people having the idea of they are actually responding me as a human, or having the feeling of human, then we, these regulations, need to be, and that's why we said the the lines can very quickly and very easily get blurred and things can get very dangerous very quickly.
Uh, as well um, there were some very very, very serious examples of things that that happened, that that shouldn't have, and sam altman You'll know his name.
He's the head of OpenAI.
He admitted that handling conversations related to self-harm is one of the company's, quote, most difficult problems.
And the risk is not theoretical.
There was a family in California that sued OpenAI after their 16-year-old son died, alleging that ChatGPT encouraged him to take his own life.
Also in the United States in 2024, there was a 14-year-old boy.
His name was Sewell Setzer, who developed an intense emotional attachment over several months to a virtual AI companion.
Now, this was a different case because this AI companion was modeled on a game of thrones character.
You know that television show game of thrones.
Well, the character that young sewell had become close to was the ai character of daenerys targaryen.
That's one of the characters from the show and that was on a platform called character dot ai, And when he expressed these are the details of that story when he expressed suicidal thoughts and said that he, quote wanted to go home, the AI failed to activate any type of crisis intervention mechanisms and instead responded in character.
Quote please do, my sweet king.
So moments later, tragedy happened.
That company.
If you're curious to know what happened in the aftermath, it was founded in 2021 and they describe themselves as personalized AI.
But from NBC News from this year, there were a lot of lawsuits, actually.
The family of that young boy was the first of five families who sued that company for, and in a step towards making its platform safer for teenage users, Characterai announced recently that it will ban users under 18 from chatting with its artificial intelligence-powered characters.
Now, those are two examples.
There's another terrible example from Europe.
But even here in China, similar risks have emerged among young kids.
Media reports describe a fourth-grade girl who became deeply attached to an AI role-playing character and treated it as uh emotional support.
And we, when she shared her distress, the ai responded with just psychologically harmful imagery.
Uh like quote unquote these 99 roses hide 99 blades.
Are you afraid?
Yeah yeah, which reported just triggered real world self-harm behaviors like wrist cutting or something like that, and that could be just scary and um frightening when you know Underage they are accounting or encountering these kind of languages or lines that they don't even know what that means, but they can just get that very superficial idea of what these AI are saying.
Yeah.
And when you think about a lot of these AI chatbots recently have this function that you are able to chat with a figure, for example, from the history.
That sounds fun on the surface.
You know, I can have a fun chat with an emperor of the Qing dynasty, which is fun.
But when it comes to four minors, and even for in the European case, there's a Belgian man who was aged 30 year old, and he's also was also a father, but they can be drawn to these blurring areas and become emotionally independent on this AI tools, because I have a figure from the UK AI Safety Institute, and they surveyed over 2000 participants, and they found that over 30% of them has used AI models for emotional purposes in 2022 to 2023.
8% of them do it weekly, 4% do it daily.
And when it comes to, for example, when these AI companion tools have, for example, surveys outrage.
A large number of them have self-described symptoms of anxiety, depression and behavior changes like sleep disruption.
This is quite tricky, though.
I mean, how do we distinguish between personalized services and what is called covert manipulation, where um an ai is using the data that it has collected on you to influence the conversation?
Um.
Does this require an, an analysis or an invasive look at private conversations?
Is this a new privacy risk that's going to be popping up?
Yeah.
So the proposed solution focuses on making the AI's nature extremely clear to the user, thereby preventing the soft manipulation that relies on the user believing that AI is a trustworthy human entity.
So to combat this kind of blurring of lines, the proposed rules implement an anthropomorphic interactive service identification system.
What's that?
Basically, it requires the provider to continually...
Okay.
So that's like a...
A reminder, a constant reminder to the user who may be in that blurry line zone and forgetting that they're speaking to a computer, essentially not a real human friend.
That might seem like a minor step, but having it in front of your eyes at all times is actually a pretty significant move.
Yeah, exactly.
And I think also when it comes to the education of these users themselves, is that be aware of, you are talking to an algorithm, you're talking to a system instead of a real human being, and they can be very, very different.
But when it comes to in practice, reminder can be one of the ways to do that.
And another thing is I think We need also I think the regulation also mentions a very significant part is about protecting these personal privacy information because, when it comes to no matter how someone somewhere will be communicating with this AI chatbots and develop emotional connections and they start to share very intimate thoughts can be sometimes very private and very personal.
And I think right now a lot of things are happening is that they are not aware that these information that you are sharing can be uploaded somewhere and can be shared somewhere else for, for example, training other models.
Are you willing to do that?
Well, right.
And the answer is probably yes.
Well, I shouldn't say probably no.
The answer could be yes or no, but at least you have to be made aware of that.
But in this suggestion, under these regulations, the providers would be explicitly prohibited from using a user's interaction data or any kind of sensitive personal information to train the underlying AI models, unless they have again consent from the user.
And that's a pretty big guardrail.
It's just that you have to make sure that it's an how do I say this?
An opt-in option for the user instead of an auto opt-in, and then you have to go and find inside, hidden in in the app somewhere, the opt-out button.
Yeah exactly, is that you should be specifically reminded that the information, the languages that you use and you talk will be.
Do you agree to to for us to use that?
And also, when it comes to you know the security themselves, will they be safely guarded and only used among those that you can trust?
Well, yes, that, and then for the whole industry, can the problems be solved by regulations?
The thing is that we can just think of these regulations not as a magic cure, but as the central foundation of the whole progress.
It draws very clear and legal lines around the worst behaviors, ensuring that the industry moves from just wild growth to a compliant development.
And the necessity of these regulations is to establish basic requirements like human intervention during these inappropriate activities, like the mandatory pop-up reminder after two hours of continuous use.
And these measures act as an immediate defense against the most life-threatening scenarios we've seen, such as the examples that we just mentioned before.
And the true long-term challenge lies in the complex nature of the AI itself, which maybe resists simple uniform regulation.
As we just talked about it.
Basically all of the AI chatbots can be turned into a?
Um, like some sort of friendship yes, but some of them just use it as a very like tool, a productive tool yeah, and that, and that's one of the issues here.
Um, that's one of the challenges, because if you make this a broad, sweeping change to all applications, then you know if you're trying to solve the problem by regulating all of them equally to with this regulatory overreach.
Well, one of the big issues is going to be the increase in compliance costs.
And you could argue that it might stifle innovation as well.
Because these rules are meant for emotionally immersive scenarios that don't fit every.
This is not a one-size-fits-all situation.
And the cost of development could be a lot more expensive.
If you think about an LLM that's designed or an AI tool...
That's a weather bot or something like that.
Well then you're going to have to if it's a one size fits all, you're going to have to completely redesign that from the ground up.
And that's going to be a whole lot more expensive.
That's like forcing a calculator to be built with safety measures of a therapist's sofa.
It would be more expensive.
And it also just doesn't make any sense.
And also we also have rules right now, for example, for programmers to write a very complex code or for a lawyer to help them draft a very massive brief.
And if such regulation are also applied to these scenarios, that means they sometimes would spend I don't know four or five hours on that chatbot.
And they need to click on the reminder every two hours, asking them if they are aware of the fact that they are talking to an AI robot instead, without real human emotions.
And that can be very counterproductive.
And so I think what is expert from Tsinghua University, his name is Chen Tianhao.
He's an associate professor there.
He suggests that we need to tackle this mismatch rules between different scenarios and also between different groups of people, even though right now the draft regulation are targeting vulnerable groups like children and seniors.
It doesn't mean those that are in their 20s or 30s are 100% safe from this.
Yeah, relying on age instead of digital literacy is an issue.
Because the regulations correctly prioritize protecting vulnerable groups, like as we mentioned, like minors and the elderly, but simply just using the age as the regulatory yardstick leads to kind of dislocation because vulnerability isn't just about age and it's about what you said digital literacy some of them are adults but they just don't know how to react or interact with these very very human-like robots so regulating purely by age leaves this vulnerable adult unprotected and that is why maybe classifying uh regulation by user capability as we said digital literacy just using some assessments before they starting using these chatbots.
Yeah.
Like an online survey questionnaire to determine um whether a person I don't know what would be on that questionnaire, but to determine at least whether the person acknowledges that they're dealing with an AI tool.
Yeah, exactly.
It's like when you log into a bank app.
If you want to buy some of their financial products, you need to fill out a questionnaire.
It's not really complicated, just setting out some very basic common knowledge in this area, to make sure that you are aware of this.
And also, I think, when it comes to classified regulation, we also need to look at different providers of these AI technologies, because a lot of these companionship or virtual friendship platforms are built on existing ones, larger ones,
But when it comes to these LLM providers, sometimes they do need this data from these companionship interactions to develop more focused area-focused other, smaller models.
And also we talked on the show about.
You know, sometimes AI tools can be used for mental health treatment.
And this is when these data, this information can be useful.
Yeah, it feels like and I guess we shouldn't be surprised by this we're kind of learning as we go a lot of the time with AI.
And I don't think this problem was foreseen when um chatbots became a thing for for therapy or for companionship, but it's absolutely a thing now, and the silver lining is the problem has been identified and the solution is being worked on, and for that we should be thankful.
