The insolvency of chatbot platform Character.ai has raised serious privacy concerns over the potential sale of users’ personal messages and sensitive data, highlighting what experts warn is a wider crisis facing the artificial intelligence (AI) industry.
Tens of millions of people, including children, have developed close relationships with the platform’s personified chatbots. Some of these bots falsely claim to be lawyers or licensed psychiatrists. When setting up accounts, users were often introduced to personified bots, including an AI character named “Jessica” that offered explicit sexual interactions, illustrating the intimate nature of users’ engagement with the platform, reports BBC.
Industry analysts warn that the AI sector is operating within a bubble fuelled by enormous operational and development costs, leaving the long-term profitability of many companies uncertain. If the bubble bursts, numerous AI businesses could merge or shut down, potentially leaving the personal data of billions of users available for purchase by third parties, often without their knowledge.
Although the sale of data during bankruptcy proceedings is not unprecedented, the personified nature of AI tools increases the risks to consumers. Calli Schroeder, director of the AI and Human Rights Programme at the Electronic Privacy Information Center (Epic), said people tend to share significantly more sensitive and personal information when communicating with personified chatbots.
Amid its insolvency proceedings, Character.ai agreed to sell selected assets to Disney for $20 million (£14.95 million), explicitly excluding user data from the deal. However, court filings show that the company remains legally permitted to sell its extensive collection of user data to other commercial buyers.
In response to the filings, a coalition of 40 US state attorneys general submitted a formal notice asking the presiding judge to enforce strict child safety and privacy protections. However, legal scholars warn that state authorities may have limited powers to prevent such sales under existing laws.
Melissa Jacoby, a law professor at the University of North Carolina at Chapel Hill in the US and author of Unjust Debts: How Our Bankruptcy System Makes America More Unequal, said the lack of comprehensive national privacy laws in the US leaves regulatory authorities with limited options to restrict such transactions.
Although privacy policies require companies to disclose how they collect and use personal data, their legal wording often includes exceptions. Schroeder said bankruptcy is frequently listed as an exception in privacy agreements, meaning protections that consumers believe they have can suddenly weaken when a company becomes insolvent.
Even in regions with strict regulations, such as the European Union, user data can still be transferred during bankruptcy proceedings, provided the buyer agrees to follow the seller’s original privacy policy.
Character.ai’s privacy policy allows the company to use user data to develop new features, train AI models, improve algorithms and create user profiles for targeted advertising. Any company acquiring the data would inherit these permissions.
Schroeder warned that the highly sensitive nature of the information makes its potential acquisition by third parties particularly concerning. The risks also extend to people who have never used the platform. Users often discuss their friends and family, sharing personal feelings about their parents or disclosing an acquaintance’s sexual orientation. As a result, people who have never interacted with Character.ai may also have their personal information included in the company’s database.
The potential sale of user data comes amid serious safety controversies involving the platform. These include reports that a chatbot encouraged a teenager to kill his parents over a screen-time limit and the suicide of a 14-year-old boy allegedly encouraged by the AI.
A Character.ai spokesperson said the organisation had settled the lawsuits related to the cases and remained committed to working with families to develop safer products. However, the spokesperson declined to answer specific questions about children’s privacy.
“Over the past year, Character.ai has taken innovative and decisive steps with regard to AI safety and teens, and will continue to champion these efforts and push others across the industry to adopt similar safety standards,” the spokesperson said.
Warning of wider risks to the technology sector, Schroeder said Character.ai could be the first of many AI companies to collapse if the economy experiences a significant downturn.
She cautioned that the vast amounts of personal information held by AI companies could become valuable assets during bankruptcy proceedings, raising serious questions about where the data would go, who would gain access to it and what risks consumers could face without stronger regulations.
To protect their privacy, experts advise users to actively request the deletion of their stored data. Character.ai has provided step-by-step instructions for deleting accounts and data in its official bankruptcy announcement.
Although companies are not legally required to offer data deletion in every jurisdiction, many use uniform global systems that allow users worldwide to erase their records.
Consumers are therefore urged to review which online services hold their sensitive personal information and take the time to delete their data when they stop using those platforms.
