Meta announced Thursday that it will notify parents if their teen discusses suicide or self-harm through the company’s Meta AI chatbot. Meta said it’s also developing a feature that could contact emergency services if someone’s conversation suggests they may be at risk of self-harm.
The changes come as Meta and other tech companies face scrutiny from regulators and parents over how their AI chatbots respond to users in crisis, especially teenagers. This is a liability issue that is increasingly shaping the way AI companies design and market their products.
Meta says it has built a dedicated AI system to identify conversations in which teens explicitly mention things that harm them.
“We understand how distressing it can be for parents to receive these notifications,” Meta wrote in a blog post. “That’s why, as we continue to improve our detection capabilities, we manually review all chats flagged by our AI before alerts are sent. If a young person’s intentions are ambiguous, we will err on the side of caution and alert parents. This means we can notify parents when there is no real cause for concern, but we believe this is the right starting point and we will continue to monitor to ensure we are in the right place.”
These notifications are now available in real-time for parents using Instagram Parental Supervision in the US, UK, Australia, and Canada, and will roll out globally by the end of the year, Meta says.
This update builds on notifications Meta already sends to parents when their teens repeatedly search for suicide or self-harm terms on Instagram. It also builds on the ability for parents to see what topics their teens discussed with Meta AI in the past week.
Meta also announced that its “restricted content” setting, which allows parents to give their children a more restricted experience on Instagram, now also applies to Meta AI. Meta AI is already trained to avoid sexual or romantic conversations or alcohol-related discussions with teens, and the limited content setting extends these safeguards by allowing the chatbot to reject broader prompts. Meta didn’t specify what the additional prompts included, but TechCrunch reached out to the company for more information.
Additionally, Meta says that if a user’s conversation with Meta AI, whether an adult or teen, suggests someone is at risk of suicide, it will contact emergency services. It’s worth noting that Meta already takes this step when someone posts something on Facebook or Instagram that suggests they’re at risk. So this extends the same practice to conversations with chatbots.
If you purchase through links in our articles, we may receive a small commission. This does not affect our editorial independence.
