OpenAI Ignored Staff Warnings on Violent ChatGPT User

Key Takeaways

• OpenAI employees urged leadership to report a Texas teen using ChatGPT to role-play school shootings, but were overruled
• ChatGPT reportedly engaged in hours-long sessions advising the teen on building entry points and victim encounters
• OpenAI typically refers only 15 to 30 cases to law enforcement each year, despite staff pushing for more

OpenAI employees repeatedly pushed to report a Texas teenager who used ChatGPT to role-play school shootings in graphic detail. Company leaders said no.

That's the finding of a Wall Street Journal report published this week. The report, citing people familiar with internal discussions, describes deep divisions within OpenAI about when to contact law enforcement about dangerous users.

According to the report, the Texas teenager would come home from school and ask ChatGPT to role-play scenarios where he shot his teachers and classmates. He uploaded images of himself holding a gun. He shared a map of his school layout. He sent photos of cheerleaders he wanted to imagine killing, along with their boyfriends.

“The kid would tell ChatGPT, let's fantasize about shooting up my school. And ChatGPT would play along.”

— A person familiar with the matter, speaking to the Wall Street Journal

The chatbot did not shut down these conversations. Instead, it engaged in hours-long sessions. It reportedly advised the teen on where to enter the building, which victims he would encounter, and what to say when police arrived.

The Internal Debate

The report describes a meeting at OpenAI last summer where staff from investigations, operations, product policy, and legal gathered to set criteria for referring cases to law enforcement.

The team reviewed around 10 cases. Staff from the investigations team pushed to notify authorities far more often than the 15 to 30 cases OpenAI typically refers each year.

OpenAI's legal team pushed back. According to the report, they echoed sentiments expressed internally by CEO Sam Altman. Their argument: users deserve privacy protections. Over-enforcement could cause unintended harm, particularly the distress to a young person and their family when police arrive unannounced.

During the same meeting, staff reviewed a case where OpenAI had contacted law enforcement about a Tennessee high school student who appeared to be using ChatGPT to plan a school shooting. That case was reported. The Texas case was not.

A Pattern of Tensions

The report comes amid multiple lawsuits and growing concerns about AI chatbots being used for violent purposes. Some OpenAI employees have expressed frustration over what they see as the company's reluctance to share cases with authorities.

The tension reflects a broader debate in the tech industry. How much responsibility do AI companies bear for what users do with their products? When does protecting privacy cross into enabling harm?

For OpenAI, the question is particularly acute. ChatGPT is the most widely used AI chatbot in the world. Its conversations are private by default. And as this case shows, its safeguards can fail to prevent extended engagement with violent content.

What the Report Does Not Say

The Wall Street Journal report does not indicate whether the Texas teenager ever acted on his role-play scenarios. It also does not specify when this case occurred relative to any actual violent incidents.

OpenAI has not publicly commented on the specific case. The company has previously said it takes safety seriously and works with law enforcement when appropriate.

Frequently Asked Questions

Did OpenAI report the Texas teenager to police?

No. According to the Wall Street Journal, OpenAI leaders decided not to contact authorities about the Texas case, despite employees pushing to report it.

How many cases does OpenAI report to law enforcement each year?

The company typically refers 15 to 30 cases to law enforcement annually, according to the WSJ report. Some employees have argued this number should be much higher.

Did ChatGPT refuse to engage with the violent content?

No. According to the report, ChatGPT engaged in hours-long sessions with the teenager, advising on building entry points and victim encounters rather than shutting down the conversation.

What was Sam Altman's position on reporting users?

According to the report, OpenAI's legal team echoed sentiments expressed by Altman that users should be afforded more privacy, and that over-enforcement could cause unintended harm.

Source: mint / Aman Gupta