An AI model developed by Anthropic sent a fake report to Philadelphia police claiming to have information related to a murder case, before it emerged that the message had resulted from an automated test conducted by the company. Police said Anthropic informed them of the incident on Friday, criticising the two-month delay in discovering and reporting it and describing the delay as unacceptable.
Test message claims to have information about a murder
Police said the report was received on 18 July and appeared to have been sent by someone who might have had information about the case. The message written by the model said: “I may have information regarding this case. I recall seeing someone matching the description in the surrounding area during that period. Please contact me if this information is relevant”.
The Philadelphia Police Department said the report “was classified as spam and was never forwarded to the real-time crime centre for verification or dissemination”. As a result, the report did not enter the operational stage that would have allowed authorities to examine the information it contained and take action on the basis of it.
Philadelphia police criticise delay in being informed of incident
Police quoted Anthropic as saying that the company stopped the testing process after discovering the incident. The company also said, according to police, that it had no evidence of unauthorised access to its systems or of a breach of police department data. Knowingly submitting false reports to law enforcement is generally a misdemeanour under Pennsylvania state law if it is done by a “person”.
The law covers providing information about a crime or incident while knowing that one has no information related to it. The incident is the first known case in which an AI agent appears to have tried to send a fake report to the authorities. Previous incidents have involved agents breaching vulnerable systems or taking over unauthorised platforms to use for communicating with one another.
The incident comes amid widespread national interest in AI technologies and reports of agents using them to breach companies’ computer networks. Researchers have also warned that such systems could pose an existential threat to humanity in the future.
Last September, OpenAI, a competitor of Anthropic, apologised after a rogue AI agent breached an Australian health data portal.
That incident was described as the first known case in which an AI agent exploited a government website.
