AI chatbots are being used and misused in all kinds of places, including the desks of U.S. intelligence analysts (CNN, September 18, 2026).
The intelligence report, circulated across the US military this spring in the midst of the war with Iran, immediately set off alarm bells: A Chinese ship in the Middle East was transporting components of a nuclear weapons program.
The US military swung into action with plans to intercept the vessel, according to four sources familiar with the episode. According to two of the sources, armed members of the US military were preparing to board the ship. Military planes were in the air, one of those sources and another source familiar with the incident said.
It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence—and that a chatbot the analyst had used inaccurately identified the material the ship was carrying….
The bot fused open-source intelligence with secret signals intelligence in government holdings and reached its fateful conclusion about the material the ship was carrying.
The analyst then used AI again to package the findings into a standard intelligence report—the kind that is trusted by military officials—and disseminated it.
So…stop doing things this way?
Why did the analyst fail to assess whether the chatbot’s “assessment” was verifiable? Did he even know what “his” report was saying? Why did officials “dig deeper” into the report—fully grasp relevant details in a way that the analyst had not—only after the mission was underway?
Jake Steckler, an Army officer and a research scholar at GovAI, tells Tech Crunch: “It’s important for service members to understand the uncertainty inherent to LLMs. But it’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions.”
“Runaway” AI?
ZeroHedge speaks of how AI “Hallucinated Nuclear Weapons Components Aboard Chinese Vessel Bound for Iran” and wonders whether a “runaway artificial intelligence” might cause World War Three à la Skynet of “Terminator” movie fame.
But it’s not evidence of “runaway artificial intelligence” when human analysts and those who read their reports rely on AI “conclusions” that they should not be relying on. It’s evidence of faulty human protocols. The AI models are not conscious. They cannot deliberate, judge, or hallucinate.
That the models sometimes mislead after churning through whatever mounds of data they churn through is well known. What the AI chatbots tell us should therefore be viewed with a skeptical eye and in many cases double-checked, especially when knowing whether what we’re being told is right or wrong is crucial.
