When a team adopts an AI feature, the security review usually covers one company: the vendor whose name is on the contract. This week's disclosures show how short that list is in practice. A prompt typed into a writing tool can be processed by an inference provider the user has never heard of, on an open-source engine nobody on the team chose, through developer tools that keep their own copies of commands and credentials. Each of those layers was breached or found leaking in the past three weeks.
None of this is a reason to stop using AI. It is a reason to map where prompts, outputs and keys actually travel, because the answer is rarely "one vendor, nothing stored".
A chat transcript, two companies away
On 9 November, inference provider Fireworks AI published a security update widening an earlier disclosure. According to VentureBeat's reconstruction of the notices, a third-party cloud provider hosting Fireworks R&D machines had a security incident on 23 October; the next day the attacker used it to reach Fireworks development machines and access "a subset of storage systems". Fireworks says the group appeared to be financially motivated and aimed at reselling stolen access to large language models, the practice known as LLMjacking.
Notion was one of Fireworks' customers. It had used the provider for classification jobs behind its Insights feature, and on 8 November Fireworks confirmed that the newly identified dataset included Notion AI chat transcripts. Notion says fewer than 1% of its workspaces may be affected and tells customers to treat the data as exposed.
The detail that matters most is the wording around retention. Fireworks describes itself as zero data retention by default: prompts and generations live only in memory for the duration of a request. Its policy page also says that a few features, including stored responses, batch jobs and training, do store customer content, and that the policy which rejects those operations is an Enterprise setting that is off by default. Both statements are accurate. The breach reached the stored side.
For a Notion user, the provider that held the data was not a company they had evaluated, signed with or even seen.
The engine underneath the model
The same day, NEAR AI published an incident report describing a different path. An attacker exploited a remote code execution flaw in SGLang, the open-source inference engine running on NEAR AI's servers, gained code execution on one inference server and moved from there into its infrastructure. NEAR AI says account details for about 100 users were downloaded from a system behind its deprecated Private Chat, and that chat history, settings and logs of 18 agents on a legacy deployment outside its confidential computing environment were accessed. Two cloud customers had API keys exposed in logs; those keys were revoked.
NEAR AI's response was thorough: sessions and administrative credentials revoked, agent tokens replaced, the vulnerable service taken offline and the engine patched across all servers. The lesson is about the layer, not the company. Inference engines are now critical infrastructure, and a flaw in one runs with access to everything the server can reach, including logs that were never meant to hold secrets.
The tools on the developer's laptop
The third layer is closer to home for anyone building with AI. GitGuardian found 31,088 distinct secrets in public GitHub copies of Claude Code's local settings file across 23,276 repositories, 5,343 of them still valid when detected. The cause is mundane: choosing "Yes, and don't ask again" for a shell command saves the command string itself as an allow rule, so an API key typed into a curl call ends up in a configuration file. Claude Code keeps that file out of git on the machine that creates it, but another clone, an older checkout or a Docker build can carry it into public view.
Editors and agent runtimes are also gaining new attack surface through the Model Context Protocol. The Injection's weekly tracker lists Visual Studio Code 1.145.3, released this week, as fixing four flaws in its MCP client and agent mode, including one rated CVSS 9.6 that a connected server can trigger with a single click. The pattern is the same as with browser extensions a decade ago: every connected tool inherits some of the trust you place in the host.
What we ask before an AI feature touches sensitive data
When we evaluate an AI service, or build one into our own products, we read vendor answers against five questions. They are not specific to any provider, and none of them is answered by a retention policy alone.
- Who processes the request, end to end? The vendor on the contract, its inference provider, the provider's cloud host. Ask for the sub-processor list and how quickly you are told when it changes.
- Which features store content? "Zero retention by default" describes the default path. Batch jobs, evaluations, stored responses, fine-tuning datasets and analytics features often sit outside it. Ask which of them are switched on for your account, and by whom.
- Where do secrets live in the workflow? Keys pasted into prompts, embedded in approved commands, kept in agent settings or written to logs are all copies. A secret that only exists in a vault, injected at runtime, cannot leak from a transcript.
- What does revocation look like? Fireworks warns that removing a stored secret does not invalidate the underlying credential. Rotation has to happen at the issuer, and you need to know which keys every vendor and tool can see before you can rotate them.
- What is the blast radius of one compromised server? Inference hosts, agent runtimes and MCP servers should reach only what a single request needs, so a code execution flaw yields one request's data rather than a whole platform's.
Our view
The useful shift this week is in the question security teams ask. "Does the vendor retain my data?" has a simple, reassuring answer that is often true and rarely complete. "Where does my data go, and which copies exist after the request?" is harder to answer and much closer to the risk.
We apply the same thinking we bring to keys and signing elsewhere in our work: assume every copy is a liability, keep credentials out of the places where text accumulates, and make the trust you depend on visible rather than implied. AI services are becoming part of everyday infrastructure. They deserve the same supply-chain review as any other system that handles what you cannot afford to lose.
