U.S. cyber agencies want AI labs to spot distillation campaigns

NSA, the FBI and CISA have issued a joint advisory that tells AI providers to watch for coordinated campaigns that extract model outputs across accounts, APIs, cloud services and resellers. The guidance gives platform-security teams a new set of behavioral signals to monitor as they protect proprietary model capabilities.

The September 8 advisory alleges that several China-based AI companies have used industrial-scale knowledge distillation against U.S. frontier models. Knowledge distillation can be a legitimate research technique: a smaller model learns from the outputs of a larger one. The agencies' claim concerns the alleged collection of restricted proprietary outputs at scale, including through access routes that obscure who is making the requests.

The named companies have not been treated here as independently proven perpetrators. The confirmed development is that three U.S. agencies published the advisory and its defensive guidance. Its practical importance lies in where the agencies place the defense: provider telemetry, account controls and information sharing, rather than a single filter at a model endpoint.

The pattern matters more than one prompt

The 18-page advisory describes campaigns that can move between native APIs, cloud providers, third-party aggregators and proxy services. It says those routes can make a high-volume operation look like scattered ordinary traffic when each service sees only one slice.

That changes the unit of detection. A suspicious prompt may be easy to rewrite or evade. A sustained operation can leave a broader trail: newly created accounts hitting maximum usage immediately, shared subscriptions used from multiple IP addresses or user agents, repeated requests across similar topics, and coordinated switching after a provider changes pricing or blocks activity.

The agencies also describe automated collection and quality checks, including attempts to distinguish a service fault from a defensive change in output quality. Those are allegations in a government assessment, not public technical evidence that outside researchers can reproduce. The document nevertheless gives security teams a concrete way to translate the risk into logs and detection rules.

Three actions for providers

The joint advisory recommends three immediate responses:

• Detect anomalous prompts, accounts, networks and usage behavior, including unusual subscription-to-usage ratios and enterprise-scale throughput.
• Change responses for suspected malicious extraction attempts in ways intended to reduce the value of collected outputs.
• Share intelligence across model providers, cloud platforms and API aggregators so distributed activity can be correlated.

Each recommendation has trade-offs. Stronger account checks can catch abuse, yet they can also make legitimate high-volume research or enterprise use harder to support. Deliberately changing outputs for suspected extraction can complicate debugging and may affect a customer who was flagged incorrectly. Cross-provider sharing can reveal patterns that no company can see alone, while creating privacy, competition and data-governance questions that need defined safeguards.

Model security now reaches the distribution layer

Many safety discussions focus on the weights of an open model or the behavior of a chatbot. The advisory focuses on an intermediate asset: the response stream delivered through a commercial service. If a model's useful behavior can be collected at sufficient volume, the value of access controls depends on the provider's ability to recognize coordinated collection before it becomes a training corpus.

The agencies map parts of the reported behavior to MITRE ATLAS, a framework for adversarial machine-learning techniques. That mapping matters because it lets security teams treat model extraction as a familiar operational problem: observe activity, correlate indicators, investigate with context, and contain an account or route without assuming every heavy user is malicious.

The advisory does not establish the scale of every allegation in public, and it cannot settle the policy disputes around model access, export controls or legitimate distillation research. It does make one operational point clear. AI labs, cloud services and API intermediaries need a shared view of patterns that move across their boundaries. The next test is whether the sector builds that coordination without turning ordinary users into collateral damage.