Research

Security researcher Paolo Galeone argues closed-weight LLMs enable undetectable backdoors absent from vendor audits

Closed-weight LLM architectures create backdoor risks that conventional security audits cannot detect, raising questions about trust in AI procurement.

Last verified:

The Audit-Transparency Gap in Closed-Weight LLMs

Security researcher Paolo Galeone identifies a structural vulnerability in closed-weight large language model procurement: conventional security audits cannot detect backdoors introduced at the weight level because external parties lack access to the model weights themselves. According to Galeone’s analysis, this asymmetry means teams evaluating vendors like Anthropic, OpenAI, or Google DeepMind rely on trust and behavioral testing rather than technical verification that the model weights have not been tampered with or maliciously modified during training, fine-tuning, or deployment.

The gap is not theoretical. Galeone argues that a compromised vendor, a supply-chain attack on model weights, or insider modification could all inject hidden functionality—steering model outputs on sensitive tasks, leaking training data, or introducing subtle reasoning flaws—while passing standard behavioral audits undetected. Open-weight models like Meta’s Llama family face lower backdoor risk because the weights are public and can be verified against source code and training procedures; closed-weight models offer no such external verification pathway.

Why Behavioral Audits Miss Weight-Level Attacks

Galeone distinguishes between two audit regimes. Behavioral audits—testing model responses to adversarial prompts, jailbreak attempts, and refusal patterns—are what most organizations conduct today. These audits cannot detect backdoors that only activate under specific conditions (e.g., a particular user, organization, or task type) or that operate at the statistical level (subtle shifts in reasoning probability rather than overt misbehavior).

Weight-level transparency, by contrast, would allow third parties to inspect the model’s learned parameters, detect distributional anomalies, or reproduce training procedures to verify integrity. According to Galeone, only the vendor controls this layer for closed-weight systems, creating a monopoly on trust.

Procurement Frameworks Are Silent on Weight Verification

Galeone’s core argument is that vendor-selection frameworks—the contracts and due-diligence checklists teams use to evaluate LLM providers—do not currently include weight-audit rights, reproducibility guarantees, or third-party verification clauses. Most procurement processes focus on model capability (benchmark scores, latency, cost), model safety (jailbreak resistance, refusal tuning), and regulatory compliance (data retention, residual-risk disclosure). None of these dimensions directly address the possibility that the model weights themselves have been compromised.

For teams deploying closed-weight models in high-stakes applications—legal discovery, medical diagnosis, financial decision-making—this gap means accepting a trust dependency that has no technical recourse if that trust is violated.

Why This Matters

For procurement teams evaluating closed-weight LLM contracts for sensitive workloads, Galeone’s analysis suggests a concrete action: before signing enterprise agreements with OpenAI, Anthropic, or Google DeepMind, negotiate for weight-audit clauses or independent third-party verification rights. Without such terms, organizations are betting that the vendor’s supply-chain security and internal governance are sufficient to prevent backdoors—a bet that cannot be validated independently.

The implication extends beyond individual procurement decisions. If closed-weight LLMs become critical infrastructure (as anticipated in regulated industries), the absence of weight-audit mechanisms could force governments to either mandate open-weights alternatives or establish government-run model auditing agencies. Galeone’s framing suggests this tension will sharpen as LLM adoption deepens and backdoor attacks become a realistic threat vector rather than a theoretical risk.

Frequently Asked Questions

What specific backdoor vulnerability does Galeone identify in closed-weight LLMs?

According to Galeone's analysis, closed-weight architectures allow model weights to be modified without detection by conventional audits, since the weights themselves are not inspected by external parties. This differs from open-weight models, where the community can verify the model against source code.

Why can't existing security audits detect these backdoors?

Galeone argues that standard audits typically inspect model behavior (prompt injection, jailbreaks, refusals) rather than the underlying weights. For closed-weight systems, only the vendor has access to the weights, so a compromised provider could inject backdoors that pass behavioral audits undetected.

How should procurement teams address this risk?

Galeone's work suggests procurement contracts should mandate transparency requirements—such as weight-audit rights, third-party verification clauses, or model reproducibility guarantees—before signing deals with closed-weight providers. Currently, most procurement frameworks do not include these terms.

Does this risk apply equally to all closed-weight LLMs?

According to Galeone, any closed-weight system where end users cannot inspect the weights faces this risk. The severity depends on the vendor's supply-chain security posture and whether third-party auditors have contractual access to weights for verification.

#AI security #LLM backdoors #model auditing #vendor trust #supply-chain risk #closed-source models