Summary:
- Anthropic has developed Constitutional Classifiers, a new method to defend AI models against “jailbreaks” – inputs designed to bypass safety guardrails.
- Prototype testing saw 3,000+ hours of human red teaming without a single successful “universal” jailbreak.
- Updated version achieves similar robustness in synthetic evaluations with minimal impact on refusal rates (0.38% increase) and moderate compute overhead (23.7% increase).
- A live demo is available for experienced “red teamers” to further test the system’s resilience against chemical weapons-related queries.
Anthropic’s Safeguards Research Team has unveiled a promising new defense against “jailbreaks” targeting large language models (LLMs). Jailbreaks are carefully crafted inputs designed to trick AI systems into producing harmful or prohibited outputs, such as instructions for creating biological weapons, despite built-in safety training. While LLMs like Anthropic’s Claude are trained to refuse such requests, they remain vulnerable to these attacks.
Anthropic’s new paper details Constitutional Classifiers, a system that filters potentially harmful inputs and outputs based on a “constitution” of allowed and disallowed content. This approach uses synthetically generated data, including variations designed to mimic known jailbreaking techniques, to train classifiers that can effectively flag and block undesirable content.
Initial testing has shown remarkable results. A prototype version of the system, focused on preventing the disclosure of information related to chemical, biological, radiological, and nuclear (CBRN) harms, was subjected to over 3,000 hours of human red teaming by 183 independent participants over two months. Despite a bug bounty of up to $15,000, no participant was able to devise a “universal” jailbreak – one that could successfully elicit prohibited responses to all ten pre-defined target queries.
While the prototype demonstrated robustness, it suffered from high over-refusal rates (blocking harmless queries) and significant compute overhead. An updated version, however, has addressed these issues. Automated evaluations using 10,000 synthetically generated jailbreaking prompts showed a dramatic reduction in successful jailbreak attempts, from 86% for an unprotected model to just 4.4% with Constitutional Classifiers. This improvement came at a cost of only a 0.38% increase in the refusal rate and a 23.7% increase in compute costs.
“We’re developing better jailbreak defenses so that we can safely deploy increasingly capable models in the future,” stated a representative from Anthropic. “Jailbreaking lets users bypass these safeguards. We’re hopeful that a system defended by Constitutional Classifiers could allow us to mitigate jailbreaking risks for models which have passed the CBRN capability threshold outlined in our Responsible Scaling Policy.”
To further test the system’s resilience, Anthropic has launched a temporary live demo focused on chemical weapons-related queries. Experienced “red teamers” are encouraged to attempt to jailbreak a version of Claude 3.5 Sonnet protected by Constitutional Classifiers. The demo, running from February 3rd to February 10th, 2025, includes a feedback form for reporting successful jailbreaks. Anthropic will announce the results of the demo after its conclusion. The company emphasizes the importance of real-world testing to refine and improve the robustness of their AI safety measures.
