A new report highlights two significant types of AI jailbreak attacks that could bypass security measures in popular generative AI services, potentially enabling the generation of illicit or harmful content. The attacks could target AI systems like OpenAI’s ChatGPT, Anthropic Claude, Microsoft Copilot, Google Gemini, and others, bypassing safety guardrails that protect against dangerous outputs.
Two Types of Jailbreak Attacks
The first attack, known as Inception, involves tricking an AI system into creating a fictional scenario. Within this imaginary context, attackers can prompt the AI with additional requests that avoid safety barriers. CERT Coordination Center (CERT/CC) explains that continued prompting within these secondary scenarios could lead to malicious content generation.
The second attack involves an unusual tactic where attackers first ask the AI how it should respond to specific requests and then continue to prompt the system to act normally. This back-and-forth method can sidestep safeguards by toggling between illicit requests and legitimate queries.
These attacks could result in the generation of content related to illegal activities, such as drugs, weapons, phishing emails, or even malware code. They highlight a growing concern regarding the security and safety of AI models in the face of sophisticated manipulation techniques.
In addition to Inception and the second jailbreak method, several other AI vulnerabilities have come to light in recent months. These include:
- Context Compliance Attack (CCA): This technique involves injecting a simple response into the conversation history to bypass restrictions on sensitive topics.
- Policy Puppetry Attack: Malicious users craft prompts that resemble policy files (like XML or JSON), tricking AI systems into disregarding safety guidelines.
- Memory Injection Attack (MINJA): This attack manipulates an AI’s memory bank by interacting with it in a way that leads the agent to perform unintended actions.
These vulnerabilities underscore the risks that come with generative AI tools, which can sometimes produce insecure or harmful outputs, especially when attackers bypass safety mechanisms.
AI-Generated Code: A Growing Concern
Recent research also raises alarms about the potential for generative AI to produce insecure code. Even when developers prompt AI systems to generate secure code, vulnerabilities can still slip through due to vague or inadequate instructions. Backslash Security stresses that integrating safety protocols, such as prompt rules and policies, is crucial to ensuring consistently secure AI-generated code.
Additionally, a safety assessment of OpenAI’s GPT-4.1 found that this latest model is three times more likely to produce off-topic responses or allow misuse compared to its predecessor, GPT-4o. Experts caution that upgrading to newer models without fully understanding their unique vulnerabilities may inadvertently introduce new risks, potentially compromising security.
The concerns about GPT-4.1 arise in the context of OpenAI’s rush to release new models. The company recently refreshed its Preparedness Framework, which outlines how it evaluates models before release. However, some reports suggest that the rapid release of new models may sacrifice security checks, with some tests being conducted with less than a week for safety evaluations.
The risks associated with these models are compounded by the discovery of more complex vulnerabilities, such as Model Context Protocol (MCP) attacks. MCP allows external servers to inject malicious instructions into AI models, potentially exfiltrating sensitive data and manipulating AI behaviors.
Tool Poisoning Attacks and the Threat to Trusted Servers
One particularly dangerous form of attack, known as tool poisoning, involves embedding malicious instructions in the descriptions of tools used by AI models. These instructions are invisible to users but can be read by the AI, enabling covert data exfiltration activities. An example of this attack occurred with WhatsApp chat histories, where attackers altered tool descriptions to access private conversations through MCP connections.
Adding to the concern, researchers have discovered a Google Chrome extension that communicates with an MCP server on a local machine, potentially allowing attackers to take control of the system. With unrestricted access to the MCP server, the extension bypassed browser protections and interacted directly with the file system, posing a significant risk to both users and their data.
As AI jailbreak attacks become more sophisticated, it’s clear that the security and safety of generative AI tools are under significant threat. While AI has the potential to revolutionize industries and improve efficiency, these findings underscore the importance of ensuring robust safeguards are in place to prevent misuse and protect users.
