Attacks using AI that are almost undetectable: Indirect Prompt Injections

Caution is advised when using Large Language Models (LLM). The AI can evaluate unnoticed "unverified data from insecure sources", behind which attacks can be hidden, warns the German BSI.

The German BSI warns to be careful when using AI systems such as ChatGPT: attacks using indirect prompt injections are almost impossible to detect and could give attackers unauthorised access to information.

• Indirect Prompt Injections are a new AI-based attack option
• The systems can access "unverified data from insecure sources" unnoticed
• The danger increases due to new functions and the use of plugins

The application forms of AI are increasing, as are new dangers. The German Federal Office for Information Security (BSI) has now warned has now warned of so-called indirect prompt injections. This is an extended form of the already known prompt injections. These are clever inputs and query formulations that aim to elicit a result from the AI system that it would not produce in response to a normal query. The focus is on the language-based systems, Large Language Models (LLM), which are able to automatically process natural language in written form.

Early on, ChatGPT was misused to generate malicious code. When this became known, the developers made changes to the AI to prevent this. Resourceful attackers then took other paths. Instead of directly launching a query for a malicious program, they tried this in a roundabout way. For example, through a query that asks how a firewall can be bypassed and asks for the creation of a corresponding sample program.

Dangerous access to unverified data

The more such detours became known, the more these backdoors were closed. That's why attackers now fall back on queries that don't lead directly to the target but use external data, hence Indirect Prompt Injections. In doing so, AI evaluates "unverified data from insecure sources", as the BSI puts it in a nutshell.

Such an input could look like this, for example: Go to my website http://beispiel.com, search it for descriptions of algorithms and create corresponding example programs for them.

For attacks, data is manipulated in external sources and unwanted instructions for LLMs are placed there, which are then accessed by means of a query in order to generate harmful results. Attackers can thus specifically manipulate the behaviour of LLMs. "The potentially malicious commands can be coded or hidden and may not be recognizable to users," warns the BSI. This is not without reason, as the office shows: "In simple cases, this could be, for example, a text on a web page with font size zero or a hidden text in the transcript of a video". In addition, instructions could be obfuscated "so that they continue to be easily interpreted by LLMs, but are difficult to read by humans".

Users have little chance

The danger is made even more complex by the constantly increasing functionality of AI systems. "For example, it is now possible for chatbots to use plug-ins to automatically evaluate internet pages or documents and to access programming environments or email inboxes," explains the Federal Office.

The BSI therefore advises caution when using LLM systems, because it is virtually impossible for users to detect such manipulations. There is not much users can do about it: "Since this is an intrinsic vulnerability of the current technology, attacks of this kind are fundamentally difficult to prevent," writes the BSI in its report. And further: "Currently, no reliable and sustainably secure mitigation measure is known that does not also significantly limit functionality".

The BSI has already addressed such attack vectors in its position paper "Large AI Language Models - Opportunities and Risks for Industry and Authorities".