COMMON WEAKNESS ENUMERATIONIncomplete

CWE-1427

Improper Neutralization of Input Used for LLM Prompting

The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.

Abstraction

Base

Structure

Simple

Likelihood of Exploit

Medium

Understand the problem

What is the problem?

The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.

Extended description

When prompts are constructed using externally controllable data, it is often possible to cause an LLM to ignore the original guidance provided by its creators (known as the "system prompt") by inserting malicious instructions in plain human language or using bypasses such as special characters or tags. Because LLMs are designed to treat all instructions as legitimate, there is often no way for the model to differentiate between what prompt language is malicious when it performs inference and returns data. Many LLM systems incorporate data from other adjacent products or external data sources like Wikipedia using API calls and retrieval augmented generation (RAG). Any external sources in use that may contain untrusted data should also be considered potentially malicious.

Terminology

Alternate terms and usages

prompt injection

attack-oriented term for modifying prompts, whether due to this weakness or other weaknesses

Root cause

How and when does the problem arise?

Lifecycle phase

Architecture and Design

LLM-connected applications that do not distinguish between trusted and untrusted input may introduce this weakness. If such systems are designed in a way where trusted and untrusted instructions are provided to the model for inference without differentiation, they may be susceptible to prompt injection and similar attacks.

Lifecycle phase

Implementation

When designing the application, input validation should be applied to user input used to construct LLM system prompts. Input validation should focus on mitigating well-known software security risks (in the event the LLM is given agency to use tools or perform API calls) as well as preventing LLM-specific syntax from being included (such as markup tags or similar).

Lifecycle phase

Implementation

This weakness could be introduced if training does not account for potentially malicious inputs.

Lifecycle phase

System Configuration

Configuration could enable model parameters to be manipulated when this was not intended.

Lifecycle phase

Integration

This weakness can occur when integrating the model into the software.

Lifecycle phase

Bundling

This weakness can occur when bundling the model with the software.

Risk

What does successful exploitation lead to?

Execute Unauthorized Code or CommandsVaries by Context

Affected security scopes: Confidentiality, Integrity, Availability

The consequences are entirely contextual, depending on the system that the model is integrated into. For example, the consequence could include output that would not have been desired by the model designer, such as using racial slurs. On the other hand, if the output is attached to a code interpreter, remote code execution (RCE) could result.
Read Application Data

Affected security scopes: Confidentiality

An attacker might be able to extract sensitive information from the model.
Modify Application DataExecute Unauthorized Code or Commands

Affected security scopes: Integrity

The extent to which integrity can be impacted is dependent on the LLM application use case.
Read Application DataModify Application DataGain Privileges or Assume Identity

Affected security scopes: Access Control

The extent to which access control can be impacted is dependent on the LLM application use case.

Defence

How is it prevented and fixed?

Architecture and Design

Architecture and design approach

Detail
LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.

Effectiveness: High

Implementation

Implementation-phase defence

Detail
LLM prompts should be constructed in a way that effectively differentiates between user-supplied input and developer-constructed system prompting to reduce the chance of model confusion at inference-time.

Effectiveness: Moderate

Architecture and Design

Architecture and design approach

Detail
LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.

Effectiveness: High

Implementation

Implementation-phase defence

Detail
Ensure that model training includes training examples that avoid leaking secrets and disregard malicious inputs. Train the model to recognize secrets, and label training data appropriately. Note that due to the non-deterministic nature of prompting LLMs, it is necessary to perform testing of the same test case several times in order to ensure that troublesome behavior is not possible. Additionally, testing should be performed each time a new model is used or a model's weights are updated.

Installation · Operation

Defence to apply across several lifecycle phases

Detail
During deployment/operation, use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.

System Configuration

Secure configuration approach

Detail
During system configuration, the model could be fine-tuned to better control and neutralize potentially dangerous inputs.

Verification

How is it detected?

Dynamic Analysis with Manual Results Interpretation

Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.

Dynamic Analysis with Automated Results Interpretation

Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.

Architecture or Design Review

Review of the product design can be effective, but it works best in conjunction with dynamic analysis.