Prompt Injection

When you send unvalidated data to system-role prompts in AI models attackers can manipulate outputs or execute unauthorized actions, compromising system integrity and data security.

In AI applications, system prompts provide pre-processing instructions or context that guides the AI responses. When these prompts are constructed using unvalidated external inputs, they become vulnerable to injection attacks. Attackers can craft inputs that, when embedded as system prompts, alter the behavior of the AI model to execute unauthorized operations or disclose sensitive information.

Example 1: The following Python code illustrates a system prompt injection to the OpenAI AI model:

  client = OpenAI()

  # Simulated attacker's input attempting to inject a malicious system prompt
  attacker_input = ...

  completion = client.chat.completions.create(
      model="gpt-3.5-turbo",
      messages=[
          {"role": "system", "content": attacker_input},
          {"role": "user", "content": "Compose a poem that explains the concept of recursion in programming."}
      ]
  )

In this example, the attacker manipulates unvalidated input to a system prompt, which can lead to a security breach.