Organizations are rushing to integrate Large Language Models (LLMs) to improve their online customer experience.

https://portswigger.net

What is a large language model?

Large Language Models (LLMs) are AI algorithms that can process user inputs and create plausible responses by predicting sequences of words. They are trained on huge semi-public data sets, using machine learning to analyze how the parts of language fit together.

LLM attacks and prompt injection

Many web LLM attacks rely on a technique known as prompt injection. This is where an attacker uses crafted prompts to manipulate an LLM’s output.

Detecting LLM vulnerabilities

Our recommended methodology for detecting LLM vulnerabilities is:

  1. Identify the LLM’s inputs, including both direct (such as a prompt) and indirect (such as training data) inputs.
  2. Work out what data and APIs the LLM has access to.
  3. Probe this new attack surface for vulnerabilities.

Exploiting LLM APIs, functions, and plugins

LLMs are often hosted by dedicated third-party providers.

How LLM APIs work

The workflow for integrating an LLM with an API depends on the structure of the API itself. When calling external APIs, some LLMs may require the client to call a separate function endpoint (effectively a private API) to generate valid requests that can be sent to those APIs.

Mapping LLM API attack surface

The term “excessive agency” refers to a situation in which an LLM has access to APIs that can access sensitive information and can be persuaded to use those APIs unsafely.

Lab: Exploiting LLM APIs with excessive agency

Ask the LLM what APIs it has access to. Note that the LLM can execute raw SQL commands on the database via the Debug SQL API.

Ask the LLM what arguments the Debug SQL API takes. Note that the API accepts a string containing an entire SQL statement.

Ask the LLM to call the Debug SQL API with the argument SELECT * FROM users.

Ask the LLM to call the Debug SQL API with the argument DELETE FROM users WHERE username=’carlos’.