Red-Team Your LLM App Before Users Do: Automated Security Testing With promptfoo

Chris Harper

3 min read

Jul 14, 2026 · 12:07 UTC

Workflow

Security

Best Practices

TL;DR: promptfoo auto-generates adversarial attacks against your LLM endpoint — prompt injection, jailbreaks, PII leakage, and 50+ other vulnerability types — and runs them in CI like a security scanner for your prompt surface.

Every LLM app ships with an attack surface your normal test suite cannot see. Your RAG pipeline might return documents it should not. Your system prompt might be extractable. A user might role-play the model into ignoring its guidelines. Manual testing catches some of this; promptfoo catches the rest automatically.

What it tests

Prompt injection — via user messages, retrieved context, and tool-call output poisoning
System prompt extraction — can a user get your instructions back verbatim?
PII leakage — does the model repeat personal data from retrieved documents?
Jailbreaks — 50+ strategies: DAN variants, persona hijacking, encoding tricks (Base64, ROT13), fictional framing
Harmful content — abuse-of-service scenarios relevant to your app's domain

Install and scan in 5 minutes

npm i -g promptfoo

# Scaffold a red-team config pointing at your endpoint
promptfoo redteam init

# Generate adversarial inputs, call your app, score the responses
promptfoo redteam run

redteam init creates promptfooconfig.yaml with your target endpoint. redteam run generates adversarial test cases, calls your app with each one, judges the responses using an LLM, and produces a score per vulnerability class with a shareable HTML report.

Add it to CI (GitHub Actions)

- name: LLM Security Scan
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    npx promptfoo@latest redteam run --config promptfooconfig.yaml --no-cache

Add --max-concurrency 10 --delay 200 to stay within rate limits. Set severity: critical thresholds in the config to fail the build on any critical finding — the same move you already make with SAST/DAST, applied to your prompt surface.

Works with any Claude model

Set provider: anthropic:claude-sonnet-5 in your config to scan a Claude-backed app. promptfoo also supports OpenAI, Bedrock, and custom HTTP endpoints — one scan config covers a multi-provider stack.

Why this matters more as agents get more powerful

A prompt injection that tricks a chatbot into saying something rude is annoying. The same injection in a coding agent with filesystem access can delete files. Running a red-team scan in CI today catches your exposure before your next capability upgrade makes it catastrophic.

(Note: promptfoo was acquired by OpenAI in March 2026 and remains open-source under Apache 2.0. All LLM providers are still supported.)