If you ship features built on LLM calls, you already know the uncomfortable part: nobody can tell you whether last week’s prompt tweak made things better or quietly broke three edge cases. Promptfoo fixes that gap by turning prompts into testable artifacts with assertions, expected outputs, and a pass/fail exit code your CI can enforce. This guide is for backend and full-stack engineers who have an LLM feature in production, or close to it, and want prompt changes reviewed with the same rigor as a database migration. By the end, you will have a working test suite, a model comparison matrix, and a GitHub Actions gate that blocks regressions before merge.
What Is Promptfoo?
Promptfoo is an open-source CLI and library for evaluating LLM prompts, models, and RAG pipelines against declarative test cases. You define prompts, providers, and assertions in a YAML config, then run promptfoo eval to score every combination. It supports deterministic checks, model-graded rubrics, custom code assertions, and cost and latency thresholds.
The tool runs locally, stores results on your machine, and works with OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and any HTTP endpoint you expose. Because it is a CLI first, it drops into existing pipelines without a hosted account or vendor lock-in. The official promptfoo documentation covers the full provider list.
Why Prompt Regressions Slip Past Normal Tests
Traditional unit tests assert on exact values. LLM outputs, however, are non-deterministic strings whose correctness depends on meaning rather than bytes. Consequently, most teams either skip testing prompts entirely or write brittle assert output === "..." checks that break on the first token of drift.
The failure mode is predictable. Someone edits a system prompt to fix one customer complaint, and the change silently degrades three other scenarios that nobody re-checked manually. Meanwhile, the model provider ships a new version, temperature settings drift between environments, and a RAG retriever starts returning slightly different chunks.
Promptfoo addresses this by separating what you assert from how strictly you assert it. You can demand an exact substring where correctness is binary, and fall back to a graded rubric where correctness is fuzzy. As a result, the same suite covers both “must include the order number” and “should sound apologetic without promising a refund date.”
If you are still shaping the prompts themselves, read prompt engineering best practices first. Testing amplifies a good prompt strategy; it does not replace one.
Installing Promptfoo and Running Your First Eval
Promptfoo requires Node.js 18 or newer. You do not need a global install — npx pins a version per run, which keeps CI reproducible.
# Scaffold a config in the current directory
npx promptfoo@latest init
# Expected output:
# ✅ Wrote promptfooconfig.yaml
# Run `npx promptfoo@latest eval` to get started!
Next, export the provider keys you plan to test against. Promptfoo reads them from the environment, so the same config works locally and in CI.
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
Then run the evaluation and open the result viewer:
# Run every prompt x provider x test combination
npx promptfoo@latest eval
# Open the local web UI to inspect outputs side by side
npx promptfoo@latest view
The terminal prints a pass/fail table, and view launches a local browser UI where you can read the raw output for every cell. Importantly, eval exits with a non-zero status when assertions fail. That single behavior is what makes promptfoo usable as a merge gate rather than a reporting dashboard.
Writing a Real Promptfoo Config
The default scaffold is intentionally thin. Here is a config closer to what a production support-reply feature actually needs.
# promptfooconfig.yaml
description: Customer support reply quality suite
prompts:
- file://prompts/support_reply.txt
providers:
- id: openai:gpt-4o-mini
config:
temperature: 0
max_tokens: 400
- id: anthropic:messages:claude-sonnet-4-5
config:
temperature: 0
max_tokens: 400
# Applied to every test case below
defaultTest:
assert:
- type: not-icontains
value: "as an AI language model"
- type: cost
threshold: 0.002 # fail if a single call exceeds $0.002
- type: javascript
value: output.length < 1200
tests:
- description: Refund request inside the 30-day window
vars:
customer_message: "I ordered the wrong size last week. Can I get my money back?"
order_age_days: 7
policy_window_days: 30
assert:
- type: icontains
value: refund
- type: llm-rubric
value: >-
The reply confirms a refund is available, references the 30-day return
window, and asks for an order number. It must NOT promise a specific
refund date or dollar amount.
- description: Refund request outside the policy window
vars:
customer_message: "I bought these boots months ago and want a refund."
order_age_days: 94
policy_window_days: 30
assert:
- type: not-icontains
value: "we will refund"
- type: llm-rubric
value: >-
The reply politely declines the refund, explains the 30-day window has
passed, and offers at least one alternative such as store credit or repair.
The prompt file itself uses Nunjucks-style variables that map to the vars block:
{# prompts/support_reply.txt #}
You are a support agent for an online footwear store.
Return policy: refunds are available within {{ policy_window_days }} days of purchase.
This order is {{ order_age_days }} days old.
Customer message: "{{ customer_message }}"
Write a reply of 3 sentences or fewer. Never invent policy details.
Two design choices matter here. First, temperature: 0 is set explicitly on every provider, because a suite that samples randomly will flake and lose the team’s trust within a week. Second, defaultTest carries the invariants that apply everywhere — refusal boilerplate, cost ceilings, length limits — so individual test cases stay focused on behavior rather than repeating hygiene checks.
Promptfoo Assertion Types That Actually Matter
Promptfoo ships dozens of assertion types, but a strong suite usually leans on a handful. The table below covers the ones worth learning first.
| Assertion type | What it checks | Cost | Best for |
|---|---|---|---|
contains / icontains | Substring presence, optionally case-insensitive | Free | Required entities, policy terms |
regex | Pattern match | Free | Ticket IDs, dates, formatting rules |
is-json / contains-json | Valid JSON, optionally against a schema | Free | Structured extraction |
javascript / python | Arbitrary code returning pass/score/reason | Free | Business rules, schema validation |
similar | Embedding cosine similarity to a reference | Cheap | Paraphrase-tolerant matching |
llm-rubric | A grader model scores against natural-language criteria | Paid | Tone, completeness, refusal behavior |
factuality | Output consistency against a reference answer | Paid | Knowledge-heavy responses |
context-faithfulness | Whether the answer is grounded in retrieved context | Paid | RAG hallucination detection |
cost / latency | Per-call spend and wall-clock time | Free | Budget and SLA guardrails |
Any assertion accepts a not- prefix, so not-icontains and not-regex work exactly as you would expect. Additionally, most accept threshold and weight, which lets you build a weighted score instead of a binary gate.
Deterministic Assertions Come First
Reach for free, deterministic checks before paying a grader model. In practice, roughly two-thirds of what you care about is deterministic: the output must be parseable JSON, must not mention a competitor, must stay under a length budget, must include the case number you passed in.
- type: is-json
value:
type: object
required: [intent, urgency, needs_human]
properties:
intent:
type: string
enum: [refund, exchange, shipping, other]
urgency:
type: integer
minimum: 1
maximum: 5
needs_human:
type: boolean
This single assertion replaces a dozen fragile substring checks. Moreover, it runs in microseconds and costs nothing, so you can afford to run it across every test case and every model variant. If you are enforcing schemas at the API layer too, OpenAI structured outputs pairs well with this pattern — validate at generation time, then verify in tests.
Model-Graded Assertions for Fuzzy Requirements
Some requirements resist regex. “Sounds empathetic,” “does not promise a delivery date,” and “explains the reason for the decline” are all real acceptance criteria that only another model can grade reliably.
defaultTest:
options:
# Use a cheaper, deterministic grader for all rubrics
provider:
id: openai:gpt-4o-mini
config:
temperature: 0
tests:
- vars:
customer_message: "My package says delivered but I never got it."
assert:
- type: llm-rubric
value: >-
The reply acknowledges the missing package, states that an
investigation will be opened, and gives a next step. It must not
state when the replacement will arrive.
threshold: 0.8
Write rubrics as checklists, not adjectives. A rubric that says “be helpful” produces noisy scores; a rubric that lists three concrete conditions produces stable ones. Furthermore, always pin the grader provider explicitly. Otherwise, upgrading your primary model silently changes the grader too, and your baseline moves for reasons unrelated to the prompt.
Custom Assertions for Business Rules
When a rule encodes real domain logic, put it in code where it can be unit tested itself.
// assertions/no_unapproved_discounts.js
// Support replies may only offer discounts from the approved list.
const APPROVED_DISCOUNTS = [10, 15, 20];
module.exports = (output, context) => {
const matches = [...output.matchAll(/(\d{1,2})\s?%\s?(?:off|discount)/gi)];
const offered = matches.map((m) => Number(m[1]));
const unapproved = offered.filter((pct) => !APPROVED_DISCOUNTS.includes(pct));
if (unapproved.length > 0) {
return {
pass: false,
score: 0,
reason: `Offered unapproved discount(s): ${unapproved.join(', ')}%`,
};
}
return { pass: true, score: 1, reason: 'All offered discounts are approved' };
};
Wire it up by file reference:
- type: javascript
value: file://assertions/no_unapproved_discounts.js
Why return a reason rather than just a boolean? Because the failure message is what a teammate reads at 5pm on a Friday when CI goes red. A custom assertion that says “Offered unapproved discount(s): 40%” saves ten minutes of digging that false does not. Python works identically through a get_assert(output, context) function.
Comparing Models and Prompt Variants Side by Side
The feature that pays for itself fastest is the comparison matrix. Because promptfoo runs every prompt against every provider, you get an N×M grid from one command.
prompts:
- file://prompts/support_reply_v1.txt # current production prompt
- file://prompts/support_reply_v2.txt # candidate with tightened constraints
providers:
- openai:gpt-4o-mini
- openai:gpt-4o
- anthropic:messages:claude-haiku-4-5
- ollama:chat:llama3.1:8b # local baseline, no API cost
# Four providers x two prompts x every test case
npx promptfoo@latest eval --max-concurrency 4 --output results.json
npx promptfoo@latest view
The UI renders a grid where each column is a prompt/provider pair and each row is a test case. Suddenly the question “can we downgrade to the cheaper model?” has an evidence-based answer instead of a hallway opinion. Notably, the cost and latency assertions turn that comparison into a hard requirement rather than a footnote.
One caveat: the latency assertion measures real request time, so it only produces meaningful numbers when the response cache is disabled. Run those comparisons with --no-cache.
Loading Test Cases From CSV
Hand-writing YAML stops scaling around thirty cases. Fortunately, promptfoo reads test cases from CSV, which lets support or QA teammates contribute cases without touching YAML.
tests: file://tests/support_cases.csv
Every column becomes a variable, and the reserved __expected column carries an assertion:
customer_message,order_age_days,policy_window_days,__expected
"Wrong size, want my money back",7,30,icontains: refund
"Bought these ages ago, refund please",94,30,not-icontains: we will refund
"Where is my order?",3,30,llm-rubric: Gives a tracking next step and does not promise a delivery date
"Can I exchange for black?",5,30,javascript: output.length < 600
For multiple assertions per row, add __expected1, __expected2, and so on. In addition, promptfoo can generate cases programmatically from a JS or Python function, which is useful when you want to sweep a parameter such as order age across a range.
Adding Promptfoo to CI With GitHub Actions
A prompt suite that only runs locally decays. Gating it on pull requests is what keeps it honest.
# .github/workflows/prompt-tests.yml
name: Prompt tests
on:
pull_request:
paths:
- 'prompts/**'
- 'promptfooconfig.yaml'
- 'tests/**'
- 'assertions/**'
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
# Reuse cached model responses across runs to cut cost and time
- uses: actions/cache@v4
with:
path: ~/.promptfoo/cache
key: promptfoo-${{ hashFiles('promptfooconfig.yaml', 'prompts/**') }}
restore-keys: promptfoo-
- name: Run promptfoo eval
run: npx promptfoo@latest eval -c promptfooconfig.yaml --output results.json
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: promptfoo-results
path: results.json
Three details make this workflow practical. First, the paths filter means the job only runs when prompts or tests actually change, so unrelated PRs are not billed for model calls. Second, the cache key hashes the config and prompt files, which means unchanged cases replay from cache while edited ones re-run for real. Third, if: always() uploads the JSON artifact even on failure, so reviewers can inspect the exact output that broke.
Promptfoo also publishes an official GitHub Action that posts a results comment directly on the pull request. That is a better fit once the suite is stable and reviewers want the diff inline. For a broader look at pipeline structure, see CI/CD for Node.js projects using GitHub Actions.
Testing RAG Pipelines and Tool Calls
Prompt testing gets more valuable, not less, once your feature is more than a single completion.
For retrieval systems, promptfoo provides assertions that grade the answer against the retrieved context rather than against a golden string:
tests:
- vars:
query: "What is the return window for sale items?"
context: file://fixtures/returns_policy_chunk.txt
assert:
- type: context-faithfulness
threshold: 0.9 # answer must be grounded in the provided context
- type: context-relevance
threshold: 0.7 # retrieved context must be relevant to the query
- type: answer-relevance
threshold: 0.8
context-faithfulness is the hallucination detector: it flags answers that assert facts the context does not support. Meanwhile, context-relevance tells you whether the retriever, not the generator, is the weak link. Splitting those two signals matters, because the fixes are completely different — one is a chunking or reranking problem, the other is a prompting problem.
For agents and tool calls, validate the call shape before validating the prose:
assert:
- type: is-valid-openai-tools-call
- type: javascript
value: |
const call = JSON.parse(output)[0];
return call.function.name === 'lookup_order';
You can also point promptfoo at your own service instead of a model provider, which tests the entire pipeline end to end:
providers:
- id: https://staging-api.example.com/v1/support-reply
config:
method: POST
headers:
Content-Type: application/json
Authorization: 'Bearer {{ env.SUPPORT_API_TOKEN }}'
body:
message: '{{ customer_message }}'
order_age_days: '{{ order_age_days }}'
transformResponse: 'json.reply'
This is the highest-fidelity option, because it exercises your retrieval, your guardrails, and your post-processing together. Speaking of guardrails, promptfoo also has a red-team mode that generates adversarial inputs for injection, PII leakage, and jailbreak categories. Pair it with the defenses described in prompt injection defense patterns and LLM guardrails for input and output validation, so the tests and the mitigations evolve together.
Controlling Cost and Flakiness
Two problems kill prompt test suites: they get expensive, and they get flaky. Both are manageable.
For cost, lean on the response cache, which is enabled by default and keyed on the prompt plus provider config. Unchanged cases cost nothing on re-runs. Additionally, keep expensive llm-rubric assertions for cases where they earn their keep, and grade with a small model rather than your flagship.
For flakiness, pin temperature: 0 on every provider and every grader. Then, before trusting a new assertion, check its stability:
# Run the suite five times to surface non-deterministic assertions
npx promptfoo@latest eval --repeat 5 --no-cache
Any assertion that flips across those runs is either badly written or measuring something genuinely unstable. Rewrite it as a checklist rubric, lower it to a threshold, or delete it. A suite with ten trustworthy assertions beats one with fifty that the team has learned to ignore.
Real-World Scenario: Catching a Regression During a Model Upgrade
Consider a mid-sized SaaS product with a support-triage feature: incoming tickets are classified into an intent, an urgency score, and a needs-human flag, then routed accordingly. A small platform team owns it, and the provider has just deprecated the model they launched on, so the upgrade is not optional.
The obvious move is to swap the model ID and spot-check a dozen tickets by hand. That check passes, because hand-picked tickets are usually the clear ones. The cases that break in production are the ambiguous ones — a ticket that mentions both a shipping delay and a refund request, or one written in a mix of two languages.
With a promptfoo suite in place, the upgrade instead runs as a comparison: old model and new model, same prompt, same eighty test cases. Over an afternoon of iteration, a pattern like this is typical — overall pass rate is comparable, but the is-json schema assertion fails on a subset of multi-intent tickets because the newer model prefers to return an array of intents rather than a single value. That is a real breaking change for downstream routing code, and no amount of eyeballing sample outputs would have surfaced it reliably.
The trade-off is honest: building that suite costs a day or two upfront and requires someone to curate ambiguous cases rather than easy ones. For a feature that touches every inbound ticket, that cost is recovered the first time an upgrade would have shipped a silent routing bug. For a prototype nobody depends on, it is premature.
Common Errors and How to Fix Them
All test cases render the same output. Your prompt file probably references a variable name that does not exist in vars. Promptfoo renders missing Nunjucks variables as empty strings rather than erroring, so check spelling on both sides.
Latency assertions always pass with impossible numbers. Cached responses return instantly. Run latency comparisons with --no-cache.
Model-graded assertions fail with an authentication error. The grader has its own provider, which defaults to an OpenAI model. Either set OPENAI_API_KEY or override the grader under defaultTest.options.provider.
Results do not change after editing a prompt. Confirm the config points at the file you edited, then clear the cache with npx promptfoo@latest cache clear.
CI passes locally but fails in the pipeline. Nine times out of ten this is an unset secret, and the provider error surfaces as a failed assertion rather than a crash. Inspect the uploaded results.json to see the actual error string.
When to Use Promptfoo
- You have prompts in production and no objective way to review changes to them
- You need to compare models or providers on your own data before committing to one
- Your team wants prompt changes gated in CI alongside code
- You are evaluating cost and latency trade-offs between model tiers
- You need lightweight RAG evaluation without adopting a full evaluation platform
When NOT to Use Promptfoo
- You are still exploring what the prompt should do — iterate in a playground first, then codify
- Your primary need is production tracing and live debugging, which suits LangSmith tracing for LLM apps better
- You want deep, Python-native metric customization inside pytest, where DeepEval’s evaluation metrics fit more naturally
- Your team has no budget for grader model calls and your requirements are all subjective
- You need per-user online evaluation on live traffic rather than offline test runs
Common Mistakes With Promptfoo
- Leaving temperature unset, which produces a flaky suite the team stops trusting
- Grading everything with
llm-rubricwhen a regex or JSON schema would be free and stricter - Writing vague rubrics such as “the reply is good” instead of concrete checklists
- Curating only easy test cases, so the suite passes while production breaks on ambiguity
- Forgetting to pin the grader provider, which makes the baseline shift during model upgrades
- Running the suite on every pull request without a
pathsfilter, which burns budget on unrelated changes - Treating a passing suite as proof of safety rather than proof of no known regressions
Conclusion
Promptfoo works because it makes prompt quality reviewable. Deterministic assertions catch the mechanical failures, model-graded rubrics cover the judgment calls, and a non-zero exit code turns both into a gate that a pull request has to clear. The practical starting point is small: write five test cases covering your riskiest scenarios, add is-json or contains assertions where correctness is binary, and wire promptfoo eval into CI this week. Expand the suite each time production surprises you.
Once the suite is running, the natural next step is understanding what happens after deployment. Start with LangSmith tracing and debugging for LLM apps to close the loop between offline tests and live behavior, and revisit prompt engineering best practices to make sure the prompts you are testing are worth defending.
