PromptFoo

PromptFoo is the open-source CLI for systematic LLM evaluation, red teaming, and side-by-side model comparison (e.g., GPT-4 vs. Claude 3) in CI/CD.

PromptFoo enables test-driven LLM development, moving past trial-and-error with a powerful CLI and library. Developers use declarative YAML configs to define test cases, running evaluations across multiple providers (OpenAI, Anthropic, Llama) simultaneously. It provides automated scoring via assertions (e.g., `is-valid-json`, `context-faithfulness`) and matrix views for side-by-side output comparison. Crucially, it secures LLM applications with built-in red teaming and vulnerability scanning, integrating cleanly into CI/CD pipelines to ensure non-regression and quality for production systems.

https://promptfoo.dev