DeepEval
Product · Developer tool by Confident AI
The open-source LLM evaluation framework for Python and TypeScript.
DeepEval is an open-source LLM evaluation framework for Python and TypeScript that provides 50+ research-backed metrics to test AI applications. It enables developers to run unit-test style evaluations locally or in CI/CD pipelines to catch regressions before deployment.
- Price
- From 0 USD
- Launched
- 2023-11-01
- Languages
- Python, TypeScript
Category: Software Testing
- Programming languages
- Python, TypeScript
- Editors and tools
- Pytest
- License
- Apache 2.0
- Deployment
- Local, CI/CD, Cloud Sync to Confident AI
- Metric Count
- 50+ research-backed metrics
- Supported Languages
- Python, TypeScript
- 50+ research-backed evaluation metrics
- Open-source under Apache 2.0 license
- Native support for Python and TypeScript
- Integrates with CI/CD for regression testing
- 14K+ GitHub stars and 2,500+ Discord members
Overview
DeepEval is an open-source LLM evaluation framework developed by Confident AI for engineering teams to validate AI quality. It provides over 50 research-backed metrics, including faithfulness, answer relevancy, and hallucination detection, to test AI applications locally or in CI/CD pipelines. The framework supports both Python and TypeScript, allowing developers to write evaluation tests that function like unit tests for LLMs. DeepEval is designed to catch regressions before deployment by comparing new test runs against established baselines. While it operates as a standalone open-source tool under the Apache 2.0 license, it also serves as the client library for the Confident AI platform to enable shared dashboards and organization-wide collaboration.
Evaluation Metrics
- 50+ Research-Backed Metrics: DeepEval includes 50+ research-backed metrics such as faithfulness, answer relevancy, contextual recall, and hallucination detection.
- Custom Metric Definitions: Developers can define fully custom metrics using Python logic or LLM-as-a-judge approaches to fit specific use cases.
- Use Case Support: The framework supports evaluation for summarization, Text-SQL, customer support chatbots, and internal RAG QA systems.
- Agentic System Evaluation: DeepEval provides tailored metrics for complex agentic workflows, including tool calls and reasoning chains.
Developer Workflows
- CI/CD Regression Gates: DeepEval integrates directly into CI/CD pipelines to run regression tests on every pull request.
- Unit Testing for LLMs: The framework works like unit testing for LLMs, allowing developers to write and run tests locally in Python or TypeScript.
- Pytest Integration Orientated: DeepEval integrates with Pytest to allow running evaluation tests as part of standard developer workflows.
- Baseline Comparisons: Users can automatically compare new test runs against a last known-good baseline to surface exact regressions.
Integrations and Ecosystem
- OpenTelemetry Native: DeepEval is OTel-native and provides SDKs that feed data directly into traces and evaluations.
- AI Stack Integrations: The framework integrates with popular AI stacks including LangChain, LlamaIndex, CrewAI, and Vercel AI SDK.
- Cloud and Gateway Support: DeepEval supports major cloud providers and gateways such as AWS, Google Cloud, Azure, and LiteLLM.
- Cloud Syncing: Test runs can be automatically synced to a shared dashboard for team-wide visibility and collaboration.
Use cases
- Engineers run automated evaluation suites in CI/CD pipelines to detect breaking changes before merging code.
- Developers use 50+ research-backed metrics to measure the faithfulness and relevancy of RAG pipeline outputs.
- QA teams build regression suites to validate AI behavior across thousands of scenarios rather than relying on manual checks.
Specifications
- Programming languages
- Python, TypeScript
- Editors and tools
- Pytest
- Integrations
- LangChain, LangGraph, LlamaIndex, Pydantic AI, CrewAI, Vercel AI SDK, LiteLLM, Portkey, OpenRouter, AWS, Google Cloud, Azure, OpenTelemetry
- License
- Apache 2.0
- Deployment
- Local, CI/CD, Cloud Sync to Confident AI
- Metric Count
- 50+ research-backed metrics
- Supported Languages
- Python, TypeScript
- GitHub Stars
- 14,000+
- Community Members
- 2,500+ (Discord)
- Integration Type
- OTel-native
How it works
- Install the Framework: Install the framework using pip install deepeval for Python or the equivalent for TypeScript.
- Write Evaluation Tests: Define evaluation tests using research-backed or custom metrics to measure LLM performance.
- Run Tests and CI/CD: Execute tests locally or integrate them into CI/CD pipelines to catch regressions before deployment.
- Sync to Cloud Dashboard: Optionally run 'deepeval login' to sync test results to a shared dashboard for team collaboration.
Pricing
- DeepEval Open Source
- Free
FAQ
What metrics are included in DeepEval?
DeepEval includes 50+ research-backed metrics such as faithfulness, answer relevancy, contextual recall, contextual precision, hallucination, bias, and toxicity.
How do I get started with DeepEval?
You can install it via pip install deepeval, write an evaluation test, and optionally connect to the cloud platform by running deepeval login.
Added by AntSpace from public sources (confident-ai.com).