DeepEval

Product · Developer tool by Confident AI

The open-source LLM evaluation framework for Python and TypeScript.

DeepEval is an open-source LLM evaluation framework for Python and TypeScript that provides 50+ research-backed metrics to test AI applications. It enables developers to run unit-test style evaluations locally or in CI/CD pipelines to catch regressions before deployment.

Price
From 0 USD
Launched
2023-11-01
Languages
Python, TypeScript

Category: Software Testing

Programming languages
Python, TypeScript
Editors and tools
Pytest
License
Apache 2.0
Deployment
Local, CI/CD, Cloud Sync to Confident AI
Metric Count
50+ research-backed metrics
Supported Languages
Python, TypeScript
  • 50+ research-backed evaluation metrics
  • Open-source under Apache 2.0 license
  • Native support for Python and TypeScript
  • Integrates with CI/CD for regression testing
  • 14K+ GitHub stars and 2,500+ Discord members

Overview

DeepEval is an open-source LLM evaluation framework developed by Confident AI for engineering teams to validate AI quality. It provides over 50 research-backed metrics, including faithfulness, answer relevancy, and hallucination detection, to test AI applications locally or in CI/CD pipelines. The framework supports both Python and TypeScript, allowing developers to write evaluation tests that function like unit tests for LLMs. DeepEval is designed to catch regressions before deployment by comparing new test runs against established baselines. While it operates as a standalone open-source tool under the Apache 2.0 license, it also serves as the client library for the Confident AI platform to enable shared dashboards and organization-wide collaboration.

Evaluation Metrics

  • 50+ Research-Backed Metrics: DeepEval includes 50+ research-backed metrics such as faithfulness, answer relevancy, contextual recall, and hallucination detection.
  • Custom Metric Definitions: Developers can define fully custom metrics using Python logic or LLM-as-a-judge approaches to fit specific use cases.
  • Use Case Support: The framework supports evaluation for summarization, Text-SQL, customer support chatbots, and internal RAG QA systems.
  • Agentic System Evaluation: DeepEval provides tailored metrics for complex agentic workflows, including tool calls and reasoning chains.

Developer Workflows

  • CI/CD Regression Gates: DeepEval integrates directly into CI/CD pipelines to run regression tests on every pull request.
  • Unit Testing for LLMs: The framework works like unit testing for LLMs, allowing developers to write and run tests locally in Python or TypeScript.
  • Pytest Integration Orientated: DeepEval integrates with Pytest to allow running evaluation tests as part of standard developer workflows.
  • Baseline Comparisons: Users can automatically compare new test runs against a last known-good baseline to surface exact regressions.

Integrations and Ecosystem

  • OpenTelemetry Native: DeepEval is OTel-native and provides SDKs that feed data directly into traces and evaluations.
  • AI Stack Integrations: The framework integrates with popular AI stacks including LangChain, LlamaIndex, CrewAI, and Vercel AI SDK.
  • Cloud and Gateway Support: DeepEval supports major cloud providers and gateways such as AWS, Google Cloud, Azure, and LiteLLM.
  • Cloud Syncing: Test runs can be automatically synced to a shared dashboard for team-wide visibility and collaboration.

Use cases

  • Engineers run automated evaluation suites in CI/CD pipelines to detect breaking changes before merging code.
  • Developers use 50+ research-backed metrics to measure the faithfulness and relevancy of RAG pipeline outputs.
  • QA teams build regression suites to validate AI behavior across thousands of scenarios rather than relying on manual checks.

Specifications

Programming languages
Python, TypeScript
Editors and tools
Pytest
Integrations
LangChain, LangGraph, LlamaIndex, Pydantic AI, CrewAI, Vercel AI SDK, LiteLLM, Portkey, OpenRouter, AWS, Google Cloud, Azure, OpenTelemetry
License
Apache 2.0
Deployment
Local, CI/CD, Cloud Sync to Confident AI
Metric Count
50+ research-backed metrics
Supported Languages
Python, TypeScript
GitHub Stars
14,000+
Community Members
2,500+ (Discord)
Integration Type
OTel-native

How it works

  1. Install the Framework: Install the framework using pip install deepeval for Python or the equivalent for TypeScript.
  2. Write Evaluation Tests: Define evaluation tests using research-backed or custom metrics to measure LLM performance.
  3. Run Tests and CI/CD: Execute tests locally or integrate them into CI/CD pipelines to catch regressions before deployment.
  4. Sync to Cloud Dashboard: Optionally run 'deepeval login' to sync test results to a shared dashboard for team collaboration.

Pricing

DeepEval Open Source
Free

FAQ

What metrics are included in DeepEval?

DeepEval includes 50+ research-backed metrics such as faithfulness, answer relevancy, contextual recall, contextual precision, hallucination, bias, and toxicity.

How do I get started with DeepEval?

You can install it via pip install deepeval, write an evaluation test, and optionally connect to the cloud platform by running deepeval login.

Try Now For Free

Added by AntSpace from public sources (confident-ai.com).

Related

More products by Confident AI