Mistral Voice Agent
AI agent · Voice agent · Customer support by Mistral AI
Build lifelike voice agents that listen, reason, and speak in 35+ languages.
Mistral Voice Agent is an end-to-end AI conversational platform for building low-latency voice assistants. It combines speech-to-text, frontier reasoning, and lifelike text-to-speech to automate customer support and digital interactions with full human-in-the-loop control.
Agent specs
- What it does
- Mistral Voice Agent is an end-to-end voice conversation system that automates the entire pipeline of speech capture, transcription, reasoning, and speech generation.
- For
- Developers building voice-enabled applications using Python or Node.js
- Actions
- Captures voice input, Transcribes speech to text, Generates model responses, Speaks answers via text-to-speech
- Integrations
- Python SDK, Node.js, PyAudio, PortAudio, SoX, FFmpeg, LiveKit
- Channels
- API, terminal
- Models
- Voxtral
- Availability
- Available via the Voice Agents API for terminal and browser-based applications.
- Languages
- English, French
Category: AI Agents
- Channels
- Website chat, API, Terminal, Mobile apps
- Latency
- Low latency transcription and speech generation
- Deployment
- Mistral Cloud, On-premises, Private Cloud
- Setup
- 10 minutes to build a complete voice agent via quickstart
- Languages
- 35+ languages and 50+ dialects supported by Voxtral TTS 3
- Primary Model Family
- Voxtral
- End-to-end voice capture, transcription, and speech generation
- Powered by Voxtral TTS 3 with 35+ languages and 50+ dialects
- Low-latency realtime transcription and turn detection
- Human-in-the-loop controls for sensitive tool-based actions
Overview
Mistral Voice Agent is an AI-powered conversational assistant platform that enables businesses to build and deploy end-to-end voice interactions. It manages the entire pipeline of capturing audio, transcribing speech with low latency, generating intelligent responses using Mistral's frontier models, and speaking back to the user. Built on the Voice Agents API, it supports advanced features like professional voice cloning, diarization, and real-time turn detection. Developers can customize agents with specific instructions and integrate them with external tools for tasks like customer support or digital concierge services. The agent is accessible via Python and Node.js SDKs and can be deployed in web browsers or local terminal environments.
Voice Interaction
- End-to-End Voice Pipeline: The agent handles the entire pipeline of capturing voice, transcribing it, generating an answer with a model, and speaking that answer.
- Real-time Transcription: The agent supports low-latency speech transcription and diarization for natural conversations.
- Speech Generation: The agent converts text into natural speech and plays it back with a single API call.
- Turn Detection: The agent supports realtime transcription for agentic use-cases with configurable latency and turn detection.
Control and Customization
- Instruction-based Behavior: The agent can be given specific instructions in natural language to shape its personality and response style.
- Customizable Voices: Users can change the agent's voice by specifying parameters such as fr_marie_neutral.
- Tool Integration: The agent can be integrated with external tools and services to perform actions beyond conversation.
- Human-in-the-Loop Approval: The agent shows every tool call and asks for approval before sensitive actions.
Deployment and Context
- Web Deployment: The agent can be deployed in web pages using the Browser guide.
- Terminal Integration: Developers can run voice agents from the terminal using Python or Node.js.
- Knowledge Grounding: The agent can be grounded in specific customer context, products, and policies to resolve routine requests.
What it does
- Voice capture
- Speech transcription
- Intelligent response generation
- Text-to-speech playback
- Tool calling and execution
- Diarization
Use cases
- Customer support teams build secure voice assistants grounded in product context to resolve routine requests faster.
- Developers create real-time audio fact-checking agents that listen to live broadcasts and verify information.
- Enterprises deploy voice-based digital concierges that use tools to schedule appointments and manage bookings.
- Organizations use voice agents for automated transcription and diarization of multi-speaker meetings.
- Engineering teams build voice-controlled terminal agents to execute commands and manage workflows through speech.
Specifications
- Channels
- Website chat, API, Terminal, Mobile apps
- Integrations
- Python SDK, Node.js SDK, PortAudio, PyAudio, LiveKit, NumPy, SoX, FFmpeg
- Latency
- Low latency transcription and speech generation
- Human control
- Asks for approval before sensitive actions and shows every tool call to the user
- Models
- Voxtral TTS 3, Voxtral Mini Transcribe 3, Mistral Large 4, Mistral Medium 3.5
- Deployment
- Mistral Cloud, On-premises, Private Cloud
- Setup
- 10 minutes to build a complete voice agent via quickstart
- Languages
- 35+ languages and 50+ dialects supported by Voxtral TTS 3
- Primary Model Family
- Voxtral
- Supported Languages
- 35+ languages
- Dialects Supported
- 50+ dialects
- Transcription Latency
- Low latency
- Deployment Options
- Cloud, On-premises, Private Cloud
- SDK Support
- Python, TypeScript, Node.js
- Audio Features
- Zero-shot voice cloning, streaming, turn detection
- Control Mechanisms
- Human-in-the-loop approval, tool call visibility
How it works
- Install SDK and Tools: Install the Mistral SDK and required audio tools like PortAudio, PyAudio, or FFmpeg depending on your environment.
- Configure API Access: Generate a Mistral API key in Studio and store it in the MISTRAL_API_KEY environment variable.
- Initialize Agent Session: Initialize an audio session with the Mistral client, providing instructions and selecting an audio interface.
- Run and Interact: Run the agent to start a voice conversation where the agent transcribes your speech and responds in real time.
Pricing
- Voxtral Mini Transcribe 3 (Inference)
- $0.0035 /Min
- Voxtral TTS 3 (Inference)
- Premier pricing (Contact for details)
FAQ
What is a Mistral Voice Agent?
Mistral Voice Agents handle the full pipeline of capturing voice, transcribing it, generating a model response, and speaking the answer back to the user.
Can I customize the agent's voice?
You can change the agent's voice by using the tts parameter and specifying a voice ID, such as fr_marie_neutral.
Can the voice agent use tools?
Yes, the Voice Agents API allows you to give your agent tools so it can interact with external systems and perform complex tasks.
Where can I deploy a Mistral Voice Agent?
Voice agents can be run in a terminal using Python or Node.js, or deployed in a web page following the Browser guide.
Added by AntSpace from public sources (docs.mistral.ai).