Voice Descriptor Model
AI model by Hume AI
Measure 190 vocal characteristics to describe how a voice sounds beyond emotional content.
The Voice Descriptor Model is a specialized AI model that measures 190 vocal characteristics across 27 families to describe speech qualities beyond emotion. It is designed for developers to create rich annotations for analysis and real-time interaction guidance across 16 languages.
Model specs
The full spec card (context, pricing per token, modalities and more) on AntSpace. Sign up free to see the full spec card.
- Languages
- Arabic, Bengali, German, English, Spanish, French, Hebrew, Indonesian, Italian, Japanese, Korean, Polish
Category: Machine Learning
- Modalities
- Audio → Structured scores
- Access
- API (Hume Expression APIs)
- Vocal Characteristics
- 190 tags
- Characteristic Families
- 27 families
- Languages Supported
- 16 languages
- Training Data Size
- 260,000+ speech recordings
- Measures 190 vocal characteristics across 27 families
- Identifies qualities like raspy, breathy, fast, and monotone
- Supports 16 languages including English, Spanish, and Japanese
- Trained on over 260,000 speech recordings
Overview
The Voice Descriptor Model is a specialized AI model from Hume AI that measures 190 vocal characteristics across 27 families to describe the qualities of speech. It goes beyond emotional labels to identify specific traits such as whether a voice is raspy, breathy, fast, monotone, or theatrical. Designed for developers and researchers, it enables the creation of rich annotations for analysis, training, and real-time interaction guidance. The model supports 16 languages and is delivered via Hume's Expression APIs to provide structured scores for recorded or streaming audio. It allows for the detection of nuanced combinations, such as identifying whispered speech or specific speaking styles that transcripts typically miss. The model was trained on a shared speech resource containing more than 260,000 recordings to ensure high agreement with human perception.
Vocal Characterization
- 190 Vocal Characteristics: The model measures 190 distinct vocal characteristics across 27 families to describe how a voice sounds beyond its emotional content.
- Vocal Quality Detection: It identifies specific vocal qualities including raspy, breathy, fast, monotone, whispering, and theatrical delivery.
Data and Integration
- Extensive Training Data: The model was trained on a shared speech resource containing more than 260,000 recordings.
- Annotated Speech Resource: The training resource includes more than 210,000 emotion-annotated recordings representing over 270 hours of speech.
- Timestamped Analysis: The model provides timestamped results that link voice descriptor scores to specific speech segments for detailed analysis.
- API Delivery: The model is delivered via Hume's Expression APIs to provide structured scores for recorded or streaming audio.
Use cases
- Developers use the model to identify specific speaking styles like whispering or theatrical delivery in recorded conversations.
- Researchers analyze vocal qualities such as raspiness or breathiness to add context to speech data beyond simple transcripts.
- AI teams use the timestamped tags to retrieve recordings with specific characteristics for training other machine learning models.
- Product teams measure vocal expression in candidate interviews to highlight specific moments for closer review.
Specifications
- Modalities
- Audio → Structured scores
- Access
- API (Hume Expression APIs)
- Languages
- Arabic, Bengali, German, English, Spanish, French, Hebrew, Indonesian, Italian, Japanese, Korean, Polish, Portuguese, Russian, Turkish, and Vietnamese
- Vocal Characteristics
- 190 tags
- Characteristic Families
- 27 families
- Languages Supported
- 16 languages
- Training Data Size
- 260,000+ speech recordings
- Annotated Recordings
- 210,000+ recordings
- Training Audio Duration
- 270+ hours
How it works
- Audio Input via API: Users send audio data to the Hume Expression APIs, either as a stream for real-time analysis or as individual clips for batch processing.
- Vocal Characteristic Measurement: The model processes the audio to identify 190 vocal characteristics and speaking styles across 27 different families.
- Structured Data Output: The API returns structured scores and timestamped tags that link specific vocal qualities to segments of the speech.
FAQ
What vocal characteristics does the model measure?
The model measures 190 characteristics across 27 families, including qualities such as raspy, breathy, fast, monotone, whispering, and theatrical.
What data was used to train the model?
The model was trained on a speech resource with more than 260,000 recordings, including over 210,000 emotion-annotated recordings.
How are the results delivered?
The model provides timestamped results that link expression scores to the corresponding speech segment, allowing users to review the source alongside its measurements.
Added by AntSpace from public sources (hume.ai).