OLMoE

AI model · Language model (LLM) · Text → Text by Ai2

The first fully open mixture-of-experts model on the Pareto frontier of performance and size.

OLMoE is a fully open mixture-of-experts (MoE) language model designed for high performance and efficiency. It is the first model to reach the Pareto frontier of performance and size while providing complete transparency through open data, code, and training logs.

Model specs

The full spec card (context, pricing per token, modalities and more) on AntSpace. Sign up free to see the full spec card.

Category: Natural Language Processing

Architecture
Mixture-of-Experts (MoE)
Training Speed Efficiency
2x faster than equivalent dense models
Open Assets
Data, code, recipes, checkpoints, logs, evaluations
Mobile Support
Open-source iOS app available
  • First mixture-of-experts model in the OLMo family
  • Reaches the Pareto frontier of performance and size
  • Available as an open-source iOS application for on-device testing
  • Released with open data, code, recipes, and training logs

Overview

OLMoE is the first mixture-of-experts (MoE) model to join the OLMo family of language models from Ai2. It is designed for researchers and developers who require a balance of high performance and computational efficiency, being the first model to reach the Pareto frontier of performance and size. OLMoE distinguishes itself by being fully open, providing not just weights but also the training data, code, recipes, evaluations, logs, and intermediate checkpoints. This transparency allows for deep scientific study and infinite customizability of the model pipeline. Additionally, Ai2 provides an open-source iOS application, enabling users to run and test OLMoE privately and securely on their own mobile devices. The model's architecture allows it to be trained twice as fast as equivalent dense models.

Architecture and Efficiency

  • Mixture-of-Experts Architecture: The model is the first in the OLMo family to use a mixture-of-experts (MoE) design for improved efficiency.
  • 2x Faster Training: OLMoE can be trained twice as fast as equivalent dense models, reducing computational requirements.
  • Pareto Frontier Performance: It is the first model to reach the Pareto frontier of performance and size in its class.

Access and Transparency

  • iOS Application: Ai2 provides open-source iOS applications that allow users to run and test the model privately on their own devices.
  • Fully Open Science: The model is released with complete open data, code, recipes, and intermediate training checkpoints.
  • Transparent Training Logs: Users can access evaluations and training logs to understand the model's development process.

Use cases

  • Researchers use the open training logs and intermediate checkpoints to study the science of language model development.
  • Developers deploy OLMoE on iOS devices to provide private and secure on-device AI capabilities to users.

Specifications

Access
Open data, code, recipes, intermediate checkpoints, and iOS app
Architecture
Mixture-of-Experts (MoE)
Training Speed Efficiency
2x faster than equivalent dense models
Open Assets
Data, code, recipes, checkpoints, logs, evaluations
Mobile Support
Open-source iOS app available

How it works

  1. Access Model Resources: Developers can access the model's open weights, training data, code, and recipes from Ai2's open ecosystem.
  2. Deployment and Testing: The model can be deployed on standard infrastructure or tested on mobile devices using the provided open-source iOS application.
  3. Research and Customization: Researchers can use the intermediate checkpoints and training logs to study the model's development or customize it for specific tasks.

FAQ

What is OLMoE?

OLMoE is the first mixture-of-experts (MoE) model to join the OLMo family, designed to reach the Pareto frontier of performance and size.

What makes OLMoE "fully open"?

The model is released with open data, code, evaluations, logs, and intermediate training checkpoints to support fully open science.

Can I run OLMoE on a mobile device?

Yes, Ai2 has released an open-source iOS app that allows users to test OLMoE on their devices privately and securely.

Get started with OLMoE

Added by AntSpace from public sources (allenai.org).

Related

More AI models by Ai2