Trinity-Large-Base
AI model · Language model (LLM) · Text → Text by Arcee AI
The 17-Trillion-Token Foundation for Open Intelligence
Trinity-Large-Base is a 398B-parameter pretrained foundation model trained on 17 trillion tokens. It provides a raw, high-performance starting point for developers and researchers to build custom AI models under a permissive Apache 2.0 license.
Model specs
The full spec card (context, pricing per token, modalities and more) on AntSpace. Sign up free to see the full spec card.
- Languages
- English
Category: Generative AI
- Modalities
- Text → Text
- Context window
- 8,192 tokens
- Parameters
- 398B total / 13B active per token
- License
- Apache 2.0
- Access
- Download via Hugging Face and GitHub
- Total Parameters
- 398B
- 17-trillion-token pretrained foundation model
- 398B parameter sparse Mixture-of-Experts architecture
- 13B active parameters per token for efficient inference
- Released under permissive Apache 2.0 license
- 8,192 token pretraining context length
Overview
Trinity-Large-Base is Arcee AI's full 17-trillion-token pretrained foundation model, serving as the primary base for the Trinity Large family of models. It is designed for researchers and developers who require a highly capable, permissively licensed starting point for further fine-tuning or specialization. The model includes mid-training anneals to optimize performance and is released under the Apache 2.0 license to support the open-model community. By providing access to raw model intelligence, Trinity-Large-Base enables organizations to build and own their AI foundations without vendor lock-in.
Foundation Capabilities
- Pretrained Foundation: The model is a full 17-trillion-token pretrained foundation model that serves as the starting point for the Trinity Large family.
- Mid-training Anneals: It includes mid-training anneals to optimize performance before the post-training phase.
- Open Weights: The model is released with open weights under the permissive Apache 2.0 license.
- Raw Intelligence Access: It provides a rare look at raw model intelligence for researchers and developers to build upon.
Architecture and Efficiency
- Sparse MoE Architecture: The model utilizes a sparse Mixture-of-Experts architecture with approximately 398 billion total parameters.
- Active Parameters: Only 13 billion parameters are active per token, ensuring efficient inference despite the large total size.
- Expert Routing: The architecture features 256 experts, with one shared expert and a 4-of-256 routing strategy.
- Causal LM Design: It is built using the AfmoeForCausalLM architecture for high-performance causal language modeling.
Use cases
- Researchers use the base model to study raw model intelligence and pretraining dynamics in large-scale MoE architectures.
- Developers fine-tune the model on domain-specific datasets to create specialized versions for technical or scientific workloads.
- Organizations deploy the model on private infrastructure to maintain full ownership and control over their AI weights and data.
Specifications
- Modalities
- Text → Text
- Context window
- 8,192 tokens
- Parameters
- 398B total / 13B active per token
- Sizes and variants
- Trinity-Large-Thinking, Trinity-Large-Preview, Trinity-Large-TrueBase, Trinity-Large-Base
- License
- Apache 2.0
- Access
- Download via Hugging Face and GitHub
- Total Parameters
- 398B
- Active Parameters per Token
- 13B
- Training Tokens
- 17 Trillion
- Architecture
- Sparse MoE (AfmoeForCausalLM)
- Pretraining Context Length
- 8,192 tokens
- Experts
- 256 (1 shared)
- Active Experts
- 4
- Routing Strategy
- 4-of-256 (1.56% sparsity)
- Dense Layers
- 6
How it works
- Access Model Weights: Access the Arcee AI Open Source Catalog or official repositories on Hugging Face and GitHub to find the model weights.
- Download and Deploy: Download the open-weight checkpoints for the 398B parameter model to your own infrastructure.
- Fine-tune or Specialize: Use the base model as a starting point for full-parameter fine-tuning or domain-specific specialization.
FAQ
What is the difference between Trinity-Large-Base and TrueBase?
Trinity-Large-Base is the full 17-trillion-token pretrained foundation model with mid-training anneals, whereas Trinity-Large-TrueBase is a 10-trillion-token pre-anneal checkpoint.
Is Trinity-Large-Base open source?
Yes, the model is released under the permissive Apache 2.0 license, allowing for broad use and modification.
What is the architecture of Trinity-Large-Base?
It is a sparse Mixture-of-Experts (MoE) model with approximately 398 billion total parameters and 13 billion active parameters per token.
How many tokens was Trinity-Large-Base trained on?
The model was trained on 17 trillion tokens of curated, high-quality data.
Added by AntSpace from public sources (arcee.ai).