Trinity-Large-Base

AI model · Language model (LLM) · Text → Text by Arcee AI

The 17-Trillion-Token Foundation for Open Intelligence

Trinity-Large-Base is a 398B-parameter pretrained foundation model trained on 17 trillion tokens. It provides a raw, high-performance starting point for developers and researchers to build custom AI models under a permissive Apache 2.0 license.

Model specs

The full spec card (context, pricing per token, modalities and more) on AntSpace. Sign up free to see the full spec card.

Languages
English

Category: Generative AI

Modalities
Text → Text
Context window
8,192 tokens
Parameters
398B total / 13B active per token
License
Apache 2.0
Access
Download via Hugging Face and GitHub
Total Parameters
398B
  • 17-trillion-token pretrained foundation model
  • 398B parameter sparse Mixture-of-Experts architecture
  • 13B active parameters per token for efficient inference
  • Released under permissive Apache 2.0 license
  • 8,192 token pretraining context length

Overview

Trinity-Large-Base is Arcee AI's full 17-trillion-token pretrained foundation model, serving as the primary base for the Trinity Large family of models. It is designed for researchers and developers who require a highly capable, permissively licensed starting point for further fine-tuning or specialization. The model includes mid-training anneals to optimize performance and is released under the Apache 2.0 license to support the open-model community. By providing access to raw model intelligence, Trinity-Large-Base enables organizations to build and own their AI foundations without vendor lock-in.

Foundation Capabilities

  • Pretrained Foundation: The model is a full 17-trillion-token pretrained foundation model that serves as the starting point for the Trinity Large family.
  • Mid-training Anneals: It includes mid-training anneals to optimize performance before the post-training phase.
  • Open Weights: The model is released with open weights under the permissive Apache 2.0 license.
  • Raw Intelligence Access: It provides a rare look at raw model intelligence for researchers and developers to build upon.

Architecture and Efficiency

  • Sparse MoE Architecture: The model utilizes a sparse Mixture-of-Experts architecture with approximately 398 billion total parameters.
  • Active Parameters: Only 13 billion parameters are active per token, ensuring efficient inference despite the large total size.
  • Expert Routing: The architecture features 256 experts, with one shared expert and a 4-of-256 routing strategy.
  • Causal LM Design: It is built using the AfmoeForCausalLM architecture for high-performance causal language modeling.

Use cases

  • Researchers use the base model to study raw model intelligence and pretraining dynamics in large-scale MoE architectures.
  • Developers fine-tune the model on domain-specific datasets to create specialized versions for technical or scientific workloads.
  • Organizations deploy the model on private infrastructure to maintain full ownership and control over their AI weights and data.

Specifications

Modalities
Text → Text
Context window
8,192 tokens
Parameters
398B total / 13B active per token
Sizes and variants
Trinity-Large-Thinking, Trinity-Large-Preview, Trinity-Large-TrueBase, Trinity-Large-Base
License
Apache 2.0
Access
Download via Hugging Face and GitHub
Total Parameters
398B
Active Parameters per Token
13B
Training Tokens
17 Trillion
Architecture
Sparse MoE (AfmoeForCausalLM)
Pretraining Context Length
8,192 tokens
Experts
256 (1 shared)
Active Experts
4
Routing Strategy
4-of-256 (1.56% sparsity)
Dense Layers
6

How it works

  1. Access Model Weights: Access the Arcee AI Open Source Catalog or official repositories on Hugging Face and GitHub to find the model weights.
  2. Download and Deploy: Download the open-weight checkpoints for the 398B parameter model to your own infrastructure.
  3. Fine-tune or Specialize: Use the base model as a starting point for full-parameter fine-tuning or domain-specific specialization.

FAQ

What is the difference between Trinity-Large-Base and TrueBase?

Trinity-Large-Base is the full 17-trillion-token pretrained foundation model with mid-training anneals, whereas Trinity-Large-TrueBase is a 10-trillion-token pre-anneal checkpoint.

Is Trinity-Large-Base open source?

Yes, the model is released under the permissive Apache 2.0 license, allowing for broad use and modification.

What is the architecture of Trinity-Large-Base?

It is a sparse Mixture-of-Experts (MoE) model with approximately 398 billion total parameters and 13 billion active parameters per token.

How many tokens was Trinity-Large-Base trained on?

The model was trained on 17 trillion tokens of curated, high-quality data.

Added by AntSpace from public sources (arcee.ai).

Related

More AI models by Arcee AI