Native Inference

Service · Managed service by Superlinked

Managed SIE clusters in your own cloud with expert engineering support.

Native Inference is a managed service for deploying the Superlinked Inference Engine (SIE) in a customer's own cloud environment. It enables enterprise teams to run 149+ task-specific models on shared GPUs with scale-to-zero autoscaling and direct engineering support.

Languages
English

Category: AI Infrastructure

Deployment
Terraform and Helm on Kubernetes (GKE, EKS, AKS)
Supported Clouds
AWS, GCP, Azure, Alibaba Cloud
Deployment Tools
Terraform, Helm, Kubernetes
Autoscaling Engine
KEDA, Prometheus
Messaging Bus
NATS JetStream
GPU Types
NVIDIA L4, A100 40GB, A100 80GB
  • Managed SIE deployment in your own cloud account
  • Engineering support for model evaluation and operations
  • Scale-to-zero autoscaling via KEDA and Prometheus
  • Supports AWS, GCP, Azure, and Alibaba Cloud
  • Shared GPU serving for 149+ task-specific models

Overview

Native Inference is Superlinked's managed deployment service for enterprise teams that need to run the Superlinked Inference Engine (SIE) within their own cloud infrastructure. It provides a private, self-hosted version of the SIE stack on AWS, GCP, Azure, or Alibaba Cloud, backed by Superlinked's engineering team for model evaluation and operational support. The solution solves the problem of high frontier model costs by serving 149 task-specific open models on shared GPUs with scale-to-zero autoscaling. It features a stateless Rust gateway for GPU-aware routing and uses KEDA to manage worker pools based on real-time demand. This allows businesses to maintain full data sovereignty while achieving near-frontier quality at a fraction of the cost of public APIs.

Enterprise Deployment

  • In-Cluster Deployment: The solution allows teams to run the Superlinked Inference Engine (SIE) cluster within their own cloud environment for full data control.
  • Engineering Support: Superlinked engineers provide direct assistance to evaluate models and operate the production deployment.
  • Cloud Infrastructure Automation: The deployment uses standardized Terraform modules and Helm charts for AWS, GCP, Azure, and Alibaba Cloud.
  • Hybrid Workload Support: Teams can keep sensitive workloads on their own cluster while using the same model catalog and SDK as the hosted API.

Inference Architecture

  • Scale-to-Zero Autoscaling: The architecture uses KEDA and Prometheus to scale worker pools from zero to a maximum of 10 nodes based on demand.
  • GPU-Aware Routing engine: A stateless Rust gateway handles GPU-aware routing to appropriate pools via specific HTTP headers or SDK parameters.
  • LRU Model Eviction: The system unloads the least recently used models when GPU memory fills up to accommodate new requests.

Operations and Management

  • Tenant Isolation: Native Inference provides dedicated worker pods to reserve capacity and isolate data for specific tenants.
  • Local Model Cache: The worker pods use a 300Gi emptyDir volume for local model caching to speed up subsequent requests.
  • Observability Stack: The deployment includes Grafana dashboards and monitoring via Prometheus for operational visibility.
  • Automated Model Management: The server handles batching, GPU sharing, and model switching automatically across 149 supported models.

Use cases

  • Enterprise teams deploy SIE in their own VPC to keep sensitive data within their security perimeter.
  • Platform engineers use Terraform and Helm to automate the deployment of multi-model GPU clusters.
  • AI developers route requests to dedicated A100 GPU pools for high-throughput document processing tasks.

Specifications

Deployment
Terraform and Helm on Kubernetes (GKE, EKS, AKS)
Supported Clouds
AWS, GCP, Azure, Alibaba Cloud
Deployment Tools
Terraform, Helm, Kubernetes
Autoscaling Engine
KEDA, Prometheus
Messaging Bus
NATS JetStream
GPU Types
NVIDIA L4, A100 40GB, A100 80GB
Model Cache Volume
300Gi emptyDir
Gateway Language
Rust
Worker Adapter Language
Python
License
Apache 2.0 (Core Engine)

How it works

  1. Infrastructure Provisioning: Superlinked provides Terraform modules to provision the VPC, Kubernetes cluster (GKE/EKS/AKS), and necessary IAM roles.
  2. Helm Installation: The SIE stack is deployed via Helm, including the Rust gateway, NATS bus, and worker pools.
  3. Model Evaluation: Superlinked engineers assist in selecting and evaluating task-specific models from the 149 available options.
  4. Application Integration: The platform team configures access rules and SDKs to route application requests to the private SIE cluster.

FAQ

What is Native Inference?

Native Inference is a managed deployment where Superlinked engineers help evaluate models and operate the SIE cluster within the customer's own cloud.

Which cloud providers are supported?

It supports AWS, GCP, Azure, and Alibaba Cloud using standardized Terraform modules and Helm charts.

Does it support scaling to zero?

Yes, it uses KEDA and Prometheus to scale worker pools from zero when idle to optimize costs.

What GPU types can be used?

It supports NVIDIA L4 (24GB), A100 40GB, and A100 80GB accelerators for different model sizes and throughput needs.

Added by AntSpace from public sources (superlinked.com).

More services by Superlinked