Encord Index

Product · SaaS / cloud software · Engineering & development by Encord

The end-to-end solution for AI data management and orchestration.

Encord Index is a multimodal data management platform that helps AI teams curate, search, and orchestrate petabyte-scale unstructured datasets. It provides embedding-based visualization and natural language search to identify high-signal data for training and fine-tuning.

Languages
English

Category: MLOps

Platforms
Web, API, Python SDK
Editions
Starter, Team, Enterprise
Maximum Data Volume
1bn+ files (Enterprise plan)
Supported Modalities
Video, Image, Audio, LiDAR, DICOM, NIfTI, PDF, Text, HTML
Cloud Integrations
AWS S3, GCS, Azure Data Lake, Oracle, OTC
Search Capabilities
Natural language, Metadata, Embedding-based similarity
  • Unify data from AWS, GCP, Azure, and Oracle into one platform
  • Search and filter petabytes of data with natural language and embeddings
  • Identify and remove poor quality data and duplicates automatically
  • Support for 1bn+ files with full lineage and versioning tracking

Overview

Encord Index is a multimodal data management and orchestration platform designed for AI teams building production-grade models. It enables businesses to centralize unstructured data from various sources, including AWS, GCP, Azure, and local storage, into a single interface for visualization and curation. The platform solves the problem of fragmented data infrastructure by providing embedding-based search, natural language queries, and automated outlier detection to find high-value training data. Encord Index supports a wide range of modalities such as video, LiDAR, DICOM, and audio, allowing teams to manage up to 1 billion+ files. By integrating directly with existing MLOps stacks via a comprehensive Python SDK, it streamlines the transition from raw data collection to training-ready datasets.

Data Management and Centralization

  • Centralized Data Hub: The platform centralizes unstructured data, eliminating the need to build fragmented traditional and vector database infrastructure.
  • Role-Based Access Control: Role Based Access Control enables organization-level user management for data access and sharing.

Curation and Visualization

  • Embedding-Based Visualization: Users can visually identify and inspect data outliers with embedding plots to find under-represented areas of datasets.
  • Semantic Search and Filtering: Advanced natural language search and granular metadata filtering allow teams to search and organize large complex datasets with ease.
  • Duplicate Detection: The platform automatically identifies image duplication to reduce storage overhead and management complexity.
  • Data Cleaning with Collections: Users can group and remove poor quality data from datasets in a few clicks using smart Collections.

Orchestration and Analytics

  • Full Lineage Tracking: The system tracks the full lineage of data from raw collection through to the labeled training set.
  • Performance Analytics: The platform provides dashboards and analytics to understand model performance and create feedback mechanisms.
  • Custom Metadata Schemas: Users can create custom metadata fields and schemas to manage and enrich data prior to annotation.

Use cases

  • Robotics teams use Encord Index to find rare edge cases in petabytes of sensor data using embedding-based search.
  • Autonomous vehicle developers curate targeted subsets of LiDAR and video data to close distribution gaps in training sets.
  • Healthcare organizations manage and visualize large volumes of DICOM and NIfTI scans for surgical AI training.
  • Defense AI teams use on-premise deployments to index and filter sensitive ISR and satellite imagery data.

Specifications

Deployment
SaaS, VPC (AWS, GCP, Azure), and On-premise options available
Platforms
Web, API, Python SDK
Integrations
AWS S3, Google Cloud Storage, Azure Data Lake Storage, Oracle, OTC cloud storage
APIs and SDKs
Comprehensive Python SDK for programmatic access to all platform functionalities
Sign-in and administration
Single sign-on (SSO), Role-based access controls (RBAC), Multiple workspaces
Limits
Supports data volumes from 500k up to 1bn+ files depending on the plan
Editions
Starter, Team, Enterprise
Maximum Data Volume
1bn+ files (Enterprise plan)
Supported Modalities
Video, Image, Audio, LiDAR, DICOM, NIfTI, PDF, Text, HTML
Cloud Integrations
AWS S3, GCS, Azure Data Lake, Oracle, OTC
Search Capabilities
Natural language, Metadata, Embedding-based similarity
Deployment Options
SaaS, VPC, On-premise, Air-gapped
Access Control
Role Based Access Control (RBAC)
Developer Tools
Python SDK, API access
Data Visualization
Pre-computed and custom embeddings plots

How it works

  1. Connect Data Sources: Connect your existing cloud storage (AWS S3, GCS, Azure) or upload data directly via the web interface or Python SDK.
  2. Explore and Visualize: Generate embeddings and use natural language search or metadata filters to explore and visualize your unstructured data.
  3. Curate and Cleanse: Group high-value data into Collections, remove outliers, and version datasets to prepare them for annotation or training.
  4. Orchestrate Workflows: Export curated datasets or route them directly into Encord's annotation workflows with full lineage tracking.

Plans

  • Starter
    • 500k Data Volume: Manage and curate up to 500,000 data files.
    • API & SDK Access: Access the platform programmatically via API and Python SDK.
    • Multi-modal Search: Use embedding-based search and natural language queries to find data.
    • Outlier Detection: Identify and remove poor quality data or outliers from your datasets.
  • Team
    • 100m Data Volume: Scale data management up to 100 million files.
    • Data Agents: Automate data tasks using integrated AI models and agents.
    • Performance Analytics: Access advanced analytics to track dataset and model performance.
  • Enterprise
    • 1bn+ Data Volume: Support for massive datasets exceeding 1 billion files.
    • VPC & On-prem Deployment: Deploy the platform within your own Virtual Private Cloud or on-premises.
    • SSO & Multiple Workspaces: Manage multiple teams and projects with dedicated workspaces and SSO.
    • Custom Embeddings: Create and use custom embeddings for specialized data search.

FAQ

How does the platform handle large video files?

Encord Index streams video natively from your cloud storage (S3, GCS, Azure) without requiring you to download footage locally or perform frame extraction.

Does Encord support on-premise or VPC deployment?

Yes, Encord offers VPC and on-premise deployment options, including AWS VPC peering and air-gapped setups, so your data stays within your own cloud perimeter.

Explore product

Added by AntSpace from public sources (encord.com).

Related

More products by Encord