Snowpark
Product · Developer tool by Snowflake
Build and run Python, Java, and Scala applications directly in Snowflake.
Snowpark is a developer framework for building data pipelines and ML models using Python, Java, and Scala directly inside Snowflake. It allows developers to run non-SQL code next to their data for better performance, security, and simplified architecture.
Category: Developer Tools
- Editors and tools
- Jupyter, VS Code, IntelliJ
- Deployment
- Fully managed Snowflake engine, Snowpark Container Services
- Supported Languages
- Python, Java, Scala
- Core Abstraction
- DataFrame
- Execution Model
- Lazy Execution with Pushdown
- Deployment Types
- UDFs, Stored Procedures, Containers
- Run Python, Java, and Scala code next to data in Snowflake
- Pushdown optimization for high-performance data processing
- Integrated Model Registry and Feature Store for ML workflows
- Zero-copy data processing without separate compute clusters
Overview
Snowpark is Snowflake's developer framework that enables data engineers, data scientists, and developers to build and run applications using Python, Java, and Scala directly within the Snowflake AI Data Cloud. It provides a set of libraries and runtimes that execute code on Snowflake's elastic compute engine, eliminating the need to move data to external systems for processing. By running code next to the data, Snowpark ensures high performance through pushdown optimization and maintains consistent security and governance. The framework supports the entire development lifecycle, from building reliable data pipelines to training and deploying machine learning models.
Developer Framework
- Multi-language Support: Developers can run Python, Java, and Scala code directly within Snowflake's elastic compute engine without moving data.
- DataFrame API: The DataFrame abstraction allows for building complex data transformations using native language constructs instead of raw SQL strings.
- Lazy Execution: Snowpark operations are executed lazily on the server, batching multiple operations into a single execution to improve performance.
- Pushdown Optimization: All data transformations and heavy lifting are pushed down to the Snowflake data cloud for efficient processing of any data size.
Application & ML Development
- UDFs and Stored Procedures: Snowpark supports custom Python, Java, or Scala code through user-defined functions (UDFs) and stored procedures.
- Machine Learning Integration: The Snowpark ML library provides features for model development, including a Model Registry and Feature Store.
- Containerized Workloads: Snowpark Container Services allows for the deployment and management of containerized applications within the Snowflake perimeter.
Use cases
- Data engineers build reliable, continuous data pipelines at scale using Python or Scala instead of complex SQL.
- Data scientists develop and train machine learning models next to their data using Snowpark ML libraries.
- Developers create interactive data applications by combining Snowpark with Streamlit for rapid UI development.
- Enterprises migrate legacy Spark workloads to Snowflake to reduce total cost of ownership and infrastructure complexity.
Specifications
- Editors and tools
- Jupyter, VS Code, IntelliJ
- Deployment
- Fully managed Snowflake engine, Snowpark Container Services
- Supported Languages
- Python, Java, Scala
- Core Abstraction
- DataFrame
- Execution Model
- Lazy Execution with Pushdown
- Deployment Types
- UDFs, Stored Procedures, Containers
- IDE Support
- VS Code, IntelliJ, Jupyter
- ML Components
- Model Registry, Feature Store, Datasets
- Client Libraries
- Java, Python, Scala API libraries
How it works
- Select Language and Library: Developers choose their preferred language (Python, Java, or Scala) and download the corresponding Snowpark library.
- Develop with DataFrame API: Code is authored using the DataFrame API in local IDEs like VS Code or directly in Snowflake Notebooks.
- Pushdown Execution: The Snowpark library translates the code into SQL and pushes the execution down to Snowflake's elastic engine.
- Deploy and Scale: Applications are deployed as UDFs, stored procedures, or containerized services within the Snowflake environment.
FAQ
What programming languages does Snowpark support?
Snowpark currently provides libraries for three primary languages: Java, Python, and Scala.
What is lazy execution in Snowpark?
Snowpark operations are executed lazily on the server, meaning data transformation is delayed until as late as possible to reduce data transfer and improve performance.
Does Snowpark support pushdown optimization?
Yes, Snowpark supports pushdown for all operations, including Snowflake UDFs, ensuring that heavy lifting is done within the Snowflake data cloud.
Added by AntSpace from public sources (docs.snowflake.com).