pgstream
Product · Self-hosted / on-prem software · Engineering & development by Xata
Open-source PostgreSQL data replication and anonymization for realistic staging.
An open-source PostgreSQL replication and anonymization engine for businesses that need to create realistic staging environments without exposing PII. It uses logical replication and deterministic transformers to maintain referential integrity in masked datasets.
- Languages
- English
Category: Data Integration
- Platforms
- Docker, Kubernetes
- License
- Apache 2.0
- Primary Language
- Golang
- Replication Method
- PostgreSQL Logical Replication
- Snapshotting
- Parallel Snapshotting
- DDL Support
- Included
- Open-source PostgreSQL replication and transformation engine
- Deterministic masking for referential integrity
- Supports Greenmask, NeoSync, and custom Golang transformers
- Apache 2.0 licensed for self-hosted or cloud deployment
Overview
pgstream is an open-source PostgreSQL data replication and transformation engine designed for creating realistic, anonymized staging environments. It enables businesses to copy production data at high throughput using logical replication and parallel snapshotting while applying configurable masking rules to sensitive information. The tool supports deterministic transformers to maintain referential integrity across complex relational datasets, ensuring that masked data remains consistent. pgstream is built in Golang and integrates with popular open-source transformer libraries like Greenmask and NeoSync. It can be deployed as a standalone Docker image or used as part of the Xata platform to power nightly anonymized snapshots for developer workflows.
Replication and Syncing
- Logical Replication: The tool uses PostgreSQL logical replication and parallel snapshotting to copy data at maximum throughput.
- DDL Support: It supports DDL statements to keep schema changes in sync during the replication process.
- Read-Replica Support: The system can create snapshots from read-replicas to minimize the performance impact on production databases.
- Automated Snapshots: It can be scheduled via CI/CD pipelines to execute nightly and create updated snapshots of production data.
Data Anonymization
- PII Masking: Configurable transformers mask PII and other sensitive data while keeping it realistic and structurally intact.
- Deterministic transformers ensure the same data is masked the same way every time to maintain referential integrity.
- Library Integrations: The tool integrates with open-source transformer libraries including Greenmask, NeoSync, and go-masker.
- Custom Golang Transformers: Users can implement custom data transformations written in the Golang programming language.
Use cases
- Engineering teams create nightly anonymized snapshots of production databases for safe testing in staging environments.
- Data platform teams integrate pgstream into CI/CD pipelines to automate the delivery of realistic datasets to developers.
- Companies handling PII use deterministic transformers to mask sensitive customer data while maintaining relational consistency across tables.
- Developers use pgstream to replicate data from production read-replicas to minimize performance impact on live applications.
Specifications
- Deployment
- Self-hosted via Docker image or run within Xata Cloud and BYOC environments
- Platforms
- Docker, Kubernetes
- Integrations
- Greenmask, NeoSync, go-masker, AWS RDS, Amazon Aurora, Google Cloud SQL, Azure Database
- License
- Apache 2.0
- Primary Language
- Golang
- Replication Method
- PostgreSQL Logical Replication
- Snapshotting
- Parallel Snapshotting
- DDL Support
- Included
- Deployment Format
- Docker Image
- Supported Transformers
- Greenmask, NeoSync, go-masker, Custom Golang
How it works
- Install with Docker: Pull the official Docker image and run it using docker-compose or within a Kubernetes cluster.
- Connect PostgreSQL: Point pgstream at your source production database or a read-replica to begin the replication process.
- Configure Transformers: Define masking rules using built-in or custom transformers to anonymize PII and sensitive fields.
- Automate Snapshots: Schedule the tool via CI/CD pipelines to create nightly anonymized snapshots for staging and development.
FAQ
Is pgstream open source?
Yes, pgstream is an open-source project licensed under Apache 2.0, providing flexibility in deployment and configuration.
How does it maintain referential integrity?
It uses deterministic transformers that ensure the same input data is masked the same way every time, keeping relational data consistent across tables.
Which PostgreSQL providers are supported?
It supports AWS RDS, Amazon Aurora, Google Cloud SQL, Azure Database, and self-hosted PostgreSQL installations.
What replication method does it use?
The tool uses PostgreSQL logical replication, including DDL statements, and parallel snapshotting for high-throughput data copying.
Added by AntSpace from public sources (xata.io).