@swampadmin/data-pipeline
v2026.02.01.0
ETL data pipeline with source connectors, S3 sinks, and integrated secret management.
Note: Requires
@swampadmin/vault-secretsfor database credential rotation.
Supports incremental extraction, partitioned writes, and automatic compaction for cost-efficient storage.
Repository
https://github.com/swamp-club/data-pipeline
Labels
dataetlpipelineanalytics
Install
$ swamp extension pull @swampadmin/data-pipelineRelease Notes
First stable release.
- Postgres source with CDC and batch modes
- S3 sink with Parquet and JSON output
- Daily sync workflow with extract → load → compact stages
- Env-based vault for local development
@swampadmin/data-pipeline/postgres-sourcev1.0.0postgres-source.ts
Global Arguments
| Argument | Type | Description |
|---|---|---|
| connection_string | string | Database connection URL |
| schema? | string | Database schema name |
connectconnect operation
| Argument | Type | Description |
|---|---|---|
| connection_string | string | Database connection URL |
| pool_size? | number | Connection pool size |
queryquery operation
| Argument | Type | Description |
|---|---|---|
| sql | string | SQL query to execute |
| params? | array | Query parameters |
streamstream operation
| Argument | Type | Description |
|---|---|---|
| table | string | Source table name |
| batch_size? | number | Rows per batch |
Resources
postgres-source.state(persistent)— Managed resource state
postgres-source.lock(ephemeral)— Concurrency lock
@swampadmin/data-pipeline/s3-sinkv1.0.0s3-sink.ts
Global Arguments
| Argument | Type | Description |
|---|---|---|
| connection_string | string | Database connection URL |
| schema? | string | Database schema name |
writewrite operation
| Argument | Type | Description |
|---|---|---|
| path | string | Destination path |
| format? | enum | Output format (parquet, json, csv) |
partitionpartition operation
| Argument | Type | Description |
|---|---|---|
| name | string | Resource name |
compactcompact operation
| Argument | Type | Description |
|---|---|---|
| name | string | Resource name |
Files
s3-sink.log(text/plain)— Operation audit log
s3-sink.json(application/json)— Structured output
Daily Data Syncdaily-sync
Daily Data Sync workflow
daily-sync-jobExecute Daily Data Sync
1.Extract@swampadmin/data-pipeline/postgres-source.stream— Extract step
2.Load@swampadmin/data-pipeline/s3-sink.write— Load step
3.Compact@swampadmin/data-pipeline/s3-sink.compact— Compact step
env-credentials
vault-providerenv-vault.ts
env-credentials secrets provider