Views, landed in S3.
Write any view to your bucket as Parquet. Files roll by hour or by size, and every batch comes with a manifest you can trust.
Overview
How it works.
Basalt writes Parquet files with Zstandard compression under Hive-style partitions. A batch is visible only when its manifest is written, and the manifest lists every file with its row count and source offsets.
Setup
Set it up in three steps.
01
Add the destination
Create an IAM role with s3:PutObject on a prefix and let Basalt assume it. Choose a roll policy by time, size or both.
# land the events view in S3 hourly basalt sink add archive --s3 s3://acme-lake/events/ --view events_raw --roll 1h
✓ role basalt-writer assumed · prefix verified
✓ first batch written · 36 files · manifest.json
02
Shape it with a view
Write plain SQL over the source. Basalt keeps the view current with every record and tracks lineage back to the topic, partition and offset.
CREATE MATERIALIZED VIEW revenue_by_min AS SELECT date_trunc('minute', ts) AS minute, currency, sum(amount) AS revenue, count(*) AS payments FROM payments WHERE status = 'captured' GROUP BY 1, 2;
03
Read it from anywhere
Every view is an endpoint the moment it exists. Call it over REST, connect with any Postgres client, or use the SDK from your app.
curl https://api.basalt.dev/v1/views/revenue_by_min?limit=5 \ -H "Authorization: Bearer $BASALT_TOKEN"
✓ 200 OK · 5 rows · served from eu-north-1 in 3.1 ms
Included
What you get.
01
Batch manifests
Each batch writes a manifest last, so readers only pick up complete batches.
02
Hive partitions
Paths follow dt=/hour= layout, ready for Athena, Spark and Trino.
03
Roll policies
Close files every hour, every 512 MB, or whichever comes first.
04
KMS encryption
SSE-S3 and SSE-KMS are supported, with your own key ARN.
Related
