What 38 regions taught us about clocks
Hybrid logical clocks, a 3 ms skew budget and the night Singapore drifted.
Ordering events across 38 regions means agreeing on time, and machines are bad at agreeing on time. Here is how we keep order without pretending clocks are perfect.
Hybrid logical clocks
Every event gets a timestamp that combines wall-clock time with a logical counter. If a message arrives from the future, the receiver moves its clock forward instead of breaking causality.
A 3 ms budget
Every node measures its offset against three time sources each second.
A node more than 3 ms off stops accepting writes and drains.
Reads stay available from replicas that are within budget.
The night Singapore drifted
In May a firmware update made one rack in Singapore drift 40 ms in six minutes. The budget caught it at 3.1 ms, the rack drained, and customers saw a slightly slower write path for eleven minutes instead of out-of-order data.
Sigrún leads the query engine team in Reykjavík. Before Basalt she worked on storage at a Nordic payments company, where she learned to distrust any latency number without a percentile next to it.
More from the engine room.
How we cut p99 from 11 ms to 4.2 ms
Three changes to the read path in v4.0: declared sort keys, a per-core allocator and a planner that stopped re-sorting partitions at query time.
Exactly-once without two-phase commit
Idempotent sinks, epoch fencing and a 64-bit sequence per partition. No coordinator, no stalls.
Time travel is just a really good index
Why AS OF queries on 30 days of history cost less than one extra replica.
