Outpost

​

Summary

Outpost lets a merchant read a customer’s meter balance from a Redis they own, right after they track an event, without a request to Polar. The SDK reduces each event into 5-minute buckets in Redis with Lua scripts, every instance of the merchant’s application shares that state through Redis pub/sub, and Polar sends snapshots of its own totals over a WebSocket. A snapshot replaces the local buckets it covers; the events Polar has not stored yet stay counted as pending. Polar remains the source of truth for billing. This document describes what ships in the SDKs today (TypeScript and Python, both built from the same Lua scripts), the Polar side that serves them, and the failure modes.

​

Context

Merchants who bill by usage check a balance before they serve a request. Polar’s meter reads come from the database and lag ingestion by seconds; an HTTP round trip per request is too slow and too many requests for the rate limit. The SDK already batches events for ingestion. Outpost adds the read side: a local copy of the usage, true to the event, that Polar corrects.

​

Current state

​

Components

​

The state in Redis

Keys carry the layout version, v1 today. A new layout takes the next version, starts from Polar’s snapshot and never reads the previous keys. <identity> is external:<external_customer_id>:<reducer_id> or customer:<customer_id>:<reducer_id>. Bucket starts are microseconds since the epoch, floored to 5 minutes. A read computes consumption as Polar’s totals, plus the pending part of the buckets before cold_until, plus every local event of the buckets from cold_until on. The balance is credits minus consumption.

​

The scripts

​

Write path

A hot bucket, at or after cold_until, counts its events whole, so an acknowledged event just stops being pending. A cold bucket keeps its events pending until a snapshot counts them: Polar received them, but its totals may not include them yet. A batch ends in one of two ways, whatever the number of attempts it took. A failed attempt changes nothing in Redis.

• Delivered, on any attempt: Metering calls acknowledge, as above.
• Dropped, after maxAttempts failed deliveries or on an error a retry cannot fix: Metering reports the batch to onDropped and calls confirm on its events, everywhere, hot or cold. They are no longer pending, Polar will never count them, and the next snapshot replaces the bucket without them.

​

Read path

The first read of a customer waits for the snapshot up to snapshotTimeout (5 s by default), then reports not_ready and the read fails. Later reads serve from memory. The memory copy follows Redis through the channel: a bucket update applies in place when its revision is the next one, anything else reloads the hash.

​

Snapshots, syncs and seals

Polar reduces every event it ingests into the same 5-minute buckets. The current bucket lives in Polar’s Redis; reducer_bucket.sync writes it to Postgres when it closes, and a late event lands in its closed bucket in Postgres at ingest. Both publish a reducer_bucket event, as does a customer meter recomputed after a credit grant, and the WebSocket handler answers it with a snapshot for that customer. Billing seals a customer’s buckets eleven minutes after its cutoff, once every event has reached Postgres; a sealed bucket is billed as it stands, and an event that arrives later starts a new generation of it. A snapshot’s cold_until is the end of the last bucket Polar has fully synced, looking back an hour at most. count, sum, min and max cover everything before it, read from Postgres in the same transaction as the confirmed ids, so they count exactly those. The SDK asks Polar which of its pending events the snapshot counts before it applies it; the ones Polar has not stored stay pending, and keep adding to the balance, for an hour at most.

​

Telemetry

The SDK records one histogram per local write and read (outpost.ingest.duration, outpost.read.duration, with the outcome), a counter of buckets written, a span per report of syncs received and applied, spans for connections and reconnections, and a span per snapshot with the local and Polar totals it replaced (outpost.snapshot.received). Every record names the meter and the customer identity, never an event’s metadata. The SDK exports every 30 seconds through the relay, which forwards to Logfire with the organization and the SDK version. The outpost dashboard in the polar Logfire project reads them. telemetry: false or POLAR_OUTPOST_TELEMETRY=0 turns it off.

​

Goals

• A read sees every event track() returned for, on every instance that shares the Redis.
• A read costs no request to Polar, and well under a millisecond on a warm customer.
• A local balance is never higher than Polar’s: local consumption counts Polar’s totals plus every event Polar has not counted yet. It converges to Polar’s own reads.
• Both SDKs behave the same: the Lua scripts are shared, and the client logic is mirrored and tested the same way.

​

Failure modes