Highport orbital control

How the station notices

Nobody has to tell Orbital Control you published; it is already watching the channel. This is what happens between your commit and your site, usually a second or two and sometimes a couple of minutes.

You write a space.highport.sites.site record to your own account. Nothing asks you to announce it. We are already subscribed to the AT Protocol network's event stream, your commit arrives on it a moment later, and we take it from there.

That has one consequence worth stating before the mechanism: the tool does not matter. The Hub at hub.highport.space writes records with putRecord, and so does a shell script, so does another client, so does a PDS admin console. All of them produce the same commit on the same stream, and all of them are indexed by the same code in the same order. There is no privileged path and no API key that makes a publish arrive faster.

There is no webhook to configure here and nothing to notify us with. You write the record in your own repository, and we find out the way everyone else on the network finds out.

What we watch

bard-indexer holds one WebSocket subscription to Jetstream v2:

wss://jetstream.us-east.bsky.network/xrpc/network.bsky.jetstream.subscribeEvents
      

jetstream.us-west.bsky.network is the fallback, and after three consecutive unproductive cycles the consumer switches to it and runs a reconciliation pass.

The subscription carries a collections filter naming exactly two things, space.highport.sites.site and space.highport.sites.tile, and no kinds filter at all. That second half is deliberate and is easy to undo by accident. Jetstream's collections filter constrains commit events only; adding a kinds filter to "tidy it up" would keep every site record indexing perfectly while silently cutting off the three event kinds that are not commits, and the visible symptom would be a suspended account whose site is still serving.

Which events matter

The account rule branches on the event's active field, not on a list of status words. A handler written against a list of known statuses fails open on a status invented later, and failing open on a takedown means a site that should have stopped serving does not.

Suspension is a status change and not a pointer edit. The snapshot, the active pointer and the DNS are all left exactly as they were, so a restoration is a status flip and not a re-index. The previous status is recorded when the suspension is applied, so a domain that was verified when it was taken down comes back verified and not active. A restoration only lifts suspensions the account path applied; an operator's takedown of the same domain survives it.

From an event to a job

Three things happen between a frame arriving and a job existing, and none of them touches your files.

The record's CID is verified at the door. Jetstream hands over the decoded record and its CID separately, so the record is re-encoded to canonical DAG-CBOR and hashed, and the result must equal the CID the commit announced. A mismatch drops that one record and the stream keeps running. It is not an error and it does not stall anything behind it.

The DID is checked against the interest set, an in-process set holding every DID that owns a domain registered here, every connected space's authority, and the configured tile authors. Three of the four event kinds arrive for the entire network, and consulting a set in memory is what keeps a database lookup out of the socket's read loop. A tile commit skips the check on a deployment that lets anyone publish a tile, because there the set is no longer a list of who may publish one. The set is rebuilt every 30 seconds, and registering a domain publishes a wake-up so a domain registered and published to in the same half-minute is not missed.

The record bytes are written before the job row. The canonical DAG-CBOR goes into the content store under its own CID first, verified against that CID and create-only, and then one transaction writes both the content-inventory row and the index_jobs row. The stream cursor advances only when that returns successfully, so a job row can never exist without the bytes it is about, and an enqueue that fails holds the cursor rather than skipping past the only copy.

Jobs deduplicate on (record URI, commit rev). Every indexer replica reads the whole stream, so a second copy of every event lands here and is absorbed. It is the ordinary case, not an error.

The site pipeline

One worker claims one job and drives it through eleven stages in order:

admit ▸ parse ▸ link ▸ validate ▸ bind ▸ resolve ▸ fetch ▸ materialize ▸ activate ▸ broadcast ▸ warm
      

Those eleven collapse onto three stored states. The state column exists to tell you which phase you are waiting on, and a write per stage would be ten extra database round trips on every publish. running covers admit through bind, fetching covers resolve and fetch, and materializing covers everything from materialize to the end.

Two kinds of failure run through all of it, and the distinction is structural. A rejection is the record's fault: it is terminal, it carries a named error, and it is reported back to you. A fault is ours or the network's: it is retried with backoff and then left failed for an operator. Nothing decides this per call site. One function asks whether the error is retryable, and one function settles the job on the answer.

1. Admit

Runs before the record body is touched at all, and the type it operates on has no field for the record, so that ordering cannot be undone by accident. In order: the record key must already be the canonical form of a domain; the DID must pass the deployment's allowlist; nothing about the record may be denylisted; the domain must have a registration in verified or active belonging to this repository; and the commit's rev must be strictly greater than the one already indexed.

Refuses with InvalidRecordKey when the key is not canonical. We do not normalize it, because serving your site at a URI you never wrote would make getRecord on our reported URI answer nothing. Refuses with DomainNotOwned when the registration is missing, held by somebody else, not serviceable, or claimed by a space; all four collapse to one name so that an unrelated identity cannot learn the state of a domain they do not hold, and the detail that separates them goes only to the owner.

A stale rev is dropped, not rejected. The job settles as done and nothing is wrong: that is what the second replica's copy of your commit looks like.

If the record was validated first and a prepared snapshot exists for exactly this (URI, record CID), admit takes the fast path and jumps straight to activation over work the preparation already did. See Validate before you write.

2. Parse

Deserializes the record. Every failure here is a rejection and never a fault, because parsing is a pure function of bytes we already hold, so a retry would produce the same answer forever.

InvalidSource when source.$type is something outside the closed union, naming the tag it found and the two it accepts. InvalidRecord for everything else in a site record. TileInvalid for a tile record that will not deserialize.

2a. Link

Only for a site whose source is a strong reference to a tile. An inline manifest skips it.

The pinned tile version is obtained from three sources in order: our content store by record CID, then our tile index by (URI, CID), then com.atproto.repo.getRecord on the author's PDS with the cid parameter asserted so a PDS that has moved on errors instead of answering with newer bytes. Whatever comes back is hashed and must equal source.cid. There is no parameter that skips that check and no source trusted enough to bypass it, our own store included.

Exactly one tile state is bindable: ready.

That window is BARD_TILE_LINK_RETRY_SECS, ten minutes by default, and it is measured from when the job was created, not counted in attempts. Publishing a tile and the site that binds it inside the same minute is ordinary, and the two commits arrive in whatever order the stream delivers them.

3. Validate

The manifest rules, the redirect rules and — for a tile job — the parameter declarations. This stage reports every rejection it can honestly produce, not just the first one, because a record with four broken entries reported one at a time is four publish cycles. The first name is what the job records, with the rest appended as a count.

Names raised here: InvalidManifest, TileInvalid, DuplicatePath, DuplicateRedirect, RedirectTargetNotFound, ReservedPathCollision. The caps behind them are in Limits and quotas: 10,000 manifest entries, 256 redirects, 2048 bytes per path, and the total size ceiling.

Two structural failures end the pass early. An entry count over the cap returns alone, so that twenty thousand path rejections do not arrive in front of the one that explains them. A key that will not parse as a path suppresses the whole redirect block, because redirect targets are checked against the key set and a set with a hole in it turns a manifest problem into a redirect problem.

3a. Bind

Resolves your parameters against what the tile declares, dereferences any by-reference values from the repository they name, and writes the two documents every tile can fetch at runtime: /_bard/params.json and /_bard/site.json. It runs for every site, inline ones included. An inline site gets {} for its parameters and a full site.json, so tile code can fetch both unconditionally rather than branching on a 404.

Refuses with UnknownParameter, MissingParameter, ParametersWithoutTile, ParametersTooLarge, ParameterTypeMismatch, ParameterConstraintViolation, ParameterPathUnrepresentable, ParameterRecordNotFound, ParameterRecordVersionUnavailable or ParameterRecordCidMismatch.

When the record names wellKnown.standardSitePublication, bind checks that too: the address has to be a site.standard.publication in your own account, read from your repository as it stands, whose url is this domain. It refuses with StandardSitePublicationInvalid, or StandardSitePublicationNotFound once the same wait a referenced record gets has run out. Proving a standard.site publication has the details.

Each blob travels with the DID it must be fetched from, and that is not the same DID for every blob: a blob inside a value you supplied inline comes from your repository, and a blob inside a record you referenced comes from that record's repository. Collapsing the two would produce a missing-blob error against a PDS that never held the bytes, and tell you to re-upload a file that is not yours.

4. Resolve

Builds the job's complete CID set in a fixed order, being manifest entries, notFound, parameter blobs and the two documents we authored, then deduplicates it by CID, and checks the whole set against the content store with 32 existence checks in flight.

Two properties fall out of this and both matter to you. Republishing a site where one file changed costs exactly one blob fetch. And an asset shared between two sites is stored once, whoever published it, because bytes that hash to a CID are the content that CID names.

An existence check that errors is never read as "missing". A store that is briefly unreachable would otherwise send every job back to every PDS it can reach for blobs it already holds.

5. Fetch

Fetches only what the previous stage found missing. Every request leaves through one outbound client with one policy: https only, resolved addresses pinned so there is no second lookup to race, private ranges refused, no cross-host redirects, no proxy, no cookies, a 5-second connect and 30-second total timeout, and a size cap enforced while streaming instead of after.

Concurrency has two bounds, taken job-first: BARD_FETCH_CONCURRENCY per job, 10 by default, and BARD_FETCH_CONCURRENCY_GLOBAL across the process, 100 by default. A blob waiting out a backoff releases the global permit first, so it does not sit on capacity other jobs are queued behind.

Every blob is verified against its CID before it is written, and that check lives inside the write itself, not at the call site, so there is no argument a future caller could pass to say it already happened. Writes are create-only, so two jobs fetching the same shared asset is a harmless no-op and not a race.

6. Materialize

Writes the snapshot, and writes the record's own canonical DAG-CBOR into the content store under its record CID. The second one is the write that gets forgotten and the one that matters longest: it is the first place stage 2a looks, and without it every tile binding on the deployment would go on resolving through the index and the author's PDS until the day an author moves repositories.

The order inside is mandatory. Record bytes first. Then the missing-blob check, asked of storage and not of what the fetch stage believed, because a resumed job has no memory and ready is a promise about the store. Then the snapshot object, before any row that could promote it. The other order leaves a row pointing at an object that is not there, which is a domain answering 500 rather than serving its previous revision.

The snapshot is immutable and keyed by the record CID, never by rev:

sites/example.com/snapshots/bafyreiale5kdmqtazvgoq27twgxyotsgtirpsefahxt2movth2ybvnn6te.cbor
      

That key is why a validated record and its later commit build the same object in the same place, and why activation is a pointer swap in both cases.

7. Activate

The cutover, and all of it is one Postgres transaction. The domain row is locked, so every activation for a domain queues instead of racing. A re-run of an activation that already happened returns the same answer. If a newer revision activated while this job was running, this snapshot is marked superseded and the job ends successfully having changed nothing.

Otherwise the current snapshot is demoted, this one is promoted, and the domain's pointer moves and its version counter increments in the same statement that owns the status, so the two cannot disagree, a verified domain becomes active on its first successful publish, and a suspended domain stays suspended even when its snapshot is re-activated. The indexing path cannot undo an operator's takedown.

sites/{domain}/current.json is written after the commit, carrying that version number, and a lower version never displaces a higher one. It is a denormalized copy read only when Postgres is unreachable; a write that loses the comparison is a success, not a failed activation.

8. Broadcast

Publishes one message on Redis so every origin replica drops its cached pointer for that domain:

{ "kind": "pointer", "domain": "example.com", "version": 42 }
      

This cannot fail the job. By the time it runs, the database already says the new revision is what the domain serves, and every replica agrees within its 60-second pointer cache lifetime whether the message lands or not. The lifetime is the guarantee; the message is the optimization. A delete sends a purge instead, which carries no version, so a replica that believes it is already current cannot filter it out.

9. Warm

Requests a fixed set of two paths, / and /index.html, on each configured origin replica, sequentially, with a 3-second timeout, so the first visitor after your publish is not the one who pays for the cold read.

The path set is fixed and nothing in your record reaches the URL. A publisher who could choose the warm set would have a way to make us issue an HTTP request of their choosing from inside the network the origins live on, on every publish. A 404 counts as warmed: the host resolved, the pointer loaded, the snapshot loaded, and a site with / and no /index.html is a normal site.

What the origin does with that snapshot from here on is The origin's rules.

How a job ends

Five settlements, and only one of them is a state you have to do something about.

failed is both the retry state and the give-up state, and the difference is the schedule, not the word. That is why "will this run again" is answered by asking when it is next due, not by reading the state name.

Rejections, by stage

Every one of these is terminal, and every one of them is about the record, not about us.

What the status method reports

space.highport.manage.getIndexStatus maps the stored state onto one value you can act on. An unfamiliar string is carried through instead of flattened into a familiar one, because "we are doing something this client does not recognize" is an answer and a wrong familiar word is not.

The tile pipeline

A tile is indexed the moment its commit arrives, before anything binds it, because once the author's PDS collects blobs nested under an untyped field our store holds the only durable copy. What a tile is and who may publish one is Tiles.

admit ▸ parse ▸ validate ▸ resolve ▸ fetch ▸ materialize ▸ ready
      

Four differences from a site, and each of them follows from a tile not being served at a domain.

• No link and no bind. A tile is the manifest, and it declares parameters instead of supplying them.
• No activation, broadcast or warm. There is no pointer to move and nothing pointing at it.
• No staleness rule. Every tile version is its own permanent row keyed by CID, so there is no ordering to violate and a replayed commit simply rewrites the row it already wrote. Admission is gated by the author policy and the per-account tile limit instead.
• A terminal fetch failure does not end the job. A site has nothing partial to serve; a tile does.

That last one is what the incomplete state is. A version whose author's PDS no longer holds one of its blobs is materialized anyway, marked incomplete, and refused at link. That tells a site owner whose record is broken, and names the missing CIDs. A tile that simply vanished would tell them nothing. The job itself still ends as a BlobUnavailable rejection, so the author hears about it too.

A retryable fetch failure still stops a tile job, and so does a CID mismatch. Those are faults, and a version that hit one never reaches incomplete; the two failure modes do not mix.

The icon

A tile's icon is a sibling field on the record, not an entry in its content manifest, and it is fetched beside the job's required set instead of inside it. A failed icon fetch is therefore a warning line and nothing else: the version still lands ready, not incomplete.

What that costs, and what fixes an icon that never arrived, is under "The icon case" in Versions, edits and the long life of a tile.

When a fetch fails

Each blob gets up to four attempts with exponential backoff, starting at 250 milliseconds and capped at 10 seconds. What happens after that depends entirely on what the repository said.

A BlobUnavailable names the role of the blob and the repository it was expected from, because the advice differs completely:

• A file in your own manifest. Re-upload it to your account and publish the record again. Uploading and then publishing promptly is the habit that avoids this.
• A file belonging to a tile you bound. You cannot fix it. The blob lives in the tile author's repository, and only they can restore it by republishing. Bind a version we already hold, or ask the author.
• A blob inside a parameter value. Re-upload it and rebind. If the value is a record that already exists, supplying it by reference instead of inline is what makes the PDS keep its blobs.

A CID mismatch is treated as a fault and not a rejection, on purpose: somebody upstream is serving wrong bytes, which is not the same as bytes being absent, and reporting it as a blob to re-upload would send you after a file that is fine.

Retries, leases and backoff

A worker claims a job for 120 seconds and renews that lease every 20 seconds while it works. The lease is not a separate column. It is the same scheduled time that says when a waiting job becomes due, which means there is exactly one answer to "when may somebody else take this". A worker that dies has its job returned to the queue by a sweep that runs every two minutes.

Attempts are counted at claim, not at failure. That is the bound on a crash loop: a job that kills its worker before the worker can report anything would otherwise be retried forever, and counting starts instead of failures makes the give-up rule hold for the failures nobody lived to record.

Job backoff is exponential with equal jitter, base 5 seconds, capped at 10 minutes, with BARD_JOB_MAX_ATTEMPTS attempts, five by default, so the reachable delays are roughly 2.5–5 s, 5–10 s, 10–20 s and 20–40 s before the job is left alone. The jitter is not decoration: replicas claim in batches from one ordered queue, so a batch that failed together against one unreachable PDS would otherwise retry together.

A job cut short by a shutdown is not a failed attempt. It is left exactly as it was claimed and settled with nothing, so a deploy does not spend anybody's attempt budget.

What a delete does

Deleting a site record is two stages, deactivate and then broadcast, and no byte is ever deleted.

The pointer is cleared and the snapshot is demoted in one transaction, the record's index row moves to superseded instead of being removed, and the purge message goes out. Your registration, your DNS and your certificate all survive, so publishing again restores service with no re-verification.

The index row is kept, not deleted, for a specific reason: the admission stage reads it to drop stale replays, so a republished record has to be strictly newer than the deleted one. Without the row, a replayed old commit could bring the site back at a revision you had already moved past.

Deleting a tile record never deactivates a site. Bindings are by content hash, the bytes are ours to keep, and every site bound to that version goes on serving exactly what it was serving. What stops is new bindings.

The owner's side of all this, meaning releasing a domain, exporting and re-indexing, is Editing, deleting and moving on.

Cursors, gaps and reconciliation

The stream position is one row per Jetstream host, keyed by the hostname. That key is not cosmetic: sequence numbers are per-instance, so a cursor taken from us-east is meaningless against us-west, and keying by host makes it structurally impossible to replay the wrong sequence space after a failover.

The cursor advances only after a job row has committed, and it is written at most every 1,000 events or every 5 seconds, whichever comes first. Coalescing is safe in exactly one direction, and the asymmetry is the whole design: a cursor that lags costs a replay, and a cursor that leads is a gap. Replays are free, because every write in the path is idempotent.

An orderly restart therefore re-reads up to a few seconds of stream, exactly as a crash does, and indexes nothing twice.

Silence and reconnection

Liveness is not measured with a ping, because a ping proves the socket is alive and the failure worth catching is a socket that is alive while the stream behind it has stopped. Instead: five minutes of complete quiet is a warning, ten minutes tears the connection down and reconnects. Every frame counts as proof of life, including one we then drop.

Those numbers work only because we send no kinds filter. Network-wide account and identity traffic keeps a healthy stream busy even when nobody is publishing a site, so five minutes of nothing really is a fault.

Reconnection backs off exponentially with equal jitter from 2 seconds to 60, and the backoff resets only when a cycle actually delivered something, so a connect-then-error loop still backs off.

If the stream refuses our stored cursor as too old, the events between where we were and where the stream now starts are gone and no reconnect brings them back. Resuming is not enough on its own, so a full reconciliation pass runs first. When the stream names the floor we resume there; when it does not, the cursor is cleared, we resume at the live tip, and the entire window is reconciled. The more expensive of the two correct answers rather than the cheap wrong one.

The nightly walk

Once a day, every domain in verified or active is read directly: one com.atproto.repo.getRecord at the publisher's PDS, comparing the CID we hold against the CID they serve. Different means we missed a commit, and a job is queued. The walk is sequential, not parallel. It is bounded by our domain count, runs on a schedule and never on a request, and being gentle with the PDSes it reads is worth more than finishing sooner.

The comparison is by record CID and not by revision, because a revision orders commits inside one repository and says nothing across a repository move.

An absent record never deactivates anything. A delete we missed and a PDS having an outage look identical from here, and inferring a deletion from an absence would let a PDS outage take a live site down.

Gap recovery does one more thing the nightly pass does not: it re-reads the exact tile versions that live sites pin, because a tile commit missed during a long outage may have left the stream entirely, while one missed in the steady state arrives seconds later and the binding site's link stage waits for it.

Garbage collection

Bytes are reclaimed by a mark-then-sweep pass, disabled by default and weekly when enabled. A blob is pinned by every active, superseded or prepared snapshot that references it, by every retained tile version, and by every live preparation. A mark records that nothing referenced a CID at one instant; the sweep re-runs the whole enumeration and deletes only what is still unreferenced a grace period later — 30 days by default. A blob that becomes referenced again, by a republish of the same bytes or by a rollback, loses its mark.

A superseded snapshot inside the retention window keeps its blobs, which is what makes rebuilding an older snapshot a restore instead of a 404.

Records in a space

A space's repository is permissioned, and its commits are not on the public stream. Nothing above applies to how those records arrive; everything above applies to what happens once they do.

Three things bring us a space's records: an inbound com.atproto.space.notifyWrite from the space host, a head check that runs every 15 minutes by default, and space.highport.manage.syncSpace, the Sync now control in the Hub. Any of them queues a space-sync job, and that is a different kind of job: it pulls a repository and then feeds records into the pipeline, instead of entering the site or tile stages itself.

The pull walks the space's operation log from our cursor under a credential we hold, or rebuilds from a full export when there is no cursor, when the host cannot deliver from where we are standing, or when our fold of the repository and the host's signed head commit disagree. It verifies that head commit, and then hands each site or tile record to the same sink the public stream feeds. From there the stages are identical, with one difference that matters at admission: a job carrying a space knows to ask whether the domain is registered to that space rather than to the publishing repository, which is why a record in a space is refused with DomainNotInSpace rather than DomainNotOwned.

Blobs come from com.atproto.space.getBlob at the authority's repository host under the same credential. Every other rule is unchanged, including the concurrency bounds, the retry schedule, the terminal 404 and the CID check inside the write, because all of it lives in the fetcher and the space plane only supplies bytes differently.

Spaces covers what a space is and how access is kept alive.

Advanced · 2 of 7