Highport orbital control

The edge

Everything arrives here first. It is the tier that answers the handshake, checks the gate, and counts the bytes.

The edge is the tier in front of the origin. It answers the TLS handshake, obtains a certificate during that handshake if the hostname does not have one yet, asks whether this reader may have this path, proxies the request to an origin replica, and writes one line to a log. Everything a visitor to a site here ever touches passes through it, and nothing on it caches.

It is a separate tier from the origin because it does a different job on a different machine. The origin's job is to turn a Host and a path into stored bytes, which is described in the origin's rules. The edge's job is everything that has to happen before that is allowed to run.

What runs here

Three processes, and only two of them are on the edge node.

Certificates live in a Redis instance the edge nodes share, and that shared storage is all that makes several of them a cluster. There is no gossip and no membership protocol. Any Caddy pointed at the same store serves the same certificates.

The origin will not answer a request that did not come through here. Caddy attaches a shared secret on every proxied request as X-Bard-Origin-Key, and the origin checks it in constant time before it canonicalizes the host, before any cache and before any storage call, so it answers identically for a domain it serves and one it has never heard of.

Everything on this page happens in the few milliseconds before a visitor sees anything, ten thousand times a day, and none of it needs you.

Why the access check runs beside the origin

The access check could have been an HTTP call into the control plane. It is deliberately not, and the reason is worth stating once: a call into bard-control on every request would put the control plane on the critical path of serving every static byte on the network, which turns a control-plane outage into a total outage. A sidecar on the same machine answers from an in-memory table refreshed from Redis, in well under a millisecond, and when it fails it fails on one node.

Two things follow from that, and both are decisions, not accidents.

The sidecar reports itself ready unconditionally, including when Redis is unreachable. Reporting unready would take it out of rotation, make the access check fail, and reach the fail-open path below. That is the correct outcome, but arrived at by an orchestrator restarting a process that was answering correctly from a slightly stale table.

The log processor runs in the same process, because the access log is a file on that machine. A design that elected a single processor across the deployment would silently stop counting every other node's traffic, the exact shape of failure this tier is built to avoid.

Certificates

Where a certificate comes from

You never request one. You point the domain at us, with an ALIAS or ANAME record at an apex or a CNAME on a subdomain, both to the address registering gave you (t-… under highport-dns.net, or go.highport.space for an older registration), and finish the registration, which is bringing your domain alongside. The first TLS connection for a hostname Caddy has not seen triggers issuance inside that handshake.

Before it obtains anything, Caddy asks Highport for permission. That question is one request:

GET /internal/tls-ask?domain=example.com HTTP/1.1
      Host: tls-ask.internal:9123

      HTTP/1.1 200 OK
      content-length: 0
      

A 200 authorizes issuance; anything else denies it and the handshake fails. The endpoint is internal-only and is never reachable from outside the deployment, because a public copy would be an oracle for which domains are registered. It answers from a single read by primary key, being the domain's status and its issuance cooldown, with no joins and nothing widened, because this runs inside a handshake and every column added to it is paid for by a visitor waiting on a connection.

Without that endpoint, anyone who pointed any domain at our addresses would cause an issuance attempt, and failed authorizations count against per-identifier limits that have no override. It is the security boundary, not a convenience: with a permission endpoint configured, Caddy applies no issuance rate limiting of its own.

What makes the answer yes

Every refusal is identical on the wire, being a 403 with no headers of its own and no body, because an endpoint that distinguished "not registered" from "wrong state" would be a domain enumerator for whoever reached it. The 500 is still a denial; the different status exists so an operator can tell "Highport said no" from "Highport is broken" in one glance at a log.

Two caching rules matter to somebody watching a new domain come up. Positive answers are cached for 30 seconds, in the process and in Redis. Denials are never cached, anywhere. A cached denial would keep refusing a domain that had just verified, and the symptom would reach the publisher as a certificate error during the exact minute they were watching. The denylist is consulted before both caches, on every ask, so a takedown reaches the endpoint within one broadcast, with no TTL added on top.

One thing is deliberately not checked: whether you have published a site. A verified domain with no site record yet is exactly the domain that needs a certificate, because the certificate has to exist before the first visitor arrives, not after.

The warm-up

The first visitor to a new domain would otherwise pay for issuance, and for a publishing service that is the wrong person to charge. So once your domain becomes serviceable we open our own HTTPS connection to it:

GET https://example.com/_bard/health
      

/_bard/health is reserved on every host we serve, so the request never touches your files, never consults your manifest, and does not show up in your traffic as a page view somebody made. The certificate is obtained during that handshake. Any HTTP response counts as success, a 404 included. A verified domain usually has nothing published yet, so a 404 is the expected answer to a warm-up that worked perfectly.

It is attempted once per registration, ever. Retrying a failed handshake means retrying an ACME authorization, and Let's Encrypt allows five authorization failures per identifier per hour with no override and no form to fill in. A domain that fails is left alone instead of spending the budget that a real visitor's handshake will need.

If the warm-up fails, the domain reports tls: failed with the underlying error, and the Hub's domain screen reads Needs attention — "The certificate could not be issued: …". The domain is not stuck. The next real visitor's handshake obtains exactly the certificate we could not, and a second pass moves the row back to issued on its next lap without asking the CA for anything.

Renewal, and the limits behind it

There is nothing to renew. Renewals coordinated by ARI are exempt from every Let's Encrypt rate limit, so a fleet in steady state spends nothing and only new domains consume budget. A second pass, which makes no handshake at all, walks every serving domain reading the certificate store, so the issue and expiry dates we report stay current after a renewal we were never told about.

The limits the permission endpoint exists to protect, and the limits applied on top of it:

There is no wildcard certificate. No DNS-01 challenge is configured on the edge, so every hostname, including every free *.sites.highport.space address, gets its own certificate through the same path as a customer domain.

When the answer is no, what the visitor sees

The one path that never depends on any of this is /.well-known/acme-challenge/*, which Caddy answers itself before any route is matched.

Releasing or suspending a domain

Releasing stops authorization the instant the status changes, and the same transaction clears your domain's pointer, so the domain stops being served at the same moment. A new certificate cannot be obtained for it after that. The certificate already in the store is not deleted; it stays there until it expires, which is why the useful statement is that we stop serving the domain and stop authorizing anything new for it.

An operator suspension takes at most one 30-second positive cache TTL to reach a replica that had cached an authorization. A denylist entry takes effect on the next ask, because the denylist is read before the caches.

Registering a released domain again from a different identity clears its certificate state, including any cooldown. Re-registering it yourself keeps it. Otherwise a cooldown that exists to protect the shared ACME budget would be clearable by releasing and re-registering.

The gate

Two things can make the edge refuse a request that the origin would happily have served: a site that asks visitors to sign in on some or all of its paths, and a site whose owner has set a bandwidth budget for anonymous readers. Both are decided here, before the origin sees the request at all.

The question Caddy asks

For every request, Caddy makes a subrequest to the sidecar on loopback:

GET /internal/site-auth HTTP/1.1
      Host: example.com
      X-Forwarded-Method: GET
      X-Forwarded-Uri: /gallery/one.jpg
      Cookie: __Host-bard-site=did:plc:examplevisitor000000000000 example.com
      Sec-Fetch-Mode: navigate
      

The path is forwarded because the gate can apply to a prefix instead of to a whole site, and the path is normalized by the same functions the origin uses. Two copies of a path rule would disagree, and the disagreement would be a file served that should not have been.

Every verdict

A refused answer is copied to the visitor exactly as the sidecar wrote it, status and headers and body, so the 402 below is what a stranger reads.

HTTP/1.1 402 Payment Required
      content-type: text/html; charset=utf-8
      cache-control: no-store
      retry-after: 10800

      <!doctype html>
      <meta charset="utf-8">
      <title>Payment required</title>
      <h1>Out of allowance</h1>
      <p>The bandwidth allowance for <code>did:plc:exampleowner00000000000000</code> is spent for this month. It resets at 2026-10-01 00:00 UTC.</p>
      

Retry-After is the number of seconds to the end of the current UTC calendar month. The 402 body is the same for an image as for a page, so a subresource on a site whose owner is out of allowance receives that markup instead of an image.

A gated request is split by whether it looks like a navigation, because sending an image into a sign-in page produces a broken image whose body is an HTML form, and that takes an afternoon to diagnose. Sec-Fetch-Mode: navigate decides it, falling back to an Accept header containing text/html.

The one path nothing can refuse

/.well-known/atproto-did is served past every refusal above. A domain published here can also be somebody's handle, and a bandwidth allowance may not revoke somebody's identity. Before this exemption existed, a domain over its budget answered handle resolution with a 401, redirected it into a sign-in form when the request asked for HTML, or answered 402 when the owner was the one over, and the resolution failed.

It is exempt from refusal and not from accounting: those bytes are still counted, because a manifest entry may legally occupy that path and an exemption that also skipped billing would be a file anyone could serve for free by naming it atproto-did. Nothing else under /.well-known/ is exempt.

Signing in happens on the publisher's own host

A gated reader is sent to /_bard/oauth/login on the domain they were already reading. Each served domain is a confidential OAuth client of its own, and the scope it asks for is the literal string atproto, which is identity and nothing else. The tokens from that exchange are discarded on the line after they are used; no refresh token is stored; what survives is a cookie holding a DID. What a visitor sees is sites that ask who you are.

The cookie is __Host-bard-site, Secure; HttpOnly; SameSite=Lax; Path=/, with a two-week Max-Age. Caddy encrypts it on the way out and decrypts it on the way in, so the browser only ever holds AES-GCM ciphertext, and the plaintext, a DID and the domain the session was minted for, exists only between Caddy and the sidecar on loopback. A cookie presented on a domain it was not minted for is anonymity, never an error.

The paths in front of the gate

The sign-in paths are matched before the gate because gating a sign-in page on being signed in is a loop with no exit, and because gating the client metadata document would answer a stranger's authorization server with a redirect to a login form, which it would then cache as this client's metadata, permanently.

A header a client cannot forge

X-Bard-Visitor and X-Bard-Payer are the sidecar's answer, and a client sending either of them is trying to bill somebody else. Three things stop it: Caddy strips both from the request before the subrequest is made; the sidecar strips them again on arrival and logs a warning if they were present; and the sidecar emits X-Bard-Visitor on every allow, using - for an anonymous reader, because a header the responder omits is a header whose inbound value survives.

Nothing on the sidecar's own listener checks a secret. Its access control is the bind address, which is why the default is loopback and why it is meant to run on the edge node and nowhere a stranger can reach.

Bandwidth

What is counted, and where

Nothing is measured on the request path. Measurement happens afterwards, from the access log, which the sidecar tails every two seconds.

Caddy records the response body's size and cannot record header bytes, so a response costs its body plus a flat 400-byte allowance for headers.

Only GET and HEAD become rows; the origin answers 405 for everything else, so a third method in the log is a request that never reached a snapshot. Every rounding error in this arithmetic runs toward under-counting, which is the right direction, and the figures on your Traffic view are labelled estimates for that reason. The allowances themselves are limits and quotas.

Because the measurement comes from a log read every two seconds, enforcement lags the traffic it is measuring instead of being instantaneous, and one large file can exceed an allowance however short that interval is made.

What happens when an allowance is exhausted

Two different things, refused in two different ways.

A site's budget for anonymous readers is spent. Anonymous readers get the gate: a 302 to that site's own sign-in for a page, a 401 for a subresource. Readers who are signed in are unaffected, because the budget only ever covered the people who were not. This is the case where a refusal is something the reader can act on, which is why it is a sign-in and not a 402.

An identity is out of its monthly allowance. That identity gets the 402 above. When the identity is the site's owner, everything on the domain answers 402 except /.well-known/atproto-did.

An identity or site that has gone over is marked in Redis with a key that holds for ten minutes by default, and the key expiring is the re-check — so raising a budget back above current usage takes effect within that hold and not immediately, and there is no second timer to disagree with it. The reset for the allowance itself is midnight UTC on the first of the month, in every deployment and on every edge node.

There is no notification of any kind before, during or after. No email, no banner. The Observatory is where an allowance becomes visible and a 402 is where it becomes a problem.

Three positions, and what the first one does not do

Enforcement is a deployment setting with exactly three values: off, report and enforce. Near-misses such as on, true, yes and enforced are refused at boot instead of read as a boolean.

off suppresses refusal only. Measurement, the ledger, the usage meters and the traffic views all keep running, and an anonymous reader under a path that requires sign-in is still refused. That gate is an audience control and not a bandwidth one, and it does not consult the mode. The switch that actually stops measurement is leaving the access-log directory unconfigured, which stops the log processor and so turns the whole feature off — a supported way to run a deployment rather than a degraded one.

Rate limits

Three limits apply to traffic for published sites. All three count requests instead of bytes, and none of the three numbers comes from a measurement. What one origin replica can absorb has not been established, so they are set loose enough not to reject real traffic.

The per-client bucket keys on the address the layer-4 balancer supplies through the PROXY protocol, never on a forwarded header. A header is something a client sets, and keying on one would let a caller evade their own bucket and poison somebody else's by naming their address.

When the check itself is unavailable

If the sidecar cannot be reached, Caddy proxies the request straight to the origin and the site keeps serving.

Caddy had to be told to do that, in a configuration file, and it is a branch that never runs while anybody is developing, because the sidecar is up on their machine. The default behavior of an access check whose responder is unreachable is a 502, which would mean every site on the network going dark because a billing dependency was down. That is the failure this tier exists to make impossible, so a 502, 503 or 504 from the sidecar is caught and the request is proxied to the origin with the origin key attached, exactly as an allowed request would have been.

What that window costs, stated instead of hidden:

• Traffic served that way is unmetered and unenforced. Nobody's allowance moves.
• A gated site is served too. A site configured to refuse everyone serves everyone for the duration. The trade is deliberate: a site going dark is worse than a site being read.
• The sign-in paths answer 404 during the window, because they are the sidecar's and the origin does not have them.
• Every affected log line carries a failopen field, so when the sidecar comes back it can count what it missed while it was gone instead of leaving a gap that looks like quiet.

A test asserts that the fail-open rule is in the file and that it names all three of 502, 503 and 504. That it actually fails open is something only the integration rig can prove.

The access log

One JSON line per response, written by Caddy and read by the sidecar seconds later. It is a privacy control before it is a data source: Caddy's default access-log format records the client's address and every request header, so turning logging on without pinning a format would ship exactly the data this design promises not to keep, on day one, with no code for anybody to review. The format is pinned in the configuration file, and a Rust test reads that file and asserts the format textually, because nothing in the test suite parses a Caddyfile.

What a line holds

{
        "level": "info",
        "ts": 1756819331.4021928,
        "logger": "http.log.access",
        "msg": "handled request",
        "request": {
          "proto": "HTTP/2.0",
          "method": "GET",
          "host": "example.com",
          "uri": "/blog/post-1?utm_source=x"
        },
        "bytes_read": 0,
        "user_id": "",
        "duration": 0.0062,
        "size": 41983,
        "status": 200,
        "visitor": "did:plc:examplevisitor000000000000 example.com",
        "payer": "visitor",
        "resp_headers": {
          "Content-Type": ["text/html; charset=utf-8"],
          "X-Bard-Cid": ["bafyreiexamplerecordcid00000000000000000000000000000000000"],
          "X-Bard-Rev": ["3lkrev"],
          "X-Bard-Did": ["did:plc:exampleowner00000000000000"],
          "X-Bard-Outcome": ["hit"],
          "Content-Encoding": ["gzip"]
        }
      }
      

The identifiers and the domain are illustrative; the key names, the nesting and the types are exact. On a real edge the request object carries those four keys and no others. visitor and payer are top-level fields, not headers, and visitor is written before the access check runs. A refused request ends the pipeline early, so with the field written afterwards every refusal would have been logged anonymously and "who was refused" could not be answered from the log at all.

The four X-Bard- response headers are the join keys that make this Highport's record of a request instead of a web server's, and they were already on the wire as support diagnostics. Nothing strips them on the way out, so every visitor sees all four. A line with no X-Bard-Did on it is skipped and not treated as an error, the ordinary outcome for every /_bard/* answer and every http:// redirect.

What is deleted, and what that costs

Five fields are removed from every line before it is written, and one is added back.

Deleting the whole header map is a correction, not a preference. It began as a list of three names, Cookie and Authorization and User-Agent, and the first time this ran behind a tunnel the log carried Cf-Connecting-Ip, X-Forwarded-For, Cf-Ipcountry and Cf-Ray: a real client address, twice, plus their country, in a file that is supposed to hold no address in any form. Nobody had enumerated those, because nobody was thinking about a tunnel when the list was written. Deleting the map is an allow-list by construction: no request header reaches the file, so no header a future proxy invents can either.

The one header that comes back is the user agent. It is written as its own top-level field, user_agent, by name, so the header map stays deleted and a header a proxy invents still cannot reach the file. It is how a request is classified as a browser, a crawler or unknown, which is the crawler line on your Traffic view. Highport keeps the string itself for 30 days, with the rest of the raw request records, so it can review crawlers and automation and check the classifier against what actually arrives. It is not shown to site owners: you see the browser/crawler split, never a raw agent, and never an agent next to a reader who signed in. Requests recorded before this was switched on in September 2026 have no agent and count as unknown.

Two things are lost by deleting the header map, and each is worth stating instead of leaving to be found.

Referrers are not kept, so there is no "where did this traffic come from" anywhere in the product.

Unique visitors are not derivable from these rows at all. Without an address there is no way to tell two anonymous requests from one visitor from one request each from two, and no field is hashed to make it possible. The user agent does not change that: a common one is shared by millions of readers, and nothing uses it as a visitor key. A visitor is counted only when a reader deliberately signed in to that domain, and every other row's visitor is null.

What you never do here

There is no configuration on this tier that belongs to a publisher. You do not obtain a certificate, install one, renew one, or receive an expiry reminder. There is no cache to purge, because a publish reaches the next request, which is how your site reaches people. There is no allowance to top up and nothing to pay: no money moves anywhere in this product, and an allowance is a grant rather than a bill.

What is yours is upstream of all of it: two DNS records, a site record, and, if you want one, a decision about who gets to read it.

Advanced · 4 of 7