Behind the scenes

How this status page works

Every dot, bar, and percentage on the front page comes from real probes hitting real endpoints — no manual switches, no curated views. This page documents exactly how those measurements are taken, smoothed, and surfaced.

1. What we monitor

Every service shown on the front page corresponds to a monitor — a single, configurable probe targeted at one specific endpoint. Three probe types are supported, each chosen to match what the underlying service actually exposes:

HTTPS — issued via cURL with follow-redirects (up to 10 hops) and a hard request timeout. A response is considered up when the final HTTP status is below 500 and the round-trip time is below the monitor's configured response threshold. Note that 4xx counts as up — a 401 or 404 means the server is alive and answering requests, it's just refusing this particular one (no auth, rate-limit, missing path). Above the threshold the response is degraded; on connection failure, TLS error, timeout, or 5xx terminal status it is down.
TCP — opens a raw socket to host:port and closes it immediately on success. Same threshold semantics for degraded/down apply, using the socket-open latency as the measurement.
Ping — sends a single ICMP echo request with a 3-second timeout via the system ping utility. Used for hosts that don't expose an application-layer endpoint we'd want to probe directly.

Each monitor carries its own interval (how often it's probed, default 60 s), response threshold (the latency line between up and degraded), and two visibility flags: whether the monitor appears publicly at all, and whether it counts toward the aggregate 30-day uptime headline.

Single vantage point. All checks are run from one place: our VPS in Hetzner's {/* TODO: fill in actual region on first deploy */} datacenter. A regional connectivity issue between that VPS and a target service looks identical to that service being down. For services with broad geographic audiences, treat the numbers on this page as “up from where we're looking,” not “up everywhere.”

2. The prober loop

A dedicated sidecar process — completely separate from the web server — wakes up every 10 seconds and asks the database which monitors are due. A monitor becomes due when the time since its last successful probe exceeds its configured interval, or when it's been touched by an operator (e.g., a manual “run check now”).

Probes run in sequence within a tick; each result is written atomically with a precise timestamp. The prober is the only writer to the per-check history — the web UI and admin panel never invent probe data.

Why a sidecar? Decoupling the probe loop from the request-serving process means probes keep running at a steady cadence regardless of frontend traffic, and a slow probe (e.g., a 30-second HTTPS timeout) never blocks a status page load.

3. Flap protection

Networks are noisy. A single slow response or a one-off TCP reset is almost never a real incident — it's a transient blip that resolves on the next tick. If the page flipped to degraded the moment one bad probe landed, the status would jitter between green and orange constantly, and the signal-to-noise ratio of every dashboard would collapse.

To prevent that, the displayed status of a monitor is derived from a small window of the most recent probes, not from any single one. The rule tolerates a single transient failure inside the window — one bad probe out of five doesn't move the dot. A service has to be consistently bad to be reported as bad.

Crucially, a service that's genuinely intermittent — flapping between healthy and unhealthy probes — shows as degraded, not green. This is the change a status page is most often quietly wrong about: an alternating up/down/up/down service responding correctly half the time is by definition not fully operational, and the page says so. A monitor only resolves to down when the recent window contains no successful probes at all.

The exact thresholds (window size, blip-tolerance count) live in the decide_displayed_status() function on the server — the truth table is right there in its docblock if you want the precise rule. The same logic runs on both the probe-write path and any operator-driven edits to probe history, so the dot can never end up disagreeing with the probes that produced it.

4. History bars & smoothing

Each service row shows a strip of coloured bars summarising the recent past. We render three resolutions on request:

1 hour — 60 one-minute buckets
1 day — 24 one-hour buckets
7 / 30 / 90 days — daily buckets, the last N of the rolling 90-day window

A bucket's colour is the worst probe outcome that landed inside it: green if every probe in that window was up, orange if any was degraded, red if any was down. Buckets with no probes at all show as “no data” (a muted grey).

Two pieces of post-processing run on top of the raw bucket grid before display:

Run smoothing. A run of bad buckets shorter than three consecutive slots is rewritten to operational, on the assumption that a single isolated bad bucket between long stretches of green is more likely a transient than a real degradation worth alarming on.
Short-gap filling. Isolated “no data” buckets sandwiched between operational neighbours — usually caused by a slow probe pushing the next one across a bucket boundary — are filled in as operational. Genuinely empty stretches (longer gaps, or gaps next to bad data) stay as no-data.

Bucket boundaries are anchored to the wall clock, not to “seconds ago” from the moment of the page load. That means refreshing mid-minute doesn't shift probes between buckets; everyone looking at the page at the same instant sees the same bar grid.

The bars and the uptime percentage are two different views of the same data — and they intentionally disagree on short outages. The bars apply the run-smoothing described above so the recent-history strip is denoised and skimmable; the uptime percentage (next section) is computed from raw probe rows with no smoothing at all. Practical consequence: a real 1–2 minute outage can be visible in the percentage and invisible in the 1-hour bar grid, because the bad minute-bucket gets smoothed back to green. This is a deliberate design choice — bars are for visual pattern recognition over time, the percentage is the authoritative number. If you need to chase down a brief failure, trust the percentage and the per-check log, not the bars.

5. Uptime % calculation

The percentage shown next to each service and the aggregate 30-day headline at the top of the page are the same calculation, run server-side, over the same probe rows. Both numbers come straight from monitor_checks — no smoothing, no operator override, no client-side recompute. The math is intentionally simple:

uptime % = (count of probes with status = up) ÷ (count of probes in the last 30 days) × 100

The per-row number reflects whichever range tab is currently selected at the top of the page — switching between 1h, 1d, 7d, 30d, 90d moves both the bars and the number together, so the two views always tell the same story over the same window. The label next to the percentage changes too (“1-hour uptime”, “24-hour uptime”, etc.) so it's always obvious which window you're looking at.

The 30-day uptime at the very top of the page is anchored at 30 days regardless of the per-row range selection. That number is the at-a-glance summary; the per-row numbers are scoped to whatever you're currently inspecting. The headline also counts only monitors flagged as contributing to uptime, so high-frequency canaries and external dependencies can be monitored without dragging it up or down.

All numbers — per-row and headline — are computed server-side from raw monitor_checks rows. No smoothing, no client-side recompute. A two-minute outage shows up in the percentage at any range that includes it, whether or not it's visible in the bars (the bars apply the run-length smoothing described in §4; the percentage doesn't).

A monitor that has never produced a probe row shows the uptime number as an em-dash rather than 100%. The page would rather say “we don't know yet” than fabricate a perfect score on no evidence.

6. Incidents & the operator overlay

When something visible is happening — a partial outage, a third-party dependency degradation, a deploy gone sideways — the on-call operator opens an incident and tags the affected services. The incident has a severity (minor or major) and a running set of timestamped updates from the response team.

Incidents change the page in two visible ways, both subject to the same rule: the incident colour only wins when it's strictly worse than what the probes are actually seeing.

Affected service dots and the history bars covering the incident window pick up the incident colour — but only where the probe-derived colour for that bucket or dot is less severe. If the probes are already reporting a service as down (red), a freshly declared minor incident (orange) does not visually downgrade the bars or the dot to orange. The probes win on ties.
The headline banner at the top reflects the worst of (probe-derived state, operator-declared severity), so an operator-only incident still shows up there even when every probe is green. The headline can be pushed up by either signal but not pulled down by either.

The reasoning: probes are observed data; incident severity is operator opinion. When they agree on the rank, the observation is what gets shown. When the operator's read of the situation is worse than what the probes have caught (because they have human context the probes lack), the operator's read wins. The reverse — operator opinion masking a real outage — is the failure mode this rule exists to prevent.

Crucially, incidents do not change the uptime numbers or the operational counter. Those remain probe-driven. An operator declaring a minor incident with no probe failures is a transparency choice (“we know about this, we're working on it”), not a falsification of the numeric record. If a real failure is happening alongside the incident, the probes will see it and the uptime % will move on its own.

7. Maintenance windows

Planned work is announced ahead of time as a maintenance window: a title, a start/end timestamp, a list of affected services, and an expected impact level. Upcoming windows appear in their own section on the front page so customers can plan around them.

Maintenance windows are an announcement layer only — they don't alter probe behaviour, suppress incidents, or change uptime calculation. If we take a service down for maintenance and that produces real probe failures, those failures count in the same way an unplanned outage would. Honest numbers over flattering numbers, every time.

8. Refresh cadence & the prober heartbeat

The front page polls a single rollup endpoint every 30 seconds in the background. The clock in the header ticks every second to make it obvious the page is live. The prober writes new probe results into the database at its own pace, so within a minute or so of any change, the page reflects it.

The prober also writes a heartbeat to the database at the end of every successful tick — separate from any individual probe result. If a probe fails, that failure gets recorded normally and the heartbeat still advances; the heartbeat is tracking the liveness of the prober itself, not the health of the services it measures.

If the heartbeat goes silent for several minutes, the page assumes the prober is dead and switches to a loss-of-signal mode: a clearly distinct gray dot on every service, an em-dash in place of every uptime number, and a yellow warning banner at the top of the page explaining how long it's been since the last successful probe. This is the one case where probe-derived colours and incident overlays are both ignored — once we've lost contact with the data source, no green / orange / red colour would mean anything honest. “Status unknown” is the only correct answer, and that's what the page says.