System Status

Services

Checks run server-side every 5 minutes; days without records are shown in grey.

Incident history

Written records: what happened, what the root cause was and what changed. Automatically detected measurement gaps are listed under "How this is measured".

1. Outage23 July 2026 at 13:00· Duration: 4 d 1 h

   Full outage — hosting provider blocked the server IP

   Impact

   The site and every service behind it returned Cloudflare 523. Nothing reached the machine: HTTP, SSH and ICMP were all dead, and even a rescue boot came up without networking.

   Root cause

   A €16.58 hosting invoice dated 2 July was missed. The provider locked the account for non-payment and blocked the primary IP address. The disk, the OS and the application were all healthy — the outage lived entirely in the network layer.

   Resolution

   The invoice was paid and an unblock request was filed through the provider's console. Access came back in stages: ports 80/443 first, SSH and ICMP roughly three hours later. While diagnosing, a temporary jump host inside the provider's own network was used via ProxyJump, which proved the machine itself was fine and the block sat at the border; that host was deleted afterwards.

   Lesson

   Monitoring that runs inside the server goes quiet together with the server: no health ping left the box during the outage, so nobody was alerted. Permanent fixes: automatic payment on file, a dedicated mail filter for critical infrastructure invoices, and independent monitoring from outside the server. The "How this is measured" section on this page exists for the same reason.

2. Near miss14 July 2026· no visitor impact

   Near miss — a restart fired mid-build and looped the service

   Impact

   No visitor impact: the reverse proxy logged zero 502s during the window. The application process briefly entered a restart loop and hit the systemd start limit.

   Root cause

   While the app was being moved to a non-root system user, the five-minute health check fired mid-build and restarted a half-written build output. The same class of bug caused the module-not-found crash loop earlier in July: the build rewrites the directory that is being served, underneath the running process.

   Resolution

   A build lock was added to both the health check and the deploy script — no script may restart the service while a build is running — and the failed state is reset before every restart. The lock was verified with a synthetic build.

   Lesson

   Nothing automated may restart the service while a build is in flight. That rule now lives in the scripts themselves; remove it and the deploy falls back into the same loop.

Background jobs

The last run of the maintenance and automation jobs that keep the site going. A job that has missed more than twice its cadence turns yellow; a job that failed turns red.

• Scheduled content publishing

  5 min cadence

  Up to date17/09, 11:25

• End-to-end smoke test

  15 min cadence

  Up to date17/09, 11:15

• Newsletter A/B winner selection

  30 min cadence

  Up to date17/09, 11:00

• Newsletter drip delivery

  once an hour

  Up to date17/09, 11:00

• Enquiry response-time check

  once an hour

  Up to date17/09, 11:10

• Daily digest report

  once a day

  Up to date17/09, 10:50

• Database backup

  once a day

  Up to date17/09, 09:00

• GDPR data cleanup

  once a day

  Up to date17/09, 09:00

Frequently asked questions