The household depends on this: Home Assistant, an MQTT broker, Zigbee devices on a PoE coordinator, a Bluetooth bridge to a battery station, and a Raspberry Pi 5 automation host. It is one home, not production at scale — but failures here have real consequences (the office loses mains power when the battery bridge lies), which makes it an honest place to practise SRE habits.

Stack: Home Assistant + Mosquitto, ZHA/Zigbee (SLZB-06M), MQTT → HMAC webhook relay, systemd user units, WireGuard, reverse proxy, openSUSE MicroOS, Python watchdogs, Zigbee LQI/RSSI instrumentation.

Failure modes I designed against

The catalyst was a 2026-09-19 incident: the Bluetooth → MQTT bridge stayed running but stopped publishing. Home Assistant kept the last value (95 %) with no unavailable transition, and the office lost power when the pack emptied. Nothing threshold-based could have caught it — the process was up.

That produced two explicit failure definitions for the bridge watchdog:

  • Silence — no MQTT messages at all (act after 180 s, 240 s tolerated per escalation rung, 300 s of traffic to call a rung done).
  • Chanting — messages arriving but no value ever changing (420 s with the last dc_output_power above a 20 W load qualifier, so a genuinely idle system is not misread as a fault).

Broker and network outages are never counted as device failure.

The escalation ladder is ordered by evidence, cheapest first: restart the bridge (stalled BLE session, ~2 s); power the adapter back on; rfkill radio power-cycle; reload hci_uart (the proven cure for a hung controller); settle-and-check; then restart on the settled radio. Deliberately not a rung: restart bluetooth.service, which is what wedges this combo chip during a live BLE session. Verdicts are published retained on bluetti/guard/status (ok / repairing:<rung> / unrecoverable), and a Home Assistant automation forces the plug on for unrecoverable.

Proof it works: the 2026-10-10 recovery

During an unrelated trim test the bridge hit a known firmware wedge at 17:34 UTC. The guard walked the full ladder on its own — restart, adapter power, rfkill, hci_uart reload, settle check, restart on the settled radio — and traffic returned at 17:57:41 UTC: about 24 minutes blind, fully self-recovered, with an audit trail of each rung.

Two lessons went straight back into the design:

  1. The zombie trap. During the rfkill step, systemd restarts produced a process that was running, had reconnected to MQTT, but whose BLE polling task had already died (POWERED_OFF). It looked healthy and published nothing. A silence watchdog is the only thing that catches that.
  2. The recovery landed inside the sunset window — the strongest window for the experiment being run — so the data was honestly reported as “telemetry gap, do not read through it”, not silently smoothed over.

Observability lessons

  • The guard cannot report its own host dying. A dead host publishes nothing, so the retained verdict stays ok. That case belongs to an off-box sentinel, which is why an HA-side automation checks telemetry staleness on the force_update power sensors.
  • Test the failsafe, not just the happy path. A first version of that HA automation used state_attr(entity, 'last_updated'), which is not a state attribute; the default collapsed the comparison to “always true” and it forced the plug on twice before being caught and corrected the same day.
  • Edge-triggered automations are not failsafes. If the system restarts with the state already past a threshold, the edge never fires; the guard’s unrecoverable verdict and the staleness sentinel cover that.
  • Know which layer is durable. Raw state history is short-lived; long-term statistics (hourly min/mean/max) are the permanent record.

The Zigbee side

The same estate has a genuinely interesting measurement problem: one coordinator cannot serve both the house and the detached office through thick masonry. The study that came out of it — 29 readings across four configurations, with occupancy treated as a variable and each configuration scored by its worst link — is written up separately in Score a Zigbee mesh by its worst link. The operational rule it produced: design to the measured floor, not the average.

What this demonstrates

  • Writing and iterating on watchdogs that model how systems fail (silence vs chanting), not just thresholds.
  • Incident handling with an audit trail, a written postmortem culture, and changes that came directly from the last incident.
  • Designing measurement protocols for noisy physical systems, with explicit confounders and a rule that survives cherry-picking.
  • Honest reporting — including the data gap caused by the recovery itself.

Provenance: ~/homelab/ (hosts.md, zigbee-network.md, mqtt-webhook-bridge.md, eb3a-ha-bridge.md, and git history including the 2026-10-10 incident record). Analysis and write-up assisted by opencode (deepseek-v4.1-flash-free); numbers and timeline checked against the logs.