Recovery #
Getting service back when something has failed.
- Console Recovery — getting back into a device when the network cannot reach it: core router, hypervisors, access switch and AP
- Troubleshoot a Service VM — a substrate service VM that is off the network, a service that is down or looping, a name the service proxy refuses, and a full or missing disk
- Omada Controller Recovery — restoring the network controller from a data snapshot, including falling back from a bad upgrade
- Backup and Restore — what is backed up (the API’s database and the state bucket, and nothing else), when, how to check it, and how to restore it into a rebuilt provisioning VM
- Rebuild the Core Router — a fresh OPNsense install, brought back from a saved configuration or from nothing
- Rebuild a Hypervisor — a fresh Proxmox install, and what each hypervisor runs back on top of it
- Repave the Builder — reinstalling the hardware Builder from a temporary builder VM, while it is still there to help
- Build the Builder — making the Builder from a bare machine, when it is already gone
What a failure takes with it #
When one component fails, this is what is rebuilt, what has to be redone after it, and who feels it. It describes the site as it works today; where something doesn’t come back on its own, the last column says so. The order of a full build is in Building Infrastructure.
| Failed | Rebuild | Then redo | Tenants feel | Doesn’t come back on its own |
|---|---|---|---|---|
| Builder | Build the Builder, or Repave the Builder if it is still there | Stage artifacts; rebuild locally built images from source | Nothing directly | The cold-fallback controller’s data snapshot |
| Core router | Rebuild the Core Router | Redeploy the Deevnet API and reconcile, which rewrites tenants’ DNS forwards | Tenant name resolution, until then | — |
| Identity VM (OpenBao, tenant DNS) | The VM; OpenBao and PowerDNS from their roles | If OpenBao’s storage was lost: the rebuild in OpenBao Drills, which brings back the same tenant CAs and Transit key from the vault, locks in the new initialization ( INC-0003) and redeploys the API. Template and fabric builds do not need OpenBao: they read the vault | Names, until republished | Tenants’ DNS records: a reconcile restores zones and keys, not records |
| Provisioning VM (Deevnet API, registry, state store) | The VM, the API and the store, then the newest backup: Backup and Restore | Reconcile, which restores tenants’ state users; tenants reapply anything newer than the backup | Applies, until it is back. Nothing at runtime | Earlier versions of state files. If no backup survives: the registry, the audit log and tenants’ state, and tenants are admitted again |
| Messaging VM (broker, log bridge) | The VM, VerneMQ and the bridge | Redeploy the API, which pins the broker’s host key | Device MQTT | Tenants’ broker accounts |
| Observability VM (logs, Grafana, downloads) | The VM and its roles | Reconcile, which restores log users and Grafana organizations; publish the downloads again | Logs and dashboards, until then | Stored logs |
| Omada controller | The VM, then its settings backup from the inventory ( Omada Controller Recovery) | Adopt the AP again if it does not reconnect | Device Wi-Fi | Wi-Fi keys issued since the last backup, until each tenant applies again |
| Management hypervisor | Rebuild a Hypervisor, then every VM on it | Every row above except the Builder and the core router | All tenants | Everything in those rows |
| Tenant hypervisor | Rebuild a Hypervisor, the fabric and egress | Reconcile each tenant, which restores its networks; tenants push their applications again | Their workloads | Tenants’ workloads: a plain apply sees no change ( After a Site Rebuild) |
A cell in the last column is a place where the site can’t yet rebuild something from code. The 2026-10 Rebuild and Access review tracks each one.
Rebuilding a site or a host from nothing is covered by Building Infrastructure. What happened in past failures is under Incident Records.