INC-0005: Core Router Hard Hang During a Provisioning VM Rebuild #

Summary #

At about 17:16 the core router stopped forwarding, in the middle of CHG-0039 step 3, the restore rehearsal. The rehearsal had destroyed the provisioning VM and was rebuilding it from its roles. The rebuild had pushed a 176 MB container image from the Builder to the new VM, and had just started a 461 MB one. Both go from management to Platform, so both cross the router. The second push never finished.

The operator lost the trusted Wi-Fi network, came back in over the travel router’s LAN, and found the router hard-hung: connected to a screen and keyboard, it did not respond. A power cycle brought it back. The provisioning VM was then restored from a dump taken before the rebuild, and came back exactly as it had been.

This is a recurrence of INC-0004’s first event. That was also a hard hang, also in the middle of a deploy pushing a container image to a Platform VM, and its cause was never established. This record is a new instance of it, with the same gap: no evidence survived.

The provisioning VM is not in a runtime path, and the incident shows it. The VM was gone from 17:14. The network was lost two minutes later, when bulk traffic crossed the router. The one thing that calls the API at runtime, the tenant egress agent, failed every five minutes for 50 minutes and left tenants’ routes as they were.

Impact #

• No routing between segments from about 17:17 to about 18:02. That includes trusted → management, the operator’s normal path to the Builder, and Wi-Fi clients’ DHCP and DNS.
• No site DNS. The Builder, asking the router for names, could not reach GitHub by name while its management link was up and the router was not.
• The Deevnet API and the state store were down from 17:14 to 18:07. About five minutes of that was planned. Tenants could not apply. No tenant tried.
• The restore rehearsal was abandoned. Nothing was restored from a backup.
• The Builder was rebooted three times by the operator. Its /tmp is memory-backed, so the rebuild’s log and the before-picture taken for the rehearsal were lost.
• Not affected: tenants’ workloads, their egress, device Wi-Fi keys, the broker and the log store were not changed. Hosts on the management segment reached each other throughout. No tenant data was lost: the registry and the state bucket came back from the dump, and three encrypted backups were on the drive.

Detection #

The operator lost the trusted network and could not reach the Builder by the usual path, about five minutes after the router stopped. Nothing alerted: the site has no monitoring ( ADR-0023 is Proposed).

Timeline #

Symptoms #

• The trusted Wi-Fi network stopped working for the operator.
• From the Builder, with its management link up: both hypervisors and the network management VM answered; 10.20.99.1 did not answer ARP; 192.168.8.106, the router’s WAN, did not answer; nothing on Platform answered.
• At the router’s console: no response to the keyboard.

Investigation #

The Builder’s journal gives the order of events. The rebuild’s steps are logged there as they ran: the clone at 17:14:33, the first image at 17:15:50, an OpenBao login through the router at 17:16:19, the second image at 17:16:32, and then nothing from the rebuild at all. The operator’s session from the travel router’s LAN follows at 17:21:23.

The half-built VM agrees. After recovery it was running the state store, started at about 17:16, and held no PostgreSQL image, no database and no API. The second push had not arrived.

The provisioning VM’s absence did not cause the loss. It had been gone for two minutes when the network failed, and it stayed gone for 51 minutes while the router was back and everything else worked. The tenant egress agent’s log shows what a missing API does at runtime: cannot reach https://api.mobile.deevnet.net:8080; leaving /etc/frr/frr.conf.local as it is.

The image push was not the only bulk traffic. Proxmox tells every new clone to upgrade all its packages on first boot. The provisioning VM upgraded 203 packages when it was first built, and the new clone started the same thing at 17:15:10, downloading through the router while the images were pushed to it. This was found afterward, and it had been an open item since CHG-0008.

A second rebuild without either flow did not hang the router. At 19:11 the same VM was rebuilt with its first-boot upgrade off and its images delivered on an ISO from its own hypervisor. The router answered every check for the nine minutes it took. One run each way suggests load is the trigger; it does not establish it.

Nothing was read from the router. It was power-cycled from a hard hang, and it keeps no log across one (INC-0004, Contributing factors).

Root cause #

Not established. The router hung hard, with its console dead, while a sustained bulk transfer crossed it from management to Platform. That has now happened twice, in INC-0004’s first event and here, and both times a deploy was pushing a container image to a Platform VM.

Two things are different from INC-0004’s later events, where the LAN NIC re0 hit watchdog timeouts and the box stayed alive. Here the box did not stay alive. Whether a hard hang is a worse form of the re0 fault or a separate one is still open, as it was in INC-0004.

INC-0004’s corrective action 2 proposed reproducing the fault deliberately, by pushing sustained traffic across VLANs with the console attached and the temperature recorded. This incident did the first half by accident and none of the second.

Recovery #

The operator power-cycled the router at its console at about 17:58. It was forwarding by about 18:02.

VM 201 was restored from the dump taken at 17:13, with qmrestore on the management hypervisor, and started at 18:06. The restore used no path through the router. The API, the state store, the registry and the state bucket were checked against what they held before the rehearsal.

The router’s configuration was not checked against inventory after the power cycle.

Contributing factors #

• INC-0004’s fix was still open. The vendor driver for the router’s Realtek NIC (INC-0004, Follow-up 3) had not been installed, and no change record for it existed.
• The change did not name the risk. CHG-0039’s risk table covered the restore rehearsal disturbing the provisioning VM. It did not cover what a rebuild of that VM does on the way: push several hundred megabytes of images through the router. INC-0004 had already recorded that as the circumstance of a hang.
• Every new VM upgrades itself through the router on first boot, unless told not to. The rebuilt VM was doing so when the router hung.
• Every image for a Platform VM crosses the router. The Builder is on management and the VMs are on Platform, so there is no way to build or rebuild one without a bulk transfer over re0.
• The operator could not log in at the Builder’s console. Every account on the Builder was password-locked, so with the network gone the console was no use, and the Builder was rebooted in the attempt to get back in.
• No monitoring, and no evidence survives a hard hang: both as in INC-0004.
• The rehearsal’s working files were in memory. The Builder’s reboot took the rebuild’s log and the before-picture with it.

Corrective actions #

Preventive actions #

Follow-ups #

Lessons learned #

• A fault left open is a risk on every later change that touches its trigger. INC-0004 recorded that a hang came during an image push across the router. A change that rebuilds a Platform VM should have read that as its own risk.
• The fallback that did not cross the router is what made recovery quick. The dump sat on the hypervisor that runs the VM. A fallback that needed the Builder to push anything to Platform would have needed the path that had just failed.
• A machine nobody can log in to at its console is not recoverable from its console.

Related changes #

• CHG-0039: Backup to an Attached SSD: its step 3 was in progress when this happened.
• CHG-0018: The Central Log Store: in progress during INC-0004’s first event, the same circumstance.

Related runbooks #

• Console Recovery: Core Router
• Backup and Restore