INC-0004: Core Router Lost: a Hard Hang, Then Recurring re0 Watchdog Timeouts #

Summary #

At about 20:14 the core router stopped answering on both its LAN (10.20.99.1, re0) and its WAN (192.168.8.106, re1). This happened in the middle of CHG-0018: the log store’s deploy stalled partway through pushing a container image to dv02obs001v01 on Platform, a path that crosses the router. The operator, working from trusted, lost the Builder and came back in over the travel router’s LAN.

At the console the router was hard-hung: no video, and the keyboard’s lock light did not respond, so the OS was not servicing USB. A power-cycle brought it back at 20:33:34. Its configuration came back intact: the firewall shows no drift against inventory, and the Kea reservations and Unbound overrides match what was there before.

The cause is not established. Nothing the deploy ran was aimed at the router. The last write to its configuration was at 19:50:49, from the opnsense_dns and opnsense_dhcp runs, 23 minutes before the hang. No log survives from the minutes before it: the root filesystem was not cleanly dismounted, and crash dumps are disabled.

It happened again, differently #

At about 21:19 the router’s LAN stopped answering again, but this time the box stayed alive. The console responded, and dmesg showed re0: watchdog timeout: the LAN NIC had stopped, not the machine. The management → Platform path then failed intermittently all night, as nms’s upload retries and obs’s server logs both record. The operator swapped the cables and rebooted, and still saw four more watchdog timeouts by the next morning.

The second event confirms the NIC driver hypothesis for that event. It does not establish that the first event, a full hard hang, had the same cause.

Impact #

• No routing between segments from about 20:14. That includes trusted → management, which is the operator’s normal path to the Builder.
• No site DNS. The router is the resolver for management and for the Builder’s enp4s0.
• Everything behind the router unreachable from the Builder: Platform (dv02obs001v01, OpenBao, the API), tenants, IoT Backend.
• CHG-0018’s deploy stopped half-done. On dv02obs001v01, VictoriaLogs was started and vmauth never deployed. The syslog port’s firewall rule was never added.
• Not affected: hosts on the management L2 segment reached each other directly. The switch, AP, both hypervisors, dv02nms001v01, dv02col001v01 and dv02bld001v01 all answered.

Detection #

The operator lost their session to the Builder and could not get back by the usual path. Claude found the router down while investigating the operator’s report at about 20:25, roughly 11 minutes after it happened. Nothing alerted, because the site has no monitoring yet (ADR-0023 is Proposed).

Timeline #

Symptoms #

• From the Builder, ip neigh shows 10.20.99.1 dev enp4s0 FAILED. Ping gives 100% loss with send errors, which is ARP failing, not a filter.
• TCP 443, 22 and 53 to 10.20.99.1 fail. DNS: communications error to 10.20.99.1#53: timed out.
• Every routed destination fails, for example 10.20.25.22:22 and 10.20.10.1:443.
• On the same segment, 10.20.99.9, .10, .21, .22, .40, .41 and .97 all answer.
• WAN side: 192.168.8.106 (re1, MAC 00:e0:4c:29:1e:4f) was in the Builder’s neighbor table as STALE, and TCP 443 to it fails. The WAN may not listen on 443 by design, so this only suggests that the whole box is down, not just its LAN.

Investigation #

Ruled out, with evidence:

• The Builder’s own network. enp4s0 is up with its declared profile. The Builder reaches every same-segment host, and its last NetworkManager event was 20:03:33, eleven minutes before the loss.
• The Claude CLI uninstall task in the workstation role. It is a no-op. The CLI is the native install at ~/.local/bin/claude, and the task reports changed whenever npm exits 0.
• The log-store deploy acting on the router. Every task it ran is listed in its log, and none targets the router. It wrote files on dv02obs001v01 and started a container there.

At the console: no video, and no response from the keyboard’s lock light. That is a hard hang, not a panic that printed and stopped, and not a reboot loop.

After the power-cycle, read through the API because SSH is not open to a_autoprov:

• boottime Tue Sep 22 0:33:34 UTC (20:33:34 EDT), and config Mon Sep 21 23:50:49 UTC (19:50:49 EDT). Nothing changed the configuration after the DNS/DHCP runs.
• The system log the API returns is the current day’s file, which in UTC begins at 20:00 EDT. It holds nothing between 20:00 and this boot, so the 14 minutes before the hang left no trace. The boot log says why:
  • WARNING: / was not properly dismounted and mount pending error: blocks 816 files 22. Unflushed writes, which would include the last minutes of logging, were lost at the hard stop.
  • Configuring crash dump device: /dev/null. Crash dumps are disabled, so a kernel panic could not have been captured either.
• Two hypotheses. Neither is confirmed, and nothing here rules either out:
  1. Thermal or load, on fanless hardware (raised by the operator). The router is a fanless ZimaBoard, and its work has grown. Since CHG-0007 it enforces 57 inter-zone rules, so every packet between segments goes through pf. It routes on a stick: management → Platform traffic enters and leaves on the same re0 trunk, so that link carries every image push twice. The evening was its heaviest yet: VM builds, image pushes, OpenBao calls, and DNS/DHCP API writes. A fanless board sheds heat only through its case, so heat built up over the evening would show in no single command. A hard lockup with no video and no keyboard response is consistent with a thermal or power stop. Against, or at least unsettled: about 50 MB of image pushes is light work for a gigabit router, so load alone should not hang a healthy box. If load was the trigger, something was already marginal.
  2. The Realtek NIC driver. Update: confirmed for the second event (re0: watchdog timeout), and not yet shown for the first. Both NICs are Realtek RTL8168/8111 on FreeBSD’s re(4) driver, which has a reputation for hanging under load. The hang came while an image was crossing re0. The first push, 26 MB, crossed it without trouble.
• Read after recovery (20:38): 44.1 °C on the only sensor exposed, one ACPI thermal zone (hw.acpi.thermal.tz0), not per-core. Load average 0.50, and 755 MB of 8 GB memory in use. That rules out memory pressure. It says nothing about the temperature at 20:14, five minutes after a cold boot.

Root cause #

For the second event and the recurrences, the immediate fault is established: the core router’s LAN NIC, re0 (Realtek RTL8168/8111 on FreeBSD’s re(4) driver), hits watchdog timeouts. Each one stops the interface that carries every VLAN. That is every routed path at the site, and the resolver. A cable swap did not stop them.

Why re0 times out is not established: the driver, the chip, the cabling, heat, or load. The cable swap may have reduced the timeouts, in the operator’s impression, but it did not stop them. Whether the first event, a hard hang with no video and no keyboard, was the same fault is also not established. A NIC watchdog timeout does not by itself stop the console, so the first event may have been a different failure, or a worse form of the same one. No evidence from it survived (see Contributing factors).

Recovery #

The operator power-cycled the router at the console at about 20:33. It booted normally, and routing, DNS and the Platform path were back by 20:35:59. Its configuration was verified read-only (see the Timeline, ~20:40).

First event: CHG-0018 Step 4 was left half-done on dv02obs001v01. It was completed at 20:44 and verified.

Second event and after: the router was rebooted by the operator, then rebooted again after a cable swap. It is reachable at 07:54 on 2026-09-22, but watchdog timeouts continue, so service is not considered restored. CHG-0018 was narrowed so that it adds no traffic across re0: nothing ships to the store, and the router’s syslog is excluded until the NIC is stable.

Contributing factors #

• No monitoring. The loss was found by the operator losing their session, about 11 minutes later (ADR-0023, Proposed).
• The router is a single point for everything routed, including the operator’s path to the Builder and the site’s DNS.
• No evidence survives a hard hang. Crash dumps go to /dev/null, and logging is local only. The last minutes of logs were lost with the unclean stop. Off-box syslog would have kept them. CHG-0018 built the store, but the router’s syslog is deliberately not sent there until the NIC is stable, because the NIC is the path.
• The shipping trial added traffic across re0 during the second event: nms’s full-journal backfill of 194,077 lines, and its overnight retries. It did not start the fault; the first event predates it. It may have made the fault more frequent. It was stopped on 2026-09-22.

Corrective actions #

Preventive actions #

Follow-ups #

Follow-up 3, the vendor driver, from OPNsense’s own plugin source (opnsense/plugins, net/realtek-re, read 2026-09-22):

• PLUGIN_NAME= realtek-re, PLUGIN_COMMENT= Realtek re(4) vendor driver, PLUGIN_DEPENDS= realtek-re-kmod.
• Its description: “This is the official driver from Realtek and can be loaded instead of the FreeBSD driver built into the GENERIC kernel if you experience issues with it (eg. watchdog timeouts), or your card is not supported.” and “Please note this driver requires a system reboot to activate.”
• It works by a loader drop-in: if_re_load="YES", if_re_name="/boot/modules/if_re.ko".

Inference, to confirm in the change record:

• OPNsense packages plugins with an os- prefix, so it installs as os-realtek-re.
• Removing the plugin removes that loader file, and the in-tree driver returns at the next reboot. That would be the rollback.

The change needs:

• console access, because a driver that fails to attach takes the LAN down
• the timeout rate measured before and after, so the result is a measurement, not an impression

Lessons learned #

To be written once the cause is known.

Related changes #

• CHG-0018: The Central Log Store: in progress when this happened.

Related runbooks #