CHG-0006: Access Switch Firmware Upgrade #

Summary #

The access switch ran firmware 1.20.1, from January 2024. Current for its hardware is 1.20.24, from May 2026: one hop, with no minimum prior version and nothing irreversible. This change is the firmware only. The switch stays standalone, configured by switch_vlans. Adopting it into the controller is a later change under ADR-0009.

It was phase 2 of CHG-0004 until the AP work was split out to go first.

Goal #

• The switch runs 1.20.24 in one image slot, with 1.20.1 kept in the other as the rollback.
• Its configuration is unchanged, and Ansible still connects.
• STP and LLDP are in a state that was chosen rather than inherited from new defaults.

Scope #

In scope: the switch’s firmware. Out of scope: adopting it into Omada, and any configuration change.

Risk and impact #

Prerequisites #

• Firmware 1.20.24 mirrored and pinned by sha256; it matched on 2026-09-10.
• Operator on site, connected to the builder through the travel router rather than the switch, per Operator Access. The SSH session survives the reboot, and its tunnel carries the switch’s web UI (https://localhost:8443) for the upload. Backup: a laptop on the operator port gi1/0/2 ( Operator Access → Backup). That path goes through the switch, so it’s fine for the backup and the upload, but it drops at the reboot along with everything else.
• The .bin downloaded to the laptop beforehand, with sha256 a01034c5409bd14c9066db857d192807bdfe85788472ed12bb5778159c7a4e0d. That is the file inside the pinned zip, checked on 2026-09-16.
• Vault decrypted, so Ansible can verify the switch before and after.
• show system-info, show image-info and the running config captured on the day. Not done. A configuration backup was taken from the web UI, but not the CLI captures. The config after the upgrade was checked against inventory instead; see Outcome.

Baseline, 2026-09-10 #

Procedure #

Follow Access Switch → Firmware upgrade:

1. Record where it starts — capture system info, image info and the running config.
2. Upload into the backup image from https://10.20.99.10 (https://localhost:8443 through the tunnel) — the .bin, with the auto-reboot box unchecked.
3. Boot from it — set the next startup image, then reboot at a moment of the operator’s choosing. Before the reboot, read Keeping the builder online.
4. Verify — see below.

Keeping the builder online during the reboot #

The builder has two uplinks. Only one of them goes through the switch.

On 2026-09-16, enp1s0 was checked on its own: it pings 1.1.1.1, reaches https://api.anthropic.com over HTTPS, and 192.168.8.1 resolves public names. An operator session to the builder over the travel router’s wireless doesn’t touch the switch, so it survives the reboot. See Operator Access.

What happened on 2026-09-16: when the switch rebooted, enp4s0 lost its link, and in the same second NetworkManager logged set 'enp1s0' (enp1s0) as default for IPv4 routing and DNS. New connections from the builder went out through enp1s0. A connection that is already open over enp4s0 doesn’t move with it. When the switch returned about 100 seconds later, enp4s0 connected again on its own (autoconnect is on), and DNS moved back to 10.20.99.1.

The default route did not move back. Neither connection pins its route metric, so on reconnecting enp4s0 was given metric 102, behind enp1s0’s 101. Internet traffic from the builder stayed on the travel router. Traffic to the site VLANs was unaffected, because it uses the specific 10.20.0.0/16 and 10.10.0.0/16 routes through enp4s0. Pinning the metrics is a follow-up.

Check it during the outage:

ip route get 1.1.1.1              # expect: via 192.168.8.1 dev enp1s0
resolvectl status | grep -E 'Link|Current DNS|Default Route'   # expect enp1s0 as the default route
getent hosts api.anthropic.com    # public names still resolve

If it doesn’t fall back (the route still points at enp4s0, or names don’t resolve), force it:

sudo nmcli con down enp4s0        # routes and DNS move to enp1s0
# ... wait until the switch has booted ...
sudo nmcli con up enp4s0          # the switch path is preferred again
ping 10.20.99.10

enp4s0 is also the builder’s only route to the switch and every site VLAN. While it’s down, ping 10.20.99.10 can’t tell you the switch is back. Watch the switch’s LEDs, or retry nmcli con up enp4s0 until it succeeds, then ping. Bring it back up before verifying.

Verification #

• Version and image slots: 1.20.24 boots, and 1.20.1 is in the other slot.
• Ansible connects.
• STP and LLDP compared with the baseline above.
• The running config compared with step 1.
• The reachability pings.
• One run of switch-vlans.yml.

Undo #

1.20.1 stays on flash. Point the boot configuration back at it and reboot — Access Switch → Rollback. The configuration is kept.

Outcome #

Done on 2026-09-16, in one operator window.

Upload and reboot. The operator reached the builder over the travel router and uploaded the .bin through the tunnel. The “reboot using the backup image” box was checked, which departs from the procedure: the switch rebooted straight after the write, not at a separately chosen moment. It did no harm. The switch came up on 1.20.24 in image1.bin, replacing the factory 1.1.3, and set image1.bin as Next Startup Image too, so the Boot Config step was already done. image2.bin still holds 1.20.1. The link was down from 08:15:32 to 08:17:11 EDT.

Verification.

No re-provisioning was needed. The configuration survived the upgrade unchanged, and the playbook runs applied nothing that wasn’t already there.

Noticed in the running config. With no capture from before the upgrade, none of these can be attributed to the new firmware:

• telnet enable is set.
• NTP is still at factory values: system-time ntp UTC+08:00 with two public servers. The switch clock read about 12 hours ahead of the builder.
• gi1/0/5 has a link but isn’t declared in inventory, so whatever is on it is in VLAN 1, which isn’t routed.
• gi1/0/13 and gi1/0/16 carry no switchport general allowed vlan 1. The other declared ports keep their default VLAN 1 membership. Inventory says nothing either way.

Follow-ups #

• Confirm whether 1.20.17 and later force a password change at first login after a reset. That can only be seen when the switch is reset, which the eventual adoption change will do.
• Move the switch firmware procedure into Lifecycle. It sits in the console-recovery page today.
• Merge net #21 and inventory #33, which declare LLDP.
• Pin the builder’s route metrics (for example enp4s0 100, enp1s0 200), so the switch path wins again after an outage.
• Find the cause of the intermittent switch-vlans.yml command timeouts seen shortly after boot.
• Decide on telnet enable. If it goes, declare it.
• Set the switch’s NTP servers and time zone from inventory.
• Identify what is on gi1/0/5, and declare the port or unplug it.
• Make VLAN 1 membership consistent across declared ports, and declare it.
• Before the next switch change, take the CLI captures. This change had to verify against inventory, because there was no capture to compare with.