CHG-0066: The Management Hypervisor Moves to Proxmox VE 9 #

Summary #

dv02hyp001p01 runs Proxmox VE 8.4.21 on Debian 12. Proxmox lists the 8.x series as “Proxmox EOL: 2026-08” and Debian 12 as “Debian EOL: 2026-07”, so the node has had no security updates from either since then. The tenant hypervisor, dv02hyp002p02, has run 9.2 since 2026-08-30. This node was the open half of the Hypervisor Platform Uplift, waiting for a window because every substrate service VM runs on it.

The upgrade is in place, along Proxmox’s own path ( Upgrade from 8 to 9): latest 8.4, the pve8to9 checklist clean, the repositories moved from Bookworm to Trixie, apt dist-upgrade, and a reboot. After it, both hypervisors run the same major version, and the automation token’s role on this node trades VM.Monitor, which also allows commands inside a guest, for the read-only VM.GuestAgent.Audit.

Goal #

• pveversion on dv02hyp001p01 reports pve-manager/9.x, running a 6.14 or later kernel.
• pve8to9 --full reports no failures.
• Every guest that was running before the change is running again, in its startup order.
• DeevnetAutomation on the node holds VM.GuestAgent.Audit and not VM.Monitor; the inventory sets proxmox_ve_major: 9 for the node.
• A template build on the node and a proxmox_vm run against one of its service VMs pass.

Scope #

In scope: Proxmox VE and Debian on dv02hyp001p01; the two boot-loader findings pve8to9 reports; the automation token’s role on the node. Out of scope: dv02hyp002p02, already on 9.2; the CPU microcode package pve8to9 suggests, which needs the non-free-firmware component (a follow-up); VMs 100 and 104 themselves, which are backed up and restarted but not otherwise touched.

Risk and impact #

Prerequisites #

• No template build or proxmox_vm run in progress on the node
• Operator at the consoles of dv02hyp001p01 and the core router for Steps 3 to 6
• Tenants told about the window
• Free space for the Step 2 backups on a storage with backup content (local: 45 GB free on 2026-11-12)
• A current CHG-0039 backup of prv’s and obs’s state
• Vault decrypted, collections built

Procedure #

Step 1: Latest 8.4, boot loader fixed #

No outage. pve8to9 --full on 2026-11-12 reported two failures and four warnings:

Run (as root on the node):

apt update
apt -o Acquire::http::Dl-Limit=1000 dist-upgrade
apt remove systemd-boot
[ -d /sys/firmware/efi ] && apt install grub-efi-amd64
pve8to9 --full

Verify: pveversion is the latest 8.4; pve8to9 --full lists only the VM.Monitor failure and the running-guests and microcode warnings.

Undo: Undo Step 1

Step 2: Back up what code cannot rebuild #

No outage. VMs 100 and 104 are not in the inventory. The node’s own configuration is copied to the Builder.

Run:

vzdump 100 104 --storage local --mode snapshot --compress zstd
tar czf /root/hyp001-etc-$(date +%F).tgz /etc/pve /etc/network/interfaces /etc/apt

Then copy the tarball (a few MB) to the Builder.

Verify: both archives are listed by pvesm list local --content backup; the tarball is on the Builder.

Undo: none needed.

Step 3: Trixie repositories and the download #

No outage. The packages are fetched now, throttled, so that Step 4 moves nothing across the router.

Run:

sed -i 's/bookworm/trixie/g' /etc/apt/sources.list /etc/apt/sources.list.d/pve-enterprise.list
# ceph.list: the node has no Ceph cluster; point it at ceph-squid on trixie, or remove it
apt update
apt -o Acquire::http::Dl-Limit=1000 dist-upgrade --download-only

pve-enterprise.list holds only a commented line; the node uses pve-no-subscription.

Verify: apt update reports no error; the download finishes and the router answered every watchdog ping.

Undo: Undo Step 3

Step 4: Stop the guests and upgrade #

Disruptive. From the node’s console, not the web UI’s: the guide says “Do not carry out the upgrade when connected via the virtual console offered by the GUI”.

Run:

pvesh create /nodes/dv02hyp001p01/stopall
apt dist-upgrade

Answers to the prompts, as the guide gives them: /etc/issue keep (“No”); /etc/lvm/lvm.conf the maintainer’s (“Yes”); /etc/default/grub keep (“No”); /etc/chrony/chrony.conf the maintainer’s (“Yes”). Record any other prompt and the answer under Outcome.

Verify: apt dist-upgrade ends without error; pve8to9 --full lists no failure but the VM.Monitor role.

Undo: none. This is the point of no return.

Step 5: Reboot #

Run: reboot, at the console.

Verify:

1. The node boots the new kernel (uname -r 6.14 or later) and pveversion reports 9.x.
2. vmbr0 is up on enp0s31f6 and the node answers at its management address. If the NIC was renamed, fix /etc/network/interfaces at the console (or pin the name with pve-network-interface-pinning), then ifreload -a.
3. The guests with onboot start in startup order: dv02idn001v01 first. OpenBao starts sealed, as after any restart.

Step 6: The automation role on 9 #

Run: set proxmox_ve_major: 9 in mobile/host_vars/dv02hyp001p01/vars.yml by pull request, then ansible-playbook playbooks/site.yml --limit dv02hyp001p01 --tags proxmox-access.

Verify: pveum role list shows DeevnetAutomation with VM.GuestAgent.Audit and no VM.Monitor; pve8to9 --full reports no failure.

Undo: revert the inventory change and run the play again.

Step 7: Tidy the sources #

Run: apt modernize-sources. The guide: “The command will also keep the old .list files around by appending .bak to them.” Remove the .bak files once apt update is clean.

Verification #

• make proxmox-fedora-pve1 builds a template on the node, with the operator at the router’s console.
• site.yml --limit dv02prv001v01 --tags vms passes.
• Every service VM answers on its service, and the API’s health check passes.

Undo #

Undo Step 1 #

Package updates within 8.4 are not reversed. If GRUB does not boot, boot the previous entry from the console and reinstall systemd-boot.

Undo Step 3 #

sed -i 's/trixie/bookworm/g' /etc/apt/sources.list /etc/apt/sources.list.d/pve-enterprise.list
apt clean && apt update

To close #

This change closes when every item below is done.

• Step 1: latest 8.4, systemd-boot meta-package removed, grub-efi-amd64 installed
• Step 2: VMs 100 and 104 backed up; the node’s configuration on the Builder
• Step 3: Trixie repositories; packages downloaded, the router up throughout
• Step 4: guests stopped; apt dist-upgrade done
• Step 5: rebooted onto 9; the network and the guests back
• Step 6: DeevnetAutomation holds VM.GuestAgent.Audit, not VM.Monitor
• Step 7: sources modernized
• Verification: a template build and a proxmox_vm run pass
• The Software Catalog lists Proxmox VE 9 on Debian 13 for the node

Outcome #

Completed after the change has run.

Departures from the plan #

Follow-ups #

• Enable non-free-firmware and install intel-microcode, as pve8to9 suggests
• Mark the Hypervisor Platform Uplift’s management-hypervisor task done