Troubleshoot a Service VM #

Finding what is wrong with one of the six substrate service VMs and getting it back: a service that is down, a name that gets no TLS handshake, a VM that is off the network, and a disk that is full or missing.

Nothing raises an alarm for any of these yet. They come to light when someone tries to use the service ( How the plane is watched). Start from what failed for the user and work inward, in the order below: the network, the VM, the service, the proxy, then the disk.

How every service VM runs #

All six VMs are built the same way, so the same checks apply to each:

• Each service is a Podman container under its own systemd unit, /etc/systemd/system/<name>.service. The unit and the container share a name.
• A failed container is restarted every 10 seconds, indefinitely (Restart=always, RestartSec=10). A service that keeps failing does not stay down. It loops, and nothing reports it.
• Data lives under /srv/<service>. Only dv02obs001v01 has a separate data disk mounted at /srv. On the other five, /srv is on the OS disk. The Omada controller keeps its data under /opt/omada-controller.
• Each host is rebuilt from code: make rebuild, then site.yml --limit <host> in deevnet.mgmt ( Build a Management-Hypervisor VM).

VM unreachable #

Proxmox reporting running does not mean the VM is on the network. In INC-0002 the controller VM was running while it answered nothing. Only the guest agent and the network showed the real state.

From the management hypervisor, dv02hyp001p01:

qm status <vmid>          # the process, nothing more
qm agent <vmid> ping      # the guest OS: every service VM runs the agent
ip neigh | grep <ip>      # INCOMPLETE: nothing on the bridge answers for that address

Capture before you reboot. For a wedged guest, open the noVNC console in the Proxmox UI and record what is on the screen before you reboot. The service VMs have no serial port. A reboot restores service and destroys the evidence. INC-0002’s cause is still unknown for that reason.

On the wrong address #

A VM on an address other than its own has taken a pool lease instead of its reservation. Either the reservation is missing, or there are two reservations for the address. A host rebuilt under a new MAC leaves its old reservation behind, because opnsense_dhcp only adds and updates. It also skips, without any message, a host whose MAC is empty, and a host whose infrastructure is set at group level rather than host level.

1. On the core router, find every reservation for the address, and delete any with a MAC other than the one in the host’s identity.yml.
2. make dhcp && make dns in deevnet.net.
3. Reboot the guest, so it asks again.

Service down #

On the VM, ask systemd first, then the container:

systemctl status <unit>
journalctl -u <unit> -n 100 --no-pager
sudo podman ps -a                     # Up, or Exited with a code
sudo podman logs --tail 100 <unit>

Re-running the play is the fix for drift. The roles converge configuration, the unit and the container. Run it with --limit <host>, using the project-local collections as Build a Management-Hypervisor VM describes. A bare ansible-playbook silently runs the system-wide copy of the roles instead.

A service that can’t read its certificate or key resets every TLS handshake. It looks like a network fault. See Certificates → Troubleshooting.

Some services have their own recovery runbooks:

• The Omada controller: Omada Controller Recovery
• OpenBao: OpenBao Drills
• The API’s database and the state store: Backup and Restore

Proxy refuses a name #

On dv02prv001v01 and dv02obs001v01, the service proxy is the only listener on 443.

A name the host doesn’t serve gets no handshake at all. The client sees the connection reset or an alert during the handshake, never an HTTP error. So a refused handshake on 443 means the name is not in the host’s service_proxy_vhosts, or the client sent no name or a different one.

# From the client: does the host serve this name, and with which certificate?
openssl s_client -connect <host-ip>:443 -servername <name>.<site>.<root> </dev/null \
  | openssl x509 -noout -subject -ext subjectAltName

Disk full #

On five of the six VMs, all of the data is on the 32 GB OS disk. When it fills up, the services fail to write, and each fails in its own way.

df -h / /srv
sudo du -xh --max-depth=2 /srv /var/lib/containers | sort -h | tail
sudo podman system df

• Old images: sudo podman image prune removes the images no container uses. Each role loads its image from a tarball, so a pruned image comes back on the next run of the play.
• The journal: sudo journalctl --vacuum-size=200M.
• Growing the OS disk: set or raise mgmt_vm.os_disk_size in the host’s vars.yml and re-run the play. PVE can grow a disk but not shrink one, and the role refuses a smaller size. Then check df -h / inside the guest. If PVE shows the new size but df doesn’t, the root filesystem has not grown yet. Cloud-init’s growpart extends it at boot.

dv02obs001v01’s logs are on its data disk. The log store’s own retention and size cap keep it from growing without limit.

Data disk missing #

dv02obs001v01 mounts its data disk at /srv with nofail, and every service on it requires that mount, so without the disk the host still boots and can be repaired, but none of its services start. Each unit has RequiresMountsFor=/srv. A disk that never appears fails the mount after about 45 seconds, and each service then fails with Dependency failed and stays inactive. It does not loop, and nothing is written to the OS disk. Unmounting /srv stops all five services first.

systemctl status srv.mount                # failed, or waiting for its device
findmnt /srv                              # nothing: the data disk is not mounted
lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT      # is a disk with serial deevnet-data present?

1. If the disk is present but unmounted, re-run the play. The data_disk role finds the disk by its serial, refuses to touch the disk backing /, refuses to mount over a populated directory, and makes a filesystem only on a disk that doesn’t have one.
2. If the disk is missing from the VM, check qm config <vmid> for scsi1. A re-run of the play attaches a disk that is declared but missing. Data on a destroyed volume is gone: qm destroy deletes every volume the VM owns.
3. Start the services. Starting any one of them mounts /srv first.

Rebuilding the whole host instead is Rebuild a Hypervisor for the hypervisor, and What a failure takes with it for what each VM loses.