Certificates on Deevnet IoTaaS

Renewing

Mon, 01 Jan 0001 00:00:00 +0000

Renewing #

certs.yml in ansible-collection-deevnet.mgmt lays down every certificate and trust anchor on the site, and is how certificates are renewed: re-run it. Nothing renews a certificate on a timer, so a certificate on a host nobody re-applies will expire (risk register R-12).

cd ansible-collection-deevnet.mgmt
ansible-playbook playbooks/certs.yml # everything
ansible-playbook playbooks/certs.yml --tags omada # one play: trust, hypervisors, router, openbao, omada
ansible-playbook playbooks/certs.yml --limit dv02obs001v01

It needs the inventory’s vaults decrypted, and deevnet.builder and deevnet.net published (make publish in each), because the trust, hypervisor and router plays are theirs.

Proxmox

Mon, 01 Jan 0001 00:00:00 +0000

Proxmox #

Each hypervisor’s web UI and API (:8006) serve a certificate signed by the Substrate CA on the control node, installed by deevnet.builder.proxmox_node_cert with Proxmox’s own command:

pvenode cert set <chain.pem> <key.pem> --force 1 --restart 1

pvenode(1) writes /etc/pve/local/pveproxy-ssl.pem and .key. It refuses a key that does not match, restoring the previous files, and never touches the node’s own pve-ssl.pem, which pveproxy falls back to when no custom certificate is present. --restart 1 restarts pveproxy, which serves the new certificate only after a restart. The node fingerprint changes with the certificate; nothing on the site pins it, and the nodes are not clustered.

Core Router

Mon, 01 Jan 0001 00:00:00 +0000

Core Router #

The core router’s web GUI and API (:443, one server) serve a certificate signed by the Substrate CA. Ansible signs it on the control node and imports it, with the Deevnet Root CA, the Site CA and the Substrate CA, over the documented Trust API (deevnet.net.opnsense_cert). OPNsense has no API for which certificate the web GUI serves, so choosing it is a manual step, made once.

Omada Controller

Mon, 01 Jan 0001 00:00:00 +0000

Omada Controller #

The controller’s web UI and Open API (:8043) and its portal (:8843) serve a certificate from the Substrate CA, through the container image’s own mechanism. Its entrypoint builds the keystore from two files in /cert on every start (mbentley/omada-controller README):

/cert/tls.crt: the full chain, root included;
/cert/tls.key.

The omada_controller role writes both into /opt/omada-controller/cert with site_cert, mounts the directory at /cert, and restarts the controller when they change. It is on where omada_tls_enabled: true (group_vars/network_controllers).

Trusting the Root

Mon, 01 Jan 0001 00:00:00 +0000

Trusting the Root #

Everything the site serves over TLS by name chains to one root, the Deevnet Root CA, deevnet-root-ca.pem. Trust it once on your computer and browsers and command-line tools verify the Deevnet API, the platform services, Proxmox, the core router and the Omada controller without flags or warnings.

Check its SHA-256 fingerprint before you trust it: F6:8A:BD:B3:1E:A5:6D:0A:88:1F:31:28:56:8A:4C:14:B0:3A:3F:5C:3F:38:CC:F1:7C:C4:F0:09:8B:EB:94:52.

curl -fsSLk -O https://downloads.mobile.deevnet.net/deevnet-root-ca.pem
openssl x509 -in deevnet-root-ca.pem -noout -fingerprint -sha256

The first fetch skips verification (-k) because you do not trust the root yet. The fingerprint check is what makes it safe.

Rotating

Mon, 01 Jan 0001 00:00:00 +0000

Rotating #

Certificates are renewed by Renewing. The CAs above them change only deliberately, in an offline ceremony, about a year before each expires ( Custody has the calendar and what each rotation reaches).

Rotate the Substrate CA with a new key and an Issuing CA ceremony #

Valid to 2031-10-04. Remove the ADR-0031 Substrate CA key block from the vault and the old request, then:

cd ansible-collection-deevnet.mgmt
ansible-playbook playbooks/substrate-ca.yml # a new key into the vault, its request to pki/mobile/
cd ../ansible-inventory-deevnet && make vault # then commit and push: the key exists nowhere else

Take the request through the Issuing CA ceremony, accept the certificate into pki/mobile/, then certs.yml. Certificates under the old Substrate CA keep chaining to the root, so clients trust both until each is reissued; ordinary renewal moves them all within the year.

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Troubleshooting #

Set R=/srv/dvnt/ansible-inventory-deevnet/pki/deevnet-root-ca.pem first.

Check the certificate is the site’s and chains to the root #

openssl s_client -connect <host>:<port> -servername <name> -showcerts -CAfile $R -no-CApath -no-CAstore </dev/null 2>&1 \
 | grep -E '^ *[0-9] s:|Verify return code'

The leaf alone, or the leaf and one CA: the service is not sending the whole chain (leaf, Substrate CA, Site CA). Clients that hold only the root cannot build it.
unable to get local issuer certificate: a certificate from something other than the Deevnet Root CA. It is either the device’s own, or one issued before a re-root. Re-run its role or certs.yml; site_cert and proxmox_node_cert reissue anything that does not chain.
Verify return code: 0 but the client still refuses: check the name. Add -verify_hostname <name>, or -verify_ip <address>, for exactly what the client dials. Every name and address comes from the inventory (site_cert_dns_names, site_cert_ip_sans); a CNAME missing from env.interfaces.<if>.dns.cnames is a name the certificate does not carry.

openssl x509 -noout -ext subjectAltName <<<"$(openssl s_client -connect <host>:<port> </dev/null 2>/dev/null)"

A check passes that should fail: OpenSSL also read the system store #

openssl verify -CAfile <root> and openssl s_client -CAfile <root> also load the host’s own trust store unless told not to. A certificate under any root that store trusts, a public CA or an outgoing Deevnet root, then verifies. Add -no-CApath -no-CAstore whenever the question is “does this chain to this root”; every check in the roles and the ceremony tools does since CHG-0033. curl --cacert and Python’s cafile use only the file given.