Custody #
The Root CA’s and every Site CA’s keys exist only as passphrase-encrypted files on two encrypted key drives, kept apart. Both passphrases, the drives’ and the key files’, are kept offline, under the holder’s own control, and never with the key media.
Two key media #
The keys live on two key media, kept apart:
- Primary key media: in the holder’s secure storage.
- Backup key media: somewhere else, so one loss (fire, theft, a bag left behind) cannot take both.
- The paper record: fingerprints and dates of every certificate the ceremonies made. It is how you recognize the right file, and how anyone checks a certificate they were handed.
Neither key media is ever plugged into a networked machine.
Opening a key drive needs only the drive and both passphrases, on any Linux machine with
cryptsetup: the pi-pki image on any Pi, or a Fedora live USB. Nothing ties a drive to the Pi
or microSD that made it (
Root and Site CA Ceremony).
Yearly check #
Every year, check that both copies still open. On the offline machine ( Root and Site CA Ceremony):
sudo deevnet-pki-media.sh keys open primary # the drive's passphrase
openssl pkey -in /mnt/keys/deevnet-root-ca.key -noout
openssl pkey -in /mnt/keys/deevnet-mobile-site-ca.key -noout # and each other site's
sudo deevnet-pki-media.sh keys close
# then the same with: keys open backup
Each openssl pkey asks for the key’s passphrase and prints nothing. Note the date in the paper record. A copy that does
not open is replaced from the other at once, by copying the file, never by generating a new key.
Rotation calendar #
Each CA is rotated a year before it expires. At the yearly check, also check the calendar:
| CA | Its key is held by | Signs | Valid for | Rotate it |
|---|---|---|---|---|
| Deevnet Root CA | the operator, offline | Site CAs | twenty years | 19 years after it was made |
| Site CA | the operator, offline | its site’s issuing CAs | ten years | 9 years after it was made |
| Substrate CA | site automation | every substrate server certificate | five years | 4 years after it was made |
| Tenant Device CA | the site’s secret store | tenant devices’ client certificates | five years | 4 years after it was made |
| Tenant Service CA | the site’s secret store | server certificates for tenants’ own services | five years | 4 years after it was made |
Rotating a CA means making a new one with the same ceremony:
- Root and Site CA Ceremony, path 2, for a Site CA;
- Issuing CA for an issuing CA.
What a rotation reaches #
A rotation reaches everything below the CA rotated, and nothing above it. A certificate works only while every CA above it is still valid. When a CA expires, everything it signed stops working with it, however long those certificates had left. A rotation therefore has to move everything below the rotated CA onto the new one before the old one expires. The year’s head start is for that.
| Rotating | Made again | Reissued | Handed out | Who notices |
|---|---|---|---|---|
| Deevnet Root CA | the root, every site’s Site CA, and every site’s issuing CAs | every certificate at every site | the new root, to every trust store that holds the old one: substrate hosts, operator computers, tenants’ computers and Pis, device firmware | everyone: each trust store must take the new root before the old one expires |
| Site CA | that site’s Site CA and all three of its issuing CAs | every certificate at that site | nothing: the root is unchanged | only that site’s services, as automation reinstalls their chains |
| Substrate CA | that site’s Substrate CA | that site’s substrate server certificates | nothing | no one: the certificates roll over at their yearly renewal |
| Tenant Device CA | that site’s Tenant Device CA | that site’s tenant device certificates | nothing | no one: devices pick up new certificates at their yearly renewal |
| Tenant Service CA | that site’s Tenant Service CA | that site’s tenant service certificates | nothing | no one: each tenant’s certificate moves to the new CA when the tenant next renews it, within the year |
A Deevnet Root CA rotation runs the old and new roots side by side.
- Make the new root.
- Make the Site CAs and issuing CAs under it, site by site.
- Install the new root beside the old one in every trust store.
- Reissue every certificate under the new chain.
- Remove the old root only once nothing uses it.
Treat it as a planned change of its own.
A Site CA rotation reaches only its own site.
- Make the new Site CA.
- Sign new Substrate, Tenant Device and Tenant Service CAs under it.
- Reissue every certificate at the site before the old Site CA expires.
Clients trust the root, not the Site CA, so nothing is handed out. Other sites are untouched.
An issuing CA rotation is absorbed by ordinary renewal. Certificates last a year and are reissued once fewer than 60 days remain ( Certificates standard 4.7). Once automation issues from the new CA, every certificate moves to it within the year, before the old CA expires. The old and new CAs both chain to the same Site CA and root, so clients trust both while they overlap.
These are planned rotations, with the old key still safe. A key that may have been exposed is replaced at once, without waiting for the calendar.
Lost key media #
A lost key media exposes nothing while the passphrases are safe, because the drive is encrypted and so is each key file on it. Copy the surviving media onto a new drive, and note it in the paper record.
Exposed key #
An exposed key is replaced, never reused, and only a re-root ends its trust. A key is exposed when the file and its passphrase may both have been seen, or, for the online issuing CAs, when the system holding it may have been compromised.
Deevnet publishes no revocation list, so a CA certificate stays valid until it expires. Making a new CA stops the site using the old one. It does not stop a thief using it: anything signed with the stolen key still chains to the trusted root. The Site CA’s name constraints bound the damage to Deevnet’s own names and private addresses; a public site cannot be impersonated.
| Exposed | It stays valid for | Do now | Then decide |
|---|---|---|---|
| Substrate CA (online, in the vault) | up to 5 years | sign a new one (
Issuing CA), rotate the vault password, reissue everything (certs.yml) | re-root, or accept the risk until it expires |
| Tenant Device CA (online, in OpenBao) | up to 5 years | sign a new one, re-enroll every device | re-root, or accept the risk until it expires |
| Tenant Service CA (online, in OpenBao) | up to 5 years | sign a new one, and have every tenant renew its certificates. A thief can certify any name under deevnet.net, a substrate name included, and every client would accept it | re-root, or accept the risk until it expires |
| Site CA (offline) | up to 10 years | new Site CA and new issuing CAs ( ceremony, path 2) | re-root, or accept the risk until it expires |
| Root CA (offline) | up to 20 years | re-root | — |
Re-root when the key may be in someone’s hands who could use it on the site’s networks. Someone in a position to intercept traffic on the site, with a stolen key, can impersonate any Deevnet service to anything that trusts the root. That is worth the re-root: a new root and Site CAs ( ceremony, path 1), new issuing CAs, and the new root handed to every machine, tenant and computer, as in a root rotation.
Accepting the risk is reasonable when the exposure is only possible. A key media misplaced and found again, a passphrase glimpsed, a vault file copied with no sign it was opened: replace the CA, note the decision and the date in the paper record and the incident record, and let the old certificate run out.
Until a re-root, a computer can refuse the exposed CA by hand. Windows (the Untrusted Certificates store) and macOS (Never Trust in Keychain Access) can distrust a single CA certificate. Site automation has no equivalent today.
Record every exposure as an incident.
Both copies lost #
Losing both copies of a key stops new signing at that level. Certificates already issued keep working until they expire, but nothing new can be signed above the issuing CAs. For a Site CA, make a new one. For the root, it is the re-root above.