Building Infrastructure #

Deevnet uses a two-phase model for substrate provisioning. These procedures apply whether building new infrastructure, recovering from failure, or replacing hardware.

The Two-Phase Model #

Phase 1: Online Preparation #

A builder node with internet access stages all required artifacts to the internal artifact server. This happens proactively—before any build or recovery.

Staged artifacts include: OS install trees, ISOs, container images, SSH keys.

Phase 2: Offline Build #

With artifacts pre-staged, the Builder can build the entire substrate without internet access. PXE boot pulls everything from local sources.

When These Procedures Apply #

In all cases, the process starts with seeding MAC addresses into inventory. Bare-metal MACs are read off the hardware and recorded; management-hypervisor VMs have no NIC to read until one is created, so their MACs are derived from the VMID instead — see Allocate VM Identity.

Stateless Substrate #

The substrate (Core Router, hypervisors, network infrastructure) is stateless. All configuration is defined in source control and applied via automation. No backup, restore, or data recovery is required for the substrate itself—just rebuild from scratch.

This means:

• No substrate snapshots or backups to maintain
• No state synchronization concerns
• Any host can be wiped and rebuilt at any time, and the Builder and management-hypervisor VMs are, once per Fedora release ( Resiliency)
• Hardware replacement is straightforward

Application tenants are different. Tenant workloads may have stateful data (databases, user files, etc.) that requires backup and recovery procedures. A tenant rebuilds itself from its own repository and state; see Tenant Operations.

Greenfield Build Sequence #

This is the site’s one build order; other pages and records link here rather than restate it. A complete build from scratch follows this sequence. Authority transitions and network segmentation are integrated steps — not separate procedures.

flowchart TD
    Z["0. Build the Builder
Fedora by hand, then its own roles"]:::manual
    A["1. Stage Artifacts
Fetch OS images, ISOs, SSH keys"]
    B["2. Seed Inventory
MAC addresses, host definitions"]
    C["3. Vault Operations
Decrypt secrets for automation"]
    D["4. Configure PXE
make bootstrap-auth"]:::transition
    E["5. Build Core Router
Manual OPNsense USB install"]:::manual
    F["6. Build Network
VLANs, firewall, DHCP, wireless
make core-auth"]:::transition
    G["7. Build Hypervisors
Manual Proxmox install, then Ansible"]:::manual
    H["8. Allocate VM Identity
VMID → MAC → DHCP reservation"]
    I["9. Build Management-Hypervisor VMs
Clone from template, or PXE netboot"]
    J["10. Verify Site
Network, DNS, DHCP, PXE validation"]
    K["11. Admit Tenants
Operator admits each tenant name"]
    L["12. Tenants Apply
Each tenant applies its own Terraform"]

    Z --> A --> B --> C --> D --> E --> F --> G --> H --> I --> J --> K --> L

    classDef default fill:#2d333b,stroke:#539bf5,color:#adbac7
    classDef transition fill:#1a3a1a,stroke:#57ab5a,color:#8ddb8c
    classDef manual fill:#3d1f00,stroke:#d29922,color:#e6c068

Legend:

flowchart LR
    T["Authority transition"]:::transition
    M["Manual step"]:::manual

    classDef transition fill:#1a3a1a,stroke:#57ab5a,color:#8ddb8c
    classDef manual fill:#3d1f00,stroke:#d29922,color:#e6c068

Within step 9, the management-plane VMs come up in the order deevnet.mgmt’s site.yml runs them: all VMs first, then OpenBao, PowerDNS, the state store, the Deevnet API, the Omada controller, the broker and log bridge, the log store, Grafana and the downloads site. The API’s play reads the messaging and observability VMs’ SSH host keys, so those VMs must exist before it runs. Certificates come from the Substrate CA through Ansible (certs.yml) and need nothing running ( ADR-0031).

Rebuild Tiers #

The site rebuilds in four tiers, each from the tiers below it. A failure in one tier is recovered by rebuilding it from the tiers beneath; what that takes is in Recovery.

OpenBao holds nothing that the vault cannot recreate. Both tenant CAs’ keys and the Transit key are the vault’s, imported on every build ( OpenBao’s Keys), and an initialization’s unseal share and Ansible login are locked into the vault before its output is deleted ( Vault Operations). It starts sealed; a build that configures it runs in a work session ( The unseal shares).

Some T2 services hold data nothing below them can recreate today:

• The Deevnet API’s database is the tenant registry, and the state store holds tenants’ Terraform state, which carries their device secrets. Both are on the provisioning VM’s OS disk, with no copy.
• The Omada controller holds tenants’ Wi-Fi keys, and the broker’s database their device accounts.

The 2026-10 Rebuild and Access review tracks closing each of these.

The manual floor #

These steps are done by a person on every full build, and the automation starts where they end:

Build Procedures #

Preparation #

• Build the Builder — Make the Ansible control node from a bare machine
• Stage Artifacts — Fetch artifacts from internet sources
• Seed Inventory — Define MAC addresses and host definitions
• Vault Operations — Decrypt secrets for automation
• Build-Time Secrets — How Packer and the fabric’s Terraform get the Proxmox token, and how to rotate it

Build #

• Configure PXE — Enter bootstrap-authoritative mode (make bootstrap-auth)
• Build Network — Core Router install, network segmentation, transition to core-authoritative (make core-auth)
• Build a Hypervisor — Install and configure the Proxmox hypervisors
• Allocate VM Identity — Derive management-VM MACs from their VMID before first boot
• Build a Management-Hypervisor VM — Put an OS on the VM, by template clone or PXE netboot

Validate #

• Verify Site — Validate site infrastructure
• Tenant Admission — Admit each tenant name; each tenant then applies its own Terraform ( Tenant Operations)

Reference #

• Authority Transition — Standalone reference for DNS/DHCP authority transitions
• Repave the Builder — Reinstall the hardware Builder from a temporary builder VM, over PXE, while the Builder is still there to help
• CHG-0001: Flat Network → VLANs — the VLAN, firewall and DHCP procedure, as recorded when the mobile site was segmented

Air-Gap Readiness #

Known Gaps #

Core Router - No automated install exists, but this is an accepted manual prerequisite for the MVP. A fresh OPNsense install from USB is performed before the automated build begins, same as factory-resetting the switch and AP. Day-2 configuration is fully automated via the deevnet.net Ansible collection. Future options (pre-imaged NVMe, alternative whitebox solutions) are tracked under Evaluations.

Hypervisors - Proxmox VE is installed by hand from the ISO staged on the artifact server. An unattended ISO with an embedded answer file exists in deevnet-image-factory but is unfinished and has not installed either node; it is on the Builder roadmap.

Post-Install Updates - See Patching for day 2 considerations.