Deevnet Infrastructure Platform

Builder

Mon, 01 Jan 0001 00:00:00 +0000

Builder #

Purpose #

The builder is responsible for provisioning and configuring all substrate infrastructure.

Every Deevnet Infrastructure Platform site needs a way to be created from scratch:

How do you provision infrastructure when no infrastructure exists yet?

The builder answers this by providing:

A self-contained provisioning role
All artifacts needed for deployment
Automation to configure every substrate component
Authority transition from bootstrap to production

Design Principles #

Self-Contained — The builder carries everything needed to stand up a substrate: IaC/CaC definitions, OS images, network boot infrastructure, and Git repositories.

Builder & Core Services

Mon, 01 Jan 0001 00:00:00 +0000

Builder & Core Services #

Builder infrastructure, network automation, and core services required to provision and rebuild the dvnt (home) site from bare metal.

The dvnt site is a permanent, always-on installation using the AOOSTAR N1 PRO as a dedicated provisioning node. It uses the same automation as dvntm with site-specific inventory (10.10.x.0/24 addressing).

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Achieve fully automated, repeatable provisioning of the dvnt (home) site from bare metal to running core services, reusing the automation built for dvntm with dvnt-specific inventory.

Builder & Core Services

Mon, 01 Jan 0001 00:00:00 +0000

Builder & Core Services ✅ #

Builder infrastructure, network automation, and core services required to provision and rebuild the dvntm (mobile) site from bare metal.

GitHub: https://github.com/deevnet
Documentation: https://deevnet.github.io/deevnet-docs/

✓ 30 | ↻ 0 | ⏳ 0 (30 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Achieve fully automated, repeatable provisioning of the dvntm (mobile) site from bare metal to running core services, with complete air-gap recovery capability.

CaribouLite SDR

Mon, 01 Jan 0001 00:00:00 +0000

CaribouLite SDR #

Purpose #

Software-defined radio (SDR) receiver for RF signal monitoring and experimentation. The CaribouLite HAT provides dual-channel SDR capability directly on the Pi’s GPIO header.

Hardware #

Component	Specification
Compute	Raspberry Pi 4 Model B, 8GB
SDR HAT	CaribouLite SDR
Storage	32GB SD card (minimum)
Antenna	SMA connector, antenna per use case

CaribouLite Specifications #

Attribute	Value
Frequency range	30 MHz – 6 GHz (with gaps)
Channels	2 (S1000: sub-1GHz, HiF: 30MHz-6GHz)
ADC	13-bit
Bandwidth	Up to 2.5 MHz per channel
Interface	GPIO header (directly mounted)

Network Position #

graph LR
 A[Core Router] <--> B[Access Switch
IoT VLAN] <--> C[CaribouLite SDR
Pi4 + SDR HAT]

Deployed on the IoT VLAN for isolation from management traffic.

Correctness

Mon, 01 Jan 0001 00:00:00 +0000

Deevnet Infrastructure Correctness #

Purpose #

This document defines what it means for Deevnet infrastructure to be correct.

Correctness is not defined by whether something “works once,” but whether it:

is deterministic,
is reproducible,
respects separation of concerns,
and remains understandable months later.

This document captures the intent, principles, and invariants of Deevnet infrastructure design.

1. Foundational Principles #

1.1 Determinism Over Convenience #

Infrastructure behavior MUST be deterministic.

Embedded Hardware

Mon, 01 Jan 0001 00:00:00 +0000

Embedded Hardware Workflow #

Template for hardware projects involving microcontrollers, custom PCBs, firmware, and physical enclosures. Follows the EVT/DVT/PVT hardware development lifecycle.

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Define what the project aims to achieve and establish boundaries.

In Scope

Core functionality and features
Target hardware platform
Key interfaces and protocols
Physical form factor requirements

Out of Scope

Features explicitly excluded
Integration points deferred to future phases
Capabilities beyond project goals

Milestone: Requirements & Constraints ⏳ #

Define what success means before building.

Networking

Mon, 01 Jan 0001 00:00:00 +0000

Substrate Networking Services #

The substrate networking layer provides foundational network services for each site. The core router serves as the segment router, firewall, and service gateway for all segments within a substrate.

For the segment model (nine segment types, trust hierarchy, and routing policy), see Network Segmentation.

Core Router Role #

Each substrate has a single core router that provides all networking services:

Function	Description
Segment routing	Inter-segment routing via VLAN interfaces
Firewall	Zone-based policy enforcement per segment
DNS	Authoritative for substrate zone, forwarding for external
DHCP	Static mappings for known hosts, dynamic pools per segment
NAT	Outbound gateway for all segments
Switching integration	VLAN trunking to access switch
Wireless integration	SSID-to-VLAN mapping via wireless AP

VLAN Routing #

The core router maintains one interface per segment, each on its own VLAN:

Networking

Mon, 01 Jan 0001 00:00:00 +0000

Tenant Networking #

Defines the network isolation model for tenant workloads.

Purpose #

Tenant networking provides:

Isolation — Tenants cannot see each other’s traffic
Controlled access — Explicit rules for shared services
Scalability — New tenants get dedicated network segments
Security boundaries — Limit blast radius of compromised workloads

VLAN Isolation Model #

Each tenant receives a dedicated VLAN:

Tenant	VLAN ID	Subnet	Purpose
`grooveiq`	100	10.100.0.0/24	IoT backend services
`vintronics`	101	10.101.0.0/24	Electronics projects
`moneyrouter`	102	10.102.0.0/24	Financial tracking

VLAN IDs and subnets are assigned from a reserved range to avoid conflicts with substrate segments (Management, Trusted, Storage, IoT, Guest).

Prerequisites & Preflight

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites & Preflight #

Vault #

All secrets are encrypted with Ansible Vault in the inventory. Decrypt before starting the migration and re-encrypt when done:

cd ansible-inventory-deevnet
make unvault # decrypt — run once before starting
# ... run migration steps ...
make vault # re-encrypt when migration is complete

Migration Artifact Capture #

Migration logs (preflight, each migration step, postcheck) are automatically captured in ansible-collection-deevnet.net/migration-logs/ with timestamps. Each make target produces a log file named YYYYMMDD-HHMMSS-<target>.log. No additional setup is required.

Stage Artifacts

Mon, 01 Jan 0001 00:00:00 +0000

Stage Artifacts #

The builder node (with internet access) stages artifacts to the artifact server before any recovery is needed.

What Gets Staged #

Artifact	Source	Role/Task
Fedora install tree	rsync from Fedora mirrors	`artifacts` role
Fedora Server ISO	download.fedoraproject.org	`artifacts` role
Proxmox VE ISO	enterprise.proxmox.com	`artifacts` role
SSH public keys	Generated locally	`artifacts` role
Container images	docker.io, etc.	`artifacts` role

Commands #

From builder node with internet:

Workstation Role

Mon, 01 Jan 0001 00:00:00 +0000

Workstation Role #

Purpose #

The workstation role configures a host as a developer/admin workstation with users, development tools, and infrastructure automation tooling.

Capabilities #

User Management #

Creates dev users with configurable primary/extra groups
Fetches SSH public keys from GitHub
Sets up home directories and shell preferences

Development Tools #

Category	Packages
Core	git, vim, tmux, golang
ISO/Image	Tools for working with disk images
X11	Display support for GUI tools

HashiCorp Tools #

Configures the HashiCorp RPM repository and installs:

Artifacts Role

Mon, 01 Jan 0001 00:00:00 +0000

Artifacts Role #

Purpose #

The artifacts server enables air-gapped provisioning for substrate hosts. Target machines fetch all installation artifacts from the local server—no internet connectivity required during provisioning.

Goals:

Air-gap capability — Substrate hosts install without upstream dependencies
Single source of truth — All provisioning artifacts in one location
Reproducibility — Known artifacts yield known outcomes

Current Capabilities #

The artifacts server provides:

Artifact Type	Description
Kickstart files	OS installation automation scripts
PXE boot artifacts	Kernel, initrd, boot configuration
Custom scripts	Post-install automation payloads
OS images	ISO images or extracted install trees

Artifacts are served via HTTP at artifacts.<site>.deevnet.net.

Authority Transition

Mon, 01 Jan 0001 00:00:00 +0000

Authority Transition Runbook #

Procedures for transitioning DNS/DHCP authority between the builder and production network infrastructure.

For the architectural model, see Core Services Architecture.

**Build context:** During a greenfield build, these transitions happen as part of the [Building Infrastructure](/docs/runbook/building-recovery/) sequence — [Configure PXE](/docs/runbook/building-recovery/build-sequence/) enters bootstrap-authoritative mode, and [Build Network](/docs/runbook/building-recovery/build-network/) transitions to core-authoritative mode. This page is the standalone reference for both directions.

Overview #

Transition	From	To	When
Promote to production	Builder-authoritative	Router-authoritative	After network infrastructure is configured and validated
Revert to bootstrap	Router-authoritative	Builder-authoritative	Before substrate rebuild or recovery

Both transitions are automated via playbooks in deevnet.builder and deevnet.net. The IP swap is the final step in each playbook and drops the SSH connection — reconnect at the new IP to verify.

Compute

Mon, 01 Jan 0001 00:00:00 +0000

Substrate Compute #

Defines the virtualization and compute model for Deevnet sites.

Overview #

Compute infrastructure provides virtualization hosts for management-plane and tenant workloads. Hypervisors run within the substrate and host:

Extended services — Observability, automation, and access tooling VMs
Tenant application VMs — Workloads deployed by tenants

Compute Hosts #

Each site includes one or more hypervisors that provide the virtualization layer. Compute hosts are provisioned through the builder and managed via the management plane.

Core Services

Mon, 01 Jan 0001 00:00:00 +0000

Core Services Architecture #

Purpose #

This document defines the foundational platform services of the management plane—DNS, DHCP, NAT, routing, firewall, and the naming/authority model that underpins all substrate operations.

These services must remain operational even if all extended services are lost.

For the builder that provisions these services, see Builder.

1. Core Services #

The Core Services provides foundational management services:

Service	Description
DNS	Authoritative for substrate zones, forwarding for external
DHCP	Static mappings for known hosts, dynamic pools
NAT	Outbound gateway for all segments
Routing	Inter-segment and gateway routing
Firewall	Inter-segment and egress rules

In production, these services are provided by dedicated network infrastructure. The Builder complements with provisioning, artifact hosting, and PXE/TFTP services.

Management

Mon, 01 Jan 0001 00:00:00 +0000

Tenant Management #

Defines the lifecycle and operational model for tenant workloads.

Purpose #

Tenant management provides:

Lifecycle control — Create, update, and destroy tenant environments
Observability — Logs, metrics, and alerting scoped to tenants
Access control — Who can manage which tenants
Operational clarity — Clear boundaries between tenants

Tenant Lifecycle #

Create #

Creating a new tenant involves:

Reserve VLAN and subnet — Allocate from tenant IP range
Configure network infrastructure — Add VLAN interface, DHCP scope, firewall zone
Provision tenant — Deploy VMs and DNS records via Terraform
Configure observability — Set up log/metric collection for tenant

Update #

Updating a tenant may include:

Naming

Mon, 01 Jan 0001 00:00:00 +0000

Deevnet Naming Standard #

Purpose #

This document defines the canonical naming conventions for the Deevnet ecosystem. The goal is to ensure that names are:

Deterministic (predictable, repeatable)
Scalable (works as the lab grows)
Readable (humans can understand intent quickly)
Environment-safe (dvnt vs dvntm is always explicit)
Service-oriented (service names are stable even if hosts move)

This naming standard applies to:

DNS zones and hostnames
Service endpoints (artifacts, PXE, DNS, etc.)
Tenants (logical workload namespaces)
Inventory naming (host identifiers)

1. Definitions #

Site #

A site is a self-contained infrastructure boundary that hosts systems and workloads.

Patch Automation

Mon, 01 Jan 0001 00:00:00 +0000

Patch Automation #

Firmware upgrades and automation improvements for the dvnt (home) site infrastructure.

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Patch Automation

Mon, 01 Jan 0001 00:00:00 +0000

Patch Automation #

Automated patching strategies, firmware upgrades, and automation improvements for substrate infrastructure components.

✓ 0 | ↻ 0 | ⏳ 16 (16 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Define and implement consistent patching strategies across all substrate components to maintain security posture while minimizing downtime. Includes firmware upgrades and automation improvements identified during the network build.

In Scope

Patching

Mon, 01 Jan 0001 00:00:00 +0000

Patching #

Day 2 maintenance and security updates for substrate hosts.

Status: Planned #

This section will document:

Online patching (hosts with internet access)
Offline patching (air-gapped site)
Local dnf mirror setup for full air-gap

Decision Required #

Post-install updates currently require internet access. Options:

Option	Pros	Cons
Accept internet required	Simple, no extra storage	Not true air-gap
Full local dnf mirror	True air-gap	~200GB per Fedora release
Hybrid (security only)	Balanced	Complex to maintain

Current State #

Install-time packages come from ISO/cdrom (air-gap ready).

SDR Platform

Mon, 01 Jan 0001 00:00:00 +0000

Pi-SDR Project #

Hardware adoption of the CaribouLite SDR HAT — software-defined radio on Raspberry Pi.

Part of deevnet-image-factory.

✓ 7 | ↻ 1 | ⏳ 1 (9 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Adopt the CaribouLite SDR HAT hardware and deploy a software-defined radio platform on Raspberry Pi for RF signal monitoring and experimentation.

Secure Identity

Mon, 01 Jan 0001 00:00:00 +0000

Deevnet Secure Identity Standard #

Purpose #

This document defines what it means for Deevnet client access to be secure.

Secure identity is not defined by whether you can “SSH in,” but whether access:

minimizes secret exposure,
is reproducible across devices and OSes,
supports least privilege and revocation,
and avoids habits that don’t scale beyond a lab.

This document captures the intent, principles, and invariants of secure client identity and access in Deevnet.

Seed Inventory

Mon, 01 Jan 0001 00:00:00 +0000

Seed Inventory #

Before any host can be PXE booted, its definition must exist in the Ansible inventory. MAC addresses, IP assignments, DNS records, and DHCP reservations are all driven from host_vars.

Repository: ansible-inventory-deevnet

When This Is Required #

Scenario	Action
Capacity expansion	Add host to `hosts.yml`, create new `host_vars/<hostname>.yml`
Hardware replacement	Update MAC address in existing `host_vars/<hostname>.yml`
Greenfield build	All hosts need both steps

Inventory Structure #

ansible-inventory-deevnet/
└── dvntm/
 ├── hosts.yml # Main inventory (hosts and group memberships)
 ├── group_vars/ # Variables by group
 └── host_vars/ # Per-host variables (MAC, IP, DNS, DHCP)
 ├── hv01.yml
 ├── hv02.yml
 └── ...

Adding a New Host (Expansion) #

1. Add to hosts.yml #

Add the hostname to appropriate groups:

Site IaC

Mon, 01 Jan 0001 00:00:00 +0000

Site IaC Workflow #

Template for full infrastructure site builds involving bare-metal provisioning, network automation, and end-to-end rebuild validation. Use this for complex, multi-layer infrastructure projects.

For simpler automation projects, see IaC/CaC.

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Define the infrastructure domain and automation goals.

In Scope

Target infrastructure components
Automation tooling
Provisioning and configuration scope
Operational boundaries

Out of Scope

VLAN Foundation

Mon, 01 Jan 0001 00:00:00 +0000

VLAN Foundation #

Create the VLAN infrastructure on the router and switch. All steps in this phase are non-disruptive — no existing traffic is affected.

Step 2: OPNsense VLAN Interfaces #

Create VLAN sub-interfaces on OPNsense. This step is non-disruptive — it only adds new interfaces without affecting existing traffic.

Run:

cd ansible-collection-deevnet.net
make migration-opnsense-vlans

Verify:

OPNsense GUI -> Interfaces -> Devices -> VLAN
Confirm 11 VLANs created on the correct parent interface
Each VLAN shows the correct tag (10, 20, 25, 30, 31, 35, 40, 50, 51, 52, 99)

Rollback: Delete VLAN interfaces via OPNsense GUI -> Interfaces -> Devices -> VLAN -> delete each entry.

Application Development

Mon, 01 Jan 0001 00:00:00 +0000

Application Development Workflow #

Template for software projects including services, tools, libraries, and applications. Covers the common phases of software development lifecycle.

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Define what the application aims to achieve and establish boundaries.

In Scope

Core functionality and features
Target platforms and environments
Key integrations and dependencies
User personas and use cases

Out of Scope

Authority Transition Gap Analysis

Mon, 01 Jan 0001 00:00:00 +0000

Authority Transition Gap Analysis #

Analysis of gaps between the authority transition runbook and the actual state of automation, performed 2026-03-26 against the segmented 10-space (10.20.x.x with per-VLAN subnets).

Scope: bootstrap-authoritative.yml, core-authoritative.yml, the bootstrap role, inventory group_vars, dnsmasq templates, and the OPNsense DNS/DHCP roles in deevnet.net.

Critical Gaps #

GAP 1: `bootstrap-authoritative.yml` doesn’t enable DHCP #

Runbook says: make bootstrap-auth enables DNS/DHCP/gateway on the builder.

Reality: The playbook calls include_role: bootstrap without overriding bootstrap_tftp_backend or bootstrap_dnsmasq_dhcp_enabled. Inventory has both set to their production values (tftpd / false), so running the playbook installs standalone tftpd with DHCP off.

Builder Cutover

Mon, 01 Jan 0001 00:00:00 +0000

Builder Cutover to Management VLAN #

Move the builder (provisioner-ph01) from the flat network to VLAN 99 with a static IP. This eliminates the DHCP dependency — the builder’s eth0 is configured with a static address before its port moves to the new VLAN. After this step, the builder has routed access to all VLANs for the rest of the migration.

Prerequisites:

Step 4 complete (trunk uplink carrying tagged VLANs)

5a — Assign and configure OPNsense VLAN interfaces #

Assign all VLAN devices to OPNsense interface slots and configure gateway IPs. The OPNsense API does not support interface assignment (GitHub #7324), so the playbook will pause and prompt you to complete a manual GUI step before continuing with automated IP configuration.

Building

Mon, 01 Jan 0001 00:00:00 +0000

Tenant Building #

Defines the provisioning model for tenant workloads.

Purpose #

Tenant building provides:

Declarative infrastructure — Define tenant environments as code
Reproducibility — Recreate tenant environments reliably
Drift detection — Identify manual changes
Lifecycle automation — Create, update, destroy via automation

Terraform-First Approach #

Unlike substrate infrastructure (automation-first), tenant workloads use Terraform:

Aspect	Substrate (Automation)	Tenant (Terraform)
Change frequency	Rare, deliberate	Frequent, agile
State model	Procedural, idempotent	Declarative, stateful
Drift detection	Manual verification	Built-in plan/apply
Lifecycle	Configure existing	Create/destroy
Use case	Infrastructure config	VM provisioning

Why Terraform for Tenants? #

Declarative definitions — Define what should exist, not how to create it
State tracking — Know exactly what’s deployed
Plan before apply — Preview changes before execution
Destroy support — Clean up tenant resources completely
Proxmox provider — Native Terraform support for VM lifecycle

Tenant Provisioning Workflow #

1. Define Tenant Infrastructure #

Create Terraform configuration for tenant VMs:

Core Services

Mon, 01 Jan 0001 00:00:00 +0000

Core Services Implementation #

How each site implements the DNS authority model defined in Core Services Architecture.

dvntm (Mobile Lab) #

Production Mode #

OPNsense Core Router is DNS/DHCP authority
Provisioner’s dnsmasq is disabled
Provisioner uses reserved IP at low end of management subnet (e.g., 192.168.10.95)

DNS records held in Core Router:

provisioner-01.mgmt.deevnet.net A 192.168.10.95
artifacts.mgmt.deevnet.net CNAME provisioner-01.mgmt.deevnet.net
pxe.mgmt.deevnet.net CNAME provisioner-01.mgmt.deevnet.net
tftp.mgmt.deevnet.net CNAME provisioner-01.mgmt.deevnet.net

Bootstrap Mode #

Provisioner’s dnsmasq is DNS/DHCP/gateway authority
Provisioner uses gateway IP (e.g., 192.168.10.1)
Core Router may not exist

DNS records held in provisioner dnsmasq:

Extended Management Plane

Mon, 01 Jan 0001 00:00:00 +0000

Extended Management Plane #

Extended management services for the dvnt (home) site — logging, telemetry, alerting, secrets, and identity.

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Extended Management Plane

Mon, 01 Jan 0001 00:00:00 +0000

Extended Management Plane #

Extended management services for the dvntm (mobile) site — logging, telemetry, alerting, secrets, and identity. Builds on the core substrate once it is operational.

GitHub: https://github.com/deevnet/ansible-collection-deevnet.mgmt
Documentation: https://deevnet.github.io/deevnet-docs/

✓ 0 | ↻ 0 | ⏳ 33 (33 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Deploy a unified management plane providing observability, security, and identity services for all substrate components, running as tenants on the Proxmox hypervisor.

Extended Services

Mon, 01 Jan 0001 00:00:00 +0000

Extended Services Architecture #

Purpose #

This document defines the extended management services layer within the Deevnet management plane.

Extended management services provide observability, automation, and access tooling. They are additive to Core Services—the minimal set of services required for the substrate to function on its own.

These services may be rebuilt entirely from the builder and core services. If the extended services tier is lost, core provisioning and network services remain operational.

IaC/CaC

Mon, 01 Jan 0001 00:00:00 +0000

IaC/CaC Workflow #

Template for Infrastructure as Code and Configuration as Code projects that automate a single system, appliance, or service. Use this for focused automation projects that don’t require full site orchestration.

For complex multi-layer infrastructure, see Site IaC.

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Define the automation target and goals.

In Scope

Target system or service
Automation tooling
Configuration scope
Deployment target

Out of Scope

Identity vs Intent

Mon, 01 Jan 0001 00:00:00 +0000

Identity vs Intent in Deevnet #

Purpose #

This document explains how identity and intent are deliberately separated in the Deevnet infrastructure model.

The goal of this separation is to ensure that:

Hosts can change purpose without renaming
Workloads can move without rewriting inventory
Configuration intent can be preserved, detached, and reused
Infrastructure history is not lost when systems are repurposed
Inventory remains truthful over time

This document complements the Naming Standard and focuses specifically on how inventory and configuration are modeled in Deevnet.

Network Segmentation

Mon, 01 Jan 0001 00:00:00 +0000

Network Segmentation #

Defines the network segmentation model for Deevnet sites.

Purpose #

Network segmentation divides each substrate into isolated broadcast domains with controlled routing between them. This provides:

Security boundaries — Limit blast radius when devices are compromised
Traffic isolation — Separate management, storage, and workload traffic
Operational clarity — Each segment has a defined purpose and trust level

Segment Model #

Each substrate implements nine segment types:

PiDP-11

Mon, 01 Jan 0001 00:00:00 +0000

PiDP-11 Project #

Hardware adoption of the PiDP-11 kit — a PDP-11 replica using simh emulation on Raspberry Pi.

Part of deevnet-image-factory.

✓ 1 | ↻ 1 | ⏳ 8 (10 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Build a functional PDP-11 replica by assembling the PiDP-11 kit and running simh emulation, capable of running period-accurate operating systems with an authentic front panel experience.

PXE Role

Mon, 01 Jan 0001 00:00:00 +0000

PXE Role #

Purpose #

The PXE boot infrastructure enables fully automated, zero-touch provisioning of bare-metal hosts. Hosts boot from the network, receive their OS installation automatically based on their MAC address, and require no human intervention.

Goals:

Zero-touch — MAC-specific configs eliminate boot menus and manual selection
UEFI-native — Modern UEFI boot with network-enabled GRUB
Decoupled services — DHCP (Core Router) and TFTP (bootstrap node) are separate
Air-gap capable — All boot artifacts served from local infrastructure

Use Cases #

Primary: Bare-Metal Provisioning #

PXE boot is the standard method for provisioning bare-metal hosts:

Storage

Mon, 01 Jan 0001 00:00:00 +0000

Substrate Storage #

Defines the shared and persistent storage model for Deevnet sites.

Status #

Storage is a planned future addition to core services. This document will be expanded as the storage architecture is defined.

Intent #

Shared storage will provide persistent volumes for substrate consumers — both management-plane services and tenant workloads — independent of any single compute host.

Stratum 1 NTP

Mon, 01 Jan 0001 00:00:00 +0000

Pi Stratum 1 NTP Server #

Raspberry Pi Zero-based Stratum 1 NTP server with GPS time source.

Part of deevnet-image-factory.

✓ 0 | ↻ 0 | ⏳ 9 (9 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Deploy a local Stratum 1 NTP server using GPS as the time source, providing accurate time synchronization for all substrate hosts independent of internet connectivity.

Vault Operations

Mon, 01 Jan 0001 00:00:00 +0000

Vault Operations #

Ansible Vault protects sensitive variables (passwords, API keys, certificates) stored in the inventory. Each environment has its own set of vault.yml files that must be encrypted at rest and decrypted only while editing.

Repository: ansible-inventory-deevnet

Setup #

After cloning the inventory repository, run the one-time hook setup:

cd ansible-inventory-deevnet
make install-hooks

This runs git config core.hooksPath hooks, pointing Git at the version-controlled hooks/ directory. The hooks stay in sync with the repo automatically — no copying required. This must be run once per clone.

Addressing

Mon, 01 Jan 0001 00:00:00 +0000

Addressing #

Defines the IP addressing convention for Deevnet sites.

Addressing Convention #

Each site is assigned a /16 block from the 10.0.0.0/8 RFC1918 space:

Site	Address Block
dvnt	10.10.0.0/16
dvntm	10.20.0.0/16

The addressing pattern is: 10.{site_id}.{vlan_id}.0/24

The second octet identifies the site
The third octet matches the VLAN ID for that segment
Each segment subnet is a /24 within the site’s /16

This creates a predictable, self-documenting address scheme where any IP immediately reveals which site and segment it belongs to.

Configure PXE

Mon, 01 Jan 0001 00:00:00 +0000

Configure PXE #

Configure PXE boot authority before provisioning hosts.

Greenfield Build (No Core Router) #

For initial site build or full recovery, the bootstrap node provides DNS/DHCP/TFTP for the management subnet.

cd ~/dvnt/ansible-collection-deevnet.builder
make bootstrap-auth

This:

Discovers the WAN interface from inventory (bootstrap_wan_interface_key)
Enables IP forwarding and masquerading on the WAN interface
Activates dnsmasq for DHCP/DNS/TFTP on the downstream (management) interface
Populates DNS host records and DHCP static reservations from inventory
Swaps the management interface IP from the reserved address to the gateway address

The IP swap is the last step — it drops the SSH connection. All configuration completes first while connectivity is stable. Reconnect at the gateway IP to verify.

ESP32 Ma Bell Bluetooth Gateway

Mon, 01 Jan 0001 00:00:00 +0000

Ma Bell Project #

Bluetooth Phone Gateway for vintage telephone integration.

Project implemented in a separate repository.

GitHub: https://github.com/cdeever/esp32-ma-bell-gateway
Documentation: https://cdeever.github.io/esp32-ma-bell-gateway/

✓ 19 | ↻ 7 | ⏳ 50 (76 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope 🔄 #

Restore full, authentic use of vintage analog telephones by bridging them to modern Bluetooth cell phones — preserving rotary dialing, ringing behavior, tones, and user experience.

Full Site Rebuild

Mon, 01 Jan 0001 00:00:00 +0000

Full Site Rebuild #

End-to-end rebuild of the dvnt (home) site from scratch. Validates that all dvnt automation is end-to-end and exercises the full build sequence.

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Full Site Rebuild

Mon, 01 Jan 0001 00:00:00 +0000

Full Site Rebuild #

End-to-end rebuild of the dvntm site from scratch. Validates air-gap recovery capability and exercises all automation built across the Builder, Patch Automation, and Extended Management Plane projects.

✓ 0 | ↻ 0 | ⏳ 6 (6 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Perform a complete tear-down and rebuild of the dvntm (mobile) site to validate that the infrastructure-as-code automation is truly end-to-end. This is the capstone event that proves the system works.

MAC Address Format

Mon, 01 Jan 0001 00:00:00 +0000

MAC Address Format Standard #

Rule #

All MAC addresses in Deevnet inventory and configuration MUST use:

lowercase hex digits (a-f, not A-F)
Colon separators (:, not -)

Examples #

Correct:

mac: "bc:24:11:2e:26:4e"
mac: "dc:a6:32:c3:b4:bc"

Incorrect:

mac: "BC:24:11:2E:26:4E" # uppercase
mac: "bc-24-11-2e-26-4e" # dashes

Rationale #

Deevnet uses lowercase to match Unix tooling conventions:

ifconfig, ip link, and most Linux utilities display MAC addresses in lowercase
Copy-paste from system output works without transformation
Consistent with what operators see when troubleshooting

IEEE 802 and RFC 7042 documentation examples use uppercase, but this is a documentation convention rather than a technical requirement. We prioritize practical alignment with the Unix ecosystem over formal documentation style.

Network Controller Role

Mon, 01 Jan 0001 00:00:00 +0000

Network Controller Role #

Purpose #

The network controller role deploys centralized management software for switches and access points as Podman containers managed by systemd.

Controllers #

Site	Controller	Managed Devices
dvntm	TP-Link Omada SDN	SG2218 switch, EAP650-Outdoor AP
dvnt	Ubiquiti UniFi Network	USW-24-G2, US-8 switches, UAP-AC-M APs

Both controllers run on the bootstrap node because they must be available for initial network configuration before VLANs exist.

Security

Mon, 01 Jan 0001 00:00:00 +0000

Security & Vulnerability Management #

Documents security posture, assumptions, and lifecycle practices.

Scope #

This section includes:

Trust boundaries and threat assumptions
Credential and key management philosophy
Vulnerability monitoring and response expectations
Patch and upgrade responsibility by layer
Security-related guardrails and invariants

This section defines what “secure enough” means for Deevnet.

Status: Planned #

Detailed security documentation is planned. Key areas to document:

Trust Boundaries #

Substrate network is trusted
Upstream/WAN is untrusted
Tenant isolation requirements

Credential Management #

SSH key distribution via artifact server
No passwords in playbooks or inventory
API tokens for service accounts

Vulnerability Response #

Monitoring sources (CVE feeds, vendor advisories)
Patch timelines by severity
Emergency response procedures

Services & Routing

Mon, 01 Jan 0001 00:00:00 +0000

Services & Routing #

Configure DHCP, interface IPs, firewall rules, and trunk PVID. After this phase, all VLANs are fully routed and served.

**Post-cutover inventory:** All `make` targets from this point forward automatically use the `dvntm-new` inventory (target IPs on the new VLAN subnets). The builder is on VLAN 99 and can only reach devices at their new addresses. No manual `-i` overrides are needed.

Step 6: Test Second Port #

Test a non-builder port on a different VLAN to validate the trunk + VLAN path end-to-end.

Inventory Lifecycle

Mon, 01 Jan 0001 00:00:00 +0000

Inventory & Lifecycle Management #

Documents how infrastructure assets are tracked, managed, and retired.

Scope #

This section includes:

Host identity and inventory sources of truth
Hardware lifecycle stages (active, standby, retired)
Image and configuration lifecycle expectations
Decommissioning and cleanup principles

This section ensures infrastructure ages intentionally, not accidentally.

Lifecycle Stages #

Stage	Description
Provisioning	Host being set up, not yet in service
Active	In production use
Standby	Available but not currently assigned
Maintenance	Temporarily offline for updates/repairs
Retired	Decommissioned, removed from inventory

Inventory Sources of Truth #

ansible-inventory-deevnet - Canonical host identity
OPNsense - Authoritative DNS/DHCP (production)
Bootstrap node - Authoritative DNS/DHCP (during provisioning)

Decommissioning #

When retiring a host:

MAC Namespace Specification

Mon, 01 Jan 0001 00:00:00 +0000

MAC Namespace Specification #

Purpose #

This document defines the deterministic MAC address strategy used across the Deevnet platform, with a primary focus on management-plane workloads.

Deterministic MAC addressing enables:

Stable DHCP reservations
Predictable IP assignments
Reproducible VM rebuilds
Clear mapping between identity layers

Formatting: All MAC addresses must follow the formatting rules in the MAC Address Format Standard (lowercase hex, colon separators).

Guiding Principles #

MAC addresses are generated outside the hypervisor
MACs are explicitly defined in inventory/code
Hypervisors must never auto-generate identity
Identity must survive VM destruction and recreation

Scope #

Workload Type	Policy
Management Plane	Mandatory deterministic MACs
Tenant Workloads	Optional, future-controlled
Ephemeral/Test VMs	May use auto-generated MACs

This specification applies primarily to the management plane.

Network Segmentation

Mon, 01 Jan 0001 00:00:00 +0000

Network Segmentation #

Defines mandatory requirements for network segmentation in Deevnet sites.

Purpose #

This standard establishes rules for how sites implement network segmentation. It complements the Substrate Networking architecture document, which describes the segment model and design rationale.

Definitions #

Term	Definition
Segment	An isolated broadcast domain (VLAN) with controlled routing to other segments
Trust boundary	The point where traffic policy changes between segments
Inter-segment traffic	Network communication that crosses segment boundaries
Segment authority	The system responsible for segment routing and policy (core router in production)

Segment Requirements #

1. Management Segment #

The management segment contains infrastructure management plane systems.

Port Migration & Wireless

Mon, 01 Jan 0001 00:00:00 +0000

Port Migration & Wireless #

Move remaining switch ports to their assigned VLANs, perform the management cutover, adopt devices in Omada, and configure AP SSIDs.

Step 10: Migrate Remaining Access Ports #

Move all remaining switch ports to their assigned VLANs as defined in host_vars/access-sw01.yml.

**DNS:** New 10.20.x.x addresses will not resolve via DNS until post-migration ([Step 11](#step-11-management-cutover) / [Post-Migration](../post-migration/)). This is expected — Ansible uses inventory IPs directly. Use IP addresses for any manual verification during this step.

**Wireless clients:** AP SSID-to-VLAN mappings are not reconfigured in this step. When the AP's port moves to its target VLAN, wireless clients may lose connectivity. SSID configuration is handled in [Step 13](#step-13-ap-ssid-configuration) after Omada adoption ([Step 12](#step-12-omada-device-adoption)).

Run:

SSL Cert Automation

Mon, 01 Jan 0001 00:00:00 +0000

SSL Cert Automation #

Automated SSL certificate provisioning and renewal for substrate services.

✓ 0 | ↻ 0 | ⏳ 16 (16 total)

Legend: ✅ Complete | 🔄 In Progress | ⏳ Planned

Project Vision & Scope #

Eliminate browser security warnings and enable secure communication between substrate services by deploying an internal Certificate Authority and automating certificate lifecycle management.

In Scope

Internal CA deployment and trust distribution
SSL certificates for infrastructure web UIs (Proxmox, OPNsense, Omada)
Automated certificate renewal
Expiration monitoring

Out of Scope

Change Management

Mon, 01 Jan 0001 00:00:00 +0000

Change Management & CI/CD #

Defines how change is introduced safely into the Deevnet ecosystem.

Scope #

This section includes:

Change classification (routine vs disruptive)
Required validation before changes are applied
Automated testing expectations by layer
CI/CD pipeline responsibilities
Guardrails that prevent unsafe changes from reaching production sites

Principles #

Automated testing and CI/CD exist to:

Validate assumptions early
Prevent regressions
Ensure changes preserve correctness

Manual changes without validation are considered defects.

Network Reference

Mon, 01 Jan 0001 00:00:00 +0000

Network Reference #

Quick reference for VLAN assignments and network configuration across Deevnet sites.

dvntm VLAN Assignments #

Segment	VLAN ID	Subnet	Gateway	DHCP
Trusted	10	10.20.10.0/24	10.20.10.1	.100-.200
Storage	20	10.20.20.0/24	10.20.20.1	Static only
Platform	25	10.20.25.0/24	10.20.25.1	Static only
IoT	30	10.20.30.0/24	10.20.30.1	.100-.200
IoT Vendor	31	10.20.31.0/24	10.20.31.1	.100-.200
IoT Backend	35	10.20.35.0/24	10.20.35.1	Static only
Guest	40	10.20.40.0/24	10.20.40.1	.50-.250
Tenant 1	50	10.20.50.0/24	10.20.50.1	Per-tenant
Tenant 2	51	10.20.51.0/24	10.20.51.1	Per-tenant
Tenant 3	52	10.20.52.0/24	10.20.52.1	Per-tenant
Management	99	10.20.99.0/24	10.20.99.1	Static only
Blackhole	999	—	—	None (unrouted)

dvnt VLAN Assignments #

Segment	VLAN ID	Subnet	Gateway	DHCP
Trusted	10	10.10.10.0/24	10.10.10.1	.100-.200
Storage	20	10.10.20.0/24	10.10.20.1	Static only
Platform	25	10.10.25.0/24	10.10.25.1	Static only
IoT	30	10.10.30.0/24	10.10.30.1	.100-.200
IoT Vendor	31	10.10.31.0/24	10.10.31.1	.100-.200
IoT Backend	35	10.10.35.0/24	10.10.35.1	Static only
Guest	40	10.10.40.0/24	10.10.40.1	.50-.250
Tenant 1	50	10.10.50.0/24	10.10.50.1	Per-tenant
Tenant 2	51	10.10.51.0/24	10.10.51.1	Per-tenant
Tenant 3	52	10.10.52.0/24	10.10.52.1	Per-tenant
Management	99	10.10.99.0/24	10.10.99.1	Static only
Blackhole	999	—	—	None (unrouted)

Segment Purpose Summary #

Segment	Trust Level	Purpose
Management	High	Infrastructure management plane (provisioners, hypervisor mgmt, switches, IPMI)
Trusted	High	User devices (workstations, laptops, personal devices)
Storage	High	Dedicated storage traffic (NAS, backup targets)
Platform	High	Shared infrastructure services (DNS, NTP, artifact mirrors, reverse proxy)
Tenant	Medium	Per-tenant workload isolation
IoT Backend	Medium	IoT application backends (MQTT, Home Assistant, data pipelines)
IoT Vendor	Very Low	Vendor-managed IoT containment zone (cloud-dependent, unauditable)
IoT	Medium	Custom-developed embedded devices with controlled firmware (Pis, sensors)
Guest	Untrusted	Transient visitor access (internet only)

Canonical Source #

VLAN definitions are maintained in Ansible inventory:

Post-Migration

Mon, 01 Jan 0001 00:00:00 +0000

Post-Migration #

After all steps complete and connectivity is verified:

Run automated post-migration validation:
```
cd ansible-collection-deevnet.net
make postcheck
```
Validates OPNsense VLANs, switch database/trunk, device reachability, gateway IPs, and builder state. All checks should show [PASS].
Run DNS and DHCP roles (must run before vault encryption):
```
cd ansible-collection-deevnet.net
make dns
make dhcp
```

Re-encrypt vault files:

cd ansible-inventory-deevnet
make vault

Remove old network config:
- Delete old 192.168.10.0/23 Kea DHCP subnet (if not already removed)
- Remove temp VLAN 99 DHCP pool (if not already removed)
- Remove old 192.168.10.0 LAN interface from OPNsense (Interfaces → LAN → clear IP or reassign)
- Remove any old static routes referencing 192.168.10.x
Ongoing switch management — use the switch target for day-2 operations:

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Troubleshooting & Known Issues #

Lost switch access after trunk configuration #

Connect via console cable
Check show interface switchport gigabitEthernet 1/0/1 for native VLAN mismatch
Revert to access mode on uplink if needed

Device not getting DHCP lease #

Kea not listening on VLAN interfaces: Check OPNsense GUI → Services → Kea DHCP → Settings → Interfaces. All VLAN interfaces must be selected. By default, Kea only listens on LAN (re0). The opnsense_dhcp role automates this, but verify with: ssh root@10.20.99.1 'cat /usr/local/etc/kea/kea-dhcp4.conf' and check the interfaces-config section.
Verify port VLAN assignment: show vlan brief
Verify DHCP subnet exists in OPNsense for that VLAN
Check OPNsense firewall rules allow DHCP on VLAN interface
Check show mac address-table to confirm device is on expected port

AP not discoverable or adoption fails in Omada #

Factory reset AP uses static fallback IP 192.168.0.254, not DHCP. Add temp IP on builder (sudo ip addr add 192.168.0.1/24 dev enp4s0) and access AP web UI at http://192.168.0.254 (admin/admin) to set inform URL.
Adoption timeout (errorCode -39002): AP can’t reach controller on required ports. Check ss -tlnp | grep 29814 — Omada must listen on TCP 29814 (not just UDP). Newer AP firmware (EAP650-Outdoor) requires TCP 29814 for v2 adoption.
Omada controller version mismatch: Controller 5.12.7 does not listen on TCP 29814. Update the controller to a version that supports v2 adoption protocol.
Firewalld missing ports: Verify sudo firewall-cmd --list-ports includes 29810-29814/udp AND 29811-29814/tcp.

Builder lost connectivity during Step 5 #

Verify ethernet cable is connected to gi1/0/16 — do not rely on WiFi for substrate access
Check port VLAN assignment: show interface switchport gigabitEthernet 1/0/16
Verify VLAN 99 interface is enabled with IP 10.20.99.1 in OPNsense
If the builder has the wrong static IP config, revert the port to VLAN 1 and re-run the builder playbook with the dvntm inventory
If the builder is unreachable, Omada adoption (Step 12) cannot proceed — but the switch and AP continue to function independently

Last resort: revert the builder port to VLAN 1 via console:

configure
interface gigabitEthernet 1/0/16
 switchport access vlan 1
end
copy running-config startup-config

Inter-VLAN routing not working #

Check default gateway on target device: Devices on VLAN 99 (management) need ip route 0.0.0.0 0.0.0.0 10.20.99.1 to route responses back to other VLANs. Without this, the device receives cross-VLAN traffic but replies are silently dropped (no return route). This was the root cause of the “builder can’t ping cross-VLAN gateways” issue — the switch had no default gateway.
Verify VLAN interfaces have IP addresses assigned in OPNsense
Check OPNsense firewall rules for inter-VLAN traffic
Verify routing table: OPNsense GUI -> System -> Routes

To Do #

Automation gaps and improvements identified during the initial migration run.

Build Network

Mon, 01 Jan 0001 00:00:00 +0000

Build Network #

Configure network infrastructure: Core Router, VLANs, firewall, DHCP, and wireless access points.

Collection: deevnet.net

Components #

Component	Role
Core Router	Firewall, DHCP, DNS, routing
Switch/VLANs	Network segmentation
Wireless AP	SSIDs, guest networks

Prerequisites #

Before the automated build-network procedures begin, the following manual steps must be completed:

Prerequisite	Method	Notes
Core Router	Fresh OPNsense install from USB	Manual installer; no PXE support
Access Switch	Factory reset to default state	Clears any prior VLAN/port config
Wireless AP	Factory reset to default state	Clears any prior SSID/network config

Additionally:

Build Management Plane

Mon, 01 Jan 0001 00:00:00 +0000

Build Management Plane #

PXE boot and install Proxmox VE hypervisors from local artifacts.

Prerequisites #

MAC addresses seeded in inventory
Network infrastructure running (or bootstrap-authoritative mode enabled)
Proxmox VE ISO staged on artifact server
DHCP reservations configured for target hosts

Procedure #

TBD

Verify Site

Mon, 01 Jan 0001 00:00:00 +0000

Verify Site #

Validation after site build is complete.

Overview #

Each build phase includes automated verification via Ansible. This page covers final validation once all components are operational.

Network Verification #

# Core Router reachable
ping gateway.dvntm.deevnet.net

# DNS resolution working
dig +short hv01.dvntm.deevnet.net
dig +short @192.168.10.1 hv01.dvntm.deevnet.net

# DHCP serving leases
# (check Core Router UI or API)

# VLAN connectivity
# (ping across segments as appropriate)

Management Plane Verification #

# Hypervisors reachable
ping hv01.dvntm.deevnet.net
ping hv02.dvntm.deevnet.net

# Proxmox API accessible
curl -k https://hv01.dvntm.deevnet.net:8006/api2/json/version

# SSH access working
ssh hv01.dvntm.deevnet.net hostname

PXE Infrastructure Verification #

# TFTP service running
systemctl status tftp.socket

# PXE configs present
ls /srv/tftp/pxelinux.cfg/

# Artifact server accessible
curl -I http://artifacts.dvntm.deevnet.net/fedora/43/mirror/

Automated Verification #

TBD - Ansible playbook for full substrate health check

Build Tenants

Mon, 01 Jan 0001 00:00:00 +0000

Build Tenants #

Provision tenant VMs and deploy application workloads.

Prerequisites #

Proxmox hypervisor(s) running
VM templates available
Application configuration in source control

Procedure #

TBD

Verify Tenants

Mon, 01 Jan 0001 00:00:00 +0000

Verify Tenants #

Validation after tenant workloads are provisioned.

Prerequisites #

Tenant VMs deployed via Build Tenants
Application services started

Application Verification #

TBD - Application-specific health checks

Backup Verification #

TBD - Verify tenant backup jobs configured and running