Tenant Fabric — Proxmox SDN Implementation #

The implementation of the tenant network model on the tenant hypervisor. The why and the options considered are recorded in ADR-0001: Tenant Network Fabric; this page is the how and the concrete technology choices.

Overview #

The tenant network is a routed overlay built with Proxmox SDN in an EVPN zone. Each tenant is a VRF-isolated virtual network with an anycast gateway hosted by the fabric. The fabric is self-contained per hypervisor: dv02hyp002p02 runs its own SDN control plane, entirely local to the node, and is not clustered with the management hypervisor.

Not chosen: VLAN-aware bridge and plain (non-EVPN) VXLAN. Both are a different paradigm with no distributed control plane, and adopting either would have to be torn out to reach a cluster. See ADR-0001, “The starter trap.”

Self-contained per hypervisor #

Proxmox SDN configuration normally lives in the cluster filesystem and is replicated cluster-wide. On a standalone node it is local to that node. dv02hyp002p02 is standalone, so its SDN fabric is its own island:

• dv02hyp002p02 runs its own SDN controller (FRR) and its own zones, VNets, and VRFs.
• A tenant lives on dv02hyp002p02 and its overlay is dv02hyp002p02’s — nothing spans to the management hypervisor.
• Rebuilding dv02hyp002p02 reconstitutes its entire fabric from code, with no peer to reconcile against.

This preserves the non-clustered, stateless, plane-separated design already established for the hypervisors, and matches the goal that each tenant’s IaC/CaC rebuilds it whole against the substrate.

Build requirements (do not violate — these keep the cluster door open) #

From ADR-0001. Each is a hard requirement because violating it reintroduces lock-in:

Perimeter transit to the core router #

The fabric hands off to the core router over a single transit VLAN per hypervisor (Phase 1 choice in ADR-0001):

• Aggregate tenant egress leaves the fabric on the transit network.
• The core router sees only the transit network — never individual tenant subnets — and provides NAT, internet egress, and tenant↔management policy there.
• Per-tenant north-south isolation is enforced inside the fabric (VRFs), not by the core router seeing each tenant.

The switch port for the tenant hypervisor therefore needs the transit VLAN and the underlay — it does not need a VLAN per tenant. New tenants require no switch change.

A future option (ADR-0001, Seam 1) is a per-VRF exit with separate transit segments per tenant, if the core router ever needs to enforce per-tenant egress policy. Not needed for Phase 1.

Provisioning with Terraform #

Tenant lifecycle is managed with the bpg/proxmox Terraform provider (the API token terraform-prov@pve exists for this). Terraform owns both the SDN objects and the VMs, so the entire tenant — network and workloads — is declarative code.

Per-tenant flow:

1. Define the tenant’s SDN objects (VRF, VNet(s), subnet, IPAM) — globally-unique numbering.
2. Clone the Packer-built Fedora template into the tenant’s VNet.
3. cloud-init applies host config and the address itself — derived from the tenant index, not leased (Proxmox has no DHCP on EVPN zones).
4. Publish the tenant’s DNS records into the substrate zone (tenant-owned).

A “tenant” becomes a small reusable Terraform module: VRF + VNet(s) + N VMs from template + DNS records, landing on the fabric the hypervisor already provides.

Trajectory: single-member fabric → cluster #

The fabric is built with exactly one member today and expands without redefinition. What changes and what stays identical is tabulated in ADR-0001 → Trajectory. The short version: the SDN objects, the VRF-per-tenant model, the anycast gateway semantics, and the tenant IaC are unchanged when members are added; only the underlay gains peers and the cluster gains a QDevice for quorum. Nothing structural is torn out to scale.

The management hypervisor (dv02hyp001p01) is on a separate path — it may form its own cluster for the management plane, independently, and does not join the tenant fabric.

Concrete allocation #

Numbering follows ADR-0002. On dv02hyp002p02:

How egress actually works #

Tenant subnets carry SNAT at the exit node. Traffic leaving a tenant is translated to the exit node’s transit address before it reaches the core router, which is what makes ADR-0001’s promise literal: the perimeter never learns tenant address space, it only ever sees 10.20.50.0/24.

For that to hold, the hypervisor’s default route must be on the transit interface rather than management — otherwise tenant egress would ride the management segment. Management stays reachable from any VLAN through source-based routing on the node: replies sourced from the management address go back out the management interface, while forwarded tenant traffic still takes the default route out transit.

Status #

Phase 1 is built and a first tenant has working egress.

• ✅ Transit and underlay VLANs exist on the switch and the perimeter; the tenant hypervisor’s port is a trunk carrying both plus management.
• ✅ dv02hyp002p02’s bridge is VLAN-aware with vmbr0.50 and vmbr0.51 up, driven from inventory by the proxmox_node_network Ansible role.
• ✅ The fabric, VTEP identity and EVPN controller are applied on dv02hyp002p02 from deevnet-tenant-fabric. That repository holds hypervisor readiness only: the underlay, this node’s VTEP identity and the EVPN controller. A tenant’s own zone, VNets, subnet and workloads are built by the Deevnet API, declared from the tenant’s own repository through the deevnet/deevnet provider ( ADR-0015, proposed; until it is deployed, tenants still apply the module by tag, ADR-0006).
• ✅ Default route moved onto transit.
• ✅ First tenant end to end — tdemo (index 1, 10.20.129.0/24), addressed by cloud-init, with internet egress through the perimeter, its names resolving through the substrate resolver, and its own repository.

How egress is enforced #

Two node-local settings the Ansible role owns, because Proxmox models neither ( ADR-0003):

• Forwarding on the transit interface. Proxmox sets ip-forward on only for the interfaces its SDN config owns, and the transit interface is node substrate. Without it, tenant traffic leaves and is answered but the replies are never forwarded back in — 100% loss in the VM while the SNAT counter climbs.
• A default route inside each tenant VRF, merged through /etc/frr/frr.conf.local. Proxmox’s own exit-node behaviour lets a VRF lookup fall through to the node’s main table, which reaches the management segment on-link and unSNATed. The explicit default is what makes every non-connected destination leave via the perimeter.