Deevnet FileSync

Build Tooling

Mon, 01 Jan 0001 00:00:00 +0000

Build Tooling #

Status: Tracking list. Development so far happens on the Deevnet build server. This page records every tool added there beyond what the Ansible-built builder provides, so that it can later become a filesync builder role, or be handled in another way.

Tool	Version installed	Installed how	Needed for	Since
`rust`, `cargo`	1.99.0 (Fedora 43 packages)	`dnf install rust cargo`	Building the Rust workspace	Prototype
`clippy`, `rustfmt`	1.99.0 (Fedora 43 packages)	`dnf install clippy rustfmt`	Linting and formatting	Prototype
MinIO container image	`quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z`	Already present in local podman storage; public pulls of MinIO images now fail	A disposable local S3 endpoint for end-to-end tests	Prototype
`hugo` (extended)	0.140.1	Already present on the build server	Building the docs site	Docs

Rust crates are fetched by Cargo from crates.io at build time and pinned in Cargo.lock. They are project dependencies, not host tooling, so they are not listed here.

Linux

Mon, 01 Jan 0001 00:00:00 +0000

Linux Client #

Status: Draft. Phase: 1–3 (MVP). Linux is the first platform and the reference platform for the sync engine. Common client requirements are on the Clients page.

Distributions and filesystems #

LNX-1 (MVP, proposed): Fedora (the current release) is the primary, tested distribution. Other distributions are expected to work but are not tested in the MVP.
LNX-2 (MVP): Roots on ext4, btrfs, and xfs are supported.
LNX-3 (MVP, proposed): Roots on network filesystems (NFS, SMB) are allowed, but get rescan only: inotify does not report changes made by other machines. The client says so when the root is mounted.
LNX-4 (MVP, proposed): Works with SELinux in enforcing mode, without policy changes.

Agent #

LNX-5 (MVP): The background agent runs as a systemd user service, starting at login.
LNX-6 (MVP, proposed): Optional “keep running while logged out”, via systemd lingering, for always-on machines.

Change detection #

LNX-7 (MVP): inotify for live changes, plus periodic full rescans as a safety net (PERF-1).
LNX-8 (MVP, proposed): When the inotify watch limit (max_user_watches) is too low for a root, the client says so, explains the fix, and falls back to rescans. It never silently misses changes.

Keys and secrets #

LNX-9 (MVP, proposed): The device private key is protected by one of: the Secret Service (GNOME Keyring / KWallet), a TPM 2.0, or a passphrase-encrypted key file for headless machines.

Packaging #

LNX-10 (MVP, proposed): Distributed as an RPM for Fedora, plus a standalone binary tarball.
LNX-11 (Non-goal, proposed): Flatpak and Snap. Their sandboxes restrict filesystem access, which conflicts with “any folder can be a root”.

Desktop integration (Phase 4) #

LNX-12 (Phase 4): The tray app uses the StatusNotifierItem standard. On GNOME this needs the AppIndicator extension, which should be documented.
LNX-13 (Phase 4): Notifications use the freedesktop notification standard.

Open Questions #

Other distributions: is Fedora-only testing right for the MVP, or does Debian/Ubuntu also matter early?
Headless/server machines: should a machine without a desktop session (e.g. a Deevnet VM) be a supported device?

Metadata Matrix

Mon, 01 Jan 0001 00:00:00 +0000

Metadata Matrix #

Status: Decided (2026-10-15). This page backs requirements META-1 to META-7.

A file is more than its bytes. Each platform keeps a different set of names, timestamps, permissions, and extra data alongside the bytes. This page sets out what each platform can represent, and records, for each kind of metadata, whether FileSync:

syncs it,
maps it to an approximation on platforms that can’t represent it exactly,
keeps it local-only (it never leaves the device), or
skips the file (refuses to sync it, and tells the user).

The hub itself is not a constraint. Metadata is stored inside the encrypted file records, so the hub can carry whatever we decide. The limits come from the receiving platform.

Prototype

Mon, 01 Jan 0001 00:00:00 +0000

Prototype #

Status: Working (2026-10-15). This is the Phase P prototype: the filesync CLI and the filesync-core library, in crates/.

What it does #

Encrypted hub on any S3-compatible store (tested against MinIO), or on a local directory.
Roots, mounted at any path, with a different path on each device.
status, sync (push now), and pull (pull now), plus run --interval N for unattended polling.
Three-way sync: changes flow both ways, conflicting edits keep both versions as a conflict copy, and an edit beats a delete.
Receive-only mounts (mount --receive-only).
Portable-name checks (Windows-invalid names and case/normalization collisions are skipped, with a warning), symlinks skipped, a per-file size limit (100 MiB by default).
Version history (history) and restore of a whole root or one path from any snapshot (restore).
Refuses to sync a root whose folder is missing, or to propagate “everything deleted” without --allow-mass-delete.

What it deliberately does not do (yet) #

Shortcut	Final design
One shared hub key, copied to each device by hand	PKI device identity, enrollment, revocation (SEC-7, SEC-9..12)
Whole-file blobs, no chunking	Content-defined chunking (PERF-3)
Blob sizes visible to the hub	Padding or chunking to hide sizes (SEC-2)
Every snapshot lists every path	Incremental snapshots or a compacted index
No garbage collection or retention	VER-2, VER-6, HUB-7
Polling only (`run --interval`)	Filesystem watching plus rescans (LNX-7)
One static S3 credential, with delete rights	Per-device, least-privilege credentials (HUB-6, HUB-7)

Build and test #

cargo build --release # target/release/filesync
cargo test # unit tests, plus two-device tests on a directory hub

Try it with two “devices” on one machine #

Each device is a config directory plus a state directory. A directory hub needs no credentials:

Windows

Mon, 01 Jan 0001 00:00:00 +0000

Windows Client #

Status: Draft. Phase: 4. Common client requirements are on the Clients page.

Versions and filesystems #

WIN-1 (Phase 4, proposed): Windows 11. Windows 10 reached end of support in October 2025.
WIN-2 (Phase 4): Roots on NTFS. ReFS, FAT32, and exFAT are not supported for roots.

Agent #

WIN-3 (Phase 4, proposed): The agent runs per user, started at sign-in, not as a Windows Service. A service runs as a different account, which complicates access to the user’s files and the user’s key store.

Change detection #

WIN-4 (Phase 4): ReadDirectoryChangesW for live changes, plus periodic rescans. The NTFS change journal (USN) is not used, because it needs administrator rights.

File system behavior #

WIN-5 (Phase 4): Long paths (over 260 characters) are handled whether or not the system’s long-path setting is enabled.
WIN-6 (Phase 4): Locked files. Office and other applications hold files open with exclusive locks. The client reads with shared access, retries files it cannot read, and delays applying an incoming change to a locked file. It never fails the whole sync because of one locked file.
WIN-7 (Phase 4): Incoming changes are written to a temporary file and swapped into place atomically, so applications never see a half-written file.
WIN-8 (Phase 4): Names follow the Metadata Matrix name policy: reserved names, forbidden characters, trailing dots and spaces, and case-only collisions.
WIN-9 (Phase 4, proposed): Tolerates antivirus scanning (Microsoft Defender) briefly locking newly written files.

Overlap with OneDrive #

WIN-10 (Phase 4): Windows 11 often redirects Documents, Desktop, and Pictures into OneDrive (“Known Folder Move”). The client detects a root inside a OneDrive-managed folder and warns (ROOT-8).

Keys and secrets #

WIN-11 (Phase 4, proposed): The device private key is held by the TPM (through the Windows platform crypto provider) where available, or else protected with DPAPI.

Packaging and UI #

WIN-12 (Phase 4, proposed): A per-user installer that does not need administrator rights.
WIN-13 (Phase 4): A notification-area (tray) icon and Windows toast notifications (UI-1).

Open Questions #

Code signing: unsigned binaries trigger SmartScreen warnings. Is a code-signing certificate worth buying for a personal tool, or are the warnings acceptable?

Cloud Hub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Hub #

Status: Draft. Requirements for the storage side of the system: the hub that every device syncs through. Core requirements are on the Requirements page.

The hub is storage only. It holds encrypted objects and runs no FileSync code. All intelligence (sync decisions, encryption, conflict handling) lives in the clients. That keeps the provider replaceable and keeps it from ever needing to understand the data.

Backend model #

BE-0 (MVP): Only one backend is active at a time. That one backend is the hub.
BE-1 (MVP): AWS S3 is the MVP backend.
BE-2 (MVP): The backend interface is narrow and primitive: put, get, list, delete, and a conditional write (compare-and-swap) for coordination. New backends implement only that interface.
BE-4 (MVP): The sync engine, crypto, and hub format do not depend on anything AWS-specific. Anything AWS-specific (authentication, request signing, provider quirks) stays inside the S3 backend adapter.
BE-3 (Later): Self-hosted storage, e.g. MinIO on Deevnet infrastructure, SFTP, or a local or NAS path.
BE-5 (Later): Other S3-compatible providers (Backblaze B2, Cloudflare R2, Wasabi), and consumer cloud drives (Google Drive, OneDrive, Dropbox) used as dumb storage.
BE-6 (Later): A data migration process moves the hub from one backend to another (e.g. AWS S3 → self-hosted) without re-uploading from every device.
BE-7 (Non-goal): Mirroring to multiple backends at the same time.

What the hub holds #

HUB-1 (MVP): Everything stored in the hub is encrypted before it leaves a device. Object names are opaque and reveal nothing about file names, paths, or roots (SEC-1, SEC-2).
HUB-2 (MVP): The hub layout and object formats are a versioned, documented specification (IND-2). The hub records its format version, and a client refuses to write to a hub whose format version it does not understand.
HUB-3 (MVP, proposed): The hub stores no plaintext metadata about the user beyond what the provider inevitably sees: object count, object sizes, and access times and IP addresses.

Coordination #

HUB-4 (MVP): Devices coordinate safely through plain object storage using conditional writes. Two devices updating at the same time never silently lose one update. (AWS S3 supports conditional writes; support among other S3-compatible providers varies, which BE-5 has to check.)
HUB-5 (MVP, proposed): No server-side compute in the MVP: no Lambda functions, no event notifications, no provider-specific services. Change detection is client polling (SYNC-3).

Access control and protection #

HUB-6 (MVP, proposed): Every device has its own credential, scoped to only the FileSync bucket (or prefix). Revoking a device (SEC-5) also removes its storage access. A receive-only device (ROOT-9), such as an Android phone, gets a read-only credential.
HUB-7 (MVP, proposed): A compromised device cannot destroy the hub. Day-to-day device credentials can add data but cannot permanently delete or overwrite history. Permanent deletion (garbage collection, VER-6) uses a separate, more privileged credential. This protects against ransomware or a stolen laptop wiping out every version.
HUB-8 (MVP, proposed): Provider-side encryption at rest (e.g. SSE-S3) is enabled as defense in depth. It is in addition to FileSync’s end-to-end encryption, never instead of it.

Setup and operation #

HUB-9 (MVP, proposed): Setting up the hub (bucket, access policies, device credentials) is documented and scriptable. The CLI can generate the policy documents; clicking through a provider console is never required.
HUB-10 (MVP): The user chooses the provider account, region, and bucket. FileSync never creates or requires an account with anyone else.

Cost #

PERF-2 (MVP): Request counts and polling frequency are kept modest and configurable, because cloud object storage bills per request as well as per GB.
HUB-11 (MVP, proposed): The docs include a cost model: expected monthly cost for a typical workload (e.g. 20 GB, 3 devices, polling every few minutes).
HUB-12 (Later, proposed): The CLI reports hub usage (stored bytes, retained versions, trash), so that retention policies can be tuned against cost.

Open Questions #

Device credentials for AWS (HUB-6): short-lived credentials via AWS IAM Roles Anywhere (the device certificate from the FileSync CA is exchanged for temporary AWS credentials, with no long-lived keys on devices), or a scoped static access key per device? Roles Anywhere fits the PKI direction well, and is AWS-only, so it would live inside the S3 adapter.
Protection mechanism (HUB-7): S3 bucket versioning plus a no-delete policy for devices, S3 Object Lock, or both? What is the privileged credential for garbage collection: the offline CA, or one designated device?
Provisioning tool (HUB-9): CLI-generated policy documents applied by hand, or an infrastructure-as-code definition (Terraform or Ansible, matching Deevnet practice)?

macOS

Mon, 01 Jan 0001 00:00:00 +0000

macOS Client #

Status: Draft. Phase: 5. Common client requirements are on the Clients page.

Versions and hardware #

MAC-1 (Phase 5, proposed): The two most recent major macOS releases, on Apple Silicon. Intel Macs are not supported.
MAC-2 (Phase 5): Roots on APFS. Both the default case-insensitive variant and the case-sensitive variant are supported.

Agent #

MAC-3 (Phase 5): The agent runs as a per-user launchd agent, started at login.

Change detection #

MAC-4 (Phase 5): FSEvents for live changes, plus periodic rescans.

Privacy permissions #

MAC-5 (Phase 5): macOS requires the user’s explicit consent before an app can read Desktop, Documents, Downloads, iCloud Drive, or removable and network volumes. The client asks for access when a root is mounted in one of these places, explains why, and reports clearly when access is denied rather than treating the folder as empty. An empty-looking root must never be synced as “everything was deleted”.

Overlap with iCloud #

MAC-6 (Phase 5): With “Desktop & Documents Folders” in iCloud Drive, those folders are managed by iCloud, and “Optimize Mac Storage” can replace files with placeholders. The client detects a root inside iCloud-managed storage and warns (ROOT-8). It skips placeholders rather than triggering downloads.

File system behavior #

MAC-7 (Phase 5): Names follow the Metadata Matrix name policy. In particular, NFC and NFD forms of the same name are the same file on APFS.
MAC-8 (Phase 5): Package/bundle directories are synced as directories (see the Metadata Matrix).

Keys and secrets #

MAC-9 (Phase 5, proposed): The device private key is held in the Secure Enclave where the key algorithm allows it (P-256 only), or in the Keychain otherwise.

Distribution and UI #

MAC-10 (Phase 5): A menu bar app with status, push now, pull now, and notifications (UI-1).

Open Questions #

Signing and notarization: distributing a macOS app without Gatekeeper warnings needs a paid Apple Developer membership and Apple’s notarization service. That is a dependency on Apple. For a personal tool, is building locally (or approving an unsigned app once) acceptable instead?

Android

Mon, 01 Jan 0001 00:00:00 +0000

Android Client #

Status: Draft. Phase: 6. Common client requirements are on the Clients page.

The Android client is an app built on the same Rust core, called from Kotlin.

Role: read-only access #

AND-10 (Phase 6): Android is a read-only device (a receive-only device, ROOT-9). It downloads and presents files from the hub; files never originate on, or are changed from, Android.

This removes most of what makes Android hard: