Unix Server Tech Knowledge Base

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Sneakernet: Production Air Gap Implementation ======

<WRAP center round tip 60%>
**Project:** Automated ZFS replication via encrypted removable media\\
**Environment:** FreeBSD with ZFS and GELI encryption\\
**Repository:** ''http://svn.dailydata.net/svn/zfs_utils/trunk''\\
**License:** BSD 2-Clause (FreeBSD License)
</WRAP>

===== Overview =====

The **sneakernet** project provides a complete automated solution for air gap server replication using ZFS datasets and encrypted transport media. This implementation was developed for a production environment requiring monthly off-site backups with split-key security architecture.

**Key Features:**
  * Automatic source/target mode detection
  * Three-drive rotation minimizing site visits
  * Multi-layer encryption (GELI + symmetric transport)
  * Split-key architecture preventing single-point compromise
  * Automated maintenance script execution
  * Comprehensive reporting and audit trails
  * Full dry-run testing capability

===== Client Requirements =====

A production deployment required the following specifications:

^ Requirement ^ Implementation ^
| Replication Schedule | Monthly updates from in-house backup server to air gap server |
| Transport Media | 3× 1.9TB SSD drives in rotation |
| Drive Rotation | One at source, one at target, one in transit — minimizes site visits |
| Security Model | Multi-layer encryption with split-key architecture |
| Location | Air gap server in unsecured location (mandatory encryption) |
| Automation | Fully automated with maintenance script execution |

===== Security Architecture =====

**Encryption Layers:**

  - **At Rest (Target):** ''GELI'' full disk encryption on air gap server
  - **In Transit:** Symmetric key encryption for all data on transport drives
  - **Maintenance Scripts:** Encrypted with same symmetric key
  - **Split-Key Design:** Target ''GELI'' key derived from:
    * Server-resident key component (stored locally)
    * Operator-carried key component (physical transport)
    * Combined via XOR bitwise operation at decrypt time
    * Target GELI key stored securely to facilitate key rotation and recovery

<wrap round tip>
**Split-key advantage:** Neither component alone can decrypt the air gap server. Compromise of a single key (server or transport) does not expose data.
</wrap>

===== Installation =====

==== Prerequisites ====

  * FreeBSD 13.0 or later
  * ZFS filesystem
  * Perl 5.28 or later
  * OpenSSL
  * ''GELI'' kernel module (for target server encryption)
  * Subversion client (for checkout)

==== Getting the Source ====

**Repository URL:** ''http://svn.dailydata.net/svn/zfs_utils/trunk''\\
**Sub-project:** ''sneakernet''

**Export the project:**
<code bash>
mkdir -p /usr/local/opt
svn export http://svn.dailydata.net/svn/zfs_utils/trunk /usr/local/opt/zfs_utils
cd /usr/local/opt/zfs_utils
</code>

==== Configuration ====

The sneakernet script uses YAML configuration files:

  - Main configuration: ''sneakernet.conf.yaml''
  - Default structure: ''sneakernet.datastructure''
  - On first run, creates config from datastructure if missing

**Key Configuration Sections:**
  * ''source'' — Source server settings (hostname, poolname)
  * ''target'' — Target server settings (hostname, poolname, GELI config)
  * ''transport'' — Transport drive settings (label, encryption key, mountpoint)
  * ''datasets'' — Dataset replication mappings

Example minimal configuration snippet:
<code yaml>
source:
  hostname: backup-primary
  poolname: tank
  
target:
  hostname: airgap-backup
  poolname: backup
  
transport:
  label: sneakernet
  fstype: ufs
  mountPoint: /mnt/sneakernet
  encryptionKey: your_hex_key_here
  
datasets:
  dataset1:
    source: tank
    target: backup
    dataset: data
</code>

===== Operation Workflows =====

==== Source Server Workflow ====

The source server performs the following operations automatically:

  - Auto-detect operating mode (source vs. target) via hostname
  - Mount transport drive using GPT label detection
  - Verify transport drive processed by target (check serial.txt)
  - Securely erase previous data from transport drive
  - Calculate incremental ZFS replication stream
  - Encrypt and write replication data to transport drive
  - Record latest snapshots sent (update status file)
  - Encrypt and write maintenance scripts to transport drive
  - Create serial.txt timestamp marker
  - Unmount transport drive
  - Email completion report to administrators

**Command:**
<code bash>
/usr/local/opt/zfs_utils/sneakernet/sneakernet
</code>

==== Target Server Workflow ====

The target server performs the following operations automatically:

  - Mount transport drive
  - Verify serial.txt exists (indicates unprocessed data)
  - Detect operator-provided secure key (USB/separate media)
  - Combine server key with operator key (XOR operation)
  - Unlock ''GELI'' encrypted disks using combined key
  - Import ZFS pool
  - Save current snapshot list to state file (enable rollback if needed)
  - Decrypt and import replication streams from transport
  - Remove serial.txt (marks data as processed)
  - Collect system statistics (pool health, disk status, capacity)
  - Decrypt and execute maintenance scripts
  - Generate detailed report and write to report drive
  - Unmount all media
  - Power off system (if ''shutdownAfterReplication'' enabled)

**Command:**
<code bash>
/usr/local/opt/zfs_utils/sneakernet/sneakernet
</code>

<wrap round info>
The script automatically detects whether it's running on the source or target server by comparing the hostname to the configuration. No mode flags required.
</wrap>

===== Three-Drive Rotation Strategy =====

The three-drive rotation minimizes operational overhead:

**Normal Operation Cycle:**

^ Month ^ Drive A ^ Drive B ^ Drive C ^ Action Required ^
| 1 | At Source (ready) | At Target | In Transit to Target | Operator: Deliver Drive C to target |
| 2 | At Source (ready) | In Transit to Source | At Target (ready) | Operator: Collect Drive B from target |
| 3 | In Transit to Target | At Source (ready) | At Target | Operator: Deliver Drive A to target |
| 4 | At Target | At Source (ready) | In Transit to Source | Operator: Collect Drive C from target |

**Benefits:**
  * Each site visit handles both delivery and pickup
  * No waiting time for drive processing
  * Reduced frequency of site access (security benefit)
  * Built-in offline backup (data exists on multiple drives)

**Drive Labeling:**

Each transport drive should be labeled with GPT labels:
<code bash>
# Label drives for easy identification
gpart add -t freebsd-ufs -l sneakernet_A /dev/ada0
gpart add -t freebsd-ufs -l sneakernet_B /dev/ada1
gpart add -t freebsd-ufs -l sneakernet_C /dev/ada2

# Create filesystems
newfs -U /dev/gpt/sneakernet_A
newfs -U /dev/gpt/sneakernet_B
newfs -U /dev/gpt/sneakernet_C
</code>

===== Key Management =====

==== Symmetric Transport Key ====

  * Unique key per deployment
  * Stored on both source and target servers
  * Used to encrypt data and scripts on transport drives
  * 256-bit AES encryption (AES-256-CBC mode)

**Generate transport key:**
<code bash>
# Generate 32-byte (256-bit) key in hex format
openssl rand 32 | xxd -p | tr -d '\n' > /secure/path/transport.key
chmod 400 /secure/path/transport.key
</code>

==== Split GELI Key ====

  * Server component: Stored on target server (never leaves facility)
  * Operator component: Carried by trusted operator (never stored at target)
  * Combined at runtime via XOR: ''final_key = server_key ⊕ operator_key''

**Generate split keys:**
<code bash>
# Generate the final GELI key (this will be stored securely off-site)
openssl rand 512 > /secure/offsite/final_geli.key

# Generate operator key
openssl rand 512 > /media/operator/operator.key

# Generate server key (XOR of final and operator keys)
# This requires the makeGeliKey utility from ZFS_Utils
/usr/local/opt/zfs_utils/utilities/makeGeliKey \
  /secure/offsite/final_geli.key \
  /media/operator/operator.key \
  /secure/server/server.key
</code>

==== Key Rotation Procedures ====

**If transport drive compromised:**
<code bash>
# Generate new symmetric key
openssl rand 32 | xxd -p | tr -d '\n' > /secure/path/new_transport.key

# Deploy as maintenance script on next run
# Old data on compromised drive remains encrypted with old key
</code>

**If operator key compromised:**
<code bash>
# Generate new operator key
openssl rand 512 > /media/operator/new_operator.key

# Retrieve final GELI key from secure off-site storage
# Regenerate server key using makeGeliKey
/usr/local/opt/zfs_utils/utilities/makeGeliKey \
  /secure/offsite/final_geli.key \
  /media/operator/new_operator.key \
  /secure/server/server.key

# Operator must use new key on next visit
</code>

<wrap round important>
**Key rotation can be automated** through maintenance scripts. New keys deployed during normal replication cycles without requiring emergency site visits.
</wrap>

===== Command-Line Options =====

<code>
Usage: sneakernet [OPTIONS]

Options:
  -n, --dryrun          Run in dry-run mode (no writes, shows what would happen)
  -v, --verbosity       Increase verbosity level (repeat for more detail)
  -V, --version         Display version number
  -h, --help            Display this help message

Verbosity Levels:
  0 = Errors and critical messages only
  1 = Standard operations
  2 = Detailed operations
  3 = Debugging information
  4 = Snapshot lists
  5 = Full detailed output
</code>

**Examples:**
<code bash>
# Test configuration without making changes
sneakernet --dryrun

# Run with detailed logging
sneakernet -vv

# Maximum verbosity for troubleshooting
sneakernet -vvvvv
</code>

===== Maintenance Scripts =====

Maintenance scripts are Perl scripts that execute on the target server after replication completes.

**Script Requirements:**
  * Must be valid Perl code
  * Return empty array for success, array of error strings for failures
  * Encrypted with transport symmetric key
  * Stored in configured ''cleanUpScriptsDir'' on transport

**Example maintenance script:**
<code perl>
#!/usr/bin/env perl
# cleanup_old_snapshots.pl
# Remove snapshots older than 90 days

use strict;
use warnings;

my @errors;
my $pool = 'backup';
my $cutoff_days = 90;

# Get list of snapshots
my @snapshots = `zfs list -t snapshot -o name -s creation -H $pool`;

foreach my $snap (@snapshots) {
    chomp $snap;
    # Parse timestamp from snapshot name and check age
    # Remove if older than cutoff
    # (implementation details omitted for brevity)
}

# Return errors array (empty = success)
return @errors;
</code>

**Deploy maintenance script:**
<code bash>
# On source server: encrypt and copy to cleanup directory
openssl enc -aes-256-cbc -salt -in cleanup_script.pl \
  -out /configured/cleanup/dir/cleanup_script.pl.enc \
  -pass file:/secure/transport.key
</code>

===== Monitoring and Reports =====

==== Source Server Reporting ====

Source server sends email reports via configured SMTP:

**Report Contents:**
  * Datasets replicated
  * Snapshot names and sizes
  * Total data transferred
  * Disk space utilization
  * Any errors or warnings

**Configuration:**
<code yaml>
source:
  report:
    subject: "Sneakernet Replication Report"
    emailTo: admin@example.com
    emailFrom: backup@example.com
</code>

==== Target Server Reporting ====

Target server writes reports to removable media:

**Report Drive Configuration:**
<code yaml>
target:
  report:
    targetDrive:
      label: report_drive
      fstype: ufs
      mountPoint: /mnt/report
</code>

**Report Contents:**
  * Timestamp of operation
  * Datasets imported
  * Pool health status
  * Disk SMART status
  * Maintenance script results
  * Any errors or warnings

===== Operational Benefits =====

This implementation balances security with operational efficiency:

**Security Advantages:**
  * No single point of key compromise
  * Lost transport drive: data remains encrypted
  * Lost operator key: server data still protected
  * Automated key rotation capability
  * Audit trail via detailed reports
  * Decryption failure = automatic abort

**Operational Advantages:**
  * Minimal site visits (monthly vs. weekly)
  * No waiting time for processing
  * Fully automated operation (no manual commands)
  * Email reports from source (connected)
  * Physical reports from target (air-gapped)
  * Automated maintenance without network access
  * Dry-run mode for testing changes

===== Troubleshooting =====

==== Common Issues ====

^ Problem ^ Solution ^
| Transport drive not detected | Verify GPT label matches config, check ''dmesg'' for device |
| Decryption fails on target | Verify transport key matches on both servers |
| Serial.txt already exists | Previous run not completed on target, investigate |
| Serial.txt missing on target | Drive already processed or not created on source |
| GELI disks won't unlock | Verify operator key present, check XOR key generation |
| Server won't shutdown | Check ''shutdownAfterReplication'' config, review logs |

==== Debug Mode ====

<code bash>
# Run with maximum verbosity and dry-run
sneakernet --dryrun -vvvvv 2>&1 | tee debug.log

# Check ZFS_Utils log file
tail -f /tmp/zfs_utils.log

# Check sneakernet log file (if configured)
tail -f /path/to/sneakernet.log
</code>

==== Log Files ====

  * Main log: Configured via ''logFile'' in YAML (default: ''sneakernet.log'')
  * ZFS_Utils log: ''/tmp/zfs_utils.log''
  * Status file: Configured via ''statusFile'' (tracks last replicated snapshots)
  * State file: Target only, records pre-update snapshot state

===== Version History =====

  * **v1.3.2** (2026-01-18) - Report drive notification, isMounted() integration
  * **v1.3.1** (2026-01-18) - CamelCase configuration keys, initialization refactoring
  * **v1.3.0** (2026-01-18) - Enhanced logging with caller tracking
  * **v1.2.x** - Serial.txt mechanism, cleanup scripts, random IVs, encryption enhancements
  * **v1.1.x** - Snapshot filtering, improved logging
  * **v1.0** (2025-12-15) - Initial release

See ''CHANGELOG.md'' in the repository for complete revision history.

===== References =====

  * [[unix:freebsd:system_builds:airgap:concepts|Air Gap Server Concepts]] — Theoretical background and best practices
  * [[https://docs.freebsd.org/en/books/handbook/disks/#disks-encrypting-geli|FreeBSD GELI Documentation]]
  * [[https://docs.freebsd.org/en/books/handbook/zfs/|FreeBSD ZFS Handbook]]
  * [[https://www.openssl.org/docs/|OpenSSL Documentation]]

===== Support =====

**Repository:** ''http://svn.dailydata.net/svn/zfs_utils/trunk''\\
**Checkout:** ''svn export http://svn.dailydata.net/svn/zfs_utils/trunk zfs_utils''\\
**Author:** R. W. Rodolico\\
**License:** BSD 2-Clause (FreeBSD License)\\
**Company:** Daily Data Inc. (https://dailydata.net)

For bug reports, feature requests, or questions, contact the repository maintainer by contact form https://dailydata.net/contact-us/