Sneakernet: Production Air Gap Implementation

The sneakernet project provides a complete automated solution for air gap server replication using ZFS datasets and encrypted transport media. This implementation was developed for a production environment requiring monthly off-site backups with split-key security architecture.

Key Features:

• Automatic source/target mode detection
• Three-drive rotation minimizing site visits
• Multi-layer encryption (GELI + symmetric transport)
• Split-key architecture preventing single-point compromise
• Automated maintenance script execution
• Comprehensive reporting and audit trails
• Full dry-run testing capability

A production deployment required the following specifications:

Encryption Layers:

1. At Rest (Target): GELI full disk encryption on air gap server
2. In Transit: Symmetric key encryption for all data on transport drives
3. Maintenance Scripts: Encrypted with same symmetric key
4. Split-Key Design: Target GELI key derived from:
   • Server-resident key component (stored locally)
   • Operator-carried key component (physical transport)
   • Combined via XOR bitwise operation at decrypt time
   • Target GELI key stored securely to facilitate key rotation and recovery

Split-key advantage: Neither component alone can decrypt the air gap server. Compromise of a single key (server or transport) does not expose data.

• FreeBSD 13.0 or later
• ZFS filesystem
• Perl 5.28 or later
• OpenSSL
• GELI kernel module (for target server encryption)
• Subversion client (for checkout)

Repository URL: http://svn.dailydata.net/svn/zfs_utils/trunk
Sub-project: sneakernet

Export the project:

mkdir -p /usr/local/opt
svn export http://svn.dailydata.net/svn/zfs_utils/trunk /usr/local/opt/zfs_utils
cd /usr/local/opt/zfs_utils

The sneakernet script uses YAML configuration files:

1. Main configuration: sneakernet.conf.yaml
2. Default structure: sneakernet.datastructure
3. On first run, creates config from datastructure if missing

Key Configuration Sections:

• source — Source server settings (hostname, poolname)
• target — Target server settings (hostname, poolname, GELI config)
• transport — Transport drive settings (label, encryption key, mountpoint)
• datasets — Dataset replication mappings

Example minimal configuration snippet:

source:
  hostname: backup-primary
  poolname: tank
  
target:
  hostname: airgap-backup
  poolname: backup
  
transport:
  label: sneakernet
  fstype: ufs
  mountPoint: /mnt/sneakernet
  encryptionKey: your_hex_key_here
  
datasets:
  dataset1:
    source: tank
    target: backup
    dataset: data

The source server performs the following operations automatically:

1. Auto-detect operating mode (source vs. target) via hostname
2. Mount transport drive using GPT label detection
3. Verify transport drive processed by target (check serial.txt)
4. Securely erase previous data from transport drive
5. Calculate incremental ZFS replication stream
6. Encrypt and write replication data to transport drive
7. Record latest snapshots sent (update status file)
8. Encrypt and write maintenance scripts to transport drive
9. Create serial.txt timestamp marker
10. Unmount transport drive
11. Email completion report to administrators

Command:

/usr/local/opt/zfs_utils/sneakernet/sneakernet

The target server performs the following operations automatically:

1. Mount transport drive
2. Verify serial.txt exists (indicates unprocessed data)
3. Detect operator-provided secure key (USB/separate media)
4. Combine server key with operator key (XOR operation)
5. Unlock GELI encrypted disks using combined key
6. Import ZFS pool
7. Save current snapshot list to state file (enable rollback if needed)
8. Decrypt and import replication streams from transport
9. Remove serial.txt (marks data as processed)
10. Collect system statistics (pool health, disk status, capacity)
11. Decrypt and execute maintenance scripts
12. Generate detailed report and write to report drive
13. Unmount all media
14. Power off system (if shutdownAfterReplication enabled)

Command:

/usr/local/opt/zfs_utils/sneakernet/sneakernet

The script automatically detects whether it's running on the source or target server by comparing the hostname to the configuration. No mode flags required.

The three-drive rotation minimizes operational overhead:

Normal Operation Cycle:

Benefits:

• Each site visit handles both delivery and pickup
• No waiting time for drive processing
• Reduced frequency of site access (security benefit)
• Built-in offline backup (data exists on multiple drives)

Drive Labeling:

Each transport drive should be labeled with GPT labels:

# Label drives for easy identification
gpart add -t freebsd-ufs -l sneakernet_A /dev/ada0
gpart add -t freebsd-ufs -l sneakernet_B /dev/ada1
gpart add -t freebsd-ufs -l sneakernet_C /dev/ada2
 
# Create filesystems
newfs -U /dev/gpt/sneakernet_A
newfs -U /dev/gpt/sneakernet_B
newfs -U /dev/gpt/sneakernet_C

• Unique key per deployment
• Stored on both source and target servers
• Used to encrypt data and scripts on transport drives
• 256-bit AES encryption (AES-256-CBC mode)

Generate transport key:

# Generate 32-byte (256-bit) key in hex format
openssl rand 32 | xxd -p | tr -d '\n' > /secure/path/transport.key
chmod 400 /secure/path/transport.key

• Server component: Stored on target server (never leaves facility)
• Operator component: Carried by trusted operator (never stored at target)
• Combined at runtime via XOR: final_key = server_key ⊕ operator_key

Generate split keys:

# Generate the final GELI key (this will be stored securely off-site)
openssl rand 512 > /secure/offsite/final_geli.key
 
# Generate operator key
openssl rand 512 > /media/operator/operator.key
 
# Generate server key (XOR of final and operator keys)
# This requires the makeGeliKey utility from ZFS_Utils
/usr/local/opt/zfs_utils/utilities/makeGeliKey \
  /secure/offsite/final_geli.key \
  /media/operator/operator.key \
  /secure/server/server.key

If transport drive compromised:

# Generate new symmetric key
openssl rand 32 | xxd -p | tr -d '\n' > /secure/path/new_transport.key
 
# Deploy as maintenance script on next run
# Old data on compromised drive remains encrypted with old key

If operator key compromised:

# Generate new operator key
openssl rand 512 > /media/operator/new_operator.key
 
# Retrieve final GELI key from secure off-site storage
# Regenerate server key using makeGeliKey
/usr/local/opt/zfs_utils/utilities/makeGeliKey \
  /secure/offsite/final_geli.key \
  /media/operator/new_operator.key \
  /secure/server/server.key
 
# Operator must use new key on next visit

Key rotation can be automated through maintenance scripts. New keys deployed during normal replication cycles without requiring emergency site visits.

Usage: sneakernet [OPTIONS]

Options:
  -n, --dryrun          Run in dry-run mode (no writes, shows what would happen)
  -v, --verbosity       Increase verbosity level (repeat for more detail)
  -V, --version         Display version number
  -h, --help            Display this help message

Verbosity Levels:
  0 = Errors and critical messages only
  1 = Standard operations
  2 = Detailed operations
  3 = Debugging information
  4 = Snapshot lists
  5 = Full detailed output

Examples:

# Test configuration without making changes
sneakernet --dryrun
 
# Run with detailed logging
sneakernet -vv
 
# Maximum verbosity for troubleshooting
sneakernet -vvvvv

Maintenance scripts are Perl scripts that execute on the target server after replication completes.

Script Requirements:

• Must be valid Perl code
• Return empty array for success, array of error strings for failures
• Encrypted with transport symmetric key
• Stored in configured cleanUpScriptsDir on transport

Example maintenance script:

#!/usr/bin/env perl
# cleanup_old_snapshots.pl
# Remove snapshots older than 90 days
 
use strict;
use warnings;
 
my @errors;
my $pool = 'backup';
my $cutoff_days = 90;
 
# Get list of snapshots
my @snapshots = `zfs list -t snapshot -o name -s creation -H $pool`;
 
foreach my $snap (@snapshots) {
    chomp $snap;
    # Parse timestamp from snapshot name and check age
    # Remove if older than cutoff
    # (implementation details omitted for brevity)
}
 
# Return errors array (empty = success)
return @errors;

Deploy maintenance script:

# On source server: encrypt and copy to cleanup directory
openssl enc -aes-256-cbc -salt -in cleanup_script.pl \
  -out /configured/cleanup/dir/cleanup_script.pl.enc \
  -pass file:/secure/transport.key

Source server sends email reports via configured SMTP:

Report Contents:

• Datasets replicated
• Snapshot names and sizes
• Total data transferred
• Disk space utilization
• Any errors or warnings

Configuration:

source:
  report:
    subject: "Sneakernet Replication Report"
    emailTo: admin@example.com
    emailFrom: backup@example.com

Target server writes reports to removable media:

Report Drive Configuration:

target:
  report:
    targetDrive:
      label: report_drive
      fstype: ufs
      mountPoint: /mnt/report

Report Contents:

• Timestamp of operation
• Datasets imported
• Pool health status
• Disk SMART status
• Maintenance script results
• Any errors or warnings

This implementation balances security with operational efficiency:

Security Advantages:

• No single point of key compromise
• Lost transport drive: data remains encrypted
• Lost operator key: server data still protected
• Automated key rotation capability
• Audit trail via detailed reports
• Decryption failure = automatic abort

Operational Advantages:

• Minimal site visits (monthly vs. weekly)
• No waiting time for processing
• Fully automated operation (no manual commands)
• Email reports from source (connected)
• Physical reports from target (air-gapped)
• Automated maintenance without network access
• Dry-run mode for testing changes

# Run with maximum verbosity and dry-run
sneakernet --dryrun -vvvvv 2>&1 | tee debug.log
 
# Check ZFS_Utils log file
tail -f /tmp/zfs_utils.log
 
# Check sneakernet log file (if configured)
tail -f /path/to/sneakernet.log

• Main log: Configured via logFile in YAML (default: sneakernet.log)
• ZFS_Utils log: /tmp/zfs_utils.log
• Status file: Configured via statusFile (tracks last replicated snapshots)
• State file: Target only, records pre-update snapshot state

• v1.3.2 (2026-01-18) - Report drive notification, isMounted() integration
• v1.3.1 (2026-01-18) - CamelCase configuration keys, initialization refactoring
• v1.3.0 (2026-01-18) - Enhanced logging with caller tracking
• v1.2.x - Serial.txt mechanism, cleanup scripts, random IVs, encryption enhancements
• v1.1.x - Snapshot filtering, improved logging
• v1.0 (2025-12-15) - Initial release

See CHANGELOG.md in the repository for complete revision history.

• Air Gap Server Concepts — Theoretical background and best practices
• FreeBSD GELI Documentation
• FreeBSD ZFS Handbook
• OpenSSL Documentation

Repository: http://svn.dailydata.net/svn/zfs_utils/trunk
Checkout: svn export http://svn.dailydata.net/svn/zfs_utils/trunk zfs_utils
Author: R. W. Rodolico
License: BSD 2-Clause (FreeBSD License)
Company: Daily Data Inc. (https://dailydata.net)

For bug reports, feature requests, or questions, contact the repository maintainer by contact form https://dailydata.net/contact-us/