Air Gap Server: Concepts and Best Practices

An Air Gap Server is a server that operates without network connectivity to protect critical backup data from remote attacks. A modified approach allows temporary network access for system updates while maintaining security boundaries.

The primary purpose is to store long-term backups with significantly reduced attack surface. Physical isolation combined with encryption provides defense-in-depth against:

• Remote network attacks (ransomware, unauthorized access)
• Physical theft or unauthorized access
• Compromised source systems

Critical: If the server must be stored in an unsecured location, full disk encryption is mandatory, not optional.

• Physical Isolation — Geographic separation from primary servers with controlled access
• Defense in Depth — Multiple security layers protect data at rest and in transit
• Data Validation — Verify integrity of all data transfers and scripts
• Automated Reporting — Track all operations for audit and monitoring
• Powered Off by Default — Server only active during updates or maintenance

Ideal Configuration:

• Store in secure facility requiring authenticated access (e.g., Network Operations Center)
• Geographic separation from primary production servers
• Documented access procedures and audit logs

Fallback for Insecure Locations:

When secure facilities are unavailable:

• Mandatory: Full disk encryption (GELI or equivalent)
• Mandatory: Documented key management procedures
• Recommended: Physical locks, tamper-evident seals
• Recommended: Motion detection or access logging

Store encryption keys in a different physical location than the server. Consider splitting keys across multiple secure locations.

This implementation uses FreeBSD with GELI disk encryption backing a ZFS filesystem.

At Rest Protection:

• Full disk encryption using GELI (minimum requirement)
• All data pools encrypted with strong passphrases or key files
• Keys never stored on the server itself

Split-Key Architecture:

For enhanced security, consider using split-key encryption where the final encryption key is derived from combining two separate key components. This enhances security by allowing the actual GELI key to be stored securely off-site, as it cannot be reconstructed without both components:

• Two-Operator Model: Each key component held by different operators
  • Requires both operators present to unlock encrypted data
  • Maximum security: no single person can access data alone
  • Higher operational overhead

• Operator + Automated Model: One key with operator, one on server
• Operator key: Physically carried by trusted operator
• Server key: Stored on automated script (never on target server)
• Keys combined via XOR or similar operation at decrypt time
• Balances security with automation needs

In Transit Protection:

• Transport media (external drives) fully encrypted
• Delta data encrypted before writing to transport media
• Encryption keys validated at both source and destination

Example GELI Setup:

# Generate a random key file (4096 bits = 512 bytes)
openssl rand 512 > /secure/path/geli.key
chmod 400 /secure/path/geli.key
 
# Initialize GELI encryption on disk using the key file
geli init -s 4096 -K /secure/path/geli.key /dev/ada0
 
# Attach encrypted device
geli attach -k /secure/path/geli.key /dev/ada0
 
# Create ZFS pool on encrypted device
zpool create backup /dev/ada0.eli

Key size of 4096 bits provides strong encryption. The key file should be stored securely and backed up to a separate location. Use -P flag to add passphrase protection in addition to key file.

Transport Media Requirements:

• Large capacity drives (match expected delta sizes)
• Encrypted filesystem (GELI, LUKS, or BitLocker) or Encryption of individual files in transit
• Labeled with GPT labels for automated mounting

Delta Monitoring:

Monitor transfer sizes to detect anomalies:

• Establish baseline delta sizes for normal operations
• Alert on deltas exceeding 150-200% of baseline
• Large deltas may indicate ransomware on source system

Data Integrity Verification:

# Generate checksum on source
zfs send pool/dataset@snapshot | tee >(sha256) > /mnt/transport/delta.zfs
 
# Verify checksum on air gap server
sha256 /mnt/transport/delta.zfs

Data Validation:

• All data encrypted with symmetric key at source
• Decryption failure automatically rejects the data
• Failed decryption indicates corruption or tampering
• Process terminates on any decryption failure

Air gap servers require special consideration for maintenance since they lack network access for updates.

Validated Script Execution:

Scripts may be deployed to perform maintenance tasks:

• ZFS scrubs and pool health checks
• Snapshot cleanup and rotation
• SMART disk monitoring
• System updates (if temporarily networked)

Script Deployment Process:

1. Scripts stored on source server and version controlled
2. Scripts encrypted with symmetric key before transfer
3. Air gap server must successfully decrypt before execution
4. Decryption failure prevents script execution and terminates process
5. Scripts run automatically during replication operations

Example Script Encryption/Decryption:

# On source server: encrypt script
openssl enc -aes-256-cbc -salt -in cleanup_script.sh \
  -out cleanup_script.sh.enc -pass file:/secure/transport.key
 
# On air gap server: decrypt and execute
openssl enc -aes-256-cbc -d -in cleanup_script.sh.enc \
  -out cleanup_script.sh -pass file:/secure/transport.key && \
  sh cleanup_script.sh || { echo "Decryption failed - aborting"; exit 1; }

Security through decryption: Scripts that cannot be decrypted with the correct symmetric key are rejected. Any decryption failure terminates the entire process to prevent execution of potentially tampered scripts.

Reporting Challenges:

• Air gap servers cannot send email reports
• No network access for remote monitoring
• Reports must be physically retrieved

Solution — Report Drive:

• Dedicated removable media for reports (USB drive, small HDD)
• Reports written to transport drive after each operation
• Administrator retrieves and processes reports manually

Report Contents:

• Timestamp of operation
• Data volumes transferred (size, snapshot names)
• Success/failure status of each operation
• Disk health (SMART status, ZFS pool health)
• Script execution results
• Any errors or warnings

Example Report Structure:

=== Air Gap Backup Report ===
Date: 2026-01-18 03:00:00
Operation: Incremental Backup
Source: production.example.com
Target: airgap-backup01

Datasets Processed:
  - pool/data: 45.2 GB transferred
    Latest: pool/data@2026-01-18_02:00:00
  - pool/databases: 12.8 GB transferred  
    Latest: pool/databases@2026-01-18_02:00:00

Pool Health: ONLINE
Disk Status: All disks PASSED SMART checks

Maintenance Scripts Executed:
  - snapshot_cleanup.sh: SUCCESS (removed 3 old snapshots)
  - zfs_scrub.sh: SUCCESS (no errors found)

System Shutdown: 2026-01-18 03:45:00
Next Expected Update: 2026-01-25

Default State: Powered Off

The air gap server should remain powered off except during:

• Scheduled data imports
• Manual maintenance operations
• Security audits

Benefits of Power-Off Strategy:

• Encrypted drives are locked (keys in memory are cleared)
• Eliminates risk of remote exploitation during off time
• Reduces hardware wear and power consumption
• Limits window of opportunity for physical attacks

Automated Shutdown:

Final script in maintenance chain should power off the system:

#!/bin/sh
# Final maintenance script - shutdown system
 
# Verify all operations completed successfully
if [ -f /var/run/backup_complete ]; then
    # Write final report
    echo "Backup completed successfully at $(date)" >> /mnt/report/status.log
 
    # Sync all filesystem buffers
    sync
 
    # Unmount transport media
    umount /mnt/transport
    umount /mnt/report
 
    # Power off system
    shutdown -p now
else
    echo "ERROR: Backup did not complete. Manual intervention required." >> /mnt/report/error.log
    # Do NOT shutdown - leave powered on for troubleshooting
fi

Do not configure automatic shutdown if backups fail. A powered-on system indicates problems requiring manual investigation.

A typical weekly backup cycle:

Day 1 (Monday) — Source Server:

1. Automated script takes ZFS snapshots of all datasets
2. Calculates incremental changes since last backup
3. Encrypts delta data to transport drive with symmetric key
4. Encrypts maintenance scripts with same symmetric key
5. Operator notified that transport drive is ready

Day 2 (Tuesday) — Physical Transport:

1. Operator removes transport drive from source server
2. Drive physically transported to air gap location
3. Transport logged in access control system

Day 3 (Wednesday) — Air Gap Server:

1. Operator inserts transport drive and powers on server
2. Server boots, mounts transport drive
3. Automated script begins:
   • Attempts to decrypt delta files with symmetric key
   • Validates delta sizes against baseline
   • Imports ZFS datasets (decryption happens during import)
   • Attempts to decrypt and run maintenance scripts
   • Any decryption failure terminates the entire process
   • Generates report to report drive
   • Powers off system (only if all operations succeed)
4. Operator retrieves report drive for later review

Day 4 (Thursday) — Report Processing:

1. Operator reviews reports from air gap server
2. Verifies all backups completed successfully
3. Archives reports for audit trail
4. Updates monitoring dashboard

Day 8 (Next Monday):

1. Process repeats with fresh delta data

[ ] Physical Security
    [ ] Secure location identified and documented
    [ ] Access procedures established
    [ ] Key storage locations determined
    
[ ] Hardware
    [ ] Air gap server procured and tested
    [ ] Transport drives procured (minimum 2 for rotation)
    [ ] Report drive procured
    [ ] All drives labeled appropriately
    
[ ] Encryption
    [ ] GELI encryption configured and tested
    [ ] Encryption keys generated and stored securely
    [ ] Key recovery procedures documented
    [ ] Transport drives encrypted
    
[ ] Software
    [ ] FreeBSD installed and hardened
    [ ] ZFS pools created and tested
    [ ] Replication scripts developed and tested
    [ ] Maintenance scripts developed and tested
    [ ] Symmetric transport keys generated and deployed
    
[ ] Procedures
    [ ] Backup schedule documented
    [ ] Transport procedures documented
    [ ] Report review procedures documented
    [ ] Key rotation schedule established
    [ ] Disaster recovery plan created
    
[ ] Testing
    [ ] Full backup cycle tested end-to-end
    [ ] Recovery procedures tested
    [ ] Failure scenarios tested
    [ ] Report generation verified
    [ ] Automated shutdown verified

Threat Model:

This design protects against:

• ✓ Remote network attacks (ransomware, unauthorized access)
• ✓ Compromised source systems
• ✓ Physical theft (with encryption)
• ✓ Unauthorized physical access (with encryption)

This design does NOT fully protect against:

• ✗ Sophisticated attackers with physical access and unlimited time
• ✗ Compromised encryption keys
• ✗ Attacks on the transport process itself
• ✗ Insider threats with authorized access

Best Practices:

• Rotate encryption keys annually
• Test recovery procedures quarterly
• Review audit logs monthly
• Update maintenance scripts as needed
• Keep offline backups of critical configuration

Common Issues:

• FreeBSD GELI Encryption Documentation
• FreeBSD ZFS Administration Guide
• OpenSSL Documentation — For symmetric encryption operations
• NIST Storage Encryption Guide

• Sneakernet Implementation Guide — Real-world deployment example
• Encryption Key Management — Key generation, storage, and rotation
• Air Gap Incident Response Plan — What to do if compromise suspected
• Disaster Recovery Procedures — Restoring from air gap backups