====== Air Gap Server: Concepts and Best Practices ======

<WRAP center round important 60%>
**Purpose:** Secure long-term backup storage isolated from network threats\\
**Key Requirements:** Full disk encryption, physical security, validated data transfer\\
**Target Environment:** FreeBSD with ZFS and GELI encryption\\
**Status:** Powered off when not actively receiving updates
</WRAP>

===== Overview =====

An **Air Gap Server** is a server that operates without network connectivity to protect critical backup data from remote attacks. A modified approach allows temporary network access for system updates while maintaining security boundaries.

The primary purpose is to store long-term backups with significantly reduced attack surface. Physical isolation combined with encryption provides defense-in-depth against:
  * Remote network attacks (ransomware, unauthorized access)
  * Physical theft or unauthorized access
  * Compromised source systems

<wrap round important>
**Critical:** If the server must be stored in an unsecured location, full disk encryption is **mandatory**, not optional.
</wrap>

===== Core Security Principles =====

  * **Physical Isolation** — Geographic separation from primary servers with controlled access
  * **Defense in Depth** — Multiple security layers protect data at rest and in transit
  * **Data Validation** — Verify integrity of all data transfers and scripts
  * **Automated Reporting** — Track all operations for audit and monitoring
  * **Powered Off by Default** — Server only active during updates or maintenance

===== Implementation Guidelines =====

==== Physical Security and Access Control ====

**Ideal Configuration:**
  * Store in secure facility requiring authenticated access (e.g., Network Operations Center)
  * Geographic separation from primary production servers
  * Documented access procedures and audit logs

**Fallback for Insecure Locations:**

When secure facilities are unavailable:
  * **Mandatory:** Full disk encryption (''GELI'' or equivalent)
  * **Mandatory:** Documented key management procedures
  * **Recommended:** Physical locks, tamper-evident seals
  * **Recommended:** Motion detection or access logging

<wrap round tip>
Store encryption keys in a different physical location than the server. Consider splitting keys across multiple secure locations.
</wrap>

==== Encryption Strategy ====

This implementation uses FreeBSD with ''GELI'' disk encryption backing a ''ZFS'' filesystem.

**At Rest Protection:**
  * Full disk encryption using ''GELI'' (minimum requirement)
  * All data pools encrypted with strong passphrases or key files
  * Keys never stored on the server itself

**Split-Key Architecture:**

For enhanced security, consider using split-key encryption where the final encryption key is derived from combining two separate key components. This enhances security by allowing the actual GELI key to be stored securely off-site, as it cannot be reconstructed without both components:

  * **Two-Operator Model:** Each key component held by different operators
    * Requires both operators present to unlock encrypted data
    * Maximum security: no single person can access data alone
    * Higher operational overhead
  
  * **Operator + Automated Model:** One key with operator, one on server
    * Operator key: Physically carried by trusted operator
    * Server key: Stored on automated script (never on target server)
    * Keys combined via XOR or similar operation at decrypt time
    * Balances security with automation needs

**In Transit Protection:**
  * Transport media (external drives) fully encrypted
  * Delta data encrypted before writing to transport media
  * Encryption keys validated at both source and destination

**Example GELI Setup:**
<code bash>
# Generate a random key file (4096 bits = 512 bytes)
openssl rand 512 > /secure/path/geli.key
chmod 400 /secure/path/geli.key

# Initialize GELI encryption on disk using the key file
geli init -s 4096 -K /secure/path/geli.key /dev/ada0

# Attach encrypted device
geli attach -k /secure/path/geli.key /dev/ada0

# Create ZFS pool on encrypted device
zpool create backup /dev/ada0.eli
</code>

<wrap round info>
Key size of 4096 bits provides strong encryption. The key file should be stored securely and backed up to a separate location. Use ''-P'' flag to add passphrase protection in addition to key file.
</wrap>

==== Data Transfer Validation ====

**Transport Media Requirements:**
  * Large capacity drives (match expected delta sizes)
  * Encrypted filesystem (''GELI'', ''LUKS'', or BitLocker) or Encryption of individual files in transit
  * Labeled with GPT labels for automated mounting

**Delta Monitoring:**

Monitor transfer sizes to detect anomalies:
  * Establish baseline delta sizes for normal operations
  * Alert on deltas exceeding 150-200% of baseline
  * **Large deltas may indicate ransomware on source system**

**Data Integrity Verification:**

<code bash>
# Generate checksum on source
zfs send pool/dataset@snapshot | tee >(sha256) > /mnt/transport/delta.zfs

# Verify checksum on air gap server
sha256 /mnt/transport/delta.zfs
</code>

**Data Validation:**
  * All data encrypted with symmetric key at source
  * Decryption failure automatically rejects the data
  * Failed decryption indicates corruption or tampering
  * Process terminates on any decryption failure

==== Script Validation and Maintenance ====

Air gap servers require special consideration for maintenance since they lack network access for updates.

**Validated Script Execution:**

Scripts may be deployed to perform maintenance tasks:
  * ''ZFS'' scrubs and pool health checks
  * Snapshot cleanup and rotation
  * SMART disk monitoring
  * System updates (if temporarily networked)

**Script Deployment Process:**
  - Scripts stored on source server and version controlled
  - Scripts encrypted with symmetric key before transfer
  - Air gap server must successfully decrypt before execution
  - Decryption failure prevents script execution and terminates process
  - Scripts run automatically during replication operations

**Example Script Encryption/Decryption:**
<code bash>
# On source server: encrypt script
openssl enc -aes-256-cbc -salt -in cleanup_script.sh \
  -out cleanup_script.sh.enc -pass file:/secure/transport.key

# On air gap server: decrypt and execute
openssl enc -aes-256-cbc -d -in cleanup_script.sh.enc \
  -out cleanup_script.sh -pass file:/secure/transport.key && \
  sh cleanup_script.sh || { echo "Decryption failed - aborting"; exit 1; }
</code>

<wrap round important>
**Security through decryption:** Scripts that cannot be decrypted with the correct symmetric key are rejected. Any decryption failure terminates the entire process to prevent execution of potentially tampered scripts.
</wrap>

==== Reporting and Audit Trail ====

**Reporting Challenges:**
  * Air gap servers cannot send email reports
  * No network access for remote monitoring
  * Reports must be physically retrieved

**Solution — Report Drive:**
  * Dedicated removable media for reports (USB drive, small HDD)
  * Reports written to transport drive after each operation
  * Administrator retrieves and processes reports manually

**Report Contents:**
  * Timestamp of operation
  * Data volumes transferred (size, snapshot names)
  * Success/failure status of each operation
  * Disk health (SMART status, ZFS pool health)
  * Script execution results
  * Any errors or warnings

**Example Report Structure:**
<code>
=== Air Gap Backup Report ===
Date: 2026-01-18 03:00:00
Operation: Incremental Backup
Source: production.example.com
Target: airgap-backup01

Datasets Processed:
  - pool/data: 45.2 GB transferred
    Latest: pool/data@2026-01-18_02:00:00
  - pool/databases: 12.8 GB transferred  
    Latest: pool/databases@2026-01-18_02:00:00

Pool Health: ONLINE
Disk Status: All disks PASSED SMART checks

Maintenance Scripts Executed:
  - snapshot_cleanup.sh: SUCCESS (removed 3 old snapshots)
  - zfs_scrub.sh: SUCCESS (no errors found)

System Shutdown: 2026-01-18 03:45:00
Next Expected Update: 2026-01-25
</code>

==== Power Management ====

**Default State: Powered Off**

The air gap server should remain powered off except during:
  * Scheduled data imports
  * Manual maintenance operations
  * Security audits

**Benefits of Power-Off Strategy:**
  * Encrypted drives are locked (keys in memory are cleared)
  * Eliminates risk of remote exploitation during off time
  * Reduces hardware wear and power consumption
  * Limits window of opportunity for physical attacks

**Automated Shutdown:**

Final script in maintenance chain should power off the system:
<code bash>
#!/bin/sh
# Final maintenance script - shutdown system

# Verify all operations completed successfully
if [ -f /var/run/backup_complete ]; then
    # Write final report
    echo "Backup completed successfully at $(date)" >> /mnt/report/status.log
    
    # Sync all filesystem buffers
    sync
    
    # Unmount transport media
    umount /mnt/transport
    umount /mnt/report
    
    # Power off system
    shutdown -p now
else
    echo "ERROR: Backup did not complete. Manual intervention required." >> /mnt/report/error.log
    # Do NOT shutdown - leave powered on for troubleshooting
fi
</code>

<wrap round important>
**Do not** configure automatic shutdown if backups fail. A powered-on system indicates problems requiring manual investigation.
</wrap>

===== Example Workflow =====

A typical weekly backup cycle:

**Day 1 (Monday) — Source Server:**
  - Automated script takes ZFS snapshots of all datasets
  - Calculates incremental changes since last backup
  - Encrypts delta data to transport drive with symmetric key
  - Encrypts maintenance scripts with same symmetric key
  - Operator notified that transport drive is ready

**Day 2 (Tuesday) — Physical Transport:**
  - Operator removes transport drive from source server
  - Drive physically transported to air gap location
  - Transport logged in access control system

**Day 3 (Wednesday) — Air Gap Server:**
  - Operator inserts transport drive and powers on server
  - Server boots, mounts transport drive
  - Automated script begins:
    * Attempts to decrypt delta files with symmetric key
    * Validates delta sizes against baseline
    * Imports ZFS datasets (decryption happens during import)
    * Attempts to decrypt and run maintenance scripts
    * Any decryption failure terminates the entire process
    * Generates report to report drive
    * Powers off system (only if all operations succeed)
  - Operator retrieves report drive for later review

**Day 4 (Thursday) — Report Processing:**
  - Operator reviews reports from air gap server
  - Verifies all backups completed successfully
  - Archives reports for audit trail
  - Updates monitoring dashboard

**Day 8 (Next Monday):**
  - Process repeats with fresh delta data

===== Pre-Implementation Checklist =====

<code>
[ ] Physical Security
    [ ] Secure location identified and documented
    [ ] Access procedures established
    [ ] Key storage locations determined
    
[ ] Hardware
    [ ] Air gap server procured and tested
    [ ] Transport drives procured (minimum 2 for rotation)
    [ ] Report drive procured
    [ ] All drives labeled appropriately
    
[ ] Encryption
    [ ] GELI encryption configured and tested
    [ ] Encryption keys generated and stored securely
    [ ] Key recovery procedures documented
    [ ] Transport drives encrypted
    
[ ] Software
    [ ] FreeBSD installed and hardened
    [ ] ZFS pools created and tested
    [ ] Replication scripts developed and tested
    [ ] Maintenance scripts developed and tested
    [ ] Symmetric transport keys generated and deployed
    
[ ] Procedures
    [ ] Backup schedule documented
    [ ] Transport procedures documented
    [ ] Report review procedures documented
    [ ] Key rotation schedule established
    [ ] Disaster recovery plan created
    
[ ] Testing
    [ ] Full backup cycle tested end-to-end
    [ ] Recovery procedures tested
    [ ] Failure scenarios tested
    [ ] Report generation verified
    [ ] Automated shutdown verified
</code>

===== Security Considerations =====

**Threat Model:**

This design protects against:
  * ✓ Remote network attacks (ransomware, unauthorized access)
  * ✓ Compromised source systems
  * ✓ Physical theft (with encryption)
  * ✓ Unauthorized physical access (with encryption)

This design does NOT fully protect against:
  * ✗ Sophisticated attackers with physical access and unlimited time
  * ✗ Compromised encryption keys
  * ✗ Attacks on the transport process itself
  * ✗ Insider threats with authorized access

**Best Practices:**
  * Rotate encryption keys annually
  * Test recovery procedures quarterly
  * Review audit logs monthly
  * Update maintenance scripts as needed
  * Keep offline backups of critical configuration

===== Troubleshooting =====

**Common Issues:**

^ Problem ^ Symptom ^ Solution ^
| Transport drive not mounting | Server unable to find ''/dev/gpt/label'' | Verify GPT label, check dmesg for device detection |
| Decryption fails | OpenSSL reports bad decrypt error | Verify correct symmetric key in use, check file integrity, investigate potential tampering or corruption |
| Large delta size | Delta exceeds baseline by 200%+ | **Do not import** — investigate source system for compromise or legitimate growth |
| Server won't shutdown | Remains powered on after backup | Check ''/var/run/backup_complete'' flag, review error logs on report drive |
| ZFS pool won't import | Import command fails | Verify encryption key, check pool status with ''zpool import -F'' |

===== References =====

  * [[https://docs.freebsd.org/en/books/handbook/disks/#disks-encrypting-geli|FreeBSD GELI Encryption Documentation]]
  * [[https://docs.freebsd.org/en/books/handbook/zfs/|FreeBSD ZFS Administration Guide]]
  * [[https://www.openssl.org/docs/|OpenSSL Documentation]] — For symmetric encryption operations
  * [[https://www.nist.gov/publications/guide-storage-encryption-technologies-end-user-devices|NIST Storage Encryption Guide]]

===== Related Documentation =====

  * [[unix:freebsd:system_builds:airgap:sneakernet_implementation|Sneakernet Implementation Guide]] — Real-world deployment example
  * [[key_management_procedures|Encryption Key Management]] — Key generation, storage, and rotation
  * [[incident_response_airgap|Air Gap Incident Response Plan]] — What to do if compromise suspected
  * [[disaster_recovery_procedures|Disaster Recovery Procedures]] — Restoring from air gap backups