This guide walks through of complete workflow for setting up and managing encrypted secrets with sops-nix. It solves "chicken and egg" problem of needing secrets before your server is fully configured.
2. Strategy 1: Generate Secrets After Install (Recommended)
3. Strategy 2: Pre-generate Secrets Locally
4. Setting Up Your Encryption Key
5. Creating and Managing Secrets
6. Service-Specific Secret Examples
7. Advanced: Multiple Recipients
The dilemma:
Two strategies to solve this:
This is simplest approach for most users.
First, install NixOS with a minimal configuration that doesn't need secrets:
# /etc/nixos/configuration.nix (initial version)
{
# ... basic system config without services that need secrets ...
# Don't configure PDS, email, or other secret-needing services yet
}
}}}
=== Step 2: Get Your Server's SSH Key ===
After NixOS is running:
# On server, get your SSH host public key
cat /etc/ssh/sshhosted25519_key.pub
# Copy this - it looks like:
# ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... root@hostname
}}}
On your *local machine*:
# Install ssh-to-age if not already installed
# macOS: brew install ssh-to-age
# Linux: download from https://github.com/Mic92/ssh-to-age/releases
# Convert SSH key to age format
ssh-to-age -i <(echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...")
# Output example:
# age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
}}}
=== Step 4: Create .sops.yaml ===
On your **local machine**, create sops configuration:
mkdir -p /etc/nixos/secrets
cat > /etc/nixos/secrets/.sops.yaml << 'EOF'
keys:
creation_rules:
key_groups:
EOF
}}}
Replace age key with your actual key from Step 3.
Now generate actual secrets. Here's how for each type:
# Generate a random 32-byte hex string
openssl rand -hex 32
# Or use pwgen
pwgen -s 64 1
# Save this - you'll encrypt it with sops
}}}
==== Generate PDS Admin Password ===
# Generate strong password
pwgen -s 32 1
# or
openssl rand -base64 24
}}}
# Generate K256 private key
openssl ecparam -name secp256k1 -genkey -noout -out /tmp/plc.key
# Convert to hex format needed by PDS
openssl ec -in /tmp/plc.key -outform DER | tail -c 32 | xxd -p -c 64
# Clean up
rm /tmp/plc.key
}}}
==== Create Email SMTP URL ===
Format: `smtps://username:password@smtp.gmail.com:465`
For Gmail:
- Create an app-specific password at https://myaccount.google.com/apppasswords
- URL encode special characters in password (replace `@` with `%40`, etc.)
=== Step 6: Create Encrypted Secrets File ===
Create secrets file with all values:
cat > /etc/nixos/secrets/bluesky-pds.yaml << 'EOF'
PDSJWTSECRET: "your-generated-jwt-secret-here"
PDSADMINPASSWORD: "your-admin-password-here"
PDSPLCROTATIONKEYK256PRIVATEKEY_HEX: "your-plc-key-hex-here"
PDSEMAILSMTP_URL: "smtps://user:pass@smtp.gmail.com:465"
PDSEMAILFROM_ADDRESS: "noreply@yourdomain.com"
EOF
# Encrypt it
sops encrypt -i /etc/nixos/secrets/bluesky-pds.yaml
}}}
Add sops configuration to your NixOS config:
{
# ... other config ...
sops.defaultSopsFile = ./secrets/bluesky-pds.yaml;
sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
# Define secrets (empty attrs = default permissions)
sops.secrets.PDS_JWT_SECRET = {};
sops.secrets.PDS_ADMIN_PASSWORD = {};
sops.secrets.PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX = {};
sops.secrets.PDS_EMAIL_SMTP_URL = {};
sops.secrets.PDS_EMAIL_FROM_ADDRESS = {};
# Create environment file from secrets
sops.templates."pds-env".content = ''
PDS_JWT_SECRET=${config.sops.placeholder.PDS_JWT_SECRET}
PDS_ADMIN_PASSWORD=${config.sops.placeholder.PDS_ADMIN_PASSWORD}
PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX=${config.sops.placeholder.PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX}
PDS_EMAIL_SMTP_URL=${config.sops.placeholder.PDS_EMAIL_SMTP_URL}
PDS_EMAIL_FROM_ADDRESS=${config.sops.placeholder.PDS_EMAIL_FROM_ADDRESS}
'';
# Now you can configure PDS
services.bluesky-pds = {
enable = true;
environmentFiles = [ config.sops.templates."pds-env".path ];
# ... other settings ...
};
}
}}}
=== Step 8: Deploy ===
cd /etc/nixos
sudo nixos-rebuild switch --flake .#snek
}}}
Verify secrets are decrypted:
sudo ls -la /run/secrets/
sudo cat /run/secrets/PDS_JWT_SECRET
}}}
== Strategy 2: Pre-generate Secrets Locally ==
Use this if you want everything ready before touching server.
=== Step 1: Generate Age Key Locally ===
# Generate a new age key pair
age-keygen -o ~/snek-server-key.txt
# This creates a file with:
# # created: 2024-01-15T10:00:00Z
# # public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
# AGE-SECRET-KEY-1...
}}}
Save public key and secret key file securely.
Create .sops.yaml using your local age key:
keys:
- &local age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
creation_rules:
- path_regex: secrets/.*\.yaml$
key_groups:
- age:
- *local
}}}
Create and encrypt secrets on your local machine.
=== Step 3: Install NixOS with Age Key ===
During NixOS installation, copy your age secret key to server:
# On server during install
mkdir -p /mnt/etc/secrets
cp ~/snek-server-key.txt /mnt/etc/secrets/age.key
chmod 600 /mnt/etc/secrets/age.key
}}}
{
sops.age.keyFile = "/etc/secrets/age.key";
# ... rest of sops config ...
}
}}}
=== Step 5: After Install, Add Server's SSH Key ===
Once installed, get server's SSH key and add it as a recipient:
# On server
cat /etc/ssh/sshhosted25519_key.pub | ssh-to-age
# Add this key to .sops.yaml as additional recipient
# Then rotate (see Rotating Secrets section)
}}}
Age uses X25519 keys:
age1..., used to encryptAGE-SECRET-KEY-1..., used to decryptYour NixOS server already has SSH host keys at /etc/ssh/sshhosted25519_key.
# Convert SSH public key to age public key
ssh-to-age -i /etc/ssh/ssh_host_ed25519_key.pub
# Convert SSH private key to age secret key (not usually needed)
ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key
}}}
=== Using Multiple Keys ===
You can have multiple recipients who can decrypt:
keys:
creation_rules:
key_groups:
}}}
# Method 1: Create then encrypt
cat > secrets/new-service.yaml << 'EOF'
API_KEY: "secret-value-here"
DATABASE_URL: "postgres://user:pass@localhost/db"
EOF
sops encrypt -i secrets/new-service.yaml
# Method 2: Use sops directly (opens in $EDITOR)
sops secrets/new-service.yaml
}}}
=== Editing Existing Secrets ===
# Decrypt, open in editor, re-encrypt automatically
sops secrets/bluesky-pds.yaml
# Make changes, save, and exit
# sops handles encryption automatically
}}}
# Decrypt and output to stdout
sops -d secrets/bluesky-pds.yaml
# Extract specific value
sops -d --extract '["PDS_JWT_SECRET"]' secrets/bluesky-pds.yaml
}}}
=== Adding New Secrets to Existing File ===
# Edit file
sops secrets/bluesky-pds.yaml
# Add new key-value pair in editor:
# NEW_SECRET: "new-value"
# Save and exit - automatically encrypted
}}}
# Edit file
sops secrets/bluesky-pds.yaml
# Change values in editor
# Save and exit
# Rebuild NixOS to pick up new values
sudo nixos-rebuild switch --flake .#snek
}}}
== Service-Specific Secret Examples ==
=== Bluesky PDS ===
# secrets/bluesky-pds.yaml
PDSJWTSECRET: "64-char-hex-string"
PDSADMINPASSWORD: "strong-admin-password"
PDSPLCROTATIONKEYK256PRIVATEKEY_HEX: "64-char-hex-private-key"
PDSEMAILSMTP_URL: "smtps://user:pass@smtp.gmail.com:465"
PDSEMAILFROM_ADDRESS: "noreply@yourdomain.com"
}}}
Generate JWT Secret:
openssl rand -hex 32
}}}
*Generate Admin Password:*pwgen -s 32 1
}}}
Generate PLC Key:
openssl ecparam -name secp256k1 -genkey -noout | openssl ec -outform DER | tail -c 32 | xxd -p -c 64
}}}
=== PostgreSQL ===
# secrets/postgresql.yaml
POSTGRES_PASSWORD: "db-password"
}}}
# secrets/external-services.yaml
GITHUB_TOKEN: "ghp_xxxxxxxx"
STRIPE_SECRET_KEY: "sk_live_xxxxxxxx"
SENDGRID_API_KEY: "SG.xxxxxxxx"
}}}
=== WireGuard/Tailscale Keys ===
# secrets/networking.yaml
TAILSCALEAUTHKEY: "tskey-auth-xxxxxxxx"
WIREGUARDPRIVATEKEY: "xxxxxxxx"
}}}
1. Get their age public key
2. Add to .sops.yaml:
keys:
- &server age1server...
- &backup age1backup...
- &newadmin age1newadmin...
creation_rules:
- path_regex: secrets/.*\.yaml$
key_groups:
- age:
- *server
- *backup
- *newadmin
}}}
3. Rotate all secrets:
# sops automatically re-encrypts with all recipients
sops rotate -i secrets/*.yaml
}}}
1. Remove key from .sops.yaml
2. Rotate all secrets (they'll be encrypted for remaining recipients only)
This re-encrypts file without changing secret values:
sops rotate -i secrets/bluesky-pds.yaml
}}}
Do this when:
- Adding/removing recipients
- You suspect key compromise
- Periodic security maintenance (monthly/quarterly)
=== Rotating Secret Values ===
This changes actual secret (e.g., password, API key):
# 1. Edit and change values
sops secrets/bluesky-pds.yaml
# 2. Deploy
sudo nixos-rebuild switch --flake .#snek
# 3. Restart affected services
sudo systemctl restart bluesky-pds
}}}
# 1. Generate new secrets
echo "PDS_JWT_SECRET: $(openssl rand -hex 32)" > /tmp/new-secrets.txt
# 2. Edit encrypted file with new values
sops secrets/bluesky-pds.yaml
# (copy-paste new values)
# 3. Rotate encryption keys
sops rotate -i secrets/bluesky-pds.yaml
# 4. Commit changes
git add secrets/
git commit -m "Rotate PDS secrets"
# 5. Deploy
sudo nixos-rebuild switch --flake .#snek
# 6. Restart services
sudo systemctl restart bluesky-pds
# 7. Verify
systemctl status bluesky-pds
journalctl -u bluesky-pds -f
}}}
== Backup and Recovery ==
=== What to Backup ===
*Critical:*
- `/etc/nixos/secrets/*.yaml` (encrypted secrets)
- `/etc/nixos/secrets/.sops.yaml` (encryption config)
- Age secret keys (if using file-based keys)
*How to backup:*# The secrets are already encrypted, safe to commit to git
cd /etc/nixos
git add secrets/
git commit -m "Backup secrets"
git push
}}}
Scenario 1: Lost Server, Have Secrets Repository
1. Provision new server
2. Install NixOS
3. Clone your config repository (includes encrypted secrets)
4. Get new server's SSH key
5. Add as recipient to .sops.yaml
6. Rotate secrets to include new recipient
7. Deploy
Scenario 2: Forgot to Backup .sops.yaml
If you have to secrets files but lost .sops.yaml:
# Check if you can still decrypt
sops -d secrets/bluesky-pds.yaml
# If yes, recreate .sops.yaml with current recipients
# (sops stores recipient info in file itself)
}}}
*Scenario 3: Lost All Age Keys*
Worst case - you can't decrypt secrets anymore:
1. Generate new secrets from scratch
2. Update services with new secrets
3. Revoke old API keys/tokens if possible
*Prevention:*
- Always have multiple recipients (backup key stored offline)
- Commit .sops.yaml to git
- Backup age secret keys to secure offline storage
=== Testing Recovery ===
Test your recovery process before you need it:
# 1. Clone your config to a temporary directory
cd /tmp
git clone https://github.com/your/config.git test-recovery
cd test-recovery
# 2. Try to decrypt secrets
sops -d secrets/bluesky-pds.yaml
# 3. If this works, your backup is good
# If not, fix your backup process
}}}
1. *Never commit unencrypted secrets* - Always use sops
2. *Use multiple recipients* - Backup key stored securely offline
3. *Rotate regularly* - Monthly for critical secrets
4. *Test recovery* - Verify you can decrypt from backup
5. *Document what each secret is for* - Add comments in YAML
6. *Use strong passwords* - For any password-type secrets
7. *Separate secrets by service* - Don't put everything in one file
8. *Version control* - Commit encrypted files to track changes
.sops.yamlls -la /run/secrets/sudo nixos-rebuild switchsops.age.sshKeyPaths or sops.age.keyFilenixos-rebuild switch/run/secrets/ has new values