← Back to Guide Index

Secrets Management: sops-nix and Secure Credential Handling

The Challenge of Secrets in NixOS

Secrets—passwords, API keys, private keys, tokens—are the most sensitive data in your system. They need to be:

This creates a tension: Nix is designed for reproducibility and stores everything in /nix/store, but secrets must not be stored there. sops-nix solves this by separating the structure of your configuration (which goes in the Nix store) from the secret values (which are decrypted at activation time).

Understanding sops (Secrets OPerationS)

sops is a tool from Mozilla that handles encrypted files. It supports multiple encryption backends (GPG, age) and file formats (YAML, JSON, ENV, BINARY). The key features:

Encryption of values, not keys: In a sops file, the structure (keys) is plaintext, but the values are encrypted. This means you can see what secrets exist without being able to read them.

Multiple keys: A file can be encrypted for multiple recipients. Any recipient can decrypt. This supports team workflows.

Key rotation: You can rotate data keys (re-encrypting the file) or add/remove recipients without changing the secret values.

Audit trail: sops can track who accessed which secrets when (with additional configuration).

age: Modern Encryption

You use *age* (pronounced "ah-gay") for encryption. It's designed to be simple and modern, in contrast to GPG's complexity.

Key pairs: age uses public-key cryptography. You have:

Key generation:

age-keygen -o key.txt
}}}

This creates a file with your private key. The public key is shown on stdout.

*Why age over GPG:*
- Simpler: No keyrings, trust databases, or complex workflows
- Modern cryptography: X25519 for encryption
- Small: Single binary, minimal attack surface
- Easy integration: Keys are just strings

== The sops-nix Integration ==

sops-nix is a NixOS module that integrates sops into NixOS activation. It provides:

*Configuration-based secrets:* Define secrets in your Nix configuration, sops-nix handles decryption and placement

*Activation-time decryption:* Secrets are decrypted when the system activates (not when Nix builds), keeping them out of the Nix store

*Permission management:* Set ownership and permissions on secrets so only the right users/services can read them

*Template support:* Create files with secrets interpolated, useful for environment files or config files containing secrets

*SSH key integration:* Can use SSH host keys as age keys, simplifying key management

== How Secrets Flow Through the System ==

Understanding the lifecycle of a secret helps clarify the architecture:

=== Phase 1: Creation (on your development machine) ===

1. You create a YAML file with your secrets in plaintext:

PDSADMINPASSWORD: super-secret-password

PDSJWTSECRET: jwt-signing-key-here

}}}

2. You run sops secrets.yaml

3. sops reads your .sops.yaml to find the encryption key (age public key)

4. sops encrypts the values, producing:

PDS_ADMIN_PASSWORD: ENC[AES256_GCM,data:abc123...,iv:...,tag:...]
PDS_JWT_SECRET: ENC[AES256_GCM,data:def456...,iv:...,tag:...]
sops:
    age:
        - recipient: age1...
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            ...
}}}

5. The file now has encrypted values but plaintext keys. You commit this to git.

=== Phase 2: Configuration (in your Nix files) ===

You write Nix configuration that references these secrets:

sops.defaultSopsFile = ./secrets/bluesky-pds.yaml;

sops.secrets.PDSADMINPASSWORD = {};

sops.templates."pds-env".content = ''

PDSADMINPASSWORD=${config.sops.placeholder.PDSADMINPASSWORD}

'';

}}}

This goes in the Nix store when you build. Note: The actual secret values are NOT in this file—just references to them.

Phase 3: Building (on build machine)

When you run nixos-rebuild:

1. Nix evaluates your configuration

2. It sees the sops configuration

3. It generates activation scripts that will handle secrets

4. The configuration (without secret values) is built into the system closure

Phase 4: Activation (on target machine)

When the system activates:

1. sops-nix activation script runs

2. It reads the encrypted YAML from /etc/nixos/secrets/ (outside the store)

3. It uses the age private key (derived from SSH host key) to decrypt

4. It writes plaintext secrets to /run/secrets/

5. It sets ownership and permissions as configured

6. It creates template files with secrets interpolated

Now your services can read secrets from /run/secrets/.

Phase 5: Runtime (services using secrets)

Your PDS systemd service has:

environmentFiles = [ config.sops.templates."pds-env".path ];
}}}

When systemd starts the PDS:
1. It loads environment variables from the file at that path
2. The file contains the actual secret values
3. The PDS process has access to secrets via environment variables
4. The file is in `/run/secrets/` which is only readable by root (or configured owner)

== The .sops.yaml Configuration ==

The `.sops.yaml` file in your secrets directory tells sops how to encrypt files:

keys:

creation_rules:

key_groups:

}}}

Keys section: Defines named keys. Here, *snek refers to your server's age public key.

Creation rules: Tells sops which keys to use for which files. The regex secrets/.\.yaml$ matches all YAML files in the secrets directory. Files matching this pattern are encrypted for the snek key.

Why YAML anchors (& and ): These are YAML features. &snek defines an anchor, snek references it. This lets you define the key once and reuse it.

SSH Key Integration

sops-nix can derive age keys from SSH keys. This is convenient because:

Your configuration:

sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
}}}

This tells sops-nix to convert your SSH Ed25519 host key to an age key for decryption.

*Security consideration:* Your SSH host key is now also your secrets key. If someone steals your SSH host key, they can decrypt your secrets. This is usually fine—SSH host keys are already highly protected—but be aware of the coupling.

== The /run/secrets Directory ==

`/run` is a tmpfs mount—data stored there is in RAM, not on disk. This is crucial for secrets:

*Advantages:*
- Secrets never touch disk unencrypted
- Secrets disappear on reboot (they're re-decrypted on next activation)
- Fast access (RAM vs disk)
- No cleanup needed—reboot clears everything

*Disadvantages:*
- Secrets don't persist across reboots (but they're re-decrypted automatically)
- Uses RAM (though secrets are typically small)
- If you accidentally delete `/run/secrets`, you need to reactivate to get them back

The directory is created by systemd-tmpfiles at boot and populated by sops-nix during activation.

== Permissions and Ownership ==

By default, secrets in `/run/secrets/` are:
- Owner: root
- Group: root
- Permissions: 0400 (read-only by owner)

You can customize this per-secret:

sops.secrets.PDSJWTSECRET = {

owner = "pds";

group = "pds";

mode = "0400";

};

}}}

This creates the secret file owned by the pds user, so the PDS service (which runs as that user) can read it directly.

Templates: Secrets in Configuration Files

Sometimes you need secrets embedded in configuration files, not as separate files. Templates handle this.

Example—Environment file:

sops.templates."pds-env".content = ''
  PDS_JWT_SECRET=${config.sops.placeholder.PDS_JWT_SECRET}
  PDS_ADMIN_PASSWORD=${config.sops.placeholder.PDS_ADMIN_PASSWORD}
'';
}}}

*How it works:*
- At build time, `config.sops.placeholder.PDS_JWT_SECRET` is a placeholder string, not the actual secret
- The template content (with placeholders) is stored in the Nix store
- At activation time, sops-nix:
  1. Decrypts the secrets
  2. Replaces placeholders with actual values
  3. Writes the result to `/run/secrets/pds-env`

This separation is crucial: the Nix store never contains secret values, only placeholders.

== Secret Rotation ==

When you need to change a secret (compromise suspected, routine rotation, etc.):

1. Decrypt the file: `sops secrets/bluesky-pds.yaml`
2. Edit the secret value
3. Save (sops re-encrypts automatically)
4. Commit the change
5. Deploy: `nixos-rebuild switch`
6. The service gets the new secret on restart

*Important:* Some services need to be restarted to pick up new secrets. Systemd services with `environmentFiles` don't automatically reload when the file changes—they need a restart.

== Backup and Recovery ==

*Backing up secrets:*
- The encrypted files in git ARE your backup
- You can also back up the age private key (derived from SSH key)
- Store backups securely—they can decrypt your secrets

*Disaster recovery:*
1. If you lose the encrypted files: Restore from git
2. If you lose the private key: You can't decrypt secrets. Prevention: have multiple recipients in `.sops.yaml`
3. If you lose both: Secrets are lost, you'll need to regenerate them

*Recommendation:* Have at least two age keys that can decrypt—your server's key and a backup key stored securely offline.

== Security Best Practices ==

*Do:*
- Keep `.sops.yaml` in git
- Use strong, unique secrets
- Rotate secrets regularly
- Limit which secrets each service can access (principle of least privilege)
- Monitor access to `/run/secrets/`
- Use different keys for different environments (dev, staging, prod)

*Don't:*
- Commit unencrypted secrets to git (even in old commits—use git-filter-repo to remove)
- Share your age private key
- Use weak passwords or predictable secrets
- Give services access to secrets they don't need
- Store secrets in environment variables that might be logged

== Common sops-nix Patterns ==

*Single secret file:*

sops.defaultSopsFile = ./secrets.yaml;

sops.secrets.mySecret = {};

}}}

Multiple secret files:

sops.secrets.databasePassword = {
  sopsFile = ./secrets/database.yaml;
};
sops.secrets.apiKey = {
  sopsFile = ./secrets/external-services.yaml;
};
}}}

*Secret with custom permissions:*

sops.secrets.databasePassword = {

owner = "postgres";

group = "postgres";

mode = "0440"; # Readable by owner and group

};

}}}

Template with multiple secrets:

sops.templates."service-config".content = ''
  DB_PASSWORD=${config.sops.placeholder.databasePassword}
  API_KEY=${config.sops.placeholder.apiKey}
  SECRET_KEY=${config.sops.placeholder.secretKey}
'';
}}}

*Binary secret (not YAML/JSON):*

sops.secrets.tlsKey = {

format = "binary";

sopsFile = ./secrets/tls.key;

};

}}}

Troubleshooting

"sops: error: cannot decrypt data key":

"Permission denied" reading secret:

"Secret not found" in template:

"File not found" during activation:

Secret value appears in Nix store:

The Mental Model

Think of sops-nix as a bridge between:

The bridge is the activation process: when your system boots or you deploy, sops-nix decrypts and places secrets where services need them. The rest of the time, secrets stay encrypted.

This architecture lets you have the benefits of version control and reproducibility without the risks of exposing secrets.