Secrets — Vault / KMS

Manage a Krypton node's non-consensus secrets without a plaintext file on disk-in-git. The headline secret is the Engine-API JWT; this artifact also provides encrypted-at-rest custody for Horcrux's key shards / ECIES keys. Two interchangeable backends are documented:

• HashiCorp Vault (KV v2 + Vault Agent) — the recommended path.
• AWS KMS (envelope-encrypted blob, or Secrets Manager + a KMS CMK) — the cloud-native alternative.

The deploy artifacts are in l1/from-source/deploy/secrets/. This page is the operator-facing version of that directory's README.md.

Vault/KMS store — Horcrux signs

This artifact is the encrypted-at-rest custodian of bytes (the JWT, env-secrets, and Horcrux's shards/ECIES keys). It does not perform consensus-key signing — that is Key management (Horcrux). Do not put signing logic here, and do not put JWT/env handling in Horcrux.

Deploy artifact — not yet run on real infra

These templates have not been stood up against a live Vault/KMS in this repo, and the public testnet is not live yet. bash -n syntax-checks the scripts; the .hcl/.ctmpl/policy files are unrendered templates with <PLACEHOLDER> values. Validate in staging before production.

What the JWT is (and why it is lower-stakes)

The Engine-API JWT is the HS256 shared secret authenticating the local CL↔EL authrpc channel (http://el:8551). It is not the validator consensus key:

• Leaking the JWT lets an attacker who can already reach :8551 (CL-only, never published) drive the local Engine API. It does not let them sign blocks or equivocate — that needs the consensus key.
• The consensus key (priv_validator_key.json) is the crown jewel; leaking it enables double-signing → slashing. That key is Horcrux's job.

So this artifact hardens the rest — a real supply-chain / disk-hygiene win (no plaintext secret committed or persisted), not the consensus-security centrepiece.

Today the JWT is openssl rand -hex 32 written to a plaintext ${JWT_PATH} (0600) and bind-mounted into both EL and CL. That plaintext file is what this artifact removes.

Vault (recommended)

KV layout

KV v2 mount secret/, one sub-tree per network (<network> = chain id; 473374 testnet, 47337 mainnet):

Path	Holds	Read by
secret/krypton/<network>/engine-jwt	hex=<64 hex chars>	node's Vault Agent → renders /jwt/jwt.hex
secret/krypton/<network>/env	misc node env-secrets	node bootstrap (optional template)
secret/krypton/<network>/horcrux/<cosigner>/ecies	cosigner ECIES keypair	that cosigner host, at provisioning
secret/krypton/<network>/horcrux/<cosigner>/shard	that cosigner's Ed25519 key shard	that cosigner host, at provisioning

Write the secrets once from an authenticated admin workstation:

NET=473374
# Engine JWT — generate the 64 hex chars and store them; never write to disk.
vault kv put "secret/krypton/${NET}/engine-jwt" hex="$(openssl rand -hex 32)"

# Horcrux at-rest material — store the files Horcrux key-gen produced (Horcrux SIGNS
# with these; Vault only stores them encrypted at rest).
vault kv put "secret/krypton/${NET}/horcrux/cosigner-1/ecies" json=@cosigner-1/ecies_keys.json
vault kv put "secret/krypton/${NET}/horcrux/cosigner-1/shard" json=@cosigner-1/${NET}_priv_validator_key.json

KV v2 path forms

KV v2 nests user data under .Data.data and exposes it at the secret/data/... API path (vs the secret/... CLI path). The template and policy use the secret/data/... / secret/metadata/... API form.

Vault Agent: render the JWT to tmpfs

The node runs a Vault Agent (vault/vault-agent.hcl) that authenticates, then renders vault/jwt.ctmpl to a tmpfs path. The node bind-mounts that file read-only to /jwt/jwt.hex — the exact path the EL (--authrpc.jwtsecret) and CL (--beacon-kit.engine.jwt-secret-path) already expect, so neither image changes.

Vault (KV v2) ──auth (AppRole/k8s)──▶ Vault Agent ──render──▶ /run/krypton-secrets/jwt.hex (tmpfs, 0600)
                                                                       │ bind-mount :ro
                                                                       ▼
                                                       EL /jwt/jwt.hex   CL /jwt/jwt.hex

The plaintext JWT exists only in Vault (encrypted at rest by Vault's seal) and in RAM (tmpfs) on the node — never in git, never on persistent disk.

# tmpfs mount (RAM-backed; gone on reboot, re-rendered by the Agent):
sudo mkdir -p /run/krypton-secrets
sudo mount -t tmpfs -o size=1m,mode=0700 tmpfs /run/krypton-secrets

Auth wiring (pick one; both are in vault-agent.hcl):

• AppRole (VMs / compose / baremetal) — role_id (non-secret) on the host; a short-lived, response-wrapped secret_id delivered out-of-band and consumed once (remove_secret_id_file_after_reading).
• Kubernetes (sidecar) — bind a role to the pod's ServiceAccount; the Agent authenticates with the projected SA token — no static secret_id at all.

The Agent auto-renews its token and re-authenticates on expiry (no cron). Set a short token_ttl with a longer token_max_ttl; static_secret_render_interval (5m in the template) controls how often KV is re-read so a rotated JWT propagates.

Least-privilege policy

vault/policy.hcl grants the node read-only on its own engine-jwt and its own horcrux/<cosigner>/* — no write, no other cosigner's shard:

vault policy write krypton-node-473374 policy.hcl

It deliberately grants no create/update/delete — secrets are written by an operator/CI role, never by the node — and no signing capability (Horcrux signs from the shard it loads).

AWS KMS alternative

Store the JWT as a KMS-encrypted blob (envelope encryption under a customer-managed CMK) and decrypt at boot into tmpfs.

# Admin workstation, once: generate + encrypt under the CMK.
KMS_KEY_ID=alias/krypton-engine-jwt AWS_REGION=us-east-1 \
  ./kms/encrypt-jwt.sh /opt/krypton/secrets/jwt.hex.enc

# Node boot (e.g. systemd ExecStartPre), before `docker compose up`:
AWS_REGION=us-east-1 \
  ./kms/decrypt-jwt.sh /opt/krypton/secrets/jwt.hex.enc /run/krypton-secrets/jwt.hex
# then point JWT_PATH=/run/krypton-secrets/jwt.hex

Both scripts pin an encryption context (purpose=krypton-engine-jwt); KMS refuses to decrypt unless the same context is supplied — defence against a blob being decrypted out of its intended purpose. The node's IAM role needs only kms:Decrypt; the admin needs kms:Encrypt / GenerateDataKey.

Alternatively, use AWS Secrets Manager + a KMS CMK for rotation hooks / CloudTrail audit without managing blobs yourself:

aws secretsmanager create-secret --name krypton/473374/engine-jwt \
  --kms-key-id alias/krypton-engine-jwt \
  --secret-string "$(openssl rand -hex 32)"

KMS for Horcrux shards. The same CMK envelope-encrypts each cosigner's ecies_keys.json and <network>_priv_validator_key.json into blobs; ship the blob to the matching cosigner host and aws kms decrypt it into that cosigner's tmpfs config dir at provisioning. Horcrux then loads it and signs — KMS never sees the signing operation.

Migration from bootstrap.sh's local JWT

bootstrap.sh step 3 currently writes openssl rand -hex 32 to a plaintext file. Migrate once (rotating the JWT just needs both EL and CL restarted together):

1. Capture into the backend — mint a fresh value (treat the old plaintext as compromised once it lived on disk): vault kv put .../engine-jwt hex="$(openssl rand -hex 32)" or ./kms/encrypt-jwt.sh ....
2. Point the node at the rendered/decrypted secret — set JWT_PATH to the tmpfs path (/run/krypton-secrets/jwt.hex). The compose mount ${JWT_PATH}:/jwt/jwt.hex:ro is unchanged.
3. Stop generating plaintext — skip bootstrap.sh step 3. (The shared deploy docs own that edit; do not modify bootstrap.sh from here.)
4. Shred the old plaintext and confirm it is not tracked in git.

Rotation

Write a new value to Vault/KMS, let the Agent re-render (or re-run decrypt-jwt.sh), then docker compose restart el cl together so both sides pick up the same new secret. A mismatch surfaces as engine-auth 401s.

Security model & honest caveats

What this gives you: no plaintext Engine JWT in git or on persistent disk (Vault sealed at rest / KMS CMK-encrypted, only ever in tmpfs RAM); least-privilege, network-scoped read access; a single audited store (Vault audit log / CloudTrail); encrypted-at-rest custody for Horcrux's shards/ECIES keys.

Honest caveats

• tmpfs ≠ unreadable. A root attacker on the node can read tmpfs and Agent memory. Vault/KMS reduce at-rest and in-git exposure plus give rotation + audit; they do not defend a fully-compromised host.
• AppRole secret_id is itself a secret — short-lived, delivered out-of-band (response-wrapped). Baking it into an image re-introduces the problem. The k8s auth method avoids a static secret_id entirely.
• Unseal/CMK = the new root of trust. Vault's unseal keys / the KMS CMK now gate every node secret — protect them (auto-unseal via a cloud KMS, MFA on the CMK key policy, separate break-glass custody).
• The JWT is lower-stakes than the consensus key. Prioritise Horcrux — that is the crown jewel.
• No API was invented here — all commands are real vault, vault agent, and aws kms / aws secretsmanager invocations; verify against the installed versions.

See also

• Key management (Horcrux) — the consensus-key signer this artifact stores material for
• Validator node · RPC / full node (L5) — the nodes whose Engine JWT this renders
• Quick start — where the plaintext JWT currently comes from
• Networks & chain IDs — <network> = 473374 / 47337