> ## Documentation Index
> Fetch the complete documentation index at: https://docs.algovoi.co.uk/llms.txt
> Use this file to discover all available pages before exploring further.

# Disaster Recovery and Key Ceremony

> HA and DR runbook for the self hosted Verifiable Compliance Suite: recovery objectives, backup scope, restore and verify, and witnessed key ceremonies.

<Note>
  **Part of the on-prem suite.** This runbook applies to the self-hosted [Verifiable Compliance Suite](/verifiable-compliance-suite). The signing steps use `algovoi-key-ceremony`, which ships in the bundle. Every attestation it produces verifies offline against your console public key, so your continuity evidence is proven, not asserted.
</Note>

## Objectives

Because the suite runs on your own infrastructure, you set the targets; these are the defaults we design and test against for a single-node deployment with nightly off-site backups.

| Objective            | Target                                              | Basis                                                                                             |
| -------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| RTO (recovery time)  | 4 hours single node, 30 minutes with a warm standby | Reinstall the bundle offline, restore the database and archive, restart services                  |
| RPO (recovery point) | 24 hours default, 1 hour with hourly database dumps | Backup cadence you choose; the archive is write-once, so no in-place data is lost between backups |
| Key recovery         | k-of-n custodians available                         | Recovery Vault reconstructs a lost key from any k of n shares with no vendor in the loop          |

RTO and RPO are properties of your backup cadence and standby posture, not of the software. Raise the cadence to lower the RPO; keep a warm standby to lower the RTO.

## What to back up

The suite state is a small, well-defined set. Everything below is captured by one signed backup manifest.

| Item                  | Path or setting                                                   | Why it matters                                                                   |
| --------------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| Console database      | `COMMAND_CENTER_DB` (SQLite file, or PostgreSQL via `pg_dump`)    | Signed audit chain, user accounts and two-factor secrets, verified-content index |
| Records Vault archive | `RV_ARCHIVE_DIR` (default `./records-vault-archive`)              | All documents, receipts, access logs, legal holds, redaction tombstones          |
| Console signing key   | `COMMAND_CENTER_KEY_FILE` (or the KMS/Vault/PKCS#11 wrapped blob) | Signs the console audit chain; required to continue the chain after recovery     |
| Archive signing key   | `RV_ARCHIVE_KEY_FILE` (default `<archive>/archive-key.json`)      | Signs every Records Vault evidence stream                                        |
| At-rest archive key   | `RV_ARCHIVE_AES_KEY_B64` (if set)                                 | Decrypts the archive at rest                                                     |
| Recovery Vault file   | `VAULT_PATH`                                                      | The Shamir-split envelopes protecting your keys                                  |

<Warning>
  The signing keys are the catastrophic single points. Keep them under a real custody mode (KMS, Vault, or PKCS#11 HSM) or split them with Recovery Vault, and never store the plaintext key beside the database backup. See [Bring your own keys](/bring-your-own-keys).
</Warning>

## Back up, then prove the backup

A backup you have not restored is a hope, not a control. Capture the set and sign a manifest that pins every file by SHA-256:

```bash theme={null}
# 1) database (PostgreSQL shown; for SQLite just copy the COMMAND_CENTER_DB file)
docker exec algovoi-postgres-1 pg_dump -U algovoi_control_plane -d algovoi_control_plane -Fc > /backup/console.dump

# 2) sign a manifest over the whole backup set with your console key
algovoi-key-ceremony backup-manifest \
  --item /backup/console.dump \
  --item "$RV_ARCHIVE_DIR" \
  --item "$VAULT_PATH" \
  --label nightly --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --base / --key "$COMMAND_CENTER_KEY_FILE" --out /backup/manifest
```

Ship `console.dump`, the archive, the vault file, and `manifest/` off-site. Keep the signing keys on a separate custody path.

## Restore and verify

On the recovery host, reinstall the bundle offline (`python algovoi_unbundle.py --license @licence.key --out ./algovoi --install bundle.algv`), restore the data, then prove byte-parity before you trust it:

```bash theme={null}
# restore the database
pg_restore -U algovoi_control_plane -d algovoi_control_plane --clean /backup/console.dump
# restore the archive + vault file to their original paths, then verify against the signed manifest
algovoi-key-ceremony verify-restore \
  --manifest /backup/manifest/backup-manifest.json --root / \
  --pub-key "$COMMAND_CENTER_KEY_FILE" --sig /backup/manifest/backup-manifest.sig.json
```

A `VERIFY-RESTORE: PASS` line means the manifest signature is valid and every restored file matches its recorded digest. The console re-verifies its own audit chain on start; a green posture confirms the chain survived the restore intact.

## Key ceremonies

Every sensitive key event should be witnessed and recorded. `algovoi-key-ceremony` turns each into a Falcon-1024-signed attestation, hash-linked to the previous one, so the whole custody history is one tamper-evident chain a reviewer replays offline.

<Steps>
  <Step title="Genesis">
    Record the generation of a signing key, naming the operator and witnesses and pinning the new key's fingerprint (kid).

    ```bash theme={null}
    algovoi-key-ceremony attest --id g1 --type genesis --target console \
      --operator "Alice (CISO)" --witness "Bob (Security)" \
      --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
      --key "$COMMAND_CENTER_KEY_FILE" --target-key "$COMMAND_CENTER_KEY_FILE" --out ./ceremony
    ```
  </Step>

  <Step title="Split into k-of-n custody">
    Split the key with [Recovery Vault](/recovery-vault), then attest to the parameters and the custodian labels. The shares themselves are never written into the evidence.

    ```bash theme={null}
    algovoi-key-ceremony attest --id s1 --type split --target console \
      --operator "Alice (CISO)" --shamir "2,3,alice:bob:carol" \
      --witness "Bob (Security)" --witness "Carol (Legal)" \
      --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
      --key "$COMMAND_CENTER_KEY_FILE" --prev ./ceremony/g1.attestation.json --out ./ceremony
    ```
  </Step>

  <Step title="Rotation">
    On schedule or after a suspected exposure, generate a fresh key. Retain the old public key so historical signatures still verify; new writes use the new key. Attest with `--type rotation`.
  </Step>

  <Step title="Recovery drill">
    Periodically prove the split works: reconstruct the key from k shares in a clean room, confirm the recovered fingerprint matches, then record the drill and its outcome with `--type recovery_drill`. This is the evidence an auditor asks for.
  </Step>

  <Step title="Disposal">
    When old key material is securely destroyed, record it with `--type disposal` so the lifecycle is closed out on the chain.
  </Step>
</Steps>

Verify the whole custody chain at any time, offline:

```bash theme={null}
algovoi-key-ceremony verify --pub-key "$COMMAND_CENTER_KEY_FILE" --chain ./ceremony
# CHAIN: PASS (N records) — every record signed, every link intact
```

## Failure-mode runbook

| Scenario                    | Action                                                                                      | Target                 |
| --------------------------- | ------------------------------------------------------------------------------------------- | ---------------------- |
| Node lost                   | Reinstall bundle offline, restore last backup, `verify-restore`, restart services           | Within RTO             |
| Database corrupted          | Restore the last good dump, let the console re-verify its audit chain                       | 1 hour                 |
| Console or archive key lost | Reconstruct from Recovery Vault shares (k custodians), attest a `recovery_drill`, resume    | k custodians available |
| KMS or Vault access lost    | Fail over to a second custody endpoint, or recover the key from Recovery Vault shares       | Within RTO             |
| A custodian leaves          | `resplit` in Recovery Vault to revoke the old shares and issue new ones, attest the change  | Same day               |
| Suspected key exposure      | Rotate the signing key, attest `rotation`, then `disposal` of the old material once retired | Same day               |

## Verify everything offline

Nothing in this runbook needs a network connection or a call to AlgoVoi. Backups verify against a manifest you signed, the audit chain re-verifies on start, and the custody chain replays under your own public key. Recovery Vault reconstructs keys from your custodians' shares alone. That is the whole point: continuity you can prove to a regulator without depending on a vendor being reachable.