> For the complete documentation index, see [llms.txt](https://services.validexis.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://services.validexis.com/celestia-infrastructure/key-custody-and-signer-change.md).

# 🔑 Key Custody and Signer Change

A validator can be slashed for double signing if two signers use the same consensus key, or if a signer loses its record of what it already signed. This page describes how VALIDEXIS holds Celestia consensus keys and how we move signing between servers without either risk.

It applies to the Mainnet and Mocha signers described in [TMKMS for Celestia v10 and Fibre](https://services.validexis.com/validator-security-our-approach-and-protection-measures/tmkms-for-remote-signing/tmkms-for-celestia-v10-and-fibre).

### Key custody

| Rule                                 | How it is enforced                                                                          |
| ------------------------------------ | ------------------------------------------------------------------------------------------- |
| One online copy per consensus key    | The key exists online only on its signer, inside a LUKS-encrypted volume                    |
| One encrypted offline backup         | Kept off any server, controlled by the responsible operator                                 |
| No key on validator or Fibre servers | Validators have no `priv_validator_key.json` in use; Fibre signs through the validator      |
| No key in transit over the internet  | Keys move only during a signer change, encrypted, over the private network or offline media |
| Passphrases never on servers         | The LUKS volume is unlocked by an operator; nothing is stored for auto-unlock               |
| Least access                         | Signer accepts key-only SSH from admin addresses; no other services or users                |

### What "signing state" means

TMKMS records the last height, round and step it signed in its `state_file`. Before signing anything new, it checks the request against that record and refuses to sign at a lower position. This is the protection against signing two different blocks at the same height.

Consequences:

* Moving only the key to a new server **without** its state removes that protection.
* Deleting or resetting the state file has the same effect.
* The exact state that must travel with the key depends on the signer implementation and version in use. We pin the TMKMS version and follow the state format of that version.

### Signer change procedure

Used for hardware replacement, server migration, or a TMKMS upgrade that requires a new instance. Always rehearsed on Mocha first.

```
1 FENCE  →  2 CARRY STATE  →  3 REVIEW  →  4 RESUME
```

**1. Fence the current signer.**

* Stop TMKMS and disable the unit so it cannot restart: `systemctl disable --now tmkms`.
* Block its connection to the validator at the firewall.
* Confirm on the validator that the signer is disconnected and that no other signer is connected.

The validator misses blocks during the change. Missing blocks is acceptable; double signing is not.

**2. Carry the complete signing state.**

* Copy the key **and** the full signing state required by the pinned TMKMS version to the new signer's encrypted volume.
* Record the last signed height/round/step from the old state file and compare it with the copy.
* Do not use chain height alone to rebuild the state.

**3. Second-operator review.**

A second operator independently confirms, before anything starts:

* the old signer is stopped, disabled and firewalled;
* the state on the new signer matches the old record;
* the new `tmkms.toml` points to the right chain and the right validator;
* only one signer will be able to reach the validator.

**4. Resume signing.**

* Start TMKMS on the new signer and watch the first signatures in its log.
* Check `cometbft_privval_signing_latency_median_seconds` and `cometbft_privval_signing_failures_total` on the validator.
* Keep the old signer disabled until its volume is wiped or the server is decommissioned.

### When state is missing or inconsistent

If the state file is missing, unreadable, or does not match the last known record, **signing stays disabled**. We do not guess the position from block height or restart with an empty state. Signing resumes only after the recovery procedure for the specific TMKMS version has been completed and independently reviewed by a second operator.

TMKMS options such as `double_sign_check_height` are treated as additional checks, never as a replacement for correct state.

### Restoring from the offline backup

The offline backup restores the **key**, not the signing state. A restore therefore always follows the full procedure above, including fencing every other signer and the state review. A restored key is never started next to a running signer.

### Sources

* [TMKMS](https://github.com/iqlusioninc/tmkms) — `state_file` and double-sign protection
* [celestia-app v10.4.0 release notes — KMS risks](https://github.com/celestiaorg/celestia-app/blob/v10.4.0/docs/release-notes/release-notes.md#key-management-systems-kms)
