> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.itential.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.itential.com/_mcp/server. # Administer etcd > Routine administration and maintenance procedures for an etcd cluster used as the shared store backend for Itential Gateway, including health checks, backup and restore, compaction, and TLS certificate rotation. An etcd cluster serves as the shared store backend for Itential Gateway when running distributed execution or active-standby high availability. Gateway requires etcd v3.5, the version these procedures target. Examples assume a three-node cluster, the smallest topology that tolerates a single-node failure while preserving write quorum. Procedures that modify state, restart processes, or alter on-disk data are marked with a warning. ## Before you begin The commands here assume the following environment variables are set: ```bash ETCD_NODES=" " ETCD_CLIENT_PORT=2379 ETCD_PEER_PORT=2380 ETCD_ENDPOINTS="https://:2379,https://:2379,https://:2379" SSH_KEY_PATH=~/.ssh/.pem ETCD_SSH_USER= export ETCDCTL_API=3 ETCDCTL_TLS="--cacert=/etc/etcd/pki/ca.crt" ``` If TLS isn't enabled in your deployment, set `ETCDCTL_TLS=""` and use `http://` URLs in `ETCD_ENDPOINTS`. A non-TLS etcd deployment isn't recommended for any environment that isn't air-gapped. `ETCDCTL_TLS` above assumes transport TLS only, on both the client and peer planes: etcd nodes present certificates but don't require clients or peers to present client certificates back. In etcd terms, `--cert-file` / `--key-file` and `--peer-cert-file` / `--peer-key-file` are set, but neither `--client-cert-auth` nor `--peer-client-cert-auth` is enabled. If your deployment uses full mTLS (`--client-cert-auth=true`), add `--cert` and `--key` flags to `ETCDCTL_TLS`, pointing at a client certificate and key trusted by the cluster's CA. Reference: [etcd v3.5 transport security](https://etcd.io/docs/v3.5/op-guide/security/). TLS private keys must never be stored in version control or distributed outside your secrets management process. In a 3-node etcd cluster, two members must remain healthy for the cluster to accept writes. Never restart, stop, or otherwise take offline more than one node at a time. A second failure during a planned outage breaks quorum and makes the cluster read-only or unavailable until a member returns. ## Deployment types RPM/DEB and container deployments differ mainly in how the process is controlled. The underlying `etcdctl` and `etcdutl` commands are identical either way. | Shape | Process control | Typical paths | | -------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | **RPM/DEB package** | `systemctl` against the `etcd` unit | Data: `/var/lib/etcd`, certs: `/etc/etcd/pki/`, config: `/etc/etcd/etcd.conf.yml`, user: `etcd:etcd` | | **Container (Docker / Compose)** | `docker` or `docker compose` against the etcd container(s) | Paths inside the container; the host-mounted volume defines data and cert locations | Confirm the actual paths against your deployment before running any command. etcd's documented data directory default is `${name}.etcd` relative to the working directory. `/var/lib/etcd` is a package convention, not an etcd default. ## Check cluster health and status etcd is a Raft-based consensus system. As long as a majority of members are healthy and reachable, the cluster accepts writes. Regular health checks confirm every member is reachable, the leader is stable, and no follower is lagging far enough behind to compromise availability if the leader fails. ### Check cluster-wide health `etcdctl endpoint health` contacts every endpoint, measures round-trip time, and reports whether each member would accept a write: ```bash etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ endpoint health --write-out=table ``` Every endpoint should report `true` under `HEALTH` with low `TOOK` latency. In v3.5, etcd also exposes HTTP `/health`, `/livez`, and `/readyz` endpoints at the address set by `--listen-metrics-urls`, useful for external monitoring. This guide uses `etcdctl endpoint health` for operator-facing checks. Reference: [etcd v3.5 monitoring](https://etcd.io/docs/v3.5/op-guide/monitoring/). ### Check per-endpoint status `etcdctl endpoint status` reveals which member holds the leader role, each member's Raft term and index, and database size on disk: ```bash etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ endpoint status --write-out=table ``` | Column | Expected value | | ------------ | ------------------------------------------------ | | `IS LEADER` | Exactly one member is `true`; the others `false` | | `RAFT TERM` | Same value across all members | | `RAFT INDEX` | Followers within a few thousand of the leader | | `DB SIZE` | Within the configured `--quota-backend-bytes` | | `ERRORS` | Empty | The default `--quota-backend-bytes` is `0`, which etcd interprets as a low space quota. Confirm the configured value before relying on absolute numbers; a `DB SIZE` approaching the quota is the most common cause of cluster read-only events. ### List cluster members and the current leader ```bash etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ member list --write-out=table etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ endpoint status --write-out=json \ | jq -r '.[] | select(.Status.leader == .Status.header.member_id) | .Endpoint' ``` Each member-list row shows the member ID, name, peer URL, client URL, and whether it's a learner (a non-voting replica being added). No member should be a learner in a steady-state cluster. Resolve the leader at runtime before any operation that will restart a node, so you can plan to handle it last. ### Verify read and write availability ```bash etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS put /healthcheck/$(date +%s) ok etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS get /healthcheck/ --prefix --keys-only | tail -5 ``` This exercises the full Raft commit path. Health-check keys accumulate, so clean them up periodically. ## Manage the etcd service In a 3-node cluster, restarting a follower is low-risk; quorum holds and the cluster keeps serving. Restarting the leader briefly pauses writes during the election that follows. Restarting two members at once breaks quorum. ### Check status on all nodes ```bash # RPM/DEB for node in $ETCD_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node 'sudo systemctl status etcd' 2>/dev/null done # Container for node in $ETCD_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node \ 'docker ps --filter name=etcd --format "table {{.Names}}\t{{.Status}}"' 2>/dev/null done ``` An etcd process that's crashed but hasn't been restarted reduces the cluster to two members without raising an alarm at the application layer until the next Gateway read fails. ### Restart on a single node Restarting the leader triggers an election that pauses writes for under a second. If you must restart the leader, prefer to step leadership down first (see [Shut down etcd safely](#shut-down-etcd-safely)) so the restart targets a follower instead. ```bash # RPM/DEB ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ 'sudo systemctl restart etcd' # Container (adjust service name to match your compose file) ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ 'docker compose -f /path/to/docker-compose.yml restart etcd' ``` Verify the node rejoined and cluster-wide health, then confirm the rejoined member caught up to the leader's Raft index before touching another member: ```bash etcdctl --endpoints=https://:$ETCD_CLIENT_PORT $ETCDCTL_TLS endpoint health etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint status --write-out=table ``` The `RAFT INDEX` of the rejoined member should be within a few hundred of the leader's; a persistent gap warrants investigation in the etcd logs. ### Start and stop on a single node ```bash # RPM/DEB ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ 'sudo systemctl stop etcd' # or: start # Container ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ 'docker compose -f /path/to/docker-compose.yml stop etcd' # or: start ``` ### Enable etcd to start on boot (RPM/DEB) ```bash for node in $ETCD_NODES; do ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node 'sudo systemctl is-enabled etcd' 2>/dev/null done ``` Enable any node that returns `disabled`: ```bash ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ 'sudo systemctl enable etcd' ``` A single offline member in a 3-node cluster leaves zero redundancy; the loss of any other member breaks quorum. For container deployments, configure the runtime's restart policy (`restart: always` in Compose, or `--restart=always` for `docker run`) to achieve the same effect. ### Review the configuration etcd has no canonical default config-file path. | Deployment | Configuration source | | ---------- | --------------------------------------------------------------------------------------------------- | | RPM/DEB | `/etc/etcd/etcd.conf.yml` (YAML), or a systemd drop-in unit defining `ETCD_*` environment variables | | Container | The Compose file or container-runtime command line | Inspect what the running process actually sees: ```bash # RPM/DEB ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ 'sudo cat /etc/etcd/etcd.conf.yml 2>/dev/null; sudo systemctl cat etcd' # Container ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ 'docker inspect etcd | jq ".[0].Args, .[0].Config.Env"' ``` | Parameter | What to check | | ----------------------------------------------------------- | ----------------------------------------- | | `data-dir` | Path to the etcd data directory | | `name` | Member name (unique across nodes) | | `initial-cluster` | Comma-separated list of all peers | | `listen-peer-urls` / `listen-client-urls` | Network bindings | | `cert-file` / `key-file` / `trusted-ca-file` | Client-plane TLS files | | `peer-cert-file` / `peer-key-file` / `peer-trusted-ca-file` | Peer-plane TLS files | | `quota-backend-bytes` | Maximum DB size before writes are refused | | `auto-compaction-mode` / `auto-compaction-retention` | Keyspace compaction policy | References: [etcd v3.5 configuration](https://etcd.io/docs/v3.5/op-guide/configuration/), [clustering](https://etcd.io/docs/v3.5/op-guide/clustering/). ## Shut down etcd safely etcd has no documented way to quiesce a member before stopping it. A safe shutdown is a sequencing exercise: keep the rest of the cluster healthy and, optionally, move leadership off the target node first to avoid a brief election pause. #### Move the leader off the target node, if applicable Stopping the leader triggers an automatic election. Transferring leadership explicitly beforehand lets you control when the election happens and ensures the new leader is a member you've already verified is healthy. Skip this step if the target node isn't the leader. ```bash # Confirm whether the target node is the leader etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ endpoint status --write-out=json \ | jq -r '.[] | "\(.Endpoint) leader=\(.Status.leader == .Status.header.member_id)"' ``` If it is, transfer leadership to a healthy follower: ```bash NEW_LEADER_ID=$(etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ member list --write-out=json \ | jq -r '.members[] | select(.name != "") | .ID' | head -1) etcdctl --endpoints=https://:$ETCD_CLIENT_PORT $ETCDCTL_TLS \ move-leader $(printf '%x' $NEW_LEADER_ID) ``` Confirm leadership moved with `endpoint status --write-out=table`. #### Verify cluster quorum before the shutdown ```bash etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint health --write-out=table ``` Every member must report healthy. If the cluster is already running with one member unhealthy, taking a second offline breaks quorum; repair the unhealthy member first. #### Stop the service ```bash # RPM/DEB ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ 'sudo systemctl stop etcd' # Container ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ 'docker compose -f /path/to/docker-compose.yml stop etcd' ``` #### Verify the cluster remains healthy ```bash etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint health --write-out=table ``` The target endpoint reports `false` (expected; it's stopped). The other two must report `true`. If any other endpoint also reports `false`, the cluster has lost quorum; start the target node again immediately and investigate. #### Verify the process exited ```bash # RPM/DEB ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ 'sudo systemctl status etcd' # Container ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ 'docker ps -a --filter name=etcd' ``` Expect `inactive (dead)` (RPM) or `Exited` (container). #### Restart the node when maintenance is complete ```bash # RPM/DEB ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ 'sudo systemctl start etcd' # Container ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ 'docker compose -f /path/to/docker-compose.yml start etcd' ``` Verify it rejoins and catches up: ```bash etcdctl --endpoints=https://:$ETCD_CLIENT_PORT $ETCDCTL_TLS endpoint health etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint status --write-out=table ``` The rejoined member's `RAFT INDEX` should be within a few hundred of the leader's. A persistent gap warrants investigation. ## Back up and restore etcd etcd is the source of truth for Gateway cluster coordination state. A corrupted data directory, a lost majority of members, or an inadvertent destructive operation against the keyspace can require restoring from a snapshot. With a current snapshot, recovery is typically a single multi-node procedure; without one, the recovery path is rebuilding Gateway cluster state from scratch. References: [etcd v3.5 disaster recovery](https://etcd.io/docs/v3.5/op-guide/recovery/), [maintenance](https://etcd.io/docs/v3.5/op-guide/maintenance/). ### What to back up | Item | Why it's required | | --------------------------------------------------------------- | ----------------------------------------------------------- | | Snapshot file (`*.db`) | The keyspace contents at a point in time | | TLS certificates and keys | Required to start a restored cluster with the same identity | | Configuration (`etcd.conf.yml`, drop-in units, or Compose file) | Defines member names, peer URLs, and tuning parameters | A snapshot alone isn't sufficient if a host is also lost; without the TLS material, a restored cluster can't be reached by clients still configured against the old CA. Back up `/etc/etcd/` (or the equivalent host-mounted volume) alongside each snapshot. ### Take and verify a snapshot `etcdctl snapshot save` produces a consistent snapshot without blocking writes. Target a single endpoint, not a comma-separated list; any healthy member works, though targeting the leader produces the most up-to-date snapshot. ```bash SNAPSHOT_DIR=/var/backups/etcd SNAPSHOT_FILE=$SNAPSHOT_DIR/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ " sudo mkdir -p $SNAPSHOT_DIR sudo ETCDCTL_API=3 etcdctl \ --endpoints=https://:$ETCD_CLIENT_PORT \ --cacert=/etc/etcd/pki/ca.crt \ snapshot save $SNAPSHOT_FILE " ``` For container deployments, prefix the snapshot command with `docker exec ` and write the file inside the container's volume so it lands on the host filesystem. A snapshot that's corrupt or truncated fails at restore time, so verify immediately after creation, while the healthy cluster is still available to compare against. Snapshot inspection lives in `etcdutl`, the offline data-management binary shipped alongside `etcdctl`: ```bash ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ "sudo etcdutl --write-out=table snapshot status $SNAPSHOT_FILE" ``` A non-zero hash and key count confirm the snapshot is intact. Compare its revision against the live cluster's revision (from `endpoint status`) to confirm it captured recent state. ### Copy the snapshot off the host and schedule regular runs A snapshot stored only on the host that produced it doesn't protect against host loss: ```bash scp -i $SSH_KEY_PATH $ETCD_SSH_USER@:$SNAPSHOT_FILE // ``` Schedule a daily snapshot, verify it, copy it off-host, and prune local copies older than your retention window: ```bash # /etc/cron.d/etcd-snapshot 0 2 * * * root /usr/local/bin/etcd-snapshot.sh >> /var/log/etcd-snapshot.log 2>&1 ``` The wrapper script should run `snapshot save`, run `snapshot status` and exit non-zero on a zero hash or key count, copy the snapshot offsite and verify the copy, delete local snapshots past the retention window, and log success or failure. Pair the cron job with monitoring that alerts when the most recent successful snapshot exceeds your recovery point objective; a silently broken cron job is not a data protection strategy. ### Restore from a snapshot Restoring overwrites the data directory of every member and starts a new cluster from the snapshot contents. All keyspace history written after the snapshot is lost. Only restore when the running cluster is unrecoverable. A restore is a four-phase process: stop etcd everywhere, restore the data directory on each member, start etcd, and verify the new cluster. #### Stop etcd on all members Because the live cluster is unrecoverable in a restore scenario, the one-node-at-a-time quorum constraint doesn't apply here. ```bash for node in $ETCD_NODES; do ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node 'sudo systemctl stop etcd' done ``` #### Move the existing data directories aside `etcdutl snapshot restore` refuses to overwrite an existing data directory. Rename it rather than deleting it, so the original is recoverable if the restore fails. ```bash for node in $ETCD_NODES; do ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node \ "sudo mv /var/lib/etcd /var/lib/etcd.preRestore-$(date +%Y%m%d-%H%M%S)" done ``` #### Restore on each member Copy the snapshot to each member, then run `etcdutl snapshot restore` on each. `--initial-cluster` and `--initial-advertise-peer-urls` must reflect your actual topology and match across all three restore commands. ```bash SNAPSHOT_FILE=/tmp/etcd-snapshot.db INITIAL_CLUSTER="=https://:2380,=https://:2380,=https://:2380" RESTORE_TOKEN="etcd-restored-$(date +%Y%m%d)" for node in $ETCD_NODES; do scp -i $SSH_KEY_PATH //etcd-snapshot.db \ $ETCD_SSH_USER@$node:$SNAPSHOT_FILE ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node " sudo etcdutl snapshot restore $SNAPSHOT_FILE \ --name=$node \ --initial-cluster='$INITIAL_CLUSTER' \ --initial-cluster-token=$RESTORE_TOKEN \ --initial-advertise-peer-urls=https://$node:2380 \ --data-dir=/var/lib/etcd sudo chown -R etcd:etcd /var/lib/etcd " done ``` The `--initial-cluster-token` value is arbitrary but must match on all three members. Use a fresh value on every restore so the restored cluster doesn't collide with any pre-existing peer state. #### Start etcd on all members ```bash for node in $ETCD_NODES; do ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node 'sudo systemctl start etcd' done ``` #### Verify the restored cluster ```bash etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint health --write-out=table etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS member list --write-out=table etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS get / --prefix --keys-only | head ``` Once verified, restart any Gateway nodes that were connected to the previous cluster so they re-establish their etcd sessions against the restored data. ## Compact and defragment the keyspace etcd retains every revision of every key by default. For a Gateway store backend, that history is unused (Gateway reads current values, never historical ones), so accumulating revisions only grows disk usage. Without compaction, the database eventually hits `--quota-backend-bytes` and refuses writes. Compaction marks old revisions as free; defragmentation reclaims the freed space on disk. Both are needed. Reference: [etcd v3.5 maintenance](https://etcd.io/docs/v3.5/op-guide/maintenance/). ### Configure auto-compaction Configure etcd to compact continuously, retaining one hour of revision history, a comfortable safety window for incident triage without letting the database bloat: ```bash --auto-compaction-mode=periodic --auto-compaction-retention=1h ``` Or in YAML: ```yaml auto-compaction-mode: periodic auto-compaction-retention: "1h" ``` Restart each member one at a time (see [Manage the etcd service](#manage-the-etcd-service)) to pick up the change, then confirm the configured value loaded: ```bash ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ 'sudo journalctl -u etcd --no-pager | grep -i "auto-compaction" | tail -5' ``` ### Run manual compaction If auto-compaction isn't configured yet, or the database grew faster than the auto-compaction interval can keep up with: ```bash CURRENT_REV=$(etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ endpoint status --write-out=json | jq -r '.[0].Status.header.revision') etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS compact $CURRENT_REV ``` ### Schedule periodic defragmentation Compaction frees revisions inside the bbolt backend file but doesn't shrink it; defragmentation rewrites the backend in place to reclaim physical disk space. It pauses requests on the targeted member while it runs, so target one member at a time, preferring followers before the leader. Schedule a weekly defrag during a low-activity window: ```bash # /etc/cron.d/etcd-defrag 0 3 * * 0 root /usr/local/bin/etcd-defrag.sh >> /var/log/etcd-defrag.log 2>&1 ``` ```bash #!/bin/bash set -euo pipefail ETCDCTL_API=3 ETCD_ENDPOINTS="https://:2379,https://:2379,https://:2379" ETCDCTL_TLS="--cacert=/etc/etcd/pki/ca.crt" LEADER=$(etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ endpoint status --write-out=json \ | jq -r '.[] | select(.Status.leader == .Status.header.member_id) | .Endpoint') for endpoint in $(echo $ETCD_ENDPOINTS | tr ',' ' '); do [ "$endpoint" = "$LEADER" ] && continue echo "=== defragmenting $endpoint ===" etcdctl --endpoints=$endpoint $ETCDCTL_TLS defrag sleep 30 done echo "=== defragmenting leader $LEADER ===" etcdctl --endpoints=$LEADER $ETCDCTL_TLS defrag ``` Confirm `DB SIZE` decreased on each member with `endpoint status --write-out=table` afterward. ### Recover from a quota alarm If etcd has already filled and refused writes, recovery is required even with auto-compaction configured: compact, defrag every member, then disarm the alarm. ```bash CURRENT_REV=$(etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS \ endpoint status --write-out=json | jq -r '.[0].Status.header.revision') etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS compact $CURRENT_REV for node in $ETCD_NODES; do echo "=== $node: defragmenting ===" etcdctl --endpoints=https://$node:$ETCD_CLIENT_PORT $ETCDCTL_TLS defrag done etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS alarm list etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS alarm disarm ``` ## Manage logs The etcd log records leader elections, member additions and removals, slow disk warnings, and any error that prevents a write from committing, the first place to look when the cluster behaves unexpectedly. Reference: [etcd v3.5 configuration logging flags](https://etcd.io/docs/v3.5/op-guide/configuration/). ### Log destinations etcd v3.5 supports two log destinations, controlled by `--log-outputs`: | Destination | Configured by | Inspection tool | Rotation | | -------------------------------- | -------------------------------------- | -------------------- | -------------------------------------------- | | journald (default under systemd) | `--log-outputs=default` (or unset) | `journalctl -u etcd` | journald (`SystemMaxUse`, `MaxRetentionSec`) | | File | `--log-outputs=/var/log/etcd/etcd.log` | `tail`, `grep` | etcd's built-in (`--enable-log-rotation`) | `stdout` and `stderr` are also valid `--log-outputs` values, typical in container deployments where the runtime captures the stream. Confirm the configured destination before relying on a specific path: ```bash ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ \ 'sudo systemctl cat etcd | grep -E "log-outputs|StandardOutput"' ``` ### View and search logs ```bash # journald journalctl -u etcd -n 100 --no-pager journalctl -u etcd -p err -n 200 --no-pager # File sudo tail -n 100 /var/log/etcd/etcd.log sudo grep -iE "ERROR|FATAL|PANIC" /var/log/etcd/etcd.log | tail -50 ``` Search across all nodes for the events most worth flagging: ```bash for node in $ETCD_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node \ 'sudo journalctl -u etcd --no-pager \ | grep -iE "elected leader|leader changed|lost leader|slow|took too long|database space exceeded|corrupt|panic|fatal" \ | tail -30' 2>/dev/null done ``` | Pattern | Meaning | | ----------------------------------- | ---------------------------------------------------- | | `elected leader` / `leader changed` | A leader election occurred | | `lost leader` | A follower lost contact with the leader | | `took too long` | A disk fsync or apply exceeded the warn threshold | | `database space exceeded` | The `--quota-backend-bytes` limit was hit | | `corrupt` | Database integrity check failed; restore from backup | | `panic` / `fatal` | The process died unexpectedly | ### Configure log rotation **journald (default):** rotation is controlled in `/etc/systemd/journald.conf`: ```ini [Journal] SystemMaxUse=2G SystemKeepFree=1G MaxRetentionSec=30day ``` Apply with `sudo systemctl restart systemd-journald`, and confirm with `journalctl --disk-usage`. **File output:** etcd v3.5 has built-in rotation when enabled: ```yaml log-outputs: /var/log/etcd/etcd.log enable-log-rotation: true log-rotation-config-json: '{"maxsize": 100, "maxage": 7, "maxbackups": 14, "localtime": false, "compress": true}' ``` Restart each member one at a time to pick up the change. If your deployment can't enable `--enable-log-rotation` (for example, a locked-down vendor wrapper), fall back to external `logrotate` with `copytruncate`, since etcd holds the file open: ``` /var/log/etcd/etcd.log { daily rotate 14 size 100M compress delaycompress missingok notifempty copytruncate su etcd etcd } ``` Save as `/etc/logrotate.d/etcd` on each node. This is the third-best option; prefer journald or `--enable-log-rotation` when possible. ## Rotate TLS certificates TLS certificates come in two planes: | Plane | Purpose | Configured by | | ------------------ | -------------------------------------------------------------- | --------------------------------------------------------------- | | Client (port 2379) | Authenticates etcd to clients (Gateway, `etcdctl`, monitoring) | `--cert-file`, `--key-file`, `--trusted-ca-file` | | Peer (port 2380) | Authenticates etcd members to each other | `--peer-cert-file`, `--peer-key-file`, `--peer-trusted-ca-file` | This guide assumes transport TLS only on both planes. If your deployment uses full mTLS, the same rotation steps apply to the client and peer cert files plus their CA trust bundles. Since v3.2, etcd reloads client-plane certificates on every new client connection: replacing `cert-file` and `key-file` on a member takes effect for the next client connection without a restart. The peer plane, where connections are long-lived between members, isn't documented as hot-reloadable; treat peer-cert changes as requiring a rolling restart. When the CA changes, deploy a bundled CA file (old and new concatenated) so every member trusts either cert during the rollout; this is an operator pattern, not an etcd-documented one, but it's the standard way to roll a CA without a window of mutual rejection. Reference: [etcd v3.5 transport security](https://etcd.io/docs/v3.5/op-guide/security/). #### Inspect current certificate configuration ```bash for node in $ETCD_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node \ 'sudo systemctl cat etcd | grep -iE "cert-file|key-file|trusted-ca-file"' 2>/dev/null done ``` For container deployments, inspect the container args/env instead of the systemd unit. #### Check current expiry Plan the rotation at least two weeks before the earliest expiry. ```bash for node in $ETCD_NODES; do echo "=== $node ===" ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node ' for cert in /etc/etcd/pki/server.crt /etc/etcd/pki/peer.crt /etc/etcd/pki/ca.crt; do [ -f "$cert" ] || continue echo "--- $cert ---" sudo openssl x509 -in "$cert" -noout -dates -subject 2>/dev/null done ' done ``` A faster wire-side check confirms what each member is actually serving: ```bash for node in $ETCD_NODES; do echo "=== $node: served client certificate ===" echo | openssl s_client -servername $node -connect $node:$ETCD_CLIENT_PORT 2>/dev/null \ | openssl x509 -noout -dates -subject done ``` #### Take a snapshot before rotating If the peer-plane rotation later requires a rolling restart and something goes wrong, a fresh snapshot eliminates the worst-case outcome (cluster fragmentation requiring restore). See [Take and verify a snapshot](#take-and-verify-a-snapshot). #### Verify new certificates locally before deploying ```bash openssl x509 -in new-server.crt -noout -text \ | grep -E "Subject:|Not Before:|Not After :|DNS:|IP Address:" openssl x509 -in new-peer.crt -noout -text \ | grep -E "Subject:|Not Before:|Not After :|DNS:|IP Address:" openssl verify -CAfile new-ca.crt new-server.crt openssl verify -CAfile new-ca.crt new-peer.crt openssl x509 -noout -modulus -in new-server.crt | openssl md5 openssl rsa -noout -modulus -in new-server.key | openssl md5 openssl x509 -noout -modulus -in new-peer.crt | openssl md5 openssl rsa -noout -modulus -in new-peer.key | openssl md5 ``` The peer certificate's SANs must include every peer URL hostname and IP other members use to reach this one; a missing SAN causes those members to reject the connection. Don't proceed if `openssl verify` errors or a modulus pair doesn't match. #### Back up existing certificate files ```bash for node in $ETCD_NODES; do ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node ' sudo cp /etc/etcd/pki/server.crt /etc/etcd/pki/server.crt.bak sudo cp /etc/etcd/pki/server.key /etc/etcd/pki/server.key.bak sudo cp /etc/etcd/pki/peer.crt /etc/etcd/pki/peer.crt.bak sudo cp /etc/etcd/pki/peer.key /etc/etcd/pki/peer.key.bak sudo cp /etc/etcd/pki/ca.crt /etc/etcd/pki/ca.crt.bak ' done ``` #### Distribute new certificate files Copy the new files to all nodes before any member starts using them. If the CA changed, deploy the bundled CA file described above; once every member holds new server/peer certs, replace it with a new-CA-only file in a follow-up rotation. ```bash for node in $ETCD_NODES; do scp -i $SSH_KEY_PATH new-server.crt new-server.key new-peer.crt new-peer.key new-ca.crt \ $ETCD_SSH_USER@$node:/tmp/ ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@$node ' sudo cp /tmp/new-server.crt /etc/etcd/pki/server.crt sudo cp /tmp/new-server.key /etc/etcd/pki/server.key sudo cp /tmp/new-peer.crt /etc/etcd/pki/peer.crt sudo cp /tmp/new-peer.key /etc/etcd/pki/peer.key sudo cp /tmp/new-ca.crt /etc/etcd/pki/ca.crt sudo chown etcd:etcd /etc/etcd/pki/*.crt /etc/etcd/pki/*.key sudo chmod 400 /etc/etcd/pki/*.key sudo chmod 444 /etc/etcd/pki/*.crt ' done ``` For container deployments, write the new files into the host-mounted PKI directory instead. #### Pick up the new certificates **Client-plane only, no CA change:** no restart required. Open a fresh connection to force the reload and confirm: ```bash for node in $ETCD_NODES; do echo "=== $node ===" echo | openssl s_client -servername $node -connect $node:$ETCD_CLIENT_PORT 2>/dev/null \ | openssl x509 -noout -dates -subject done ``` **Peer-plane changed, or CA changed:** a rolling restart is required. For each member in sequence: move the leader off it if applicable, verify quorum, restart, then confirm it rejoined and is presenting the new certificate before moving to the next member. ```bash ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ 'sudo systemctl restart etcd' echo | openssl s_client -servername -connect :$ETCD_CLIENT_PORT 2>/dev/null \ | openssl x509 -noout -dates -subject etcdctl --endpoints=https://:$ETCD_CLIENT_PORT $ETCDCTL_TLS endpoint health etcdctl --endpoints=$ETCD_ENDPOINTS $ETCDCTL_TLS endpoint health --write-out=table ``` #### Update references to the old CA Update local scripts, environment files, monitoring configuration, and Gateway's `GATEWAY_STORE_ETCD_CA_CERTIFICATE_FILE` (if the CA changed) to reference the new file. If a bundled CA was deployed, schedule the follow-up rotation to replace it with a new-CA-only file. ### Rollback ```bash ssh -i $SSH_KEY_PATH $ETCD_SSH_USER@ ' sudo cp /etc/etcd/pki/server.crt.bak /etc/etcd/pki/server.crt sudo cp /etc/etcd/pki/server.key.bak /etc/etcd/pki/server.key sudo cp /etc/etcd/pki/peer.crt.bak /etc/etcd/pki/peer.crt sudo cp /etc/etcd/pki/peer.key.bak /etc/etcd/pki/peer.key sudo cp /etc/etcd/pki/ca.crt.bak /etc/etcd/pki/ca.crt sudo systemctl restart etcd ' ``` If the cluster fully fragmented because rotation was attempted on multiple nodes simultaneously, restore from the snapshot taken before the rotation using [Restore from a snapshot](#restore-from-a-snapshot). ## Related resources * [etcd v3.5 op-guide](https://etcd.io/docs/v3.5/op-guide/) * [Configure etcd shared database](./configure-etcd-database)