> For the complete documentation index, see [llms.txt](https://services.validexis.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://services.validexis.com/monitoring/monitoring-celestia-validators-and-fibre.md).

# 📡 Monitoring Celestia Validators and Fibre

This page describes how VALIDEXIS monitors its Celestia Mainnet Beta and Mocha stack: two validators, two Fibre servers and two TMKMS signers. It extends our general [Node-exporter + Prometheus + Grafana](https://services.validexis.com/monitoring/node-exporter-+-prometheus-+-grafana) guide with the Celestia- and Fibre-specific signals.

### Stack

| Layer        | Tool                                               | What it collects                                                          |
| ------------ | -------------------------------------------------- | ------------------------------------------------------------------------- |
| Hosts        | node-exporter → Prometheus                         | CPU, memory, disks, SSD wear, network on all six servers                  |
| Validators   | celestia-app Prometheus endpoint                   | Consensus, peers, `cometbft_privval_*` remote-signing metrics             |
| Fibre        | OpenTelemetry (OTLP/HTTP) → collector → Prometheus | `fibre.server.*` upload, download, store and signing metrics              |
| Dashboards   | Grafana                                            | Per-network dashboards plus the official Fibre dashboards                 |
| Alerts       | Prometheus alert rules → Alertmanager              | Routed to on-call with escalation                                         |
| Outside view | Off-site monitor in another city                   | Chain progress and public Fibre reachability; it observes, it never signs |

### Telemetry from Fibre

Fibre pushes traces and metrics over OTLP/HTTP:

```bash
fibre start --home /var/lib/fibre --otel-endpoint http://<otel_collector>:4318
```

Use the collector's base URL; Fibre appends `/v1/traces` and `/v1/metrics`. The celestia-app repository ships a reference [collector configuration](https://github.com/celestiaorg/celestia-app/blob/v10.4.0-mocha/observability/docker/otel-collector/config.yml) and the [Fibre dashboard](https://github.com/celestiaorg/celestia-app/blob/v10.4.0-mocha/observability/docker/grafana/dashboards/fibre.json) and [Fibre runtime dashboard](https://github.com/celestiaorg/celestia-app/blob/v10.4.0-mocha/observability/docker/grafana/dashboards/fibre-runtime.json) we import into Grafana.

Key Fibre server metrics (OTLP names; Prometheus converts dots to underscores and may add unit suffixes):

| Metric                                                   | Why we watch it                                             |
| -------------------------------------------------------- | ----------------------------------------------------------- |
| `fibre.server.upload_shard.duration` (`success`)         | Upload latency and error rate                               |
| `fibre.server.upload_shard.bytes`                        | Shard bytes received — real load on the server              |
| `fibre.server.download_shard.duration` (`success`)       | Serving reads to clients                                    |
| `fibre.server.store.put.duration` / `store.get.duration` | Shard disk health                                           |
| `fibre.server.sign.duration` (`success`)                 | Payment-promise endorsement through the validator and TMKMS |
| `fibre.server.prune.duration`                            | Retention is being enforced                                 |

### Two different signing measurements

| Measurement                   | Metric                                                           | Requirement    | Our alert                             |
| ----------------------------- | ---------------------------------------------------------------- | -------------- | ------------------------------------- |
| Node → remote signer latency  | `cometbft_privval_signing_latency_median_seconds` (celestia-app) | Median ≤ 10 ms | Median > 5 ms                         |
| Fibre payment-promise signing | `fibre.server.sign.duration` (Fibre)                             | —              | Error rate and p95 tracked separately |

The 10 ms requirement applies to the first one only. Celestia's documentation states this explicitly; mixing the two would hide a slow signer.

### Alert rules

| Alert                  | Condition                                           | Severity           |
| ---------------------- | --------------------------------------------------- | ------------------ |
| Remote-signing latency | Median node → KMS signing latency above 5 ms        | Critical           |
| Remote-signing failure | `cometbft_privval_signing_failures_total` increases | Critical           |
| Missed blocks          | More than 20 missed blocks in one hour              | Critical           |
| Chain stalled          | No new block for 60 seconds                         | Critical           |
| Fibre errors           | Upload, download or sign error rate above baseline  | Warning → Critical |
| Fibre unreachable      | Off-site probe fails on the public Fibre port       | Critical           |
| Disk usage             | Any data or shard disk above 70%                    | Warning            |
| Network utilisation    | Public port above 70% sustained                     | Warning            |
| SSD wear               | Wear level above 80%                                | Warning            |

A Prometheus rules file implementing these alerts is published in [validexisinfra/Celestia/monitoring](https://github.com/validexisinfra/Celestia/tree/main/monitoring).

### Escalation and response

1. Alerts go to the on-call operator as a push notification.
2. Critical alerts that are not acknowledged within **15 minutes** escalate to a phone call.
3. We start work on a critical alert within **30 minutes**, around the clock, and reply to emergency email within **two hours**.
4. Foundation upgrade notices are applied within 24 hours, or sooner when an announced activation schedule requires it.

These are standing operating commitments, not a contractual SLA or a recovery-time guarantee.

### The outside view

Monitoring that lives in the same datacenter goes dark with it. Our off-site monitor runs in another city and checks only what an outsider can see: whether the chain advances and whether our registered Fibre host answers on its public port. It holds no keys and has no access to the private network.

Public, independent views complement it: the community dashboards listed in the [Fibre monitoring](https://docs.celestia.org/operate/consensus-validators/fibre/metrics/#community-dashboards) guide, and our own [VALIDEXIS Celestia platform](https://celestia.validexis.com/), which publishes external Fibre checks next to our server telemetry.

### Sources

* [Fibre monitoring](https://docs.celestia.org/operate/consensus-validators/fibre/metrics/) · [Validator metrics](https://docs.celestia.org/operate/consensus-validators/metrics)
* [celestia-core privval metrics](https://github.com/celestiaorg/celestia-core/blob/main/privval/metrics.go)
* Alert rules and collector template: [github.com/validexisinfra/Celestia/tree/main/monitoring](https://github.com/validexisinfra/Celestia/tree/main/monitoring)
