Added exporter for Grandmaster, borderClocl and client clock
This commit is contained in:
370
exporters/README.md
Normal file
370
exporters/README.md
Normal file
@@ -0,0 +1,370 @@
|
||||
# PTP Prometheus Exporters
|
||||
|
||||
Python Prometheus exporters for the PTP topology:
|
||||
|
||||
```text
|
||||
GT-U7
|
||||
│
|
||||
\/
|
||||
RPi5 Grandmaster
|
||||
Grandmaster
|
||||
│
|
||||
\/
|
||||
Linux boundary clock
|
||||
│ │
|
||||
\/ \/
|
||||
NEXUS / C93180LC-EX
|
||||
Nexus boundary clock
|
||||
│
|
||||
\/
|
||||
Linux downstream clients
|
||||
```
|
||||
|
||||
There are two exporter programs:
|
||||
|
||||
- `linux_ptp_exporter.py` — runs locally on Linux grandmasters, Linux boundary clocks, and Linux clients.
|
||||
- `nexus_ptp_exporter.py` — runs on a Linux monitoring/Prometheus host and polls the Nexus over SSH.
|
||||
|
||||
The Linux exporter uses the **read-only** `ptp4l` management socket (`/var/run/ptp4lro`) and `pmc`. It does not alter PTP state.
|
||||
|
||||
## Metrics worth graphing
|
||||
|
||||
For Linux PTP nodes:
|
||||
|
||||
```text
|
||||
ptp_master_offset_nanoseconds
|
||||
ptp_mean_path_delay_nanoseconds
|
||||
ptp_steps_removed
|
||||
ptp_gm_present
|
||||
ptp_port_state
|
||||
ptp_port_messages_total
|
||||
ptp_port_service_events_total
|
||||
ptp_phc2sys_offset_nanoseconds
|
||||
ptp_service_up
|
||||
```
|
||||
|
||||
On the RPi5 grandmaster, additionally:
|
||||
|
||||
```text
|
||||
ptp_gm_chrony_system_time_offset_seconds
|
||||
ptp_gm_chrony_last_offset_seconds
|
||||
ptp_gm_chrony_rms_offset_seconds
|
||||
ptp_gm_chrony_root_dispersion_seconds
|
||||
ptp_gm_chrony_source
|
||||
```
|
||||
|
||||
For NYATER:
|
||||
|
||||
```text
|
||||
ptp_nexus_clock_locked
|
||||
ptp_nexus_offset_from_master_nanoseconds
|
||||
ptp_nexus_mean_path_delay_nanoseconds
|
||||
ptp_nexus_steps_removed
|
||||
ptp_nexus_port_state
|
||||
ptp_nexus_port_messages_total
|
||||
ptp_nexus_parent_info
|
||||
```
|
||||
|
||||
## 1. Install on a Linux PTP node
|
||||
|
||||
Debian/Raspberry Pi OS/Proxmox example:
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install python3-venv linuxptp
|
||||
|
||||
sudo useradd --system --home /opt/ptp-exporter --shell /usr/sbin/nologin ptp-exporter
|
||||
sudo mkdir -p /opt/ptp-exporter
|
||||
sudo chown ptp-exporter:ptp-exporter /opt/ptp-exporter
|
||||
|
||||
sudo -u ptp-exporter python3 -m venv /opt/ptp-exporter/venv
|
||||
sudo -u ptp-exporter /opt/ptp-exporter/venv/bin/pip install prometheus-client
|
||||
|
||||
sudo install -m 0755 linux_ptp_exporter.py /opt/ptp-exporter/linux_ptp_exporter.py
|
||||
```
|
||||
|
||||
linuxptp's default `/var/run/ptp4lro` is a read-only management socket intended for monitoring and is normally mode `0666`, so the exporter does not need write access to `/var/run/ptp4l`.
|
||||
|
||||
If `ptp4l` on your distro uses `/var/run/ptp/ptp4lro` instead, pass that with `--uds`.
|
||||
|
||||
### RPi5 grandmaster
|
||||
|
||||
Install:
|
||||
|
||||
```bash
|
||||
sudo install -m 0644 systemd/ptp-exporter-grandmaster.service \
|
||||
/etc/systemd/system/ptp-exporter.service
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now ptp-exporter
|
||||
```
|
||||
|
||||
The unit runs:
|
||||
|
||||
```bash
|
||||
linux_ptp_exporter.py \
|
||||
--role grandmaster \
|
||||
--chrony \
|
||||
--ptp4l-service ptp4l-gm.service \
|
||||
--phc2sys-service phc2sys-gm.service
|
||||
```
|
||||
|
||||
This exports both the PTP side and the GNSS/PPS -> chrony side.
|
||||
|
||||
### Linux boundary clock
|
||||
|
||||
Install:
|
||||
|
||||
```bash
|
||||
sudo install -m 0644 systemd/ptp-exporter-boundary.service \
|
||||
/etc/systemd/system/ptp-exporter.service
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now ptp-exporter
|
||||
```
|
||||
|
||||
<!-- This is especially useful on WOLF because it exports both:
|
||||
|
||||
```text
|
||||
RPi5 -> eno4 PHC via ptp4l/pmc
|
||||
eno4 PHC -> Mellanox PHC via recent phc2sys journal samples
|
||||
``` -->
|
||||
|
||||
<!-- It also exports the per-port linuxptp packet counters from `PORT_STATS_NP`. -->
|
||||
|
||||
### Downstream Linux client
|
||||
|
||||
Install:
|
||||
|
||||
```bash
|
||||
sudo install -m 0644 systemd/ptp-exporter-client.service \
|
||||
/etc/systemd/system/ptp-exporter.service
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now ptp-exporter
|
||||
```
|
||||
|
||||
## 2. Journal permissions
|
||||
|
||||
The exact `pmc` metrics do not require journal access.
|
||||
|
||||
`ptp_phc2sys_*` metrics are parsed from the latest `phc2sys -m` systemd journal lines. The included services use:
|
||||
|
||||
```text
|
||||
SupplementaryGroups=systemd-journal
|
||||
```
|
||||
|
||||
so the exporter can read those lines without running as root.
|
||||
|
||||
If your distribution does not have the `systemd-journal` group, remove that line. The exporter will continue working; only the `ptp_phc2sys_*` metrics will be absent.
|
||||
|
||||
## 3. Test a Linux exporter
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:9559/metrics | grep '^ptp_'
|
||||
```
|
||||
|
||||
Useful direct checks:
|
||||
|
||||
```bash
|
||||
pmc -u -s /var/run/ptp4lro -b 0 'GET CURRENT_DATA_SET'
|
||||
pmc -u -s /var/run/ptp4lro -b 0 'GET TIME_STATUS_NP'
|
||||
pmc -u -s /var/run/ptp4lro -b 0 'GET PORT_PROPERTIES_NP'
|
||||
pmc -u -s /var/run/ptp4lro -b 0 'GET PORT_STATS_NP'
|
||||
pmc -u -s /var/run/ptp4lro -b 0 'GET PORT_SERVICE_STATS_NP'
|
||||
```
|
||||
|
||||
`PORT_STATS_NP` is particularly useful because it exposes real per-port Sync, Follow_Up, Delay_Req, Delay_Resp, Announce, Management, etc. RX/TX counters.
|
||||
|
||||
## 4. Nexus exporter
|
||||
|
||||
Python cannot run directly on NX-OS, so `nexus_ptp_exporter.py` should run on a Linux monitoring host.
|
||||
|
||||
Install:
|
||||
|
||||
```bash
|
||||
sudo useradd --system --home /opt/ptp-exporter --shell /usr/sbin/nologin ptp-exporter
|
||||
sudo mkdir -p /opt/ptp-exporter
|
||||
sudo chown ptp-exporter:ptp-exporter /opt/ptp-exporter
|
||||
|
||||
sudo -u ptp-exporter python3 -m venv /opt/ptp-exporter/venv
|
||||
sudo -u ptp-exporter /opt/ptp-exporter/venv/bin/pip install prometheus-client paramiko
|
||||
|
||||
sudo install -m 0755 nexus_ptp_exporter.py /opt/ptp-exporter/nexus_ptp_exporter.py
|
||||
```
|
||||
|
||||
Use SSH key authentication if possible. The account only needs permission to run:
|
||||
|
||||
```text
|
||||
show ptp clock
|
||||
show ptp parent
|
||||
show ptp brief
|
||||
show ptp counters all
|
||||
```
|
||||
|
||||
Example manual start:
|
||||
|
||||
```bash
|
||||
/opt/ptp-exporter/venv/bin/python \
|
||||
/opt/ptp-exporter/nexus_ptp_exporter.py \
|
||||
--host 10.255.254.255 \
|
||||
--target-name NYATER \
|
||||
--username prometheus \
|
||||
--key-file /opt/ptp-exporter/.ssh/id_ed25519 \
|
||||
--port 9560
|
||||
```
|
||||
|
||||
`10.255.254.255` above is only an example based on the PTP source/loopback shown in the current topology. Use whichever management-reachable address you actually want SSH to connect to.
|
||||
|
||||
The exporter checks the system SSH known-hosts database by default. Do not use `--accept-new-host-key` for a permanent deployment unless you explicitly want trust-on-first-use behavior.
|
||||
|
||||
### Password authentication
|
||||
|
||||
If required:
|
||||
|
||||
```bash
|
||||
export NEXUS_PTP_PASSWORD='...'
|
||||
|
||||
nexus_ptp_exporter.py \
|
||||
--host <NYATER-address> \
|
||||
--username <user> \
|
||||
--password-env NEXUS_PTP_PASSWORD
|
||||
```
|
||||
|
||||
Do not put the password directly in `ExecStart=`.
|
||||
|
||||
## 5. Prometheus scrape configuration
|
||||
|
||||
See `examples/prometheus.yml`.
|
||||
|
||||
Typical jobs:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: ptp-linux
|
||||
static_configs:
|
||||
- targets:
|
||||
- GrandMaster:9559
|
||||
- BoundaryClock:9559
|
||||
- server-a:9559
|
||||
- server-b:9559
|
||||
|
||||
- job_name: ptp-nexus
|
||||
static_configs:
|
||||
- targets:
|
||||
- prometheus-host:9560
|
||||
```
|
||||
|
||||
The Nexus target is the **exporter host**, not NEXUS itself.
|
||||
|
||||
## 6. Suggested PromQL
|
||||
|
||||
### Offset from immediate master
|
||||
|
||||
```promql
|
||||
ptp_master_offset_nanoseconds
|
||||
```
|
||||
|
||||
or Nexus:
|
||||
|
||||
```promql
|
||||
ptp_nexus_offset_from_master_nanoseconds
|
||||
```
|
||||
|
||||
### Absolute value of offset
|
||||
|
||||
```promql
|
||||
abs(ptp_master_offset_nanoseconds)
|
||||
```
|
||||
|
||||
### 5-minute worst-case Linux offset
|
||||
|
||||
```promql
|
||||
max_over_time(abs(ptp_master_offset_nanoseconds)[5m])
|
||||
```
|
||||
|
||||
### 5-minute RMS-like standard deviation
|
||||
|
||||
```promql
|
||||
stddev_over_time(ptp_master_offset_nanoseconds[5m])
|
||||
```
|
||||
|
||||
### Ensure GM is present
|
||||
|
||||
```promql
|
||||
ptp_gm_present == 1
|
||||
```
|
||||
|
||||
### Ensure Nexus is locked
|
||||
|
||||
```promql
|
||||
ptp_nexus_clock_locked == 1
|
||||
```
|
||||
|
||||
### Check WOLF / client port states
|
||||
|
||||
```promql
|
||||
ptp_port_state{state="client"} == 1
|
||||
```
|
||||
|
||||
### Check Nexus upstream redundancy
|
||||
|
||||
```promql
|
||||
ptp_nexus_port_state{state=~"slave|passive"}
|
||||
```
|
||||
|
||||
### PTP packet rate
|
||||
|
||||
```promql
|
||||
rate(ptp_port_messages_total[5m])
|
||||
```
|
||||
|
||||
For NEXUS:
|
||||
|
||||
```promql
|
||||
rate(ptp_nexus_port_messages_total[5m])
|
||||
```
|
||||
<!--
|
||||
## 7. Suggested alerts
|
||||
|
||||
Examples are in `examples/alerts.yml`.
|
||||
|
||||
The useful first alerts are:
|
||||
|
||||
- exporter scrape failed;
|
||||
- grandmaster disappeared;
|
||||
- Nexus unlocked;
|
||||
- master offset exceeds a threshold for several minutes;
|
||||
- a client leaves CLIENT/SLAVE state;
|
||||
- WOLF upstream port leaves CLIENT/SLAVE state;
|
||||
- PTP packet counters stop increasing unexpectedly.
|
||||
|
||||
Do not start with extremely tight offset alerts such as 100 ns until you have collected a few days of normal behavior. First establish the actual distribution of your own hardware. -->
|
||||
|
||||
## 8. Security / firewall
|
||||
|
||||
The exporters listen on all addresses by default. Restrict TCP/9559 and TCP/9560 to the Prometheus server with the host firewall, or start them with a specific management address:
|
||||
|
||||
```text
|
||||
--listen <management-IP>
|
||||
```
|
||||
|
||||
The HTTP endpoint has no authentication; network-level restriction is recommended.
|
||||
|
||||
## 9. Notes about PTP semantics
|
||||
|
||||
`ptp_master_offset_nanoseconds` is the local clock's offset from its **immediate PTP master**, not a proof of absolute UTC accuracy.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
MOMI GNSS/PPS
|
||||
-> RPi CLOCK_REALTIME
|
||||
-> RPi PHC
|
||||
-> WOLF eno4 PHC
|
||||
-> WOLF Mellanox PHC
|
||||
-> NYATER
|
||||
-> final client
|
||||
```
|
||||
|
||||
Each stage can contribute error. Grafana should therefore show both the per-hop PTP offsets and the RPi chrony/GNSS health.
|
||||
Reference in New Issue
Block a user