From a148392e70ef6b0875ed06ff878fda317bf5e71c Mon Sep 17 00:00:00 2001 From: YuruC3 Date: Mon, 7 Sep 2026 00:27:05 +0200 Subject: [PATCH] my shitty notes I've asked AI to clean up and reorder so that they are readable It is just work in progress so I don't really car --- NEXUS.md | 238 +++++++++++++++++++++++++++ ShowCommands.md | 168 +++++++++++++++++++ boundaryClockAndClient.md | 294 ++++++++++++++++++++++++++++++++++ clientConfig.md | 206 ++++++++++++++++++++++++ conf.md | 253 +++++++++++++++++++++++++++++ exporters/examples/alerts.yml | 51 ++++++ 6 files changed, 1210 insertions(+) create mode 100644 NEXUS.md create mode 100644 ShowCommands.md create mode 100644 boundaryClockAndClient.md create mode 100644 clientConfig.md create mode 100644 conf.md create mode 100644 exporters/examples/alerts.yml diff --git a/NEXUS.md b/NEXUS.md new file mode 100644 index 0000000..df696f5 --- /dev/null +++ b/NEXUS.md @@ -0,0 +1,238 @@ +# Cisco Nexus C93180LC-EX PTP Boundary Clock + +NYATER is the second boundary clock in the PTP chain. + +```text +RPi5 GM + │ + ▼ +WOLF-HUNTER BC + │ │ + │ two QSFP+ │ + ▼ ▼ +Eth1/5 Eth1/6 + NYATER + C93180LC-EX BC + │ + ├─ PTP client-facing ports + └─ PTP client-facing ports +``` + +With the current design: + +```text +RPi5 stepsRemoved 0 +WOLF-HUNTER stepsRemoved 1 +NYATER stepsRemoved 2 +Linux client stepsRemoved 3 +``` + +The current Nexus configuration is stored in `NYATER.ios` in the root of the PTP directory. + +## 1. Base configuration + +`NYATER.ios` contains: + +```text +feature ptp + +ptp device-type boundary-clock +ptp domain 0 +ptp source 10.255.254.255 + +interface Ethernet1/5 + ptp + +interface Ethernet1/6 + ptp +``` + +`10.255.254.255` is the address selected as the Nexus PTP source address. In this setup it is the `lo0` address; using the loopback gives PTP packets originated by NYATER a stable source address independent of which physical upstream path is selected. + +Before applying the PTP config, verify that the address really exists on NYATER: + +```text +show ip interface brief | include 10.255.254.255 +show running-config interface loopback0 +``` + +## 2. Apply the base PTP configuration + +Enter configuration mode and apply the contents of `NYATER.ios`, or paste the equivalent configuration manually: + +```text +configure terminal + +feature ptp + +ptp device-type boundary-clock +ptp domain 0 +ptp source 10.255.254.255 + +interface Ethernet1/5 + ptp + +interface Ethernet1/6 + ptp + +end +copy running-config startup-config +``` + +`Ethernet1/5` and `Ethernet1/6` are the two links from WOLF-HUNTER's Mellanox ports. + +## 3. Expected upstream behavior + +When both WOLF-HUNTER links are healthy, NYATER should select one as the active upstream PTP port and leave the other as the redundant path. + +Expected output: + +```text +show ptp brief + +PTP port status +----------------------------------- +Port State +--------------------- ------------ +Eth1/5 Slave +Eth1/6 Passive +``` + +The exact choice may be reversed; `Eth1/6 Slave` and `Eth1/5 Passive` is equally valid. + +The important points are: + +```text +- one WOLF-facing port is Slave +- the other WOLF-facing port is Passive +- PTP Clock state is Locked +- grandmaster identity is still the RPi5 +- steps removed is 2 +- domain is 0 +- UTC offset received in Announce packets is 37 +``` + +## 4. Add a downstream PTP client + +For a normal Linux server connected directly to a physical Nexus port, enable PTP on that port: + +```text +configure terminal + +interface Ethernet1/ + ptp + +end +copy running-config startup-config +``` + +For example: + +```text +interface Ethernet1/10 + ptp +``` + +No separate per-client PTP source address is required. The existing global: + +```text +ptp source 10.255.254.255 +``` + +continues to be used by NYATER. + +Once the downstream Linux client is active and the Nexus is locked upstream, that client-facing port should normally appear as `Master`. + +Example: + +```text +show ptp brief + +Eth1/5 Slave +Eth1/6 Passive +Eth1/10 Master +``` + +Repeat only the interface-level `ptp` command for every additional direct downstream client port. + +## 5. Verification commands + +The most useful operational commands are: + +```text +show ptp clock +show ptp brief +show ptp packet +``` + +Your healthy output should resemble: + +```text +PTP Device Type : boundary-clock +PTP Source IP Address : 10.255.254.255 +Clock Domain : 0 +PTP Clock state : Locked +Steps removed : 2 +``` + +`show ptp packet` is useful for confirming that Announce packets still carry the RPi5 grandmaster identity and the expected UTC offset: + +```text +gm_id= +utc_offset=37 +``` + +It also exposes Sync, Follow_Up, Delay_Req and Delay_Resp traffic and is useful when investigating offset or path-delay behavior. + +## 6. Current known-good state + +A known-good state observed on NYATER had: + +```text +PTP Device Type : boundary-clock +PTP Source IP Address : 10.255.254.255 +Clock Domain : 0 +PTP Clock state : Locked +Steps removed : 2 +Eth1/5 : Slave +Eth1/6 : Passive +gm_id : 88a29efffe826ce6 +utc_offset : 37 +``` + +Use that as a topology sanity check. The active/passive physical ports may swap, but the grandmaster identity should not change merely because the redundant path changes. + +## 7. What the `ptp source` address does + +The `ptp source 10.255.254.255` setting selects the source IP address used by PTP/UDP packets originated by NYATER. It does **not** mean the Nexus gets its time from `lo0`. + +The actual timing reference is still selected by PTP/BMCA through the WOLF-HUNTER-facing ports: + +```text +RPi5 -> WOLF-HUNTER -> NYATER +``` + +## 8. Troubleshooting + +If NYATER is not locked: + +```text +show ptp clock +show ptp brief +show ptp packet +show interface Ethernet1/5 +show interface Ethernet1/6 +``` + +Check that: + +```text +- `feature ptp` is enabled +- domain is 0 everywhere +- both WOLF-facing ports have `ptp` +- one upstream port receives Announce/Sync packets +- the received gm_id is the RPi5 identity +- WOLF-HUNTER itself is synchronized upstream +``` + +If a downstream client is not synchronizing, first confirm its Nexus port has the interface-level `ptp` command and appears as `Master` in `show ptp brief`. diff --git a/ShowCommands.md b/ShowCommands.md new file mode 100644 index 0000000..f77822b --- /dev/null +++ b/ShowCommands.md @@ -0,0 +1,168 @@ +# On Grand Master + +```bash +sudo pmc -u -b 0 'GET CURRENT_DATA_SET' +sudo pmc -u -b 0 'GET PARENT_DATA_SET' +sudo pmc -u -b 0 'GET TIME_STATUS_NP' +sudo pmc -u -b 0 'GET PORT_DATA_SET' +``` + + +Additionally + +chronyc tracking + +and + +chronyc sources -v + +can be used + + + +# On Boundary clocks + + +```bash +sudo pmc -u -b 0 'GET CURRENT_DATA_SET' +sudo pmc -u -b 0 'GET PARENT_DATA_SET' +sudo pmc -u -b 0 'GET TIME_STATUS_NP' +sudo pmc -u -b 0 'GET PORT_DATA_SET' +``` + + +# On clients + + +```bash +sudo pmc -u -b 0 'GET TIME_STATUS_NP' +sudo pmc -u -b 0 'GET CURRENT_DATA_SET' +``` + + +# On NYATER + + +```bash +show ptp clock +show ptp brief +show ptp parent +show ptp packet +``` + + + +# Extras + + + +``` +journalctl -fu ptp4l- +journalctl -fu phc2sys- +``` + + +# Description + + + + +## GET TIME_STATUS_NP + +Typical output includes values like: + +master_offset -34 +ingress_time ... +cumulativeScaledRateOffset ... +scaledLastGmPhaseChange ... +gmTimeBaseIndicator ... +lastGmPhaseChange ... +gmPresent true +gmIdentity 88a29efffe826ce6 + + +The interesting bits are: + +master_offset +gmPresent +gmIdentity + + +**master_offset** is in nanoseconds, so something like: + +master_offset -34 + +means the local PTP clock is about 34 ns from its master. + + + +## CURRENT_DATA_SET + + +This gives you: + +stepsRemoved +offsetFromMaster +meanPathDelay + +So on my topology I had: + +RPi5 GM stepsRemoved 0 +WOLF-HUNTER BC stepsRemoved 1 +NYATER Nexus BC stepsRemoved 2 +Linux client stepsRemoved 3 + +On a client you'll get something along the lines of: + +stepsRemoved 3 +offsetFromMaster -21 +meanPathDelay 318 + +Again, the time quantities are effectively nanosecond-scale values. + +This is probably the closest thing to a quick **chronyc tracking** + + +## PARENT_DATA_SET + +Look for: + +parentPortIdentity +grandmasterIdentity +grandmasterClockQuality +grandmasterPriority1 +grandmasterPriority2 + +You should see your RPi5's MAC: + +grandmasterIdentity 88a29efffe826ce6 + + + +That confirms the hierarchy: + + +## GET PORT_DATA_SET + +Per-port state + + +This is especially useful because you can see whether each port is: + +SLAVE / CLIENT +MASTER / SERVER +PASSIVE +LISTENING +FAULTY + +On boundary clocl you should roughly have: + +eno4 (port to GrandMaster) SLAVE +enp5s0 (port to another boundaryClock) MASTER +enp5s0d1 (port to another boundaryClock) MASTER + +while upstream device from enp5s0 and enp5s0d1 has: + +Eth1/5 Slave +Eth1/6 Passive + diff --git a/boundaryClockAndClient.md b/boundaryClockAndClient.md new file mode 100644 index 0000000..0e12fb4 --- /dev/null +++ b/boundaryClockAndClient.md @@ -0,0 +1,294 @@ +```text id="97dya4" +eno4 upstream toward RPi5 grandmaster +enp4s0/enp4s0d1 downstream toward N9K +``` + +This design is: + +```text id="c1c9so" +RPi5 GM + -> ptp4l-client on eno4 + -> eno4 PHC + -> phc2sys-client + -> CLOCK_REALTIME on WOLF-HUNTER + -> phc2sys-downstream + -> enp4s0/enp4s0d1 PHC + -> ptp4l-downstream + -> N9K / downstream clients +``` + +`ptp4l` can use multiple `-i` interfaces, `clientOnly` is the client/slave mode, and `serverOnly` prevents a port from becoming a client. `phc2sys` is the tool that syncs PHCs and `CLOCK_REALTIME`; `-w` waits for `ptp4l`, while `-O 37` provides the UTC/TAI offset explicitly. ([Linux PTP Project][1]) + +## 1. Upstream PTP client config: `eno4` + +Create: + +```bash id="o4uyiz" +sudo nano /etc/linuxptp/ptp4l-upstream-client.conf +``` + +```conf id="fxd4o4" +[global] +verbose 1 +time_stamping hardware + +domainNumber 0 +network_transport UDPv4 +delay_mechanism E2E + +# Upstream side: only act as a PTP client/slave +clientOnly 1 + +summary_interval 1 + +# Separate UDS socket so multiple ptp4l instances do not collide +uds_address /var/run/ptp4l-upstream + +[eno4] +``` + +## 2. Downstream PTP server config: `enp4s0/enp4s0d1` + +Create: + +```bash id="zhy8l3" +sudo nano /etc/linuxptp/ptp4l-downstream-server.conf +``` + +```conf id="qqo3eb" +[global] +verbose 1 +time_stamping hardware + +domainNumber 0 +network_transport UDPv4 +delay_mechanism E2E + +# Downstream side: serve time toward N9K/clients +serverOnly 1 + +# Make this preferred on the downstream PTP segment +priority1 128 +priority2 128 + +# This host is fed by upstream PTP, not directly by GNSS +timeSource 0x40 + +# Conservative advertised quality for a PTP-fed downstream server. +# You can tune these later after measuring. +clockClass 6 +clockAccuracy 0x23 +offsetScaledLogVariance 0xFFFF + +utc_offset 37 +summary_interval 1 + +# Separate UDS socket +uds_address /var/run/ptp4l-downstream + +[enp4s0] + +[enp4s0d1] +``` + +Small caveat: this split-service design works, but it is not a “pure” PTP boundary clock because the upstream and downstream `ptp4l` instances are separate. A single `ptp4l` boundary clock preserves PTP parent/stepsRemoved behavior better, but your split design avoids the mixed-PHC/JBOD issue. + +## 3. `ptp4l` upstream client service + +Create: + +```bash id="l54yfe" +sudo nano /etc/systemd/system/ptp4l-upstream-client.service +``` + +```ini id="1pnwq3" +[Unit] +Description=PTP upstream client on eno4 +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/sbin/ptp4l -i eno4 -f /etc/linuxptp/ptp4l-upstream-client.conf -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +## 4. `phc2sys` upstream: `eno4 PHC -> CLOCK_REALTIME` + +This syncs the **boundary server’s own Linux system clock**. + +Create: + +```bash id="r430d0" +sudo nano /etc/systemd/system/phc2sys-upstream-to-system.service +``` + +```ini id="eplzh9" +[Unit] +Description=Sync WOLF-HUNTER system clock from upstream eno4 PHC +After=ptp4l-upstream-client.service +Requires=ptp4l-upstream-client.service + +[Service] +Type=simple +ExecStart=/usr/sbin/phc2sys -s eno4 -c CLOCK_REALTIME -w -z /var/run/ptp4l-upstream -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +## 5. `phc2sys` downstream: `CLOCK_REALTIME -> enp4s0 PHC` + +First check whether `enp4s0` and `enp4s0d1` share the same PHC: + +```bash id="ha7hlv" +readlink -f /sys/class/net/enp4s0/device/ptp +readlink -f /sys/class/net/enp4s0d1/device/ptp +ethtool -T enp4s0 | grep 'Hardware timestamp provider' +ethtool -T enp4s0d1 | grep 'Hardware timestamp provider' +``` + +If both are the same PHC, one downstream `phc2sys` service is enough. + +Create: + +```bash id="m3hpvj" +sudo nano /etc/systemd/system/phc2sys-system-to-downstream.service +``` + +```ini id="j9tgaw" +[Unit] +Description=Sync downstream enp4s0 PHC from WOLF-HUNTER system clock +After=phc2sys-upstream-to-system.service ptp4l-downstream-server.service +Requires=phc2sys-upstream-to-system.service ptp4l-downstream-server.service + +[Service] +Type=simple +ExecStart=/usr/sbin/phc2sys -s CLOCK_REALTIME -c enp4s0 -O 37 -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +If `enp4s0d1` is on a **different** PHC from `enp4s0`, create a second downstream PHC sync service too: + +```bash id="tlsjdk" +sudo nano /etc/systemd/system/phc2sys-system-to-downstream-d1.service +``` + +```ini id="70k638" +[Unit] +Description=Sync downstream enp4s0d1 PHC from WOLF-HUNTER system clock +After=phc2sys-upstream-to-system.service ptp4l-downstream-server.service +Requires=phc2sys-upstream-to-system.service ptp4l-downstream-server.service + +[Service] +Type=simple +ExecStart=/usr/sbin/phc2sys -s CLOCK_REALTIME -c enp4s0d1 -O 37 -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +## 6. `ptp4l` downstream server service + +Create: + +```bash id="nl7f0c" +sudo nano /etc/systemd/system/ptp4l-downstream-server.service +``` + +```ini id="46l106" +[Unit] +Description=PTP downstream server on enp4s0 and enp4s0d1 +After=network-online.target phc2sys-upstream-to-system.service +Wants=network-online.target +Requires=phc2sys-upstream-to-system.service + +[Service] +Type=simple +ExecStart=/usr/sbin/ptp4l -i enp4s0 -i enp4s0d1 -f /etc/linuxptp/ptp4l-downstream-server.conf -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +## 7. Enable in order + +```bash id="ze5i6s" +sudo systemctl daemon-reload + +sudo systemctl enable --now ptp4l-upstream-client.service +sudo systemctl enable --now phc2sys-upstream-to-system.service +sudo systemctl enable --now ptp4l-downstream-server.service +sudo systemctl enable --now phc2sys-system-to-downstream.service +``` + +Only enable this one if `enp4s0d1` has a different PHC: + +```bash id="e1i9bw" +sudo systemctl enable --now phc2sys-system-to-downstream-d1.service +``` + +## 8. Disable other system-clock discipliners on WOLF-HUNTER + +On this boundary host, do not let chrony/NTP/timesyncd also adjust `CLOCK_REALTIME` while `phc2sys-upstream-to-system` is doing it: + +```bash id="zzbz88" +sudo systemctl disable --now chrony 2>/dev/null +sudo systemctl disable --now systemd-timesyncd 2>/dev/null +sudo systemctl disable --now ntp 2>/dev/null +``` + +## 9. Verify + +```bash id="j1o8oo" +systemctl status ptp4l-upstream-client.service +systemctl status phc2sys-upstream-to-system.service +systemctl status ptp4l-downstream-server.service +systemctl status phc2sys-system-to-downstream.service +``` + +Logs: + +```bash id="s3klj9" +journalctl -u ptp4l-upstream-client.service -f +journalctl -u phc2sys-upstream-to-system.service -f +journalctl -u ptp4l-downstream-server.service -f +journalctl -u phc2sys-system-to-downstream.service -f +``` + +Expected behavior: + +```text id="a8sadn" +ptp4l-upstream-client: + eno4 should go SLAVE / UNCALIBRATED -> SLAVE + +phc2sys-upstream-to-system: + CLOCK_REALTIME phc offset ... s2 ... + +ptp4l-downstream-server: + enp4s0/enp4s0d1 should assume MASTER/server role + +phc2sys-system-to-downstream: + enp4s0 sys offset ... s2 ... +``` + +Also useful: + +```bash id="n0wowv" +pgrep -a ptp4l +pgrep -a phc2sys +``` diff --git a/clientConfig.md b/clientConfig.md new file mode 100644 index 0000000..6e71a32 --- /dev/null +++ b/clientConfig.md @@ -0,0 +1,206 @@ +# Prereqs + +Here I assume that your NIC is called enp4s0d1 + +Then check if the client supports **hardware-transmit** and **hardware-receive**. + +```bash +ethtool -T enp4s0d1 +``` + +If the client does support these two, continue on + + +# Prepare PTP configs + +First, install the **linuxptp** package. + +Then make sure that no other time synchronization services, like chrony or ntpd or systemd-timesynced, are running. + +```bash +sudo systemctl stop ntpd +sudo systemctl stop chrony +sudo systemctl stop systemd-timesyncd +sudo systemctl disable ntpd +sudo systemctl disable chrony +sudo systemctl disable systemd-timesyncd +``` + + +## Create PTP config file + +sudo vim /etc/linuxptp/ptp4l-client.conf + +```bash +[global] +verbose 1 + +# Hardware timestamping using the NIC PHC +time_stamping hardware + +# Must match your grandmaster +domainNumber 0 +network_transport UDPv4 +delay_mechanism E2E + +# Force this machine to be a PTP client, never grandmaster +clientOnly 1 + +# Useful while tuning +summary_interval 1 + +# Repeat these square brackets for each interface +[enp4s0] +``` + + + + +# Make PTP persistent + +sudo vim /etc/systemd/system/ptp4l-client.service + +```bash +[Unit] +Description=PTP client on enp4s0 +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/sbin/ptp4l -i enp4s0 -f /etc/linuxptp/ptp4l-client.conf -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +sudo vim /etc/systemd/system/phc2sys-client.service + +```bash +[Unit] +Description=Sync system clock from enp4s0 PHC +After=ptp4l-client.service +Requires=ptp4l-client.service + +[Service] +Type=simple +ExecStart=/usr/sbin/phc2sys -s enp4s0 -c CLOCK_REALTIME -w -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + + + +Then reload daemon and activate both PTP client and phc2sys + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now ptp4l-client.service +sudo systemctl enable --now phc2sys-client.service +``` + + + +# Verification + +Verify if PTP and phc2sys works with these commands + +```bash +journalctl -u ptp4l-client -f +journalctl -u phc2sys-client -f +``` + + + +and also verify the status of PTP with these commands + +```bash +pmc -u -b 0 'GET PORT_DATA_SET' +pmc -u -b 0 'GET TIME_STATUS_NP' +pmc -u -b 0 'GET CURRENT_DATA_SET' +``` + + + + +# Boundary clock server + +First, create a ptp4l-boundary-server.conf + +sudo vim /etc/linuxptp/ptp4l-boundary-server.conf + +```bash +[global] +time_stamping hardware +domainNumber 0 +network_transport UDPv4 +delay_mechanism E2E +# boundary_clock_jbod 1 + +# Let BMCA decide: +# upstream-facing port should become SLAVE/client +# downstream-facing port should become MASTER/server +priority1 200 +priority2 128 + +summary_interval 1 + +[eno1] +# toward RPi5 grandmaster + +[ens1f0] +# toward Nexus +``` + + +Here's systemd service configs for a boundary clock server + +sudo vim /etc/systemd/system/ptp4l-boundary-server.service + +```bash +[Unit] +Description=PTP client on , +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/sbin/ptp4l -i -i -f /etc/linuxptp/ptp4l-boundary-server.conf -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +sudo vim /etc/systemd/system/phc2sys-boundary-server.service + +```bash +[Unit] +Description=Sync system clock from enp4s0 PHC +After=ptp4l-boundary-server.service +Requires=ptp4l-boundary-server.service + +[Service] +Type=simple +ExecStart=/usr/sbin/phc2sys -a -rr -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + + +Then reload daemon and activate both PTP client and phc2sys + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now ptp4l-boundary-server.service +sudo systemctl enable --now phc2sys-boundary-server.service +``` diff --git a/conf.md b/conf.md new file mode 100644 index 0000000..b9336fb --- /dev/null +++ b/conf.md @@ -0,0 +1,253 @@ +# Install + +```bash +sudo apt install gpsd gpsd-clients pps-tools chrony linuxptp +``` + +# GPSD + +change */etc/default/gpsd* to have + +```bash +# Default settings for the gpsd init script and the hotplug wrapper. + +# Start the gpsd daemon automatically at boot time +START_DAEMON="true" + +# Use USB hotplugging to add new USB devices automatically to the daemon +USBAUTO="true" + +# Devices gpsd should collect to at boot time. +# They need to be read/writeable, either by user gpsd or the group dialout. +DEVICES="/dev/ttyAMA0 /dev/pps0" + +# Other options you want to pass to gpsd +GPSD_OPTIONS="-n" +``` + +and then restart gpsd.service + +# Chrony + +The PTP server can also be used to server NTP + +edit /etc/chrony.conf to have the following + +```bash +# This directive specifies the location of the file containing ID/key pairs for +# NTP authentication. +keyfile /etc/chrony/chrony.keys + +# This directive specifies the file into which chronyd will store the rate +# information. +driftfile /var/lib/chrony/chrony.drift + +# Save NTS keys and cookies. +ntsdumpdir /var/lib/chrony + +# Uncomment the following line to turn logging on. +#log tracking measurements statistics + +# Log files location. +logdir /var/log/chrony + +# Stop bad estimates upsetting machine clock. +maxupdateskew 100.0 + +# This directive enables kernel synchronisation (every 11 minutes) of the +# real-time clock. Note that it can't be used along with the 'rtcfile' directive. +rtcsync + +# Step the system clock instead of slewing it if the adjustment is larger than +# one second, but only in the first three clock updates. +makestep 1 3 + +# Get TAI-UTC offset and leap seconds from the system tz database. +# This directive must be commented out when using time sources serving +# leap-smeared time. +leapseclist /usr/share/zoneinfo/leap-seconds.list + +# Include configuration files found in /etc/chrony/conf.d. +#confdir /etc/chrony/conf.d + + +# ACL +# I have set it to RFC1918 because I'm using my firewall to explicitly allow access to NTP server +allow 192.168.0.0/16 +allow 10.0.0.0/8 +allow 172.16.0.0/12 + +# Explicitly tell clients that we are stratum 1 +local stratum 1 + + +# Onboard GPS clock + +#refclock SHM 0 refid NMEA offset 0.000 precision 1e-3 poll 3 noselect +refclock SHM 0 refid NMEA offset 0.0 precision 1e-1 poll 3 noselect +refclock PPS /dev/pps0 refid PPS lock NMEA poll 3 + +``` + +then restart chrony.service + +# PTP + + + + + + + + + + + + +sudo mv /etc/linuxptp/ptp4l.conf /etc/linuxptp/ptp4l.conf.original + +Then edit the ptp4l.conf file + +sudo vim /etc/linuxptp/ptp4l.conf + + +```bash +[global] +verbose 1 + +# Use NIC hardware timestamping / PHC +time_stamping hardware + +# PTP domain +domainNumber 0 + +# Normal IEEE 1588 over UDP/IPv4 +network_transport UDPv4 +delay_mechanism E2E + +# Force this node to serve, not become a client/slave (won't revert to slave). +serverOnly 1 + +# BMCA priority; lower wins +priority1 128 +priority2 128 + +# Advertise GNSS/PPS-backed grandmaster quality +# clockClass=6 for GNSS reference +# other classes = https://documentation.nokia.com/srlinux/24-10/books/network-synchronization/ieee-1588-ptp.html +clockClass 6 +clockAccuracy 0x23 +offsetScaledLogVariance 0xFFFF + +# timeSource is where time comes from (this value doesn't discipline anything, it is only informational) + +# | Value | Meaning | Use case | +# | -----: | --------------------- | --------------------------------------- | +# | `0x10` | `ATOMIC_CLOCK` | Direct atomic/rubidium/cesium reference | +# | `0x20` | `GPS` | GPS/GNSS-disciplined time source | +# | `0x30` | `TERRESTRIAL_RADIO` | Radio time source | +# | `0x40` | `PTP` | Synced from another external PTP source | +# | `0x50` | `NTP` | Synced from NTP | +# | `0x60` | `HAND_SET` | Manually set by human/operator | +# | `0x90` | `OTHER` | Other external source | +# | `0xA0` | `INTERNAL_OSCILLATOR` | Free-running/local oscillator/default | + +timeSource 0x20 + +# Useful while testing +summary_interval 1 + +[eth0] +``` + +To test manually run these commands in separate windows. First run the **ptp4l** command, and then the **phc2sys** command + +```bash +sudo ptp4l -i eth0 -f /etc/linuxptp/ptp4l.conf -m +sudo phc2sys -s CLOCK_REALTIME -c /dev/ptp0 -w --step_threshold=0.5 -m + +# -s CLOCK_REALTIME source = Linux system clock, disciplined by chrony/PPS +# -c /dev/ptp0 destination = NIC PTP hardware clock +# -w wait for ptp4l and get UTC/PTP offset from it +# -m print logs +``` + + + +## Verifying + +You can use the following commands to see if the PTP server works correctly. + +```bash +sudo pmc -u -b 0 'GET PORT_DATA_SET' +sudo pmc -u -b 0 'GET GRANDMASTER_SETTINGS_NP' +sudo pmc -u -b 0 'GET TIME_STATUS_NP' +``` + +The first command should return MASTER as portState + +The second command should return 0x20 as the timeSource and 0x23 as the clockAccuracy. Just like we have set in /etc/linuxptp/ptp4l.conf + +The third command returns the gmIdentity which is the MAC address of the NIC + + + +## Making the PTP persistent + +In order to make the PTP persistent, create a systemd service + +sudo vim /etc/systemd/system/ptp4l-gm.service + +```bash +[Unit] +Description=PTP grandmaster on eth0 +After=network-online.target chrony.service +Wants=network-online.target +Requires=chrony.service + +[Service] +Type=simple +ExecStart=/usr/sbin/ptp4l -i eth0 -f /etc/linuxptp/ptp4l.conf -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +and a second phyc2sys service + +sudo vim /etc/systemd/system/phc2sys-gm.service + +```bash +[Unit] +Description=Sync eth0 PHC from chrony-disciplined system clock +After=ptp4l-gm.service chrony.service +Requires=ptp4l-gm.service chrony.service + +[Service] +Type=simple +ExecStart=/usr/sbin/phc2sys -s CLOCK_REALTIME -c /dev/ptp0 -O 37 --step_threshold=0.5 -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +### Enable the services + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now ptp4l-gm.service +sudo systemctl enable --now phc2sys-gm.service +``` + + + + + + + + + diff --git a/exporters/examples/alerts.yml b/exporters/examples/alerts.yml new file mode 100644 index 0000000..b30223e --- /dev/null +++ b/exporters/examples/alerts.yml @@ -0,0 +1,51 @@ +groups: + - name: ptp + rules: + - alert: PTPExporterDown + expr: up{job=~"ptp-linux|ptp-nexus"} == 0 + for: 2m + labels: + severity: warning + annotations: + summary: "PTP exporter is unreachable on {{ $labels.instance }}" + + - alert: LinuxPTPScrapeFailed + expr: ptp_exporter_scrape_success{stage!="ok"} == 0 + for: 2m + labels: + severity: warning + annotations: + summary: "linuxptp PMC scrape failed on {{ $labels.instance }}" + + - alert: PTPGrandmasterMissing + expr: ptp_gm_present == 0 + for: 30s + labels: + severity: critical + annotations: + summary: "PTP grandmaster is not present on {{ $labels.instance }}" + + - alert: NexusPTPUnlocked + expr: ptp_nexus_clock_locked == 0 + for: 30s + labels: + severity: critical + annotations: + summary: "Nexus PTP clock is not locked" + + # Start conservatively and tighten this after observing normal behavior. + - alert: PTPMasterOffsetHigh + expr: abs(ptp_master_offset_nanoseconds) > 1000 + for: 5m + labels: + severity: warning + annotations: + summary: "PTP offset exceeds 1 us on {{ $labels.instance }}" + + - alert: NexusPTPMasterOffsetHigh + expr: abs(ptp_nexus_offset_from_master_nanoseconds) > 1000 + for: 5m + labels: + severity: warning + annotations: + summary: "Nexus PTP offset exceeds 1 us"