diff --git a/BoundaryClock/README.md b/BoundaryClock/README.md new file mode 100644 index 0000000..19ea7c1 --- /dev/null +++ b/BoundaryClock/README.md @@ -0,0 +1,235 @@ +# Linux PTP Boundary Clock + +In this case a server bridges the Raspberry Pi 5 grandmaster's RJ45 connection into the two QSFP+ links that feed NEXUS (the C93180LC-EX). + +The physical PTP topology is: + +```text +RPi5 Grandmaster + │ + │ RJ45 + \/ + eno4 + Linux boundary clock + │ + ├─ enp4s0 ── QSFP+ ──> NEXUS Eth1/5 + └─ enp5s0d1 ─ QSFP+ ──> NEXUS Eth1/6 +``` + +`eno4` is one of the servers's onboard RJ45 NICs. `enp4s0` and `enp5s0d1` are the two ports on the Mellanox ConnectX-3 Pro card. + +It runs one multi-port `ptp4l` process as a real boundary clock. `phc2sys -a -r` follows the `ptp4l` port states and synchronizes the separate PHCs plus the Linux system clock. + +## Files in this directory + +| File | Install as | Purpose | +|---|---|---| +| `ptp4l-boundary.conf` | `/etc/linuxptp/ptp4l-boundary.conf` | Three-port boundary-clock configuration | +| `ptp4l-boundary.service` | `/etc/systemd/system/ptp4l-boundary.service` | Starts `ptp4l` on all three interfaces | +| `phc2sys-boundary.service` | `/etc/systemd/system/phc2sys-boundary.service` | Synchronizes the separate PHCs and `CLOCK_REALTIME` | + +## 1. Install the required packages + +```bash +sudo apt update +sudo apt install linuxptp ethtool +``` + +## 2. Verify interface names and hardware timestamping + +The bundled configuration expects exactly these interfaces: + +```text +eno4 upstream to RPi5 +enp4s0 downstream to NEXUS +enp5s0d1 downstream to NEXUS +``` + +Verify them: + +```bash +ip -br link show eno4 enp4s0 enp5s0d1 +``` + +Check hardware timestamping on all three ports: + +```bash +ethtool -T eno4 +ethtool -T enp4s0 +ethtool -T enp5s0d1 +``` + +Each PTP port needs hardware TX/RX/raw timestamp support and a PHC. + +Check which PHC belongs to each interface: + +```bash +for i in eno4 enp4s0 enp5s0d1; do + echo "=== $i ===" + ethtool -T "$i" | grep -E 'PTP Hardware Clock|SOF_TIMESTAMPING_(TX|RX|RAW)_HARDWARE' + readlink -f /sys/class/net/"$i"/device/ptp/ptp* 2>/dev/null || true +done +``` + +The expected layout is normally: + +```text +eno4 -> onboard NIC PHC +enp4s0 + enp5s0d1 -> Mellanox PHC +``` + +Because the upstream onboard NIC and Mellanox card use different PHCs, the configuration intentionally contains: + +```text +boundary_clock_jbod 1 +``` + +Do not remove it unless you have verified that every active `ptp4l` port is backed by the same PHC. + +## 3. Install the boundary-clock files + +From this directory: + +```bash +sudo install -D -m 0644 ptp4l-boundary.conf /etc/linuxptp/ptp4l-boundary.conf +sudo install -D -m 0644 ptp4l-boundary.service /etc/systemd/system/ptp4l-boundary.service +sudo install -D -m 0644 phc2sys-boundary.service /etc/systemd/system/phc2sys-boundary.service +``` + +The PTP config is intentionally asymmetric: + +```text +[eno4] + BMCA may select this as CLIENT/SLAVE to the RPi5 + +[enp4s0] + serverOnly 1 + +[enp5s0d1] + serverOnly 1 +``` + +Both Mellanox links therefore advertise the server toward NEXUS, while only `eno4` can become the upstream client port. + +## 4. Disable competing Linux clock synchronization + +`phc2sys-boundary.service` uses `-r`, so the server's `CLOCK_REALTIME` is also disciplined from PTP. Do not leave chrony, ntpd, or systemd-timesyncd simultaneously controlling the system clock. + +For example: + +```bash +sudo systemctl disable --now chrony 2>/dev/null || true +sudo systemctl disable --now systemd-timesyncd 2>/dev/null || true +sudo systemctl disable --now ntp 2>/dev/null || true +sudo systemctl disable --now ntpd 2>/dev/null || true +``` + +## 5. Enable the services + +```bash +sudo systemctl daemon-reload +sudo systemctl enable --now ptp4l-boundary.service +sudo systemctl enable --now phc2sys-boundary.service +``` + +The bundled `phc2sys` command is: + +```bash +phc2sys -a -r -m +``` + +Here: + +```text +-a follow ptp4l port states and automatically choose PHC source/sinks +-r also synchronize CLOCK_REALTIME as a sink +``` + +Use one `-r`, not `-rr`, for this design. System's system clock should not become an alternate PTP source. + +## 6. Expected steady-state port roles + +With the RPi5 grandmaster reachable on `eno4`, the server should converge to: + +```text +eno4 CLIENT / SLAVE +enp4s0 SERVER / MASTER +enp5s0d1 SERVER / MASTER +``` + +The grandmaster identity should still be the RPi5 identity, and the server should report approximately: + +```text +stepsRemoved = 1 +``` + +The two Mellanox-facing ports are separate redundant paths toward the same Nexus. On NEXUS, one of those Nexus ports should become `Slave` and the other `Passive`. + +## 7. Verification + +Service state: + +```bash +systemctl status ptp4l-boundary +systemctl status phc2sys-boundary +``` + +Live logs: + +```bash +journalctl -fu ptp4l-boundary +journalctl -fu phc2sys-boundary +``` + +Query the PTP datasets: + +```bash +sudo pmc -u -b 0 'GET PORT_DATA_SET' +sudo pmc -u -b 0 'GET CURRENT_DATA_SET' +sudo pmc -u -b 0 'GET PARENT_DATA_SET' +sudo pmc -u -b 0 'GET TIME_STATUS_NP' +``` + +Things to check: + +```text +- eno4 has selected the RPi5 as its upstream master/server +- both Mellanox ports are master/server ports +- grandmasterIdentity is the RPi5 identity +- stepsRemoved is 1 +- phc2sys offsets settle near zero +``` + +If you want to inspect each interface's PTP hardware again: + +```bash +ethtool -T eno4 +ethtool -T enp4s0 +ethtool -T enp5s0d1 +``` + +## 8. Failure behavior + +The two NEXUS links are redundant from the Nexus BMCA perspective. + +Normal state on NEXUS: + +```text +one link Slave +other link Passive +``` + +If the selected link fails, the passive link should be eligible to become the new `Slave` port without changing the RPi5 grandmaster identity. + +If `eno4` loses the RPi5 completely, the server no longer has its intended upstream reference. Investigate `ptp4l-boundary` and `phc2sys-boundary` before trusting time downstream. + +## 9. Important filename/interface consistency + +The actual second Mellanox interface is `enp5s0d1`. The `ptp4l-boundary.service` already uses that name, and the configuration in this bundle should also contain: + +```text +[enp5s0d1] +serverOnly 1 +``` + +If the OS renames any NIC after a hardware/firmware change, update both `ptp4l-boundary.conf` and `ptp4l-boundary.service` before restarting PTP. diff --git a/BoundaryClock/phc2sys-boundary.service b/BoundaryClock/phc2sys-boundary.service new file mode 100644 index 0000000..438651f --- /dev/null +++ b/BoundaryClock/phc2sys-boundary.service @@ -0,0 +1,15 @@ +# /etc/systemd/system/phc2sys-boundary.service + +[Unit] +Description=Synchronize PTP Boundary Clock PHCs +After=ptp4l-boundary.service +Requires=ptp4l-boundary.service + +[Service] +Type=simple +ExecStart=/usr/sbin/phc2sys -a -r -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target diff --git a/BoundaryClock/ptp4l-boundary.conf b/BoundaryClock/ptp4l-boundary.conf new file mode 100644 index 0000000..70948da --- /dev/null +++ b/BoundaryClock/ptp4l-boundary.conf @@ -0,0 +1,38 @@ +# /etc/linuxptp/ptp4l-boundary.conf + +[global] + +time_stamping hardware + +domainNumber 0 +network_transport UDPv4 +delay_mechanism E2E + +# Needed when the two NICs have different PHCs. +boundary_clock_jbod 1 + +# If both interfaces somehow share the same PHC, remove: +# boundary_clock_jbod 1 + +# We want the RPi GM to beat this clock in BMCA. +priority1 200 +priority2 128 + +summary_interval 1 + + +# Upstream toward RPi5. +# This is allowed to become the PTP client according to BMCA. +[eno4] + + +# Never accept another master from this direction. +# Downstream link 1 to Nexus +[enp4s0] +serverOnly 1 + + +# Downstream link 2 to Nexus +[enp5s0d1] +serverOnly 1 + diff --git a/BoundaryClock/ptp4l-boundary.service b/BoundaryClock/ptp4l-boundary.service new file mode 100644 index 0000000..6bce302 --- /dev/null +++ b/BoundaryClock/ptp4l-boundary.service @@ -0,0 +1,20 @@ +# /etc/systemd/system/ptp4l-boundary.service + +[Unit] +Description=PTP Boundary Clock +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +ExecStart=/usr/sbin/ptp4l \ + -i eno4 \ + -i enp4s0 \ + -i enp5s0d1 \ + -f /etc/linuxptp/ptp4l-boundary.conf \ + -m +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target