configs for a boundary clock
This commit is contained in:
235
BoundaryClock/README.md
Normal file
235
BoundaryClock/README.md
Normal file
@@ -0,0 +1,235 @@
|
||||
# Linux PTP Boundary Clock
|
||||
|
||||
In this case a server bridges the Raspberry Pi 5 grandmaster's RJ45 connection into the two QSFP+ links that feed NEXUS (the C93180LC-EX).
|
||||
|
||||
The physical PTP topology is:
|
||||
|
||||
```text
|
||||
RPi5 Grandmaster
|
||||
│
|
||||
│ RJ45
|
||||
\/
|
||||
eno4
|
||||
Linux boundary clock
|
||||
│
|
||||
├─ enp4s0 ── QSFP+ ──> NEXUS Eth1/5
|
||||
└─ enp5s0d1 ─ QSFP+ ──> NEXUS Eth1/6
|
||||
```
|
||||
|
||||
`eno4` is one of the servers's onboard RJ45 NICs. `enp4s0` and `enp5s0d1` are the two ports on the Mellanox ConnectX-3 Pro card.
|
||||
|
||||
It runs one multi-port `ptp4l` process as a real boundary clock. `phc2sys -a -r` follows the `ptp4l` port states and synchronizes the separate PHCs plus the Linux system clock.
|
||||
|
||||
## Files in this directory
|
||||
|
||||
| File | Install as | Purpose |
|
||||
|---|---|---|
|
||||
| `ptp4l-boundary.conf` | `/etc/linuxptp/ptp4l-boundary.conf` | Three-port boundary-clock configuration |
|
||||
| `ptp4l-boundary.service` | `/etc/systemd/system/ptp4l-boundary.service` | Starts `ptp4l` on all three interfaces |
|
||||
| `phc2sys-boundary.service` | `/etc/systemd/system/phc2sys-boundary.service` | Synchronizes the separate PHCs and `CLOCK_REALTIME` |
|
||||
|
||||
## 1. Install the required packages
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install linuxptp ethtool
|
||||
```
|
||||
|
||||
## 2. Verify interface names and hardware timestamping
|
||||
|
||||
The bundled configuration expects exactly these interfaces:
|
||||
|
||||
```text
|
||||
eno4 upstream to RPi5
|
||||
enp4s0 downstream to NEXUS
|
||||
enp5s0d1 downstream to NEXUS
|
||||
```
|
||||
|
||||
Verify them:
|
||||
|
||||
```bash
|
||||
ip -br link show eno4 enp4s0 enp5s0d1
|
||||
```
|
||||
|
||||
Check hardware timestamping on all three ports:
|
||||
|
||||
```bash
|
||||
ethtool -T eno4
|
||||
ethtool -T enp4s0
|
||||
ethtool -T enp5s0d1
|
||||
```
|
||||
|
||||
Each PTP port needs hardware TX/RX/raw timestamp support and a PHC.
|
||||
|
||||
Check which PHC belongs to each interface:
|
||||
|
||||
```bash
|
||||
for i in eno4 enp4s0 enp5s0d1; do
|
||||
echo "=== $i ==="
|
||||
ethtool -T "$i" | grep -E 'PTP Hardware Clock|SOF_TIMESTAMPING_(TX|RX|RAW)_HARDWARE'
|
||||
readlink -f /sys/class/net/"$i"/device/ptp/ptp* 2>/dev/null || true
|
||||
done
|
||||
```
|
||||
|
||||
The expected layout is normally:
|
||||
|
||||
```text
|
||||
eno4 -> onboard NIC PHC
|
||||
enp4s0 + enp5s0d1 -> Mellanox PHC
|
||||
```
|
||||
|
||||
Because the upstream onboard NIC and Mellanox card use different PHCs, the configuration intentionally contains:
|
||||
|
||||
```text
|
||||
boundary_clock_jbod 1
|
||||
```
|
||||
|
||||
Do not remove it unless you have verified that every active `ptp4l` port is backed by the same PHC.
|
||||
|
||||
## 3. Install the boundary-clock files
|
||||
|
||||
From this directory:
|
||||
|
||||
```bash
|
||||
sudo install -D -m 0644 ptp4l-boundary.conf /etc/linuxptp/ptp4l-boundary.conf
|
||||
sudo install -D -m 0644 ptp4l-boundary.service /etc/systemd/system/ptp4l-boundary.service
|
||||
sudo install -D -m 0644 phc2sys-boundary.service /etc/systemd/system/phc2sys-boundary.service
|
||||
```
|
||||
|
||||
The PTP config is intentionally asymmetric:
|
||||
|
||||
```text
|
||||
[eno4]
|
||||
BMCA may select this as CLIENT/SLAVE to the RPi5
|
||||
|
||||
[enp4s0]
|
||||
serverOnly 1
|
||||
|
||||
[enp5s0d1]
|
||||
serverOnly 1
|
||||
```
|
||||
|
||||
Both Mellanox links therefore advertise the server toward NEXUS, while only `eno4` can become the upstream client port.
|
||||
|
||||
## 4. Disable competing Linux clock synchronization
|
||||
|
||||
`phc2sys-boundary.service` uses `-r`, so the server's `CLOCK_REALTIME` is also disciplined from PTP. Do not leave chrony, ntpd, or systemd-timesyncd simultaneously controlling the system clock.
|
||||
|
||||
For example:
|
||||
|
||||
```bash
|
||||
sudo systemctl disable --now chrony 2>/dev/null || true
|
||||
sudo systemctl disable --now systemd-timesyncd 2>/dev/null || true
|
||||
sudo systemctl disable --now ntp 2>/dev/null || true
|
||||
sudo systemctl disable --now ntpd 2>/dev/null || true
|
||||
```
|
||||
|
||||
## 5. Enable the services
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now ptp4l-boundary.service
|
||||
sudo systemctl enable --now phc2sys-boundary.service
|
||||
```
|
||||
|
||||
The bundled `phc2sys` command is:
|
||||
|
||||
```bash
|
||||
phc2sys -a -r -m
|
||||
```
|
||||
|
||||
Here:
|
||||
|
||||
```text
|
||||
-a follow ptp4l port states and automatically choose PHC source/sinks
|
||||
-r also synchronize CLOCK_REALTIME as a sink
|
||||
```
|
||||
|
||||
Use one `-r`, not `-rr`, for this design. System's system clock should not become an alternate PTP source.
|
||||
|
||||
## 6. Expected steady-state port roles
|
||||
|
||||
With the RPi5 grandmaster reachable on `eno4`, the server should converge to:
|
||||
|
||||
```text
|
||||
eno4 CLIENT / SLAVE
|
||||
enp4s0 SERVER / MASTER
|
||||
enp5s0d1 SERVER / MASTER
|
||||
```
|
||||
|
||||
The grandmaster identity should still be the RPi5 identity, and the server should report approximately:
|
||||
|
||||
```text
|
||||
stepsRemoved = 1
|
||||
```
|
||||
|
||||
The two Mellanox-facing ports are separate redundant paths toward the same Nexus. On NEXUS, one of those Nexus ports should become `Slave` and the other `Passive`.
|
||||
|
||||
## 7. Verification
|
||||
|
||||
Service state:
|
||||
|
||||
```bash
|
||||
systemctl status ptp4l-boundary
|
||||
systemctl status phc2sys-boundary
|
||||
```
|
||||
|
||||
Live logs:
|
||||
|
||||
```bash
|
||||
journalctl -fu ptp4l-boundary
|
||||
journalctl -fu phc2sys-boundary
|
||||
```
|
||||
|
||||
Query the PTP datasets:
|
||||
|
||||
```bash
|
||||
sudo pmc -u -b 0 'GET PORT_DATA_SET'
|
||||
sudo pmc -u -b 0 'GET CURRENT_DATA_SET'
|
||||
sudo pmc -u -b 0 'GET PARENT_DATA_SET'
|
||||
sudo pmc -u -b 0 'GET TIME_STATUS_NP'
|
||||
```
|
||||
|
||||
Things to check:
|
||||
|
||||
```text
|
||||
- eno4 has selected the RPi5 as its upstream master/server
|
||||
- both Mellanox ports are master/server ports
|
||||
- grandmasterIdentity is the RPi5 identity
|
||||
- stepsRemoved is 1
|
||||
- phc2sys offsets settle near zero
|
||||
```
|
||||
|
||||
If you want to inspect each interface's PTP hardware again:
|
||||
|
||||
```bash
|
||||
ethtool -T eno4
|
||||
ethtool -T enp4s0
|
||||
ethtool -T enp5s0d1
|
||||
```
|
||||
|
||||
## 8. Failure behavior
|
||||
|
||||
The two NEXUS links are redundant from the Nexus BMCA perspective.
|
||||
|
||||
Normal state on NEXUS:
|
||||
|
||||
```text
|
||||
one link Slave
|
||||
other link Passive
|
||||
```
|
||||
|
||||
If the selected link fails, the passive link should be eligible to become the new `Slave` port without changing the RPi5 grandmaster identity.
|
||||
|
||||
If `eno4` loses the RPi5 completely, the server no longer has its intended upstream reference. Investigate `ptp4l-boundary` and `phc2sys-boundary` before trusting time downstream.
|
||||
|
||||
## 9. Important filename/interface consistency
|
||||
|
||||
The actual second Mellanox interface is `enp5s0d1`. The `ptp4l-boundary.service` already uses that name, and the configuration in this bundle should also contain:
|
||||
|
||||
```text
|
||||
[enp5s0d1]
|
||||
serverOnly 1
|
||||
```
|
||||
|
||||
If the OS renames any NIC after a hardware/firmware change, update both `ptp4l-boundary.conf` and `ptp4l-boundary.service` before restarting PTP.
|
||||
Reference in New Issue
Block a user