236 lines
6.0 KiB
Markdown
236 lines
6.0 KiB
Markdown
# Linux PTP Boundary Clock
|
|
|
|
In this case a server bridges the Raspberry Pi 5 grandmaster's RJ45 connection into the two QSFP+ links that feed NEXUS (the C93180LC-EX).
|
|
|
|
The physical PTP topology is:
|
|
|
|
```text
|
|
RPi5 Grandmaster
|
|
│
|
|
│ RJ45
|
|
\/
|
|
eno4
|
|
Linux boundary clock
|
|
│
|
|
├─ enp4s0 ── QSFP+ ──> NEXUS Eth1/5
|
|
└─ enp5s0d1 ─ QSFP+ ──> NEXUS Eth1/6
|
|
```
|
|
|
|
`eno4` is one of the servers's onboard RJ45 NICs. `enp4s0` and `enp5s0d1` are the two ports on the Mellanox ConnectX-3 Pro card.
|
|
|
|
It runs one multi-port `ptp4l` process as a real boundary clock. `phc2sys -a -r` follows the `ptp4l` port states and synchronizes the separate PHCs plus the Linux system clock.
|
|
|
|
## Files in this directory
|
|
|
|
| File | Install as | Purpose |
|
|
|---|---|---|
|
|
| `ptp4l-boundary.conf` | `/etc/linuxptp/ptp4l-boundary.conf` | Three-port boundary-clock configuration |
|
|
| `ptp4l-boundary.service` | `/etc/systemd/system/ptp4l-boundary.service` | Starts `ptp4l` on all three interfaces |
|
|
| `phc2sys-boundary.service` | `/etc/systemd/system/phc2sys-boundary.service` | Synchronizes the separate PHCs and `CLOCK_REALTIME` |
|
|
|
|
## 1. Install the required packages
|
|
|
|
```bash
|
|
sudo apt update
|
|
sudo apt install linuxptp ethtool
|
|
```
|
|
|
|
## 2. Verify interface names and hardware timestamping
|
|
|
|
The bundled configuration expects exactly these interfaces:
|
|
|
|
```text
|
|
eno4 upstream to RPi5
|
|
enp4s0 downstream to NEXUS
|
|
enp5s0d1 downstream to NEXUS
|
|
```
|
|
|
|
Verify them:
|
|
|
|
```bash
|
|
ip -br link show eno4 enp4s0 enp5s0d1
|
|
```
|
|
|
|
Check hardware timestamping on all three ports:
|
|
|
|
```bash
|
|
ethtool -T eno4
|
|
ethtool -T enp4s0
|
|
ethtool -T enp5s0d1
|
|
```
|
|
|
|
Each PTP port needs hardware TX/RX/raw timestamp support and a PHC.
|
|
|
|
Check which PHC belongs to each interface:
|
|
|
|
```bash
|
|
for i in eno4 enp4s0 enp5s0d1; do
|
|
echo "=== $i ==="
|
|
ethtool -T "$i" | grep -E 'PTP Hardware Clock|SOF_TIMESTAMPING_(TX|RX|RAW)_HARDWARE'
|
|
readlink -f /sys/class/net/"$i"/device/ptp/ptp* 2>/dev/null || true
|
|
done
|
|
```
|
|
|
|
The expected layout is normally:
|
|
|
|
```text
|
|
eno4 -> onboard NIC PHC
|
|
enp4s0 + enp5s0d1 -> Mellanox PHC
|
|
```
|
|
|
|
Because the upstream onboard NIC and Mellanox card use different PHCs, the configuration intentionally contains:
|
|
|
|
```text
|
|
boundary_clock_jbod 1
|
|
```
|
|
|
|
Do not remove it unless you have verified that every active `ptp4l` port is backed by the same PHC.
|
|
|
|
## 3. Install the boundary-clock files
|
|
|
|
From this directory:
|
|
|
|
```bash
|
|
sudo install -D -m 0644 ptp4l-boundary.conf /etc/linuxptp/ptp4l-boundary.conf
|
|
sudo install -D -m 0644 ptp4l-boundary.service /etc/systemd/system/ptp4l-boundary.service
|
|
sudo install -D -m 0644 phc2sys-boundary.service /etc/systemd/system/phc2sys-boundary.service
|
|
```
|
|
|
|
The PTP config is intentionally asymmetric:
|
|
|
|
```text
|
|
[eno4]
|
|
BMCA may select this as CLIENT/SLAVE to the RPi5
|
|
|
|
[enp4s0]
|
|
serverOnly 1
|
|
|
|
[enp5s0d1]
|
|
serverOnly 1
|
|
```
|
|
|
|
Both Mellanox links therefore advertise the server toward NEXUS, while only `eno4` can become the upstream client port.
|
|
|
|
## 4. Disable competing Linux clock synchronization
|
|
|
|
`phc2sys-boundary.service` uses `-r`, so the server's `CLOCK_REALTIME` is also disciplined from PTP. Do not leave chrony, ntpd, or systemd-timesyncd simultaneously controlling the system clock.
|
|
|
|
For example:
|
|
|
|
```bash
|
|
sudo systemctl disable --now chrony 2>/dev/null || true
|
|
sudo systemctl disable --now systemd-timesyncd 2>/dev/null || true
|
|
sudo systemctl disable --now ntp 2>/dev/null || true
|
|
sudo systemctl disable --now ntpd 2>/dev/null || true
|
|
```
|
|
|
|
## 5. Enable the services
|
|
|
|
```bash
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now ptp4l-boundary.service
|
|
sudo systemctl enable --now phc2sys-boundary.service
|
|
```
|
|
|
|
The bundled `phc2sys` command is:
|
|
|
|
```bash
|
|
phc2sys -a -r -m
|
|
```
|
|
|
|
Here:
|
|
|
|
```text
|
|
-a follow ptp4l port states and automatically choose PHC source/sinks
|
|
-r also synchronize CLOCK_REALTIME as a sink
|
|
```
|
|
|
|
Use one `-r`, not `-rr`, for this design. System's system clock should not become an alternate PTP source.
|
|
|
|
## 6. Expected steady-state port roles
|
|
|
|
With the RPi5 grandmaster reachable on `eno4`, the server should converge to:
|
|
|
|
```text
|
|
eno4 CLIENT / SLAVE
|
|
enp4s0 SERVER / MASTER
|
|
enp5s0d1 SERVER / MASTER
|
|
```
|
|
|
|
The grandmaster identity should still be the RPi5 identity, and the server should report approximately:
|
|
|
|
```text
|
|
stepsRemoved = 1
|
|
```
|
|
|
|
The two Mellanox-facing ports are separate redundant paths toward the same Nexus. On NEXUS, one of those Nexus ports should become `Slave` and the other `Passive`.
|
|
|
|
## 7. Verification
|
|
|
|
Service state:
|
|
|
|
```bash
|
|
systemctl status ptp4l-boundary
|
|
systemctl status phc2sys-boundary
|
|
```
|
|
|
|
Live logs:
|
|
|
|
```bash
|
|
journalctl -fu ptp4l-boundary
|
|
journalctl -fu phc2sys-boundary
|
|
```
|
|
|
|
Query the PTP datasets:
|
|
|
|
```bash
|
|
sudo pmc -u -b 0 'GET PORT_DATA_SET'
|
|
sudo pmc -u -b 0 'GET CURRENT_DATA_SET'
|
|
sudo pmc -u -b 0 'GET PARENT_DATA_SET'
|
|
sudo pmc -u -b 0 'GET TIME_STATUS_NP'
|
|
```
|
|
|
|
Things to check:
|
|
|
|
```text
|
|
- eno4 has selected the RPi5 as its upstream master/server
|
|
- both Mellanox ports are master/server ports
|
|
- grandmasterIdentity is the RPi5 identity
|
|
- stepsRemoved is 1
|
|
- phc2sys offsets settle near zero
|
|
```
|
|
|
|
If you want to inspect each interface's PTP hardware again:
|
|
|
|
```bash
|
|
ethtool -T eno4
|
|
ethtool -T enp4s0
|
|
ethtool -T enp5s0d1
|
|
```
|
|
|
|
## 8. Failure behavior
|
|
|
|
The two NEXUS links are redundant from the Nexus BMCA perspective.
|
|
|
|
Normal state on NEXUS:
|
|
|
|
```text
|
|
one link Slave
|
|
other link Passive
|
|
```
|
|
|
|
If the selected link fails, the passive link should be eligible to become the new `Slave` port without changing the RPi5 grandmaster identity.
|
|
|
|
If `eno4` loses the RPi5 completely, the server no longer has its intended upstream reference. Investigate `ptp4l-boundary` and `phc2sys-boundary` before trusting time downstream.
|
|
|
|
## 9. Important filename/interface consistency
|
|
|
|
The actual second Mellanox interface is `enp5s0d1`. The `ptp4l-boundary.service` already uses that name, and the configuration in this bundle should also contain:
|
|
|
|
```text
|
|
[enp5s0d1]
|
|
serverOnly 1
|
|
```
|
|
|
|
If the OS renames any NIC after a hardware/firmware change, update both `ptp4l-boundary.conf` and `ptp4l-boundary.service` before restarting PTP.
|