RPi5-PTP-server/BoundaryClock/README.md

# Linux PTP Boundary Clock

In this case a server bridges the Raspberry Pi 5 grandmaster's RJ45 connection into the two QSFP+ links that feed NEXUS (the C93180LC-EX).

The physical PTP topology is:

```text
RPi5 Grandmaster
      │
      │ RJ45
      \/
    eno4
 Linux boundary clock
      │
      ├─ enp4s0   ── QSFP+ ──> NEXUS Eth1/5
      └─ enp5s0d1 ─ QSFP+ ──> NEXUS Eth1/6
```

`eno4` is one of the servers's onboard RJ45 NICs. `enp4s0` and `enp5s0d1` are the two ports on the Mellanox ConnectX-3 Pro card.

It runs one multi-port `ptp4l` process as a real boundary clock. `phc2sys -a -r` follows the `ptp4l` port states and synchronizes the separate PHCs plus the Linux system clock.

## Files in this directory

| File | Install as | Purpose |
|---|---|---|
| `ptp4l-boundary.conf` | `/etc/linuxptp/ptp4l-boundary.conf` | Three-port boundary-clock configuration |
| `ptp4l-boundary.service` | `/etc/systemd/system/ptp4l-boundary.service` | Starts `ptp4l` on all three interfaces |
| `phc2sys-boundary.service` | `/etc/systemd/system/phc2sys-boundary.service` | Synchronizes the separate PHCs and `CLOCK_REALTIME` |

## 1. Install the required packages

```bash
sudo apt update
sudo apt install linuxptp ethtool
```

## 2. Verify interface names and hardware timestamping

The bundled configuration expects exactly these interfaces:

```text
eno4       upstream to RPi5
enp4s0     downstream to NEXUS
enp5s0d1   downstream to NEXUS
```

Verify them:

```bash
ip -br link show eno4 enp4s0 enp5s0d1
```

Check hardware timestamping on all three ports:

```bash
ethtool -T eno4
ethtool -T enp4s0
ethtool -T enp5s0d1
```

Each PTP port needs hardware TX/RX/raw timestamp support and a PHC.

Check which PHC belongs to each interface:

```bash
for i in eno4 enp4s0 enp5s0d1; do
    echo "=== $i ==="
    ethtool -T "$i" | grep -E 'PTP Hardware Clock|SOF_TIMESTAMPING_(TX|RX|RAW)_HARDWARE'
    readlink -f /sys/class/net/"$i"/device/ptp/ptp* 2>/dev/null || true
done
```

The expected layout is normally:

```text
eno4                  -> onboard NIC PHC
enp4s0 + enp5s0d1     -> Mellanox PHC
```

Because the upstream onboard NIC and Mellanox card use different PHCs, the configuration intentionally contains:

```text
boundary_clock_jbod 1
```

Do not remove it unless you have verified that every active `ptp4l` port is backed by the same PHC.

## 3. Install the boundary-clock files

From this directory:

```bash
sudo install -D -m 0644 ptp4l-boundary.conf /etc/linuxptp/ptp4l-boundary.conf
sudo install -D -m 0644 ptp4l-boundary.service /etc/systemd/system/ptp4l-boundary.service
sudo install -D -m 0644 phc2sys-boundary.service /etc/systemd/system/phc2sys-boundary.service
```

The PTP config is intentionally asymmetric:

```text
[eno4]
    BMCA may select this as CLIENT/SLAVE to the RPi5

[enp4s0]
    serverOnly 1

[enp5s0d1]
    serverOnly 1
```

Both Mellanox links therefore advertise the server toward NEXUS, while only `eno4` can become the upstream client port.

## 4. Disable competing Linux clock synchronization

`phc2sys-boundary.service` uses `-r`, so the server's `CLOCK_REALTIME` is also disciplined from PTP. Do not leave chrony, ntpd, or systemd-timesyncd simultaneously controlling the system clock.

For example:

```bash
sudo systemctl disable --now chrony 2>/dev/null || true
sudo systemctl disable --now systemd-timesyncd 2>/dev/null || true
sudo systemctl disable --now ntp 2>/dev/null || true
sudo systemctl disable --now ntpd 2>/dev/null || true
```

## 5. Enable the services

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now ptp4l-boundary.service
sudo systemctl enable --now phc2sys-boundary.service
```

The bundled `phc2sys` command is:

```bash
phc2sys -a -r -m
```

Here:

```text
-a   follow ptp4l port states and automatically choose PHC source/sinks
-r   also synchronize CLOCK_REALTIME as a sink
```

Use one `-r`, not `-rr`, for this design. System's system clock should not become an alternate PTP source.

## 6. Expected steady-state port roles

With the RPi5 grandmaster reachable on `eno4`, the server should converge to:

```text
eno4       CLIENT / SLAVE
enp4s0     SERVER / MASTER
enp5s0d1   SERVER / MASTER
```

The grandmaster identity should still be the RPi5 identity, and the server should report approximately:

```text
stepsRemoved = 1
```

The two Mellanox-facing ports are separate redundant paths toward the same Nexus. On NEXUS, one of those Nexus ports should become `Slave` and the other `Passive`.

## 7. Verification

Service state:

```bash
systemctl status ptp4l-boundary
systemctl status phc2sys-boundary
```

Live logs:

```bash
journalctl -fu ptp4l-boundary
journalctl -fu phc2sys-boundary
```

Query the PTP datasets:

```bash
sudo pmc -u -b 0 'GET PORT_DATA_SET'
sudo pmc -u -b 0 'GET CURRENT_DATA_SET'
sudo pmc -u -b 0 'GET PARENT_DATA_SET'
sudo pmc -u -b 0 'GET TIME_STATUS_NP'
```

Things to check:

```text
- eno4 has selected the RPi5 as its upstream master/server
- both Mellanox ports are master/server ports
- grandmasterIdentity is the RPi5 identity
- stepsRemoved is 1
- phc2sys offsets settle near zero
```

If you want to inspect each interface's PTP hardware again:

```bash
ethtool -T eno4
ethtool -T enp4s0
ethtool -T enp5s0d1
```

## 8. Failure behavior

The two NEXUS links are redundant from the Nexus BMCA perspective.

Normal state on NEXUS:

```text
one link   Slave
other link Passive
```

If the selected link fails, the passive link should be eligible to become the new `Slave` port without changing the RPi5 grandmaster identity.

If `eno4` loses the RPi5 completely, the server no longer has its intended upstream reference. Investigate `ptp4l-boundary` and `phc2sys-boundary` before trusting time downstream.

## 9. Important filename/interface consistency

The actual second Mellanox interface is `enp5s0d1`. The `ptp4l-boundary.service` already uses that name, and the configuration in this bundle should also contain:

```text
[enp5s0d1]
serverOnly 1
```

If the OS renames any NIC after a hardware/firmware change, update both `ptp4l-boundary.conf` and `ptp4l-boundary.service` before restarting PTP.