Hi,
I ran a deeper investigation with a simple experiment.
- I ran my test twice - with and without traffic.
- I ran it on two different setups (both master and slave are using HW
timestamping):
- Master (Banana Pi with embedded KSZ9477 switch) <--> Slave (Banana
Pi with embedded KSZ9477 switch)
- Master (Intel NUC) <--> Slave (Banana Pi with embedded KSZ9477
switch)
- On each test, I trace after the ptp4l logs (master offset and path
delay).
- On each test, I've also added additional info: the difference between
the current t1-timestamps and the previous t1-timestamps (and the same for
t2, t3 and t4). Then, if the timestamps are accurate enough (and if there
is no switches or other devices in the path between master and slave) we
should see that the difference between <current_t1 - previous_t1> -
<current_t2 - previous_t2> should be very small (same for t3 and t4).
So now let's have a look at the results.
*First Setup: NUC vs KSZ *
Here I trace after offset and path delay:
*----------- no traffic yet -----------*
ptp4l[2307.831]: master offset -547 s2 freq -17060 path delay
7378
ptp4l[2308.892]: master offset -869 s2 freq -17307 path delay
7378
ptp4l[2309.831]: master offset -296 s2 freq -17050 path delay
7378
ptp4l[2310.832]: master offset 77 s2 freq -16857 path delay
7357
ptp4l[2311.832]: master offset 236 s2 freq -16753 path delay
7357
ptp4l[2312.832]: master offset 260 s2 freq -16716 path delay
7378
ptp4l[2314.080]: master offset 278 s2 freq -16678 path delay
7378
ptp4l[2314.898]: master offset 61 s2 freq -16781 path delay
7378
ptp4l[2315.832]: master offset 123 s2 freq -16738 path delay
7357
ptp4l[2316.923]: master offset 141 s2 freq -16715 path delay
7357
ptp4l[2317.832]: master offset 95 s2 freq -16728 path delay
7349
ptp4l[2318.832]: master offset 115 s2 freq -16707 path delay
7349
ptp4l[2319.832]: master offset -528 s2 freq -17080 path delay
7253
ptp4l[2320.971]: master offset -832 s2 freq -17315 path delay
7253
ptp4l[2321.832]: master offset -324 s2 freq -17094 path delay
7253
ptp4l[2323.004]: master offset -575 s2 freq -17277 path delay
7253
ptp4l[2323.868]: master offset 10 s2 freq -16984 path delay
7226
*----------- start traffic -----------*
ptp4l[2324.832]: master offset 343 s2 freq -16784 path delay
7226
ptp4l[2325.832]: master offset 495 s2 freq -16658 path delay
7226
ptp4l[2326.832]: master offset 2754 s2 freq -15253 path delay
5022
ptp4l[2327.832]: master offset 1291 s2 freq -15856 path delay
5022
ptp4l[2328.833]: master offset 431 s2 freq -16243 path delay
5022
ptp4l[2329.832]: master offset 1036 s2 freq -15836 path delay
5022
ptp4l[2330.832]: master offset 1484 s2 freq -15464 path delay
5022
ptp4l[2331.845]: master offset 260 s2 freq -16050 path delay
5022
ptp4l[2332.833]: master offset -396 s2 freq -16417 path delay
5022
ptp4l[2333.861]: master offset -691 s2 freq -16634 path delay
5022
ptp4l[2334.832]: master offset 164 s2 freq -16191 path delay
4151
ptp4l[2335.877]: master offset 5863 s2 freq -12755 path delay
875
ptp4l[2336.832]: master offset 2143 s2 freq -14401 path delay
875
ptp4l[2337.832]: master offset -165 s2 freq -15570 path delay
875
ptp4l[2338.832]: master offset -1295 s2 freq -16265 path delay
875
ptp4l[2339.832]: master offset 3157 s2 freq -13724 path delay
875
ptp4l[2340.833]: master offset 4187 s2 freq -12792 path delay
875
ptp4l[2341.832]: master offset 248 s2 freq -14735 path delay
875
ptp4l[2342.832]: master offset 2175 s2 freq -13555 path delay
875
ptp4l[2343.987]: master offset -951 s2 freq -15212 path delay
875
ptp4l[2344.832]: master offset -1805 s2 freq -15819 path delay
875
ptp4l[2345.832]: master offset 1465 s2 freq -14039 path delay
850
Let's summarize what we see:
- Even with no traffic the offset correction is not so good (with HW
timestaming I would expect to get sub 100 nsec offsets).
- When traffic starts, path delays change (no switch is in between
master and slave) and offsets are getting much bigger.
Here in the logs below, we are still with the same setup (NUC vs KSZ), but
now I measure the difference between <current_t1 - previous_t1> -
<current_t2 - previous_t2>, which should be with small jitter, *but it's
NOT* (and I measure the same for t3 and t4).
diff_t1t2_err(23), diff_t3t4_err(-151)
diff_t1t2_err(7), diff_t3t4_err(127)
diff_t1t2_err(30), diff_t3t4_err(-1157)
diff_t1t2_err(884), diff_t3t4_err(-749)
diff_t1t2_err(-277), diff_t3t4_err(1059)
diff_t1t2_err(450), diff_t3t4_err(-301)
diff_t1t2_err(-781), diff_t3t4_err(-450)
diff_t1t2_err(462), diff_t3t4_err(-25)
diff_t1t2_err(-1036), diff_t3t4_err(697)
diff_t1t2_err(-129), diff_t3t4_err(885)
diff_t1t2_err(493), diff_t3t4_err(-1243)
diff_t1t2_err(0), diff_t3t4_err(1030)
diff_t1t2_err(879), diff_t3t4_err(-1959)
diff_t1t2_err(-445), diff_t3t4_err(603)
diff_t1t2_err(320), diff_t3t4_err(-298)
diff_t1t2_err(86), diff_t3t4_err(969)
diff_t1t2_err(-1061), diff_t3t4_err(736)
diff_t1t2_err(625), diff_t3t4_err(-1494)
diff_t1t2_err(326), diff_t3t4_err(-269)
diff_t1t2_err(82), diff_t3t4_err(1847)
diff_t1t2_err(-838), diff_t3t4_err(315)
diff_t1t2_err(-626), diff_t3t4_err(-1781)
diff_t1t2_err(1500), diff_t3t4_err(-593)
diff_t1t2_err(85), diff_t3t4_err(97)
diff_t1t2_err(-1466), diff_t3t4_err(2093)
diff_t1t2_err(658), diff_t3t4_err(-556)
diff_t1t2_err(-1320), diff_t3t4_err(1614)
diff_t1t2_err(1315), diff_t3t4_err(-2557)
diff_t1t2_err(0), diff_t3t4_err(944)
diff_t1t2_err(0), diff_t3t4_err(-1255)
diff_t1t2_err(0), diff_t3t4_err(799)
diff_t1t2_err(901), diff_t3t4_err(-1219)
diff_t1t2_err(-1843), diff_t3t4_err(2344)
diff_t1t2_err(432), diff_t3t4_err(-941)
diff_t1t2_err(-320), diff_t3t4_err(-199)
diff_t1t2_err(376), diff_t3t4_err(1095)
diff_t1t2_err(0), diff_t3t4_err(-358)
diff_t1t2_err(1308), diff_t3t4_err(-2081)
diff_t1t2_err(118), diff_t3t4_err(556)
diff_t1t2_err(0), diff_t3t4_err(389)
diff_t1t2_err(-485), diff_t3t4_err(-154)
diff_t1t2_err(-902), diff_t3t4_err(986)
diff_t1t2_err(793), diff_t3t4_err(-200)
diff_t1t2_err(349), diff_t3t4_err(-553)
diff_t1t2_err(-1861), diff_t3t4_err(1571)
We can see here huge jitters. It can reach above 2 usec.
*Second Setup: KSZ vs KSZ *
Here I trace after offset and path delay:
*----------- no traffic yet -----------*
ptp4l[3343.167]: master offset 62 s2 freq -16440 path delay
22
ptp4l[3344.167]: master offset 71 s2 freq -16428 path delay
22
ptp4l[3345.167]: master offset 75 s2 freq -16419 path delay
22
ptp4l[3346.167]: master offset 61 s2 freq -16419 path delay
22
ptp4l[3347.167]: master offset 49 s2 freq -16421 path delay
22
ptp4l[3348.167]: master offset 30 s2 freq -16427 path delay
22
ptp4l[3349.167]: master offset -30 s2 freq -16459 path delay
22
ptp4l[3350.167]: master offset -10 s2 freq -16451 path delay
22
ptp4l[3351.167]: master offset 9 s2 freq -16441 path delay
22
ptp4l[3352.167]: master offset 19 s2 freq -16434 path delay
22
ptp4l[3353.168]: master offset 13 s2 freq -16436 path delay
22
ptp4l[3354.167]: master offset 17 s2 freq -16432 path delay
22
ptp4l[3355.168]: master offset 9 s2 freq -16435 path delay
22
ptp4l[3356.170]: master offset 5 s2 freq -16436 path delay
22
*----------- start traffic -----------*
ptp4l[3357.168]: master offset -37 s2 freq -16461 path delay
22
ptp4l[3358.168]: master offset -40 s2 freq -16467 path delay
22
ptp4l[3359.168]: master offset -22 s2 freq -16459 path delay
22
ptp4l[3360.168]: master offset -10 s2 freq -16455 path delay
22
ptp4l[3361.168]: master offset -114 s2 freq -16517 path delay
22
ptp4l[3362.268]: master offset -165 s2 freq -16560 path delay
22
ptp4l[3363.168]: master offset -177 s2 freq -16583 path delay
22
ptp4l[3364.168]: master offset -161 s2 freq -16592 path delay
22
ptp4l[3365.168]: master offset -129 s2 freq -16588 path delay
22
ptp4l[3366.168]: master offset -109 s2 freq -16589 path delay
22
ptp4l[3367.169]: master offset -56 s2 freq -16568 path delay
22
ptp4l[3368.168]: master offset 73 s2 freq -16497 path delay
22
ptp4l[3369.180]: master offset 18 s2 freq -16523 path delay
22
ptp4l[3370.169]: master offset -27 s2 freq -16548 path delay
22
ptp4l[3371.169]: master offset -47 s2 freq -16562 path delay
22
ptp4l[3372.169]: master offset -53 s2 freq -16570 path delay
22
ptp4l[3373.169]: master offset 5 s2 freq -16542 path delay
22
ptp4l[3374.169]: master offset -21 s2 freq -16556 path delay
22
ptp4l[3375.169]: master offset -31 s2 freq -16564 path delay
22
Let's summarize what we see:
- Even with heavy traffic, path delay is stable and the offsets are
almost in the sub 100-150 nsec units.
Here in the logs below, we are still with the same setup (KSZ vs KSZ), but
now I measure the difference between <current_t1 - previous_t1> -
<current_t2 - previous_t2>, which should be with small jitter, but it's NOT
(and I measure the same for t3 and t4).
diff_t1t2_err(-3), diff_t3t4_err(4)
diff_t1t2_err(2), diff_t3t4_err(3)
diff_t1t2_err(0), diff_t3t4_err(1)
diff_t1t2_err(-7), diff_t3t4_err(-1)
diff_t1t2_err(11), diff_t3t4_err(-4)
diff_t1t2_err(14), diff_t3t4_err(-10)
diff_t1t2_err(-1), diff_t3t4_err(-3)
diff_t1t2_err(-5), diff_t3t4_err(-4)
diff_t1t2_err(0), diff_t3t4_err(4)
diff_t1t2_err(5), diff_t3t4_err(-5)
diff_t1t2_err(8), diff_t3t4_err(-4)
diff_t1t2_err(-23), diff_t3t4_err(24)
diff_t1t2_err(-7), diff_t3t4_err(11)
diff_t1t2_err(0), diff_t3t4_err(-3)
diff_t1t2_err(-1), diff_t3t4_err(-7)
diff_t1t2_err(20), diff_t3t4_err(-13)
diff_t1t2_err(-3), diff_t3t4_err(0)
diff_t1t2_err(5), diff_t3t4_err(-2)
diff_t1t2_err(-9), diff_t3t4_err(6)
diff_t1t2_err(-3), diff_t3t4_err(0)
diff_t1t2_err(2), diff_t3t4_err(0)
diff_t1t2_err(9), diff_t3t4_err(-8)
diff_t1t2_err(3), diff_t3t4_err(0)
diff_t1t2_err(2), diff_t3t4_err(-11)
diff_t1t2_err(11), diff_t3t4_err(-4)
diff_t1t2_err(-5), diff_t3t4_err(2)
diff_t1t2_err(-8), diff_t3t4_err(12)
diff_t1t2_err(-2), diff_t3t4_err(6)
diff_t1t2_err(8), diff_t3t4_err(-9)
diff_t1t2_err(-1), diff_t3t4_err(-9)
As we can see the jitter is very low (sub 50 nsec).
*Test results:*
- In the KSZ <--> KSZ test, the path delay and offsets were very small
(path delay stable, offsets are below 100 nsec).
- In the NUC <--> KSZ test, the path delay and offsets were very high
(path delay veray between 7.5 usec to <1 usec. offsets can reach up to 5 or
6 usec).
*Conclusions:*
I guess it's something with the NUC :)
*Question:*
Are you familiar with a PTP HW issue with Intel NUC or maybe some driver
issue ?
Here is a bit more details:
uname -r
4.15.0-144-generic
ethtool -i enp0s25
driver: e1000e
version: 3.2.6-k
firmware-version: 0.2-4
expansion-rom-version:
bus-info: 0000:00:19.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
ethtool -T enp0s25
Time stamping parameters for enp0s25:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
off (HWTSTAMP_TX_OFF)
on (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
none (HWTSTAMP_FILTER_NONE)
all (HWTSTAMP_FILTER_ALL)
ptpv1-l4-sync (HWTSTAMP_FILTER_PTP_V1_L4_SYNC)
ptpv1-l4-delay-req (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ)
ptpv2-l4-sync (HWTSTAMP_FILTER_PTP_V2_L4_SYNC)
ptpv2-l4-delay-req (HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ)
ptpv2-l2-sync (HWTSTAMP_FILTER_PTP_V2_L2_SYNC)
ptpv2-l2-delay-req (HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ)
ptpv2-event (HWTSTAMP_FILTER_PTP_V2_EVENT)
ptpv2-sync (HWTSTAMP_FILTER_PTP_V2_SYNC)
ptpv2-delay-req (HWTSTAMP_FILTER_PTP_V2_DELAY_REQ)
Thanks,
Joseph
On Sat, Jun 5, 2021 at 11:52 PM Joseph Matan <joseph.matan...@gmail.com>
wrote:
> Hi Richard,
>
> First, my setup description was wrong, so please let me update it:
>
> Master <--> Slave (with embedded KSZ switch)
>
> - Master is using HW timestamps.
> - Slave has an embedded ptp switch (KSZ9477) which provides the Slave
> device HW timestamps.
>
> 1. Is your answer still valid after my setup description update ?
> 2. To use TC, do I need to run both master and slave in P2P mode ?
>
> Thanks,
> Joseph
>
> On Fri, Jun 4, 2021 at 4:27 PM Richard Cochran <richardcoch...@gmail.com>
> wrote:
>
>> On Thu, Jun 03, 2021 at 10:17:41PM +0300, Joseph Matan wrote:
>> > When there is only ptp traffic, the path delay is almost constant (very
>> low
>> > jitter).
>> > The weird thing (to me anyway...) is that under heavy traffic the path
>> > delay is getting smaller - how does that make sense?
>>
>> Because the network traffic affects the residence time in the switch.
>>
>> > If the switch is not ptp-aware I was expecting the path delay to get
>> bigger.
>>
>> A TC will add the (still variable) residence time into the correction
>> field, and as a result the path delay will remain constant.
>>
>> HTH,
>> Richard
>>
>
_______________________________________________
Linuxptp-users mailing list
Linuxptp-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linuxptp-users
