Re: [Linuxptp-devel] SyncE support
On Tue, Mar 16, 2021 at 11:38:17PM +0100, Frantisek Rysanek wrote: > Still I'm curious what news the i225 may bring in terms of e.g. GPIO > timestamping resolution. Does it get better than those 8 ns > I got used to with the i210 etc. I have not had a chance to play with one yet. If the frequency of the clock followed the increase in the symbol rate of 2.5GBASE-T, it would have a ~4.8ns resolution. > the packet timestamping is done by the NIC HW, so unless you have an > application where you need accurate timing all the way to the > software-based system clock, the determinism of PCIe bus latencies > should not matter much... Right, but in most cases I see people are interested in the accuracy and stability of the system clock. There don't seem to be many applications using hardware timestamps of the PHC. In some cases a sub-microsecond accuracy is required. The CPU<->NIC connection is the most problematic link in the chain. You can buy switches with good PTP support and compensate for asymmetric errors in hardware timestamping to get the PHC accurate to few nanoseconds, but due to the huge PCIe latency with an unknown asymmetry it's difficult to tell how accurate the system clock actually is. With the same NIC but different CPUs/boards you can get very different results. > As for getting the Linux system timebase precise, I guess the random > component to ISR latency has several contributing factors, the PCI-e > bus latency being only one of them... Normally there should be no interrupts involved. The time critical part is just a single read of a 32-bit PCI register. With the I210 that takes about 1700 ns and the asymmetry in my measurements was up to about 100 ns. The shortest delay I saw with faster NICs was about 450 ns, asymmetry unknown. > also, the crystal oscillator > serving as a "stable" local reference for the PC chipset clock > synth is in reality not very stable. I don't have a precise figure, > but if I take the typical i210 NIC 25 MHz xtals for a benchmark, > I've seen instability translating into ppb deviations of about +/- 10 > to +/- 20 ppb, within the 1s period of ptp4l sampling - and a PC > chipset is likely to have an even worse oscillator... I'd say that's normal for an uncompensated XO in a computer case where the temperature can change rapidly. The PTP and phc2sys update rate need to be higher to get better stability. Some PTP profiles use 128 messages/second. -- Miroslav Lichvar ___ Linuxptp-devel mailing list Linuxptp-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-devel
Re: [Linuxptp-devel] SyncE support
On 17 Mar 2021 at 13:04, Miroslav Lichvar wrote: > > Right, but in most cases I see people are interested in the accuracy > and stability of the system clock. There don't seem to be many > applications using hardware timestamps of the PHC. > > In some cases a sub-microsecond accuracy is required. The CPU<->NIC > connection is the most problematic link in the chain. You can buy > switches with good PTP support and compensate for asymmetric errors in > hardware timestamping to get the PHC accurate to few nanoseconds, but > due to the huge PCIe latency with an unknown asymmetry it's difficult > to tell how accurate the system clock actually is. With the same NIC > but different CPUs/boards you can get very different results. > > > As for getting the Linux system timebase precise, I guess the random > > component to ISR latency has several contributing factors, the PCI-e > > bus latency being only one of them... > > Normally there should be no interrupts involved. The time critical > part is just a single read of a 32-bit PCI register. With the I210 > that takes about 1700 ns and the asymmetry in my measurements was up > to about 100 ns. The shortest delay I saw with faster NICs was about > 450 ns, asymmetry unknown. > Thanks for all the data and clarifications :-) That's interesting... I'm not sure the PTM (that you have mentioned previously) will help with asymmetry. You characterize the mechanism as "NTP-like". I'm not a member of the the PCI-SIG, and the only clear text about the PTM that can be found in Google is some change notice by the PCI-SIG from back in 2013, referring to PTM v1.0a. https://fdocuments.in/document/precision-time-measurement-ptm.html That paper clearly says (I'm quoting): "Therefore ((t4 - t1) - (t3 - t2)) effectively gives the round trip message transit time between the two components, and that quantity divided by 2 approximates the Link delay - the time difference between t1 and t2. It is assumed that the Link transit times from PTM Requester to PTM Responder and back again are symmetric, which is typically a good assumption." The current revision listed at the PCI-SIG website is v4.0. Which makes me wonder if this assumption is still considered valid :-) I understand that apart from the NIC (peripheral device / endpoint function in general), PTM must also be supported by the root complex and any PCI-e switches along the path... so a PTM-capable NIC alone will not even suffice for this simple NTP-style mechanism to function, if the host chipset is unaware of PTM. To account for asymmetry, the mechanism would have to resemble PTP = there would have to be a correction field in the PCI-e messages and the PCI-e switches would have to work like PTP TC switches :-) Frank ___ Linuxptp-devel mailing list Linuxptp-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-devel
Re: [Linuxptp-devel] SyncE support
On Wed, Mar 17, 2021 at 04:49:01PM +0100, Frantisek Rysanek wrote: > I understand that apart from the NIC (peripheral device / endpoint > function in general), PTM must also be supported by the root complex > and any PCI-e switches along the path... so a PTM-capable NIC alone > will not even suffice for this simple NTP-style mechanism to > function, if the host chipset is unaware of PTM. Right, all those components are expected to support PTM. It's implemented in the hardware. My understanding is that (at least some) existing CPUs already support it. I have no idea about switches. I guess even if they didn't support it, it could still perform better than what we currently have. As a workaround, at least for testing, the NIC can be inserted in the 16x slot which is connected directly to the CPU. In servers with fast NICs and CPUs with large number of PCIe lanes that could be the usual case. > To account for asymmetry, the mechanism would have to resemble PTP = > there would have to be a correction field in the PCI-e messages and > the PCI-e switches would have to work like PTP TC switches :-) The switches are supposed to work like NTP server and client at the same time (boundary clock in the PTP terminology), so all PCIe links have hardware timestamping on both ends. BTW, at least in theory, a network using boundary clocks should perform better than a similar network using transparent clocks, assuming the servos are well configured and the sync interval is short enough to minimize the time errors in the chain. Divide and conquer :). I think transparent clocks are meant to be the simpler and cheaper variant. -- Miroslav Lichvar ___ Linuxptp-devel mailing list Linuxptp-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linuxptp-devel
