Hi, most NICs don't support timestamping TCP packets. It works for TX on some NICs, but RX is more difficult: of the Intel NICs, only some of the igb (1 Gbit/s) family and the X550 support this.
For RX: I've implemented it for the 82580 igb NIC, but I'm not sure if it still works since the driver refactoring in MoonGen. The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's datasheet is inconsistent about the registers here. For TX (which seems to be your use case): It might work depending on your HW, you can test it: 1. call buf:enableTimestamps() on the buf you are interested in 2. send the packet 3. get the timestamp with queue:getTimestamp(maxWaitMicros) Note that the timestamp is kept in a register on the NIC. It stores only one TX timestamp at a time, irregardless of the number of queues etc. You have to read this register via queue:getTimestamp() before another packet can be timestamped. Our default measureLatency() function might be helpful: https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 <https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64> Paul > Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <[email protected]>: > > Hi Paul, > > If I am not wrong this [1] script enables only timestamps for PTP or UDP > packets. Is this similar functionality available for TCP packets ? > > I am generating multiple TCP flows and I just want to time-stamp first packet > of each flow. Is this possible using the NICs hardware time-stamping > capability ? > > > [1] : > https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua > > <https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua> > > On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <[email protected] > <mailto:[email protected]>> wrote: > Hi, > > the examples in "timestamping-tests.lua" are only meant as a demonstration of > the different timestamping capabilities (and/or as a starting point for a > custom script). > > In your case, you could use a device counter to print the whole throughput of > the device. You can use the default stats task to do that by adding the > following call in the master task: > > stats.startStatsTask({rxDev, txDev}) > > I'll also add the call to the example script in the repository later today as > having this is probably a good idea :) > > Paul > > > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <[email protected] > > <mailto:[email protected]>>: > > > > Hello Emmerich, > > > > MoonGen is really helpful in measuring performance of network devices. > > I wonder if we could get some information about packet loss > > while running timestamps-software.lua? > > > > Thanks, > > Tu > > > > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <[email protected] > > <mailto:[email protected]><mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > > > Hi, > > > > Ajinkya D Kadam: > > I was reading through your paper and I think this tool will be much more > > helpful to me. Btw I am using quad X710 and dual X520 NICs. > > Is this [1] the right code to look at if i want to see how you have > > achieved hardware based time stamping ? > > > > Yes, run this example script with two directly connected ports for a simple > > demo and test of your hardware's capabilities. It will work with both of > > your NICs. > > > > In addition, I want to confirm my understanding of why MoonGen is better > > than PktGen in time stamping context. > > PktGen reads the value of rdtsc which it then appends to packet, this > > adds more delay and hence the precision is bad. > > > > Software timestamping by writing the TSC to the packet is also supported > > (but the API is less nice, see issue #153): > > > > See examples/timestamping-tests/timestamps-software.lua for an example. > > > > The main problem is that there is unpredictable jitter from the NIC and > > PCIe transfer and other random errors. Especially if you are running this > > at higher packet rates. > > This leads to the 200-300ns random error that I've previously mentioned. > > > > > > In case of MoonGen how does this work ? I am not sure. Could you please > > elaborate ? > > > > MoonGen enables the hardware timestamping feature of the NIC and uses it. > > The NIC will store the timestamp in a register which needs to be read > > before another packet can be timestamped, this limits the throughput of > > timestamped packets. However, I've found that you rarely need to timestamp > > *all* packets in a packet generator. You'll have to use software > > timestamping if you really need that. > > > > > > Paul > > > > > > Thanks, > > Ajinkya > > > > > > [1] > > https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua > > > > <https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua> > > > > ᐧ > > > > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <[email protected] > > <mailto:[email protected]><mailto:[email protected] > > <mailto:[email protected]>> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > Hi, > > > > > > Ajinkya D Kadam: > > > > If yes I would like to modify the pktgen code so that each > > transmitting and > > received packet is timestamped. Right now I am exploring the > > example > > applications like rxtx_callbacks which timestamps packets in > > DPDK, Is this > > the right direction to go ? > > > > > > Check out my packet generator MoonGen > > https://github.com/emmericp/MoonGen <https://github.com/emmericp/MoonGen> > > <https://github.com/emmericp/MoonGen > > <https://github.com/emmericp/MoonGen>> > > > > It uses the hardware timestamping features (PTP) to do latency > > measurements in the nanosecond-range. > > > > However, if you will run into hardware limitations if you want to > > timestamp *all* packets. This is sometimes supported on RX (e.g., > > i310, X550) but I don't know a NIC that supports this on TX. > > > > As for the precision that is achievable: ~10ns (depending on the > > NIC) with hardware support. Software timestamping will typically > > result in a standard deviation of 200-300ns under load and there > > will be huge outliers. > > > > > > Paul > > Chair of Network Architectures and Services Department of Informatics TU München Boltzmannstr. 3 85748 Garching bei München, Germany
