Hi Paul, I was able to read the timestamp value of the packets which are transmitted using MoonGen.
I tried looking how I can synchronize the timestamp(read from the NIC) of the transmitted packet with the packets received on the controller side ? I read about packet time referencing, epoch time however it doesn't seem to do what I want. Is there a way at all which will synchronize the timestamp of those packets ?? On Wed, Dec 14, 2016 at 7:00 PM, Ajinkya D Kadam <[email protected]> wrote: > Thanks a lot for your time and inputs. > > I have added my comments inline. > > On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich <[email protected]> > wrote: > >> Hi, >> >> Ajinkya D Kadam <[email protected]>: >> I want packets to be generated using different IP address and each packet >> should be timestamped at two places. (Please refer the topology attached) >> Once when it is sent from the server (to switch port 1) *i.e t1* and >> next time when it reaches the controller *i.e t2*. I am trying to >> measure the time taken by switch to generate packet in message. Also the >> reason why I want to work with different IP address is the switch doesn't >> currently understand mac addresses. So I can program the flows in the >> switch only using IP address information. >> >> >> I believe that I can generate packets from different IP addresses (the >> above script doesn't have that functionality yet) >> >> >> example for varying IP addresses: https://github.com/ >> emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96 >> > > This is what I was looking for. Thanks so much. > >> but what my concern is, right now the timestamped logic I have written in >> line 48-50 >> <https://gist.github.com/ajinkyakadam/641b70904dbaf7f6f3bc08cf6de9748f#file-hardware-timestamp-lua-L48> >> returns a "nil" as timestamped value when I try to print it. The NIC that >> is being used here is Intel X710. Can you please suggest what might be >> possibly wrong here ? >> >> >> You have to set the timestamp flag on the buffer that you want to >> timestamp via buf:enableTimestamps(). See my previous email for example >> code. >> Also, you only need to call queue:enableTimestamps() once before the main >> loop. >> >> > Ahh..Got the mistake. Thanks for the correction. > >> Next, another solution is to use ptp packets which are timestamped and >> then encapsulate them with packets differing in source IP. However the PTP >> packets timestamped at the server side >> have nanosecond resolution like "26155" (which is nice, also check >> wireshark output attached) however the packets i am capturing at controller >> using libpcap are timestamped according to the unix timestamp which counts >> the number of seconds passed from january 1st 1970. Is there a way in which >> I can receive these packets with the same resolution as the one PTP packets >> are timestamped ? If not is there any other way which I should look for? >> (The controller NIC is X520 Intel NIC) >> >> >> I'd advise against measuring latencies by taking timestamps on different >> devices, that's very complicated to get right (and typically involves a GPS >> receiver for time synchronization). >> (However, some fancy stuff with PTP might work fine, depending on your >> requirements on accuracy and precision). >> Also, using libpcap will lower your accuracy and precision by several >> microseconds if not backed by hardware timestamping (libpcap actually >> supports this for NICs that expose this in the driver, the challenge will >> just be to sync the clocks of the NICs, but I'm not an expert on this). >> For the granularity in the pcap format: You can try using the pcapng >> format which supports storing timestamps with nanosecond granularity. >> >> If I were running such a test, I would use a different test setup: >> * only one server attached to the switch >> > > Agreed. I am sorry the topology that I have shown actually has only one > server. So the controller, Moongen all are running on the same machine. I > drew it that way to make things more clearer. Sorry about that. The purpose > of keeping the same machine is to keep the synchronization intact as you > have said. As the machine is the same can I now just use libpcap feature to > read timestamps ? I read the X710 datasheet > <http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf>. > It seems that "PRTTSYN_TXTIME" is used to store the transmit timestamp and > "PRTTSYN_RXTIME" for received timestamps. Do I need to read these > registers using libpcap ? > > >> * receive the controller traffic via an interface bound to DPDK, do >> whatever you need to do there, then bridge it to the host (e.g., KNI or a >> separate cable) >> > > I believe I can bind the DPDK port to OVS bridge and then run the > controller there. I will try doing this tonight. > > >> >> > >> > This is more complicated, but avoids a lot of pitfalls and provides a nice >> centralized control over the whole setup from your test application. >> You can use a NIC that is capable of timestamping arbitrary RX packets >> for the "bridge" port (e.g., 82580). >> >> > I have a 82571 NIC. I will look into it if this can support RX packet > timestamping. > > >> Paul >> >> >> >> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <[email protected]> >> wrote: >> >>> Hi, >>> >>> most NICs don't support timestamping TCP packets. >>> It works for TX on some NICs, but RX is more difficult: of the Intel >>> NICs, only some of the igb (1 Gbit/s) family and the X550 support this. >>> >>> For RX: >>> I've implemented it for the 82580 igb NIC, but I'm not sure if it still >>> works since the driver refactoring in MoonGen. >>> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's >>> datasheet is inconsistent about the registers here. >>> >>> For TX (which seems to be your use case): >>> It might work depending on your HW, you can test it: >>> >> >> >>> 1. call buf:enableTimestamps() on the buf you are interested in >>> 2. send the packet >>> 3. get the timestamp with queue:getTimestamp(maxWaitMicros) >>> >>> Note that the timestamp is kept in a register on the NIC. It stores only >>> one TX timestamp at a time, irregardless of the number of queues etc. >>> You have to read this register via queue:getTimestamp() before another >>> packet can be timestamped. >>> >>> Our default measureLatency() function might be helpful: >>> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64 >>> >>> >>> Paul >>> >>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <[email protected]>: >>> >>> Hi Paul, >>> >>> If I am not wrong this [1] script enables only timestamps for PTP or UDP >>> packets. Is this similar functionality available for TCP packets ? >>> >>> I am generating multiple TCP flows and I just want to time-stamp first >>> packet of each flow. Is this possible using the NICs hardware time-stamping >>> capability ? >>> >>> >>> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64 >>> 073b57dd8ca82692d3858c/examples/hardware-timestamping.lua >>> >>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> the examples in "timestamping-tests.lua" are only meant as a >>>> demonstration of the different timestamping capabilities (and/or as a >>>> starting point for a custom script). >>>> >>>> In your case, you could use a device counter to print the whole >>>> throughput of the device. You can use the default stats task to do that by >>>> adding the following call in the master task: >>>> >>>> stats.startStatsTask({rxDev, txDev}) >>>> >>>> I'll also add the call to the example script in the repository later >>>> today as having this is probably a good idea :) >>>> >>>> Paul >>>> >>>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <[email protected]>: >>>> > >>>> > Hello Emmerich, >>>> > >>>> > MoonGen is really helpful in measuring performance of network devices. >>>> > I wonder if we could get some information about packet loss >>>> > while running timestamps-software.lua? >>>> > >>>> > Thanks, >>>> > Tu >>>> > >>>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> > >>>> > Hi, >>>> > >>>> > Ajinkya D Kadam: >>>> > I was reading through your paper and I think this tool will be much >>>> more >>>> > helpful to me. Btw I am using quad X710 and dual X520 NICs. >>>> > Is this [1] the right code to look at if i want to see how you have >>>> > achieved hardware based time stamping ? >>>> > >>>> > Yes, run this example script with two directly connected ports for a >>>> simple demo and test of your hardware's capabilities. It will work with >>>> both of your NICs. >>>> > >>>> > In addition, I want to confirm my understanding of why MoonGen is >>>> better >>>> > than PktGen in time stamping context. >>>> > PktGen reads the value of rdtsc which it then appends to packet, this >>>> > adds more delay and hence the precision is bad. >>>> > >>>> > Software timestamping by writing the TSC to the packet is also >>>> supported (but the API is less nice, see issue #153): >>>> > >>>> > See examples/timestamping-tests/timestamps-software.lua for an >>>> example. >>>> > >>>> > The main problem is that there is unpredictable jitter from the NIC >>>> and PCIe transfer and other random errors. Especially if you are running >>>> this at higher packet rates. >>>> > This leads to the 200-300ns random error that I've previously >>>> mentioned. >>>> > >>>> > >>>> > In case of MoonGen how does this work ? I am not sure. Could you >>>> please >>>> > elaborate ? >>>> > >>>> > MoonGen enables the hardware timestamping feature of the NIC and uses >>>> it. The NIC will store the timestamp in a register which needs to be read >>>> before another packet can be timestamped, this limits the throughput of >>>> timestamped packets. However, I've found that you rarely need to timestamp >>>> *all* packets in a packet generator. You'll have to use software >>>> timestamping if you really need that. >>>> > >>>> > >>>> > Paul >>>> > >>>> > >>>> > Thanks, >>>> > Ajinkya >>>> > >>>> > >>>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db6407 >>>> 3b57dd8ca82692d3858c/examples/hardware-timestamping.lua >>>> > >>>> > ᐧ >>>> > >>>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich < >>>> [email protected]<mailto:[email protected]> >>>> > <mailto:[email protected]>> wrote: >>>> > >>>> > Hi, >>>> > >>>> > >>>> > Ajinkya D Kadam: >>>> > >>>> > If yes I would like to modify the pktgen code so that each >>>> > transmitting and >>>> > received packet is timestamped. Right now I am exploring the >>>> > example >>>> > applications like rxtx_callbacks which timestamps packets in >>>> > DPDK, Is this >>>> > the right direction to go ? >>>> > >>>> > >>>> > Check out my packet generator MoonGen >>>> > https://github.com/emmericp/MoonGen >>>> > <https://github.com/emmericp/MoonGen> >>>> > >>>> > It uses the hardware timestamping features (PTP) to do latency >>>> > measurements in the nanosecond-range. >>>> > >>>> > However, if you will run into hardware limitations if you want to >>>> > timestamp *all* packets. This is sometimes supported on RX (e.g., >>>> > i310, X550) but I don't know a NIC that supports this on TX. >>>> > >>>> > As for the precision that is achievable: ~10ns (depending on the >>>> > NIC) with hardware support. Software timestamping will typically >>>> > result in a standard deviation of 200-300ns under load and there >>>> > will be huge outliers. >>>> > >>>> > >>>> > Paul >>>> >>>> >>> >>> Chair of Network Architectures and Services >>> Department of Informatics >>> TU München >>> Boltzmannstr. 3 >>> 85748 Garching bei München, Germany >>> >>> >>> >>> >> <topology.jpg><PTPTimestamp.png><libpcapTimestamping.png> >> >> >> Chair of Network Architectures and Services >> Department of Informatics >> TU München >> Boltzmannstr. 3 >> 85748 Garching bei München, Germany >> >> >> >> >
