Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/19/2012 11:52 PM, Brian Utterback wrote: On 12/19/12 16:49, Brian Utterback wrote: Generally, the PPS signal does not go over the PCI bus. The kernel gets its PPS signal via the serial port. You would therefore like the controller to have its own PPS signal input, but I don't see one in the datasheet. So you are back to worrying about the sync of the kernel clock and the controller clock. It might not matter too much, but it will kind of depend on how the receive timestamp is obtained from the card. The receive timestamp and transmit timestamp have got to be on the same time source or you could run into problems. Brian Actually, the controller does have the equivalent of a PPS input. There is a provision for detecting a level change on a single input line. The controller timestamp of the level change is stored in a register. The driver can read the register and if it knows what the timestamp should have been when that signal came in, it can calculate the offset. There is then a register that it can write the offset to, which results in the controller adjusting its clock by that amount. There is also a facility in the controller to generate a periodic clock based signal. Pretty nifty all told, if the driver makes use of it. Yes, you found it before I had a chance to add here. Nifty is a good work to describe this.:) Not only can you do a one-shot adjustment, but you can control the ns-fraction added at each 8ns tick of the clock using (I hope I remember the name correctly now) TIMINCA to steer and lock the clock to the PPS. The benefit compared to a serial line-connected PPS is of course the lack of a interrupt to read the time register so there should be less jitter in this reading. Generating a PPS output from a second SDP (more or less a GPIO pin) for tracability will give you a clue for how good the whole scheme works in the end. // jwalck -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQ0sdyAAoJEFwg9i9GDX+nUCUP/1ThYa9+S90v/+sIr1fG1yk5 U8MyHMz1T6ZXgXo+jtOsAX4BGPXN06PbkMwAeVzVB9fY8OhD+qfmsQn7RoHHR4B2 e0yJJF12ZZh2aDMbYI2zGuEUF/A31D0rf5xMdt+mCOyMjIjatFj6qJNUvf1E2PpU dbJWLy3GS80l7CjSubmK9lJzMWv2E+0X4aPNKl+FuI3vzxYTVgzQE6Uxpode5ccJ SR5LiarGV4aiBuhD5o45I4HootsMlSAqmyTYlf6kS6hMEidszjpf2yhFCh8W8HOx vDTn++JSYbQm4RAGy4JYUOmEQIKMisLLYCjGrNRzXh4K04hQbj6anola75WadvMA W9ZQFS9d+XaViV84/lX2Qp/bQBjR5G2KgTEZgXSFkNJWmbwfl1t+zQVo1Ux/HDTc mQBUIWC4li6hFjtNPa7kWf4To12GowRtRDq8gKTM4RpVXCXmlOB2/pa+0b+phdnw fmgFYmb6tZAJdc+Z+2Y2wW1dNP4wgmiwD4GeAJp7uz0l0/d5WuYhPCuWr6O6MbB4 9BmdrmeD5N+/iABDqXckyrLMGIfbMPvVJ3dUAq7fLwHFd37ql/akS6Vr6Zj7SSew BxYdRKE3xTy+N4sdG2cmL2F1qnHbaPGlDdITVt7/RUwK4kMr+emgT6MU+ddpHrr3 D4XetYm+LFNviCRf3+rE =1cgw -END PGP SIGNATURE- ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 2012-12-20 10:32, Hal Murray wrote: In article beoAs.13296$ks4.12...@newsfe11.iad, unruh un...@invalid.ca writes: Pick the best one? How would you know what the best one was? Read TSC, read clock, read TSC. Subtract the TSC values to get an upper limit on how long it took to read the clock. Best is fastest. Not sure what you mean by a good clock. It certainly will not be an accurate clock. It may be one whose drift rate is not too bad, although I suspect it will change with temperature. I was thinking that the error in a clock on the network card after reading by the CPU would be better than one done with interrupts on the CPU. Yes, I was assuming something like a sane crystal and yes it would probably drift with temperature. Luckily there is never a need to read the SYSTIM counter directly, since you read H/W timestamps in CMSGs. BR Ulf Samuelsson ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 2012-12-19 17:05, Brian Utterback wrote: On 12/19/12 10:12, Ulf Samuelsson wrote: Now, if you have a PPS signal available and can provide it to both the network controller and the kernel, then you don't have this problem since the PPS signal will sync the time to an accuracy better than the jitter that was introduced. Even without the PPS signal to the kernel, your system might be usable, since the only timestamp used in the kernel will be for the originate/transmit timestamp, and this timestamp will be in sync with the the network controller timestamp by virtue of the use of launchtime. But you will have to be sure that the kernel clock is always a little ahead of the network controller clock, enough so that the actual delay in the stack doesn't cause the packet to reach the controller after the designated launchtime, but not so far ahead that the timestamp wraps (i.e. .5 second). Also, not so far ahead that you get too large a back log in the controller of packets waiting to be sent The desired launchtime is compared to the network controller timestamp counter in H/W, so again there is no need to synchronize with the system time. Yes there is. The ntpd program has to set a timestamp in the outgoing packet and then specify the launchtime when it writes the packet. The goal here is to have the timestamp written in the packet exactly match the time the packet actually hits the wire. So, the timestamp in the packet must be a little in the future when it is written so that by the time the controller gets it the packet can be delayed until the right time. Since ntpd cannot access the clock in the controller, The Network controller will timestamp the message in H/W as it arrives and will add a CMESG with the timestamp, and this is supported in ntpd since some time. You add a small delay to the receive timestamp to make your launchtime. systime does not get involved at any stage, and can be way off. (Note that I am only interested in Stratum 1 server functionality) BR Ulf Samuelsson this requires that the kernel time be relatively close to the controller time. If you can guarantee that they agree to within some upper bound and then add that maximum error to the timestamp written to packet, plus the maximum delay in the stack, then your scheme should work anyway. The key is, what is that maximum error and delay? If they get too close to .25 seconds then it will fail because of the timestamp wrap restrictions of the launchtime register. So you are dependent on how accurately you can synchronize the kernel and network controller's clocks. But I think that the required tolerance should easily be obtainable, so you should be good to go. But remember that the greater that additive term is, the more packets you might need to queue. I couldn't find the spec I saw originally that made me think that there was a three packet limit, but even if that isn't the limit, there is very likely to be some limit. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 2012-12-19 22:49, Brian Utterback wrote: On 12/19/12 14:05, unruh wrote: On 2012-12-19, Hal Murrayhal-use...@ip-64-139-1-69.sjc.megapath.net wrote: In article50d1c5b9.8020...@oracle.com, Brian Utterbackbrian.utterb...@oracle.com writes: No, you are missing the point. You have two clocks in this scenario, the kernel clock and the network controller clock. If one gets a good time then you have to set the other from it. This means that this time will have to travel over the PCI bus which will introduce jitter. Now, if you have a PPS signal available and can provide it to both the network controller and the kernel, then you don't have this problem since the PPS signal will sync the time to an accuracy better than the jitter that was introduced. Doesn't the PPS signal to the kernel have to go over the same PCI bus? I'd guess that you would get better results from a network card. That's assuming it has a good clock. All you have to do is read a counter. There is no interrupt latency. You can also read it several times and pick the best one. Pick the best one? How would you know what the best one was? Not sure what you mean by a good clock. It certainly will not be an accurate clock. It may be one whose drift rate is not too bad, although I suspect it will change with temperature. Generally, the PPS signal does not go over the PCI bus. The kernel gets its PPS signal via the serial port. You would therefore like the controller to have its own PPS signal input, but I don't see one in the datasheet. There are a couple of GPIOs on the part, and you can capture when they toggle. Then you can tweak the clock up and down, until you capture GPIO the toggle at the same time as the SYSTIM counter ns part wraps around from 999,999,999 to 0. Then you have synchronized the SYSTIM with the PPS signal. BR Ulf Samuelsson So you are back to worrying about the sync of the kernel clock and the controller clock. It might not matter too much, but it will kind of depend on how the receive timestamp is obtained from the card. The receive timestamp and transmit timestamp have got to be on the same time source or you could run into problems. Brian ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 19 Dec, 2012, at 11:05 , unruh un...@invalid.ca wrote: On 2012-12-19, Hal Murray hal-use...@ip-64-139-1-69.sjc.megapath.net wrote: Doesn't the PPS signal to the kernel have to go over the same PCI bus? I'd guess that you would get better results from a network card. That's assuming it has a good clock. All you have to do is read a counter. There is no interrupt latency. You can also read it several times and pick the best one. Pick the best one? How would you know what the best one was? If you can sample both the system clock and the network card clock with a read of the counter you can sample with this instruction sequence (though it may take some work to find the magic instructions which do the operations in the order written): read clock_A read clock_B read clock_A Better, less noisy samples will be associated with smaller values of (A_after - A_before). You can eliminate the source of the largest errors in individual samples if you can disable interrupts on the processor core while doing that. You can also take independent samples as often as you need to, if that improves the filtered result. That is, you aren't limited to one sample per second the way the PPS signal is, if 5 or 20 samples per second improve the accuracy of the filtered result you can just take that many. The biggest problem with this isn't jitter, which can be mitigated by taking and filtering additional samples. It is instead the fact that the minimal value of (A_after - A_before) is probably going to be on the order of 100 ns if clock_B is a PCIe peripheral and it will be difficult to know exactly where in that interval clock_B is sampled, leading to a (constant) ambiguity of +/- 50 ns. When I did an FPGA implementation of this for PCI-X I had the read clock_B part capture two timestamps instead, one captured speculatively at the very start of the bus cycle, which was returned to the read, and one at the very end of the bus cycle, which was stored so that it could be read later. That reduced the ambiguity to about +/- 7 ns for that card (though even +/- 50 ns is pretty good). This is a very good way to do time transfers between a pair of clocks. What is needed is a system call interface which recognises that systems often have more than one clock, only one of which will be the system clock, and which provides an API to compare these clocks and independently adjust them. Not sure what you mean by a good clock. It certainly will not be an accurate clock. It may be one whose drift rate is not too bad, although I suspect it will change with temperature. There are two contributors to the ultimate synchronisation quality of a clock, the stability of the frequency source driving it and the speed/accuracy with which you can measure that frequency and adjust for the errors. Dr. Mills's book calls this the Allan intercept; lower is better. The frequency source on the network card may be no better than your system clock but the hardware PPS timestamps allow a speedy and precise determination of that frequency which allows you to adjust for changes in that frequency before a big time offset accumulates (it moves the Allan intercept lower and to the left). A clock which can be kept more accurate is a good clock whether that accuracy is achieved by a more stable frequency source driving the clock or by more accurate frequency measurements against a good reference. Don't fixate on the quality of the crystal alone, there is more to it than that. Dennis Ferguson ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 19 Dec, 2012, at 13:49 , Brian Utterback brian.utterb...@oracle.com wrote: On 12/19/12 14:05, unruh wrote: On 2012-12-19, Hal Murrayhal-use...@ip-64-139-1-69.sjc.megapath.net wrote: Doesn't the PPS signal to the kernel have to go over the same PCI bus? ... Generally, the PPS signal does not go over the PCI bus. The kernel gets its PPS signal via the serial port. You would therefore like the controller to have its own PPS signal input, but I don't see one in the datasheet. The interrupt from the serial port is transmitted across some bus (often by message signalling these days), and it seems to me that a PCIe-attached serial port is probably one of the better situations you could hope for now. The legacy I/O peripherals on many Intel/AMD motherboards, including the serial ports, are instead often attached to the processor support chipset via an LPC bus connection and the latter, being designed as an economical replacement for the ISA bus, is just dead slow. Hardware PPS sampling, and synchonising pairs of internal clocks with PIO operations, is much, much better than anything involving interrupts. So you are back to worrying about the sync of the kernel clock and the controller clock. It might not matter too much, but it will kind of depend on how the receive timestamp is obtained from the card. The receive timestamp and transmit timestamp have got to be on the same time source or you could run into problems. The size of the problem with multiple time sources is limited by the precision with which one clock can be synchronised with another. With a good arrangement for sampling those clocks against each other this problem can be minuscule compared to NTP's requirements. Dennis Ferguson ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
Ulf Samuelsson u...@invalid.emagii.com wrote: Rick Jones skrev 2012-12-13 02:57: It would seem that at least one NIC has the ability to allow one to schedule the time at which a packet will be transmitted. This is called launch time support. Someone in Linux-land has started asking about the prospect of enabling support of that in the stack, ... Yes, My post regarding the new i210 chip from Intel. I am sitting facing Jonatan right now... I suppose I might have known that the world was thusly small :) But while I do not always follow the advice, a wise engineer once told me the correct spelling is ass-u-me :) rick jones -- Don't anthropomorphize computers. They hate that. - Anonymous these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH... ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/20/2012 6:25 AM, Ulf Samuelsson wrote: Yes there is. The ntpd program has to set a timestamp in the outgoing packet and then specify the launchtime when it writes the packet. The goal here is to have the timestamp written in the packet exactly match the time the packet actually hits the wire. So, the timestamp in the packet must be a little in the future when it is written so that by the time the controller gets it the packet can be delayed until the right time. Since ntpd cannot access the clock in the controller, The Network controller will timestamp the message in H/W as it arrives and will add a CMESG with the timestamp, and this is supported in ntpd since some time. You add a small delay to the receive timestamp to make your launchtime. systime does not get involved at any stage, and can be way off. (Note that I am only interested in Stratum 1 server functionality) You have to guarantee that the actual processing delay from controller to controller is never more than .5 seconds minus your small delay, otherwise the timestamp will be off. It would be a good idea to synchronize the system clock so that you can detect that kind of situation. Also, if you are only going to be a stratum 1 server, from where are you going to get the actual time? If you are only going to use the controller timestamps then the controller better be disciplined to some source of time. Some how something needs to be setting the time on the controller. Brian. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
Brian Utterback brian.utterb...@oracle.com wrote: On 12/20/2012 6:25 AM, Ulf Samuelsson wrote: Yes there is. The ntpd program has to set a timestamp in the outgoing packet and then specify the launchtime when it writes the packet. The goal here is to have the timestamp written in the packet exactly match the time the packet actually hits the wire. So, the timestamp in the packet must be a little in the future when it is written so that by the time the controller gets it the packet can be delayed until the right time. Since ntpd cannot access the clock in the controller, The Network controller will timestamp the message in H/W as it arrives and will add a CMESG with the timestamp, and this is supported in ntpd since some time. You add a small delay to the receive timestamp to make your launchtime. systime does not get involved at any stage, and can be way off. (Note that I am only interested in Stratum 1 server functionality) You have to guarantee that the actual processing delay from controller to controller is never more than .5 seconds minus your small delay, otherwise the timestamp will be off. It would be a good idea to synchronize the system clock so that you can detect that kind of situation. We found that an unoptimized system can handle about 50,000 packets per second. Have not digged that deep into ntpd, But so far, I have not seen any trace of multithreading operation, so I assume that this is a single core doing the job. Processing a packet should not take more than 1 second / 50,000 = 20 us in the normal case. Then again, if you send 100s of thousands of packets per second to the server, this will cause the system to basically halt. I measured the server replies to less than 100 per second in this case. This could cause false replies, unless the H/W drops a packet which is queued past its launchtime. Also, if you are only going to be a stratum 1 server, from where are you going to get the actual time? If you are only going to use the controller timestamps then the controller better be disciplined to some source of time. Some how something needs to be setting the time on the controller. I am not 100 % clear on how it is planned to be done. If it can be accomplished, then the 1 PPS pulse should be synchronized with UTC. Then you only have to set the seconds counter in SYSTIM. This can easily be done from any source. Wait until you get the 1 PPS interrupts and then check time. You have one second until seconds counter is updated again. You will be in unsynchronized state until that happens, so your timestamps are not to be trusted. Brian. -- Best Regards, Ulf Samuelsson ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 2012-12-19 04:07, E-Mail Sent to this address will be added to the
BlackLists wrote:
Ulf Samuelsson wrote: BlackLists wrote:
Ulf Samuelsson wrote:
In my proposal, I add a new flag to socket.c:sendto.
While the launchtime is proposed to configurable using
Kconfig, this adds a small performance penalty to every
other packet sent over Ethernet.
An alternative would be to extend the kernel with a new
syscall, which would not impose a penalty to sendto.
Then sendto would not be burdened with launchtime
which for most applications is unneeded.
I do have a feeling that proposing adding syscall's
is not the easy route forward.
Isn't it already supported in kernels 2.6.30 and above?
I'd start by looking at how the 1588 PTP group is already
supporting the features of those Ethernet MACs.
{82576, 82580, i210, i211, i350 ? and perhaps
IXP46X, 82574, 82575, 80579, 82599 ?}
I think TI also has a chip with similar features?
Launch time is only supported H/W wise in the i210/11.
Not in earlier chips.
The latest version of the igb driver does not support launch time
E1000_TQAVCTRL_LAUNCH_ARB, E1000_TQAVCTRL_LAUNCH_TIMER_ENABLE ?
In the header files, yes.
Restrict your search to *.c and you will find NADA...
BR
Ulf
so I do not see how the kernel can support this.
Please state where the launch time code is in the kernel,
if you believe otherwise.
Rx Timestamp is supported, but that is not the issue here.
I'm sorry, I must have been mistaken.
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/18/2012 7:05 PM, Ulf Samuelsson wrote: Brian Utterback brian.utterb...@oracle.com wrote: On 12/13/2012 5:00 AM, Jonatan Walck wrote: This is going to be very hard to get it to be useful. Looking at the specs for the card, the timestamp you give is relative to a clock that is internal to the controller, and is only accurate to the nearest second. That is, it is like the PPS in that it is assumed that the clock is in sync to within .5 seconds to avoid aliasing the timestamp. Brian. The internal clock of the network controller is the PHC for IEEE1588, it has a 1 ns resolution, and can be steered with a 32 bit fractional of 1 ns. see SYSTIML and TIMINCA in the I210 datasheet. // jwalck I know that. The problem is that there is going to be jitter introduced when you set the clock from the kernel. That is generally the problem with IEEE 1588, getting the time from the controller to the kernel and vice versa. If you have to go across a PCI bus for instance that will introduce jitter. Brian No, you capture the time for the 1 PPS pulse in the network controller. Then you tweak the count rate of the timestamp counter up or down. Eventually you will have synchronized the timestamp counter with the 1 PPS pulse, If you run the network contoller from a clocksource derived from the Cesium clock You should get within a few ns from the pulse, It is an inconvenient approach since you can easily synch your timestamp with a 1 PPS pulse using a few counters if you understand the issue. Unfortunately, those counters are not available in the Intel chips, so we have to use it they way they want us to use it. Since launch time will be the arrival time of the client request (through H/W timestamping) + a constant delay, any delays in the PCIe bus will not affect the precision. You just have to make sure that the constant delay added, is longer than the processing time inside the server. No, you are missing the point. You have two clocks in this scenario, the kernel clock and the network controller clock. If one gets a good time then you have to set the other from it. This means that this time will have to travel over the PCI bus which will introduce jitter. Now, if you have a PPS signal available and can provide it to both the network controller and the kernel, then you don't have this problem since the PPS signal will sync the time to an accuracy better than the jitter that was introduced. Even without the PPS signal to the kernel, your system might be usable, since the only timestamp used in the kernel will be for the originate/transmit timestamp, and this timestamp will be in sync with the the network controller timestamp by virtue of the use of launchtime. But you will have to be sure that the kernel clock is always a little ahead of the network controller clock, enough so that the actual delay in the stack doesn't cause the packet to reach the controller after the designated launchtime, but not so far ahead that the timestamp wraps (i.e. .5 second). Also, not so far ahead that you get too large a back log in the controller of packets waiting to be sent. Of course, this also all depends on the controller providing receive timestamps as well. Otherwise you will replace jitter with systematic asymmetric delays, which are worse. Brian ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 2012-12-19 14:48, Brian Utterback wrote: On 12/18/2012 7:05 PM, Ulf Samuelsson wrote: Brian Utterback brian.utterb...@oracle.com wrote: On 12/13/2012 5:00 AM, Jonatan Walck wrote: This is going to be very hard to get it to be useful. Looking at the specs for the card, the timestamp you give is relative to a clock that is internal to the controller, and is only accurate to the nearest second. That is, it is like the PPS in that it is assumed that the clock is in sync to within .5 seconds to avoid aliasing the timestamp. Brian. The internal clock of the network controller is the PHC for IEEE1588, it has a 1 ns resolution, and can be steered with a 32 bit fractional of 1 ns. see SYSTIML and TIMINCA in the I210 datasheet. // jwalck I know that. The problem is that there is going to be jitter introduced when you set the clock from the kernel. That is generally the problem with IEEE 1588, getting the time from the controller to the kernel and vice versa. If you have to go across a PCI bus for instance that will introduce jitter. Brian No, you capture the time for the 1 PPS pulse in the network controller. Then you tweak the count rate of the timestamp counter up or down. Eventually you will have synchronized the timestamp counter with the 1 PPS pulse, If you run the network contoller from a clocksource derived from the Cesium clock You should get within a few ns from the pulse, It is an inconvenient approach since you can easily synch your timestamp with a 1 PPS pulse using a few counters if you understand the issue. Unfortunately, those counters are not available in the Intel chips, so we have to use it they way they want us to use it. Since launch time will be the arrival time of the client request (through H/W timestamping) + a constant delay, any delays in the PCIe bus will not affect the precision. You just have to make sure that the constant delay added, is longer than the processing time inside the server. No, you are missing the point. You have two clocks in this scenario, the kernel clock and the network controller clock. If one gets a good time then you have to set the other from it. This means that this time will have to travel over the PCI bus which will introduce jitter. No, all timestamps in our approach are coming from the network controller so the system time can be way off without anyone caring. Since we are a Stratum 1 server, we just need to get the seconds counter right at init. Until that is done, we are unsynchronized. Now, if you have a PPS signal available and can provide it to both the network controller and the kernel, then you don't have this problem since the PPS signal will sync the time to an accuracy better than the jitter that was introduced. Even without the PPS signal to the kernel, your system might be usable, since the only timestamp used in the kernel will be for the originate/transmit timestamp, and this timestamp will be in sync with the the network controller timestamp by virtue of the use of launchtime. But you will have to be sure that the kernel clock is always a little ahead of the network controller clock, enough so that the actual delay in the stack doesn't cause the packet to reach the controller after the designated launchtime, but not so far ahead that the timestamp wraps (i.e. .5 second). Also, not so far ahead that you get too large a back log in the controller of packets waiting to be sent The desired launchtime is compared to the network controller timestamp counter in H/W, so again there is no need to synchronize with the system time. Of course, this also all depends on the controller providing receive timestamps as well. Otherwise you will replace jitter with systematic asymmetric delays, which are worse. Yes, the launch time will be the H/W receive timestamp + a constant delay. Brian ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/19/12 10:12, Ulf Samuelsson wrote: Now, if you have a PPS signal available and can provide it to both the network controller and the kernel, then you don't have this problem since the PPS signal will sync the time to an accuracy better than the jitter that was introduced. Even without the PPS signal to the kernel, your system might be usable, since the only timestamp used in the kernel will be for the originate/transmit timestamp, and this timestamp will be in sync with the the network controller timestamp by virtue of the use of launchtime. But you will have to be sure that the kernel clock is always a little ahead of the network controller clock, enough so that the actual delay in the stack doesn't cause the packet to reach the controller after the designated launchtime, but not so far ahead that the timestamp wraps (i.e. .5 second). Also, not so far ahead that you get too large a back log in the controller of packets waiting to be sent The desired launchtime is compared to the network controller timestamp counter in H/W, so again there is no need to synchronize with the system time. Yes there is. The ntpd program has to set a timestamp in the outgoing packet and then specify the launchtime when it writes the packet. The goal here is to have the timestamp written in the packet exactly match the time the packet actually hits the wire. So, the timestamp in the packet must be a little in the future when it is written so that by the time the controller gets it the packet can be delayed until the right time. Since ntpd cannot access the clock in the controller, this requires that the kernel time be relatively close to the controller time. If you can guarantee that they agree to within some upper bound and then add that maximum error to the timestamp written to packet, plus the maximum delay in the stack, then your scheme should work anyway. The key is, what is that maximum error and delay? If they get too close to .25 seconds then it will fail because of the timestamp wrap restrictions of the launchtime register. So you are dependent on how accurately you can synchronize the kernel and network controller's clocks. But I think that the required tolerance should easily be obtainable, so you should be good to go. But remember that the greater that additive term is, the more packets you might need to queue. I couldn't find the spec I saw originally that made me think that there was a three packet limit, but even if that isn't the limit, there is very likely to be some limit. -- blu Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. - Martin Golding ---| Brian Utterback - Solaris RPE, Oracle Corporation. Ph:603-262-3916, Em:brian.utterb...@oracle.com ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On Wed, Dec 19, 2012 at 11:05:57AM -0500, Brian Utterback wrote: On 12/19/12 10:12, Ulf Samuelsson wrote: The desired launchtime is compared to the network controller timestamp counter in H/W, so again there is no need to synchronize with the system time. Yes there is. The ntpd program has to set a timestamp in the outgoing packet and then specify the launchtime when it writes the packet. The goal here is to have the timestamp written in the packet exactly match the time the packet actually hits the wire. So, the timestamp in the packet must be a little in the future when it is written so that by the time the controller gets it the packet can be delayed until the right time. Since ntpd cannot access the clock in the controller, this requires that the kernel time be relatively close to the controller time. ntpd can read the clock, much more slowly than the system clock, but still fast enough to send tens of thousands of packets per second. I think it makes more sense to have one loop controlling just the PHC and another, much tighter, syncing the system clock from the PHC, rather than trying to sync the system clock through the PHC. -- Miroslav Lichvar ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 2012-12-19, Hal Murray hal-use...@ip-64-139-1-69.sjc.megapath.net wrote: In article 50d1c5b9.8020...@oracle.com, Brian Utterback brian.utterb...@oracle.com writes: No, you are missing the point. You have two clocks in this scenario, the kernel clock and the network controller clock. If one gets a good time then you have to set the other from it. This means that this time will have to travel over the PCI bus which will introduce jitter. Now, if you have a PPS signal available and can provide it to both the network controller and the kernel, then you don't have this problem since the PPS signal will sync the time to an accuracy better than the jitter that was introduced. Doesn't the PPS signal to the kernel have to go over the same PCI bus? I'd guess that you would get better results from a network card. That's assuming it has a good clock. All you have to do is read a counter. There is no interrupt latency. You can also read it several times and pick the best one. Pick the best one? How would you know what the best one was? Not sure what you mean by a good clock. It certainly will not be an accurate clock. It may be one whose drift rate is not too bad, although I suspect it will change with temperature. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/19/12 14:05, unruh wrote: On 2012-12-19, Hal Murrayhal-use...@ip-64-139-1-69.sjc.megapath.net wrote: In article50d1c5b9.8020...@oracle.com, Brian Utterbackbrian.utterb...@oracle.com writes: No, you are missing the point. You have two clocks in this scenario, the kernel clock and the network controller clock. If one gets a good time then you have to set the other from it. This means that this time will have to travel over the PCI bus which will introduce jitter. Now, if you have a PPS signal available and can provide it to both the network controller and the kernel, then you don't have this problem since the PPS signal will sync the time to an accuracy better than the jitter that was introduced. Doesn't the PPS signal to the kernel have to go over the same PCI bus? I'd guess that you would get better results from a network card. That's assuming it has a good clock. All you have to do is read a counter. There is no interrupt latency. You can also read it several times and pick the best one. Pick the best one? How would you know what the best one was? Not sure what you mean by a good clock. It certainly will not be an accurate clock. It may be one whose drift rate is not too bad, although I suspect it will change with temperature. Generally, the PPS signal does not go over the PCI bus. The kernel gets its PPS signal via the serial port. You would therefore like the controller to have its own PPS signal input, but I don't see one in the datasheet. So you are back to worrying about the sync of the kernel clock and the controller clock. It might not matter too much, but it will kind of depend on how the receive timestamp is obtained from the card. The receive timestamp and transmit timestamp have got to be on the same time source or you could run into problems. Brian -- blu Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. - Martin Golding ---| Brian Utterback - Solaris RPE, Oracle Corporation. Ph:603-262-3916, Em:brian.utterb...@oracle.com ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/19/12 16:49, Brian Utterback wrote: Generally, the PPS signal does not go over the PCI bus. The kernel gets its PPS signal via the serial port. You would therefore like the controller to have its own PPS signal input, but I don't see one in the datasheet. So you are back to worrying about the sync of the kernel clock and the controller clock. It might not matter too much, but it will kind of depend on how the receive timestamp is obtained from the card. The receive timestamp and transmit timestamp have got to be on the same time source or you could run into problems. Brian Actually, the controller does have the equivalent of a PPS input. There is a provision for detecting a level change on a single input line. The controller timestamp of the level change is stored in a register. The driver can read the register and if it knows what the timestamp should have been when that signal came in, it can calculate the offset. There is then a register that it can write the offset to, which results in the controller adjusting its clock by that amount. There is also a facility in the controller to generate a periodic clock based signal. Pretty nifty all told, if the driver makes use of it. -- blu Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. - Martin Golding ---| Brian Utterback - Solaris RPE, Oracle Corporation. Ph:603-262-3916, Em:brian.utterb...@oracle.com ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
Rick Jones skrev 2012-12-13 02:57: It would seem that at least one NIC has the ability to allow one to schedule the time at which a packet will be transmitted. This is called launch time support. Someone in Linux-land has started asking about the prospect of enabling support of that in the stack, with his stated use case being to improve the jitter of NTP-based timekeeping. The initial posting can be found at: http://www.spinics.net/lists/netdev/msg219770.html rick jones Yes, My post regarding the new i210 chip from Intel. I am sitting facing Jonatan right now... One of the main points that Jonatan and I are thinking about is whether launch time should be supported: * inside the kernel, or * NTPD should be rewritten to become a userspace application. -- Allows use of accelerated networking using netmap, pf-ring or similar Would be interested in people's opinion! Anyone working on NTPD as userspace right now? If people think that the kernel approach is the best, then the question is how to implement it. In my proposal, I add a new flag to socket.c:sendto. While the launchtime is proposed to configurable using Kconfig, this adds a small performance penalty to every other packet sent over Ethernet. An alternative would be to extend the kernel with a new syscall, which would not impose a penalty to sendto. Then sendto would not be burdened with launchtime which for most applications is unneeded. I do have a feeling that proposing adding syscall's is not the easy route forward. Regardless of which route to go, the driver will need fixes as well. In my proposal, each driver supporting launchtime would have to be extended with a operation similar to ndo_start_xmit but with a launchtime parameter. Later in the stack, one would have to select between the normal ndo_start_xmit and the new driver operation. I do not think that everything in the networking stack can be duplicated to support launchtime, so this will always introduce another small performance penalty. BR Ulf Samuelsson ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
Ulf Samuelsson wrote:
In my proposal, I add a new flag to socket.c:sendto.
While the launchtime is proposed to configurable using
Kconfig, this adds a small performance penalty to every
other packet sent over Ethernet.
An alternative would be to extend the kernel with a new
syscall, which would not impose a penalty to sendto.
Then sendto would not be burdened with launchtime
which for most applications is unneeded.
I do have a feeling that proposing adding syscall's
is not the easy route forward.
Isn't it already supported in kernels 2.6.30 and above?
I'd start by looking at how the 1588 PTP group is already
supporting the features of those Ethernet MACs.
{82576, 82580, i210, i211, i350 ? and perhaps
IXP46X, 82574, 82575, 80579, 82599 ?}
I think TI also has a chip with similar features?
--
E-Mail Sent to this address blackl...@anitech-systems.com
will be added to the BlackLists.
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
E-Mail Sent to this address will be added to the BlackLists
Null@BlackList.Anitech-Systems.invalid wrote:
Ulf Samuelsson wrote:
In my proposal, I add a new flag to socket.c:sendto.
While the launchtime is proposed to configurable using
Kconfig, this adds a small performance penalty to every
other packet sent over Ethernet.
An alternative would be to extend the kernel with a new
syscall, which would not impose a penalty to sendto.
Then sendto would not be burdened with launchtime
which for most applications is unneeded.
I do have a feeling that proposing adding syscall's
is not the easy route forward.
Isn't it already supported in kernels 2.6.30 and above?
I'd start by looking at how the 1588 PTP group is already
supporting the features of those Ethernet MACs.
{82576, 82580, i210, i211, i350 ? and perhaps
IXP46X, 82574, 82575, 80579, 82599 ?}
I think TI also has a chip with similar features?
Launch time is only supported H/W wise in the i210/11. Not in earlier
chips.
The latest version of the igb driver does not support launch time
so I do not see how the kernel can support this.
Please state where the launch time code is in the kernel, if you believe
otherwise.
Rx Timestamp is supported, but that is not the issue here.
--
Best Regards, Ulf Samuelsson
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
Brian Utterback brian.utterb...@oracle.com wrote: On 12/13/2012 5:00 AM, Jonatan Walck wrote: This is going to be very hard to get it to be useful. Looking at the specs for the card, the timestamp you give is relative to a clock that is internal to the controller, and is only accurate to the nearest second. That is, it is like the PPS in that it is assumed that the clock is in sync to within .5 seconds to avoid aliasing the timestamp. Brian. The internal clock of the network controller is the PHC for IEEE1588, it has a 1 ns resolution, and can be steered with a 32 bit fractional of 1 ns. see SYSTIML and TIMINCA in the I210 datasheet. // jwalck I know that. The problem is that there is going to be jitter introduced when you set the clock from the kernel. That is generally the problem with IEEE 1588, getting the time from the controller to the kernel and vice versa. If you have to go across a PCI bus for instance that will introduce jitter. Brian No, you capture the time for the 1 PPS pulse in the network controller. Then you tweak the count rate of the timestamp counter up or down. Eventually you will have synchronized the timestamp counter with the 1 PPS pulse, If you run the network contoller from a clocksource derived from the Cesium clock You should get within a few ns from the pulse, It is an inconvenient approach since you can easily synch your timestamp with a 1 PPS pulse using a few counters if you understand the issue. Unfortunately, those counters are not available in the Intel chips, so we have to use it they way they want us to use it. Since launch time will be the arrival time of the client request (through H/W timestamping) + a constant delay, any delays in the PCIe bus will not affect the precision. You just have to make sure that the constant delay added, is longer than the processing time inside the server. -- Best Regards, Ulf Samuelsson ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
Ulf Samuelsson wrote: BlackLists wrote:
Ulf Samuelsson wrote:
In my proposal, I add a new flag to socket.c:sendto.
While the launchtime is proposed to configurable using
Kconfig, this adds a small performance penalty to every
other packet sent over Ethernet.
An alternative would be to extend the kernel with a new
syscall, which would not impose a penalty to sendto.
Then sendto would not be burdened with launchtime
which for most applications is unneeded.
I do have a feeling that proposing adding syscall's
is not the easy route forward.
Isn't it already supported in kernels 2.6.30 and above?
I'd start by looking at how the 1588 PTP group is already
supporting the features of those Ethernet MACs.
{82576, 82580, i210, i211, i350 ? and perhaps
IXP46X, 82574, 82575, 80579, 82599 ?}
I think TI also has a chip with similar features?
Launch time is only supported H/W wise in the i210/11.
Not in earlier chips.
The latest version of the igb driver does not support launch time
E1000_TQAVCTRL_LAUNCH_ARB, E1000_TQAVCTRL_LAUNCH_TIMER_ENABLE ?
so I do not see how the kernel can support this.
Please state where the launch time code is in the kernel,
if you believe otherwise.
Rx Timestamp is supported, but that is not the issue here.
I'm sorry, I must have been mistaken.
--
E-Mail Sent to this address blackl...@anitech-systems.com
will be added to the BlackLists.
___
questions mailing list
questions@lists.ntp.org
http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/13/2012 03:52 AM, Brian Utterback wrote: On 12/12/2012 8:57 PM, Rick Jones wrote: It would seem that at least one NIC has the ability to allow one to schedule the time at which a packet will be transmitted. This is called launch time support. Someone in Linux-land has started asking about the prospect of enabling support of that in the stack, with his stated use case being to improve the jitter of NTP-based timekeeping. The initial posting can be found at: http://www.spinics.net/lists/netdev/msg219770.html rick jones This is going to be very hard to get it to be useful. Looking at the specs for the card, the timestamp you give is relative to a clock that is internal to the controller, and is only accurate to the nearest second. That is, it is like the PPS in that it is assumed that the clock is in sync to within .5 seconds to avoid aliasing the timestamp. Brian. The internal clock of the network controller is the PHC for IEEE1588, it has a 1 ns resolution, and can be steered with a 32 bit fractional of 1 ns. see SYSTIML and TIMINCA in the I210 datasheet. // jwalck -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQyadZAAoJEFwg9i9GDX+n8wkQALi88DpPl70SL83HYlnELL2T Sp8cypMmBVhneUxRkdeM0I+ZmNtwAt1vCxpVwWGKClPWxW4qXBI9+zwLb61+oR+G YuGjwju+A2ALz9R6pU/r+YJtATzKf0YTNZq+fJa34CTAbsnXrunDk49KUFGSXiyl 7ScGbi2t4y6FtdkL40sKeYqcP1JGPdA104qy/2TIkURHdwVELMfRaO2X4s8CFAOj rBxQ/ANnpWSsy/jsuZa1AADdDpanVo9juip+43C6jehothEo0o8085QoWIQtuXpn EH30EczFMREKKGmhzT1X9DosX8jaDDpA4USg3JYs7dSCggH5H524RzATn3aPQ2Ab /P3o2O9pXjmtEwjMNseO5843HnawjeB27VqQXQiAdLwGhU/qTgQhAkFOijRCqi+r n7b/W1Q52OQK0X4S63aCNWxBuho3rFH+NJ3hE+NEWnmdRk1Xbfq2Pzr8sC3O8fSv 3FjCm9Ruu3pMD9kHJBtM+s+iJ6pIlRkg2nC1bW2xXmEb87wU7Wn+aDCOfgZ/W1dP aKE16aKvClGqwW5qZ5jPVM6TTkwr5F6pNyj/Ig/kWR7QM+PL0Oq+N+A9lw54z/ey aAcLs/q2dhshrhmx+cPm/n9UHDawOzXdcYzmPhheDcrzFrrpm+qF2LElbOyDDqLR eQnHwtksjnaN1HD3VHUe =/UMK -END PGP SIGNATURE- ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/13/2012 5:00 AM, Jonatan Walck wrote: This is going to be very hard to get it to be useful. Looking at the specs for the card, the timestamp you give is relative to a clock that is internal to the controller, and is only accurate to the nearest second. That is, it is like the PPS in that it is assumed that the clock is in sync to within .5 seconds to avoid aliasing the timestamp. Brian. The internal clock of the network controller is the PHC for IEEE1588, it has a 1 ns resolution, and can be steered with a 32 bit fractional of 1 ns. see SYSTIML and TIMINCA in the I210 datasheet. // jwalck I know that. The problem is that there is going to be jitter introduced when you set the clock from the kernel. That is generally the problem with IEEE 1588, getting the time from the controller to the kernel and vice versa. If you have to go across a PCI bus for instance that will introduce jitter. Brian ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/13/2012 02:23 PM, Brian Utterback wrote: On 12/13/2012 5:00 AM, Jonatan Walck wrote: This is going to be very hard to get it to be useful. Looking at the specs for the card, the timestamp you give is relative to a clock that is internal to the controller, and is only accurate to the nearest second. That is, it is like the PPS in that it is assumed that the clock is in sync to within .5 seconds to avoid aliasing the timestamp. Brian. The internal clock of the network controller is the PHC for IEEE1588, it has a 1 ns resolution, and can be steered with a 32 bit fractional of 1 ns. see SYSTIML and TIMINCA in the I210 datasheet. // jwalck I know that. The problem is that there is going to be jitter introduced when you set the clock from the kernel. That is generally the problem with IEEE 1588, getting the time from the controller to the kernel and vice versa. If you have to go across a PCI bus for instance that will introduce jitter. Brian I'm pondering two different use cases and senarios to make this work, and work well; In a generic case: If a client or server (same principle) runs NTP and lets propose run a PPL locking the PHC clock to the local/kernel clock (I admit this creates new challenges), you wouldn't have to move time over the bus at every single query. One could fetch a recieve timestamp from the network controller (actually it'll come along with the packet already, so you don't have to fetch anything) and use that as if it was set from the clock in the kernel. When sending, one would base the wanted time sometime in the future based on the local clock, make it a timed transmit and just let the network controller decide when it is time to put the packet on the wire. In a very specific case, like the one I'm currently working with: Right by the NTP server is UTC, with a PPS*. Connect 1-PPS to the NICs SDP, it can sample it's internal clock without interrupts at the rising edge of the PPS. Use those samples for a PLL to steer the network controllers clock to UTC, using TIMINCA and associates. Now forget about the clocks from the kernel. Just let NTP use the receive timestamps, and set timed transmits whenever in the future that's far enough ahead for the computer to have time to do whatever it is busy doing and putting the packet in the tx queue, and let LaunchTime (as Intel calls it) do the job. I'm interested in both of these cases, but as I mentioned above focusing on the latter. // jwalck * How this is done, time transmission methods from the national time standard and local atomic frequency standards is out of scope for this discussion. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQyeK4AAoJEFwg9i9GDX+nUdsQALjDtOO/HBZFlTHBVc9fyIJi 5ljsoGcz3mQI7sQGpO1GN4VWz0k+rpjFVrwOUiKcDnalpO361+HvvpHVllMtT9wp pysMode7lCVyQtLxxU1tsJuCkCC5LQ6KkECX1Hb3SZlCXKXfUlO74qXC35nPTpJ4 KLoBJ8OFpr8+obdgdR7jCrUZnwAudSImsfdPFRTsKnSNa+2OA7Ja5+L52DE5hSK/ loHyCiq19OMaSAUmQBvyV6rBJJB/flrOqF8Apo9bjdUN1Wym83HbadQcBYgtsCkY Ak81akJymFIP/WMVg8JSM9uVWLHxfvmVB4JXHx6CMi6qnZSwW2FuH+cbCOxvHDCw HhS9USDI8l2JbsYyM94kOzO2mLHiGH4GmEKDBgkZbawu77tCg+FEXJ04ghg6DWZ7 fKW+8iyirapJl4JrRDMiUzf6ZaDTmf0jlbXfZ88aa5crw3hMW9MjMdU54BnqnnxV MbMfuo1RaVomNxT1RtMMg9tEI70+TmpZd+00l7NGWHliZ7h9QA2w0oVuAsRfaqCW dn2DVtVquaOcaDtzzEPrfVzXEz3DtV1KnwkmyxIUk/pRZ4cH8cvO8vsrbqej2XOd vUyUI1q2MdFEwjR6jVjufc5dWqz+S3B/eib3/4KfCMgsRoViBvIfj4yTA06moUoJ fo02sjPJkzsJGRY4R7qs =HMi4 -END PGP SIGNATURE- ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/13/12 09:14, Jonatan Walck wrote: I'm pondering two different use cases and senarios to make this work, and work well; In a generic case: If a client or server (same principle) runs NTP and lets propose run a PPL locking the PHC clock to the local/kernel clock (I admit this creates new challenges), you wouldn't have to move time over the bus at every single query. One could fetch a recieve timestamp from the network controller (actually it'll come along with the packet already, so you don't have to fetch anything) and use that as if it was set from the clock in the kernel. When sending, one would base the wanted time sometime in the future based on the local clock, make it a timed transmit and just let the network controller decide when it is time to put the packet on the wire. In a very specific case, like the one I'm currently working with: Right by the NTP server is UTC, with a PPS*. Connect 1-PPS to the NICs SDP, it can sample it's internal clock without interrupts at the rising edge of the PPS. Use those samples for a PLL to steer the network controllers clock to UTC, using TIMINCA and associates. Now forget about the clocks from the kernel. Just let NTP use the receive timestamps, and set timed transmits whenever in the future that's far enough ahead for the computer to have time to do whatever it is busy doing and putting the packet in the tx queue, and let LaunchTime (as Intel calls it) do the job. I'm interested in both of these cases, but as I mentioned above focusing on the latter. So you have an interesting case of the kernel clock being mostly undisciplined and just using the controller clock. That might work, but it would be exceedingly odd. Also, as I read the specs, I think that there is a maximum of three packets that can be queued for launchtime, you might need to take that into account. Brian. -- blu Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. - Martin Golding ---| Brian Utterback - Solaris RPE, Oracle Corporation. Ph:603-262-3916, Em:brian.utterb...@oracle.com ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/13/2012 03:34 PM, Brian Utterback wrote: On 12/13/12 09:14, Jonatan Walck wrote: I'm pondering two different use cases and senarios to make this work, and work well; In a generic case: If a client or server (same principle) runs NTP and lets propose run a PPL locking the PHC clock to the local/kernel clock (I admit this creates new challenges), you wouldn't have to move time over the bus at every single query. One could fetch a recieve timestamp from the network controller (actually it'll come along with the packet already, so you don't have to fetch anything) and use that as if it was set from the clock in the kernel. When sending, one would base the wanted time sometime in the future based on the local clock, make it a timed transmit and just let the network controller decide when it is time to put the packet on the wire. In a very specific case, like the one I'm currently working with: Right by the NTP server is UTC, with a PPS*. Connect 1-PPS to the NICs SDP, it can sample it's internal clock without interrupts at the rising edge of the PPS. Use those samples for a PLL to steer the network controllers clock to UTC, using TIMINCA and associates. Now forget about the clocks from the kernel. Just let NTP use the receive timestamps, and set timed transmits whenever in the future that's far enough ahead for the computer to have time to do whatever it is busy doing and putting the packet in the tx queue, and let LaunchTime (as Intel calls it) do the job. I'm interested in both of these cases, but as I mentioned above focusing on the latter. So you have an interesting case of the kernel clock being mostly undisciplined and just using the controller clock. That might work, but it would be exceedingly odd. Also, as I read the specs, I think that there is a maximum of three packets that can be queued for launchtime, you might need to take that into account. Brian. Since one wants the timestamps as close to the wire as possible, I don't find doing it in the network controller when it has a good enough clock that odd.:) The three packet limit is only for 82580 iirc, I350 and I210 widens the whole rx queue and can save timestamps for each and every incoming packet. // jwalck -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQyen0AAoJEFwg9i9GDX+n+u8QAIzTEwbzHFX1KKfmhtJb/qMD rlTJOUcNCEp09HeY7XjDxoJiVrVqhVg+3817H7oU/Zmr+8WMMPETC6eCNQzhyczi NNwhc2L9IOUM+P1mkp34zQEAZKDXGDEas3xE+ViKGJWQeeSMotRx8JuwuGKqCHhA zNovxcp8A7ajo2EcBs10mzn46/Hr0N4JSKSny6LO/VAX+TdhjmN2I3SfjLad3DUL spdre9U8fQ3VGHwCuiwTr3GYxrZwsrdBfXqYgQQC+OqvLNKWwJNr8njSUU7DtiC+ qcL6o3gbwhdbzGrzm1z1D18uLMoDZ52HGYe0xdi5dF0eYj9kkz9rgDJvY11tW23m G3QUtZ/4SCc4tTteydg63CCX7uqNVn/JiggESGawZs9sgnTBcJfa5eroCGJWiAmD vZDC8BxP86EJ37ch+5HKlZf+NW15G/x0gnltVfZjTCxt7TP++txXaQje+nh/o+H0 80bJx6kwtN+csFBwFDWNSHckUNRYQl8sLubXPNqGNgnydEYK7i8v2GsGmwGBbPEF S8YU+Kevk/+86xaH6X79caNF35o8AAdCdnxw83bW5PCIQdk6pOGcQIOqC+wOLlPU yzSXNRerrbMYgyBXJUW+ZOWACGHggzJTQORXlOwgc2ECG5PWGlCzvYH7s4cIiV6q 3bPG4KR+4AexlXd+xZKE =wO+t -END PGP SIGNATURE- ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/13/2012 9:45 AM, Jonatan Walck wrote: The three packet limit is only for 82580 iirc, I350 and I210 widens the whole rx queue and can save timestamps for each and every incoming packet. I might be wrong, but from the flowchart on the data sheet, it looks like it only buffers three packets in the launchtime outgoing queue. Brian. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On Thu, Dec 13, 2012 at 08:23:47AM -0500, Brian Utterback wrote: The internal clock of the network controller is the PHC for IEEE1588, it has a 1 ns resolution, and can be steered with a 32 bit fractional of 1 ns. see SYSTIML and TIMINCA in the I210 datasheet. // jwalck I know that. The problem is that there is going to be jitter introduced when you set the clock from the kernel. That is generally the problem with IEEE 1588, getting the time from the controller to the kernel and vice versa. If you have to go across a PCI bus for instance that will introduce jitter. From what I have seen, with multiple readings and some filtering, the jitter is very small, somewhere in nanoseconds or couple tens of nanoseconds. Even if the delay was highly asymmetric, with 2us RTT the error would be only 1 us, which is still much better than the delays causing the error in the TX timestamp on Ethernet. The phc2sys program from the linuxptp project can be used to synchronize the system clock to the PHC or the PHC to the system clock. It can do that via PPS or filtered clock readings. -- Miroslav Lichvar ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/13/2012 03:55 PM, Brian Utterback wrote: On 12/13/2012 9:45 AM, Jonatan Walck wrote: The three packet limit is only for 82580 iirc, I350 and I210 widens the whole rx queue and can save timestamps for each and every incoming packet. I might be wrong, but from the flowchart on the data sheet, it looks like it only buffers three packets in the launchtime outgoing queue. Brian. Sorry, I missunderstood you earlier and replied about rx timestamping, not LaunchTime. You might be correct here, and yes that'd have to be taken into consideration. Right now I cannot find the flow chart showing this limitation though, do you have a pointer? // jwalck -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQykn5AAoJEFwg9i9GDX+n4AYQALXRdXQFYIzIAumQhxSTA8JU 72Xdlu/+LRGKd5/+YdceZ9J0d5hNXhLtDRKP4DClPekXaTcSekTBmiWPJWipSZvT dridaBYo3r0Eunf7TV7iBJ/idTBRGGTnleVwKzwsSdU7B9wYVtKd5fWhFAUAEPzC SBH7xdp2r/98k78MUpIMhhSr39F+F9oP1FaH6fD08QXeXsso6Fkyj/knQ0QXyyIg LkZ1gsPv69KSgXjqYqV3mYgbRLynlh0r6MkprQOjlsQGbUynVPb8JCfradvX7zsv fE85417fNtMKLX6NaxA4PP8KApGRxLV5L64B0R2hksRE2e+mvGOpo2b11bWnPh6g V6+zKs8XrZvLqY/+ro10TZEeXp9TjMkgHzFB++8GNFvpCr+VQA/XXn8CbNL1wTvD f29qOomZRCHYZamMCu2yaPGGu2b5wQdQQtv/Z1EK1s5hZU7vPAKfaPcJtAl/DUBK FVI053lXy7zOK2jucRufQS25alNxunvCK9Sxdcg+XzP69jVNtaCxMOoKDYvLGrBT byi1lG6rHefc0Dq4FXSpy7NFZ7DTqvyEmJKziQs1Cjp7tS/dV3I4xhTshD1T29DZ WDsuVMTEso6C0aZqalpvfXBXlEKF1NPPA+xLGaooJSvgHWQ/HIhISY5ps9hPEpti 3Z++7YylQSqRSxrXNjny =rtTe -END PGP SIGNATURE- ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/13/2012 04:05 PM, Miroslav Lichvar wrote: The phc2sys program from the linuxptp project can be used to synchronize the system clock to the PHC or the PHC to the system clock. It can do that via PPS or filtered clock readings. I've done some intial test with linuxptp a few weeks enough but not taken the time to dig deep enough. phc2sys sounds just like what I've been wanting and considering doing myself, a big thanks for pointing it out! // jwalck -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQykrmAAoJEFwg9i9GDX+n31gP/j9lScbkjagRkDz3W6/1qMpU gT2JyDNfvMkNLH8XPnoEGHDc4goHsIBQqoI1g6q+jqM7iG0TuT4L3hD0uH6W47NM 2+wne7J4LK4KEbdIIFlY56JsxtzCxwK3XibVOCywc06TR6DlRUfYd9NKOMIb66ul JqcDbSIroGBPacazhzwVidsRm4cqRCQ2c0UKsVaXvMEpFpx7WIchUDbCOoKXyQAw mc+E5ghmHn0FqOXhVvRGwQxrYvqJd4ZCqtrqPLy4KIIiW5wxlHVrJTBcfyNsBnoh kWBO5rY3Pnn6pZxavWLCRIk3TRIS8a2Q0GU4Ib3PZI50PtUPAohDV9OPQEkgXwgz jEwP/8TrX13/gUCrfi5wjETTzqP1QH38BJ4FPHj9nZWAJ4MrDA0XNL+QMad+5qbV WusK2N00bQ07LJjFmBD+x3Q/diCJ7D/95j4gEH7+/65lgFa5iGaogyXAsTJ+6gO9 h7ZD4b5Ja6MA1UciqeF8fxuSQTTiV5N0HUG1xMNZBf5NLF0tGTR+QWKtOJXpdv6z 1WdiUA9CggTf1hAfprJKOwpHb8+y+q59w5BBTYRowxu69oksU0ygh0xeaSEJe+VQ WfF0ndEgxoLn9HChnhiDNlMkaMRsCo6gOSR0PMBlHeASaxse5A6sMtTjXpvv4+zw 9sxxEC9+l7FX70pjZDXh =WCPx -END PGP SIGNATURE- ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/13/2012 6:14 AM, Jonatan Walck wrote: ... you wouldn't have to move time over the bus at every single query. One could fetch a receive timestamp from the network controller (actually it'll come along with the packet already, so you don't have to fetch anything) and use that as if it was set from the clock in the kernel. I think 1588 Precision Time Protocol is already doing this kind of stuff with these adapters. -- E-Mail Sent to this address blackl...@anitech-systems.com will be added to the BlackLists. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
[ntp:questions] A proposal to use NIC launch time support to improve NTP
It would seem that at least one NIC has the ability to allow one to schedule the time at which a packet will be transmitted. This is called launch time support. Someone in Linux-land has started asking about the prospect of enabling support of that in the stack, with his stated use case being to improve the jitter of NTP-based timekeeping. The initial posting can be found at: http://www.spinics.net/lists/netdev/msg219770.html rick jones -- Wisdom Teeth are impacted, people are affected by the effects of events. these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH... ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] A proposal to use NIC launch time support to improve NTP
On 12/12/2012 8:57 PM, Rick Jones wrote: It would seem that at least one NIC has the ability to allow one to schedule the time at which a packet will be transmitted. This is called launch time support. Someone in Linux-land has started asking about the prospect of enabling support of that in the stack, with his stated use case being to improve the jitter of NTP-based timekeeping. The initial posting can be found at: http://www.spinics.net/lists/netdev/msg219770.html rick jones This is going to be very hard to get it to be useful. Looking at the specs for the card, the timestamp you give is relative to a clock that is internal to the controller, and is only accurate to the nearest second. That is, it is like the PPS in that it is assumed that the clock is in sync to within .5 seconds to avoid aliasing the timestamp. Brian. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
