Re: [ntp:questions] Accuracy of NTP - Advice Needed
"Paul Sobey" wrote in message news:alpine.lnx.2.00.1112221841020.9...@vulcan.the-annexe.net... Dear All, I work for a firm which requires clocks to be synchronised to quite a high degree of accuracy. We have an existing ntp-based infrastructure but want to improve on it to the point where the bulk of our hosts are synchronised to single digit microseconds of each other if possible. We have about 400 hosts in production, spread across about 15 sites. [] I appreciate these may appear to be silly questions with obvious answers - I am grateful in advance for your patience, and any research sources you may direct me to. Many thanks, Paul Paul, What OS are your hosts running? If it's Windows, millisecond, not microsecond accuracy will be what you can get at best when syncing over the network. If you really need microsecond, I suspect you will be looking at a GPS receiver or two at each site, and distributing the PPS (pulse per second) signal to each host, and praying that the hosts have a serial port connection! Cheers, David ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
Paul Sobey wrote: We have an existing ntp-based infrastructure but want to improve on it to the point where the bulk of our hosts are synchronised to single digit microseconds of each other if possible. We have about 400 hosts in production, spread across about 15 sites. There are very few applications where this sort of accuracy is meaningful over distances of more than single figure kilometres. You will need to consider latency between external events and the application program reading the time, and the resolution of the time presented, by the OS, to application programs. I hear from many vendors and industry colleagues that 'ntp just isn't suitable for high precision work and anything less than 1-2ms precision requires ptp or direct connection to gps clock'. I find these numbers For microsecond accuracy, I would say that NTP needs direct (PPS) connection to a GPS clock. On a very well managed LAN, with very good temperature control and a constant power dissipation on the machines, you might achieve it over LANs. On realistic internet connections, I don't think any technology will achieve it. (I guess you could use the internet to coordinate atomic clocks.) microseconds. Further tuning the ntp config by adding the minpoll 4, maxpoll 6 and burst keywords result in ntpd reported accuracy dropping to within 10-20 microseconds (as reported by ntpq -p and borne out by loopstats). Further improvements can be made running ntpd in the RT Setting tight minpoll and maxpoll gives you poor frequency stability and reduces the tolerances to wander in the round trip asymmetry. priority class. My questions to you all, if you've read through the above waffle are: - what is a sensible expected accuracy of ntpd if pointed at several stratum 1 time sources across a low jitter gigabit network (we'd probably spread them over several UK and US sites for resiliency but all paths are low jitter and highly deterministic latency) You need to consider more than this; for example, ethernet switches can be a major source of degradation. - are there any obvious tunables to improve accuracy other than minpoll/burst and process scheduling class, and how agressive can the polling cycles be sensible made? Very good temperature control, and maintaining a constant load on the machine. It is also essential that you calibrate the system. This either means using GPS or a portable atomic clock. - can ntpd's own reported offset (ntpq -p or loopstats) be trusted (assuming high priority means it gets scheduled as desired)? I've quoted our apparent numbers at several people and the response is always 'pfft you can't trust ntpd to know its own offset' - but nobody can ever tell me why It can be trusted as what it actually measures. It is not a measurement of the error from true time. If it were, and it could be trusted, ntpd would be remiss not to use it to correct the time to a point where the remaining offset was no longer a good measure. The offset on a locked up system should be several times larger than the RMS error in the actual system time. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
Richard B. Gilbert wrote: On 12/22/2011 9:17 PM, Chris Adams wrote: The securities traders (especially HFT) want it. I suspect the OP is in that group. That level of timekeeping has been discussed here before. I think that the radio astronomers are some of the most demanding. They need far better than just PPS accuracy, they normally use common-view GPS in the same mode as is used for comparing UTC master clocks, i.e. with eventual accuracy in the low ns range. This is sufficient to phase-lock multiple radio observatories synthesizing a _very_ long baseline receiver. Terje -- - "almost all programming can be viewed as an exercise in caching" ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] ntp server pool advice
ben slimup wrote: thanks all, let s say, i have one site in california and another one in new york. all clients are independent to each other and connected to adsl. all clients are spread out all over the internet. six ntp servers are located in each site ( CA and NY) and need to connect the internet to send time to all clients. If your clients are on the internet, then you are _NOT_ using private addresses for them! (Or is this a corporate VPN connected via private ADSL links?) lets say that i have control to clients config file meaning that i will config 4 external ip address. how servers will send the ntp traffic over the internet to synch all clients? Servers never send out traffic, they just reply to requests... To Terje i m not an ISP what kind of info you will need? Types of IP addresses available for both servers and clients. Can clients and/or servers communicate with both private and public addresses? Terje Regards ;) From: "terje.mathisen at tmsw.no"@ntp.org To: questions@lists.ntp.org Date: Thu, 22 Dec 2011 19:09:07 +0100 Subject: Re: [ntp:questions] ntp server pool advice ben slimup wrote: Hi Terje, if i do not use nat how can i route private adresse to internet ?, i do not want to use ipv6. What is your setup? Are you an ISP, or do you have some control over the client configuration? Assuming you are using private addresses for all your clients, by far the simplest setup would be to setup your 4-6 private NTP servers in the same private address range, and keep all communication private. If I were you I would allow your internal servers to use multiple external (pool.ntp.org?) as backup for your GPS sources. also i m planning to 2 boxes with 3 card on each site, how can i load balance between site if i m do not use round robin? Proper NTP (i.e. if you control the clients!) is to use multiple (at least 4!) servers from each client. If you just have to publicize an official server list, then I would use the same setup as the ntp pool, i.e. ntp.yourdomain.com would return multiple addresses, probably in random or round robin order. We need more detail to help you! Terje Thank for your support From: "terje.mathisen at tmsw.no"@ntp.org To: questions@lists.ntp.org Date: Thu, 22 Dec 2011 12:44:07 +0100 Subject: Re: [ntp:questions] ntp server pool advice ben slimup wrote: Thank for prompt answer Chris, Unfortunately, this ntp network should give time to specific clients devices and not anyone on the public network. according to your advice, better not using load balancer, thats good how to load balance between ntp server if i do not use round robin? if all client choosing the same server then the ntp server will be overload. is it a problem if for example client 1 poll or synch with server 1 , and then with server 2 , etc...? or udp roundtrip comes each time from different ntp server? how many ntp servers should be needed to handle that much request knowing that each card handle 10,000 request per sec? First, each client should have at least 4 configured servers, so you can use the same ntp.conf file for all of them. Second, if you really can handle 10K requests/second per card, then that means that you can handle 640K clients per card, with worst-case polling. I.e. servers capable of 10K/second should handle your expected load just fine, even though a proper (FreeBSD-based) 1U server with a GPS will serve even more clients with better time performance. Terje much appreciate your expertize cheers From: albertson.ch...@gmail.com Date: Wed, 21 Dec 2011 19:43:53 -0800 Subject: Re: [ntp:questions] ntp server pool advice To: slimu...@hotmail.com On Wed, Dec 21, 2011 at 5:07 PM, ben slimup wrote: Dear all, Thank you very much for support, i do not have 1000,000 client, i need those ntp servers to serve a load between 10 to 100 clients over a public network with an accuracy of 100ms those clients will use dns round robin to resolve 4 external ip, 2 IPs on each site. i have 4 servers with 4 ntp server slot card each ( meinberg M900) 1 ntp server card can support 10,000 request. First off the good news. 100ms is an "easy" spec to meet you can do this without a lot of effort. Don't let the outside world "see" your meinberg servers. Build out a layer of "statum 2" servers and expose those to your clients. 1M clients is a lot for the little 386 class CPU that is in the meinberg box. I still don't understand, Why do all those NTP clients need to go to your NTP servers. Why can't they use any they like? Are your servers doing something special? Also know that EACH client needs to be configured to see multiple NTP servers. practically three servers is a minimum but others will argue for more for five A would not use load balancing for NTP servers.With NTP it does not matter at all if a server crashes. The clients are all configure to use five servers and if one crashes they will do fine using four. If you expose four,
Re: [ntp:questions] Accuracy of NTP - Advice Needed
Paul Sobey wrote: Our internal testing to this point is that a stock ntpd pointed against a stratum 1 clock on a low contention gigabit ethernet (stratum 1 source and client less than 1ms apart) reports its own accuracy at approx 200 microseconds. Further tuning the ntp config by adding the minpoll 4, maxpoll 6 and burst keywords result in ntpd reported accuracy dropping to within 10-20 microseconds (as reported by ntpq -p and borne out by loopstats). Further improvements can be made running ntpd in the RT priority class. Good, you've done your homework! :-) My questions to you all, if you've read through the above waffle are: - what is a sensible expected accuracy of ntpd if pointed at several stratum 1 time sources across a low jitter gigabit network (we'd probably spread them over several UK and US sites for resiliency but all paths are low jitter and highly deterministic latency) Gbit and low jitter is not quite compatible: 100 Mbit switches were using cut-through, while (afaik) all Gbit and up switches use store & forward, leading to higher latency and jitter. - are there any obvious tunables to improve accuracy other than minpoll/burst and process scheduling class, and how agressive can the polling cycles be sensible made? - can ntpd's own reported offset (ntpq -p or loopstats) be trusted (assuming high priority means it gets scheduled as desired)? I've quoted our apparent numbers at several people and the response is always 'pfft you can't trust ntpd to know its own offset' - but nobody can ever tell me why You can use ntpd's internal numbers to verify the maximum possible offset (half the round trip time), you should be able to use statistics to show that the jitter is quite low as well. I appreciate these may appear to be silly questions with obvious answers - I am grateful in advance for your patience, and any research sources you may direct me to. The best (and probably only possible) solution that does give you single-digit us is to route a PPS signal to each and every server, then use the network for approximate (~100 us) timing, with the PPS doing the last two orders of magnitude. Terje -- - "almost all programming can be viewed as an exercise in caching" ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
Paul Sobey wrote: Our internal testing to this point is that a stock ntpd pointed against a stratum 1 clock on a low contention gigabit ethernet (stratum 1 source and client less than 1ms apart) reports its own accuracy at approx 200 microseconds. Further tuning the ntp config by adding the minpoll 4, maxpoll 6 and burst keywords result in ntpd reported accuracy dropping to within 10-20 microseconds (as reported by ntpq -p and borne out by loopstats). Further improvements can be made running ntpd in the RT priority class. Good, you've done your homework! :-) I've been trying! It's a challenging subject to get to grips with! A long way to go I think... My questions to you all, if you've read through the above waffle are: - what is a sensible expected accuracy of ntpd if pointed at several stratum 1 time sources across a low jitter gigabit network (we'd probably spread them over several UK and US sites for resiliency but all paths are low jitter and highly deterministic latency) Gbit and low jitter is not quite compatible: 100 Mbit switches were using cut-through, while (afaik) all Gbit and up switches use store & forward, leading to higher latency and jitter. There are several varieties of cut-through gig/10GB switch available now - but noting that store and forward adds latency is a good point. Presumably layer 3 switching (routing) is always store and forward since we're effectively writing a new packet. It's all done in hardware though - negligible jitter. - are there any obvious tunables to improve accuracy other than minpoll/burst and process scheduling class, and how agressive can the polling cycles be sensible made? - can ntpd's own reported offset (ntpq -p or loopstats) be trusted (assuming high priority means it gets scheduled as desired)? I've quoted our apparent numbers at several people and the response is always 'pfft you can't trust ntpd to know its own offset' - but nobody can ever tell me why You can use ntpd's internal numbers to verify the maximum possible offset (half the round trip time), you should be able to use statistics to show that the jitter is quite low as well. At the risk of pushing my luck, can you expand on this? I appreciate these may appear to be silly questions with obvious answers - I am grateful in advance for your patience, and any research sources you may direct me to. The best (and probably only possible) solution that does give you single-digit us is to route a PPS signal to each and every server, then use the network for approximate (~100 us) timing, with the PPS doing the last two orders of magnitude. Our problem will be that running coax around many sites to lots of machines, many of which don't have serial ports (think blades), is both highly time consuming and maintenance intensive. If we have to do it then we will but I'd like a clear idea as to the whys before I start down that particular path. In particular at this stage I'm trying to understand more about the theoretical accuracies obtainable under ideal conditions, and most important, how to independently verify the results of any tweaks we might apply. Say I have coalesence turned on a nic and I disable it - I'd like to be able to determine the effect, if any of that change. Is it possible for ntpd (or ptpd) to accurately determine its own accuracy, if that makes sense? If not what techniques might I use to independently measure? On a related note, I'm aware that there are various methods for querying the system time, some of which involve a context switch, and some of which can be done cheaply in userspace. I'm not sure whether the same is true for setting the time. Is anyone aware of how much ntpd operation involves context switches, which obviously would place quite a high ceiling on accuracy since we're at the mercy of the OS scheduler? Cheers, Paul ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
I hear from many vendors and industry colleagues that 'ntp just isn't suitable for high precision work and anything less than 1-2ms precision requires ptp or direct connection to gps clock'. I find these numbers For microsecond accuracy, I would say that NTP needs direct (PPS) connection to a GPS clock. On a very well managed LAN, with very good temperature control and a constant power dissipation on the machines, you might achieve it over LANs. On realistic internet connections, I don't think any technology will achieve it. (I guess you could use the internet to coordinate atomic clocks.) Thanks for this reponse. Assuming I give NTP a direct connection to a PPS clock, can you advise how I might determine what my accuracy actually is? This is the piece I'm very unsure of - I had hoped I could simply use the offset estimations from loopstats, now it seems I was mistaken in that assumption, but I'm still unclear as to why, and what numbers I can actually use to gain a measure. You need to consider more than this; for example, ethernet switches can be a major source of degradation. Is this true even with the kind of architecture I've described? I'm aware that switches and networks come in varying flavours, but I think I'm being fair when I describe ours as low latency and low jitter. Most paths have very little contention on. - are there any obvious tunables to improve accuracy other than minpoll/burst and process scheduling class, and how agressive can the polling cycles be sensible made? Very good temperature control, and maintaining a constant load on the machine. It is also essential that you calibrate the system. This either means using GPS or a portable atomic clock. What does calibrate the system mean? Is this 'use a GPS clock as an ntp source' or some other technique I haven't heard of? - can ntpd's own reported offset (ntpq -p or loopstats) be trusted (assuming high priority means it gets scheduled as desired)? I've quoted our apparent numbers at several people and the response is always 'pfft you can't trust ntpd to know its own offset' - but nobody can ever tell me why It can be trusted as what it actually measures. It is not a measurement of the error from true time. If it were, and it could be trusted, ntpd would be remiss not to use it to correct the time to a point where the remaining offset was no longer a good measure. The offset on a locked up system should be several times larger than the RMS error in the actual system time. Understood, at least in part. I have a nice Christmas reading list of man pages and white papers! Cheers, Paul ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
What OS are your hosts running? If it's Windows, millisecond, not microsecond accuracy will be what you can get at best when syncing over the network. A mixture of linux flavours and solaris. For the linux hosts I have a little more control over which version of ntpd I deploy. For the solaris ones, the stock version is ancient, not sure I have much control over that one but it could be raised if necessary. If you really need microsecond, I suspect you will be looking at a GPS receiver or two at each site, and distributing the PPS (pulse per second) signal to each host, and praying that the hosts have a serial port connection! Well that's the rub - some of them don't :) If nothing else it might inform new hardware purchases though. Some of these sites vary in their willingness to allow GPS antenas on roofs as well, joy. Cheers, Paul ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
What OS are your hosts running? If it's Windows, millisecond, not microsecond accuracy will be what you can get at best when syncing over the network. A mixture of linux flavours and solaris. For the linux hosts I have a little more control over which version of ntpd I deploy. For the solaris ones, the stock version is ancient, not sure I have much control over that one but it could be raised if necessary. I read that folk who want the best accuracy will run FreeBSD rather than Linux, but also that recent versions of Linux may not be as bad. How much the NTP version will affect the accuracy on such systems I don't know, nor how well Solaris performs. If you really need microsecond, I suspect you will be looking at a GPS receiver or two at each site, and distributing the PPS (pulse per second) signal to each host, and praying that the hosts have a serial port connection! Well that's the rub - some of them don't :) If nothing else it might inform new hardware purchases though. Some of these sites vary in their willingness to allow GPS antenas on roofs as well, joy. Cheers, Paul My gut feeling is that without GPS, microsecond accuracy is out of reach, and likely without a direct PPS connection to the box as well. It may not apply here, but is the requirement at all realistic? Was it dreamt up off the top of someone's head? What happens if you say that (for example) 100 microseconds is the best you can get. It's not high-energy physics or radio astronomy, is it? Not sure I can help any further, except to point you to my "best" PC, which is a dual-core Intel Atom box, doing nothing other than time serving, connected to a roof-mounted GPS, running FreeBSD. It's in a domestic, non-temperature controlled, environment. http://www.satsignal.eu/mrtg/pixie_ntp.html The zero line of the graph is at +20 microseconds, as MRTG doesn't plot negative numbers. The long term drift on the "yearly" graph is an MRTG issue. The "spike" of several microseconds you see on Wednesday is just the heating coming on here. Windows systems do not do as well: http://www.satsignal.eu/mrtg/performance_ntp.php You were planning on temperature controlled rooms for these systems, I suppose? Cheers, David ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On 2011-12-23, Paul Sobey wrote: >>> I hear from many vendors and industry colleagues that 'ntp just isn't >>> suitable for high precision work and anything less than 1-2ms precision >>> requires ptp or direct connection to gps clock'. I find these numbers >> >> For microsecond accuracy, I would say that NTP needs direct (PPS) connection >> to a GPS clock. On a very well managed LAN, with very good temperature >> control and a constant power dissipation on the machines, you might achieve >> it over LANs. On realistic internet connections, I don't think any >> technology will achieve it. (I guess you could use the internet to >> coordinate atomic clocks.) > > Thanks for this reponse. Assuming I give NTP a direct connection to a PPS > clock, can you advise how I might determine what my accuracy actually is? > This is the piece I'm very unsure of - I had hoped I could simply use the > offset estimations from loopstats, now it seems I was mistaken in that > assumption, but I'm still unclear as to why, and what numbers I can > actually use to gain a measure. With a good GPS receiver, AND a system which directly time tags the interrupt (no driver in between like a serial driver) you can assume that the time stamps on the gps signal are a reasonable estimate of the true offset (main problem is that interrupt latency). You could have the system put out a pulse the instant it receives the interrupt pulse (eg toggle a line on a parallel port) and attack an oscilliscope on the two lines (gps PPS line and the output port line ) to see what the time lag is. > >> You need to consider more than this; for example, ethernet switches can be a >> major source of degradation. > > Is this true even with the kind of architecture I've described? I'm aware > that switches and networks come in varying flavours, but I think I'm being > fair when I describe ours as low latency and low jitter. Most paths have > very little contention on. Not the problem. The routers have delays. Teh NICs have delays (some purposely wait to collect a reasonable amount of data before sending it out on the line). I used to have all 100MbS NICS and switches. I now have mixed 100Mbs and 1Gbs system and I see a really terrible degredation of the ntp network performance. While it used to be a very reliable 120us roundtrip on the ntp packets, it is now bimodal ( some about half 120-140, half 300 and a few up to 10ms delays). This change occured when the routers were replaced by Gbrouters. And there is very little contention on this network. > >>> - are there any obvious tunables to improve accuracy other than >>>minpoll/burst and process scheduling class, and how agressive can the >>>polling cycles be sensible made? >> >> Very good temperature control, and maintaining a constant load on the >> machine. >> >> It is also essential that you calibrate the system. This either means using >> GPS or a portable atomic clock. > > What does calibrate the system mean? Is this 'use a GPS clock as an ntp > source' or some other technique I haven't heard of? It means using a pps source to measure the offsets of your system. > >>> - can ntpd's own reported offset (ntpq -p or loopstats) be trusted >>>(assuming high priority means it gets scheduled as desired)? I've quoted It can be trusted to tell you what the offsets are. It cannot be trusted to tell you what the difference between your system time and true time is. >>>our apparent numbers at several people and the response is always 'pfft >>>you can't trust ntpd to know its own offset' - but nobody can ever tell >>>me why >> >> It can be trusted as what it actually measures. It is not a measurement of >> the error from true time. If it were, and it could be trusted, ntpd would >> be >> remiss not to use it to correct the time to a point where the remaining >> offset was no longer a good measure. The offset on a locked up system >> should >> be several times larger than the RMS error in the actual system time. > > Understood, at least in part. I have a nice Christmas reading list of man > pages and white papers! > > Cheers, > Paul ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
Paul Sobey wrote: Gbit and low jitter is not quite compatible: 100 Mbit switches were using cut-through, while (afaik) all Gbit and up switches use store & forward, leading to higher latency and jitter. There are several varieties of cut-through gig/10GB switch available now - but noting that store and forward adds latency is a good point. Presumably layer 3 switching (routing) is always store and forward since we're effectively writing a new packet. It's all done in hardware though - negligible jitter. Negligible only in a statistical way, when running on a lightly loaded network (close to zero collisions/arbitration for access to the same port) You can use ntpd's internal numbers to verify the maximum possible offset (half the round trip time), you should be able to use statistics to show that the jitter is quite low as well. At the risk of pushing my luck, can you expand on this? NTPD statistics will easily give you an order of magnitude better accuracy than that defined by half the RTT, so if you have ping times well below 1 ms, you should get actual offsets in the dual-digit us range. The best (and probably only possible) solution that does give you single-digit us is to route a PPS signal to each and every server, then use the network for approximate (~100 us) timing, with the PPS doing the last two orders of magnitude. Our problem will be that running coax around many sites to lots of machines, many of which don't have serial ports (think blades), is both highly time consuming and maintenance intensive. If we have to do it then we will but I'd like a clear idea as to the whys before I start down that particular path. In particular at this stage I'm trying to understand more about the theoretical accuracies obtainable under ideal conditions, and most important, how to independently verify the results of any tweaks we might apply. Say I have coalesence turned on a nic and I disable it - That is just one of the things you _have_ to do in order to get rid of jitter. If you have ethernet cards that support hw time sync (I don't remember the spec number), then you can indeed get low us over network links, but that isn't ntp any longer, and you'll still have to replace the blade infrastructure. I'd like to be able to determine the effect, if any of that change. Is it possible for ntpd (or ptpd) to accurately determine its own accuracy, if that makes sense? If not what techniques might I use to independently measure? On a related note, I'm aware that there are various methods for querying the system time, some of which involve a context switch, and some of which can be done cheaply in userspace. I'm not sure whether the same is A fast system time query will load the time as of the last hw clock tick, along with the corresponding RDTSC (or similar, constant-rate highres clock source), load the current RDTSC value, then re-read the OS tick. If the OS tick has been updated in the meantime, restart the process. Finally, scale the RDTSC delta by the (OS-provided) frequency and add to the tick time: The result is the current system time in ~100 clock cycles, with sub-us precision. true for setting the time. Is anyone aware of how much ntpd operation involves context switches, which obviously would place quite a high ceiling on accuracy since we're at the mercy of the OS scheduler? NTP does not set the time, it just tells the OS how much the tick rate should be tuned by, i.e. effectively how short/long each tick really is. Terje -- - "almost all programming can be viewed as an exercise in caching" ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On Fri, Dec 23, 2011 at 17:02, Terje Mathisen <"terje.mathisen at tmsw.no"@ntp.org> wrote: > A fast system time query will load the time as of the last hw clock tick, > along with the corresponding RDTSC (or similar, constant-rate highres clock > source), load the current RDTSC value, then re-read the OS tick. > > If the OS tick has been updated in the meantime, restart the process. > > Finally, scale the RDTSC delta by the (OS-provided) frequency and add to the > tick time: The result is the current system time in ~100 clock cycles, with > sub-us precision. That's a good description of most OS clocks. There are exceptions where no high-resolution counter is used, such as often seen with ntpd on Windows Vista and later. On such systems, reading the clock can be as fast as reading 64 bits from shared memory, possibly twice if partial updates are visible. Cheers, Dave Hart ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
I see a common misconception here. Most of the concepts about NTP can be explained using a common wristwatch. One is that more frequent checking to a standard keeps the time closer to the standard. That would be true only if you set your watch to match the standard each time. NTP does not do that. It plays with the "fast and slow" lever and adjusts the rate. To set a rate the longer you wait between checks the better. It takes time to see it you are running fast or slow. Frequent jumping the clock to match a standard while ignoring the rate adjustment produces a sawtooth shape error function. Making small adjustments to the rate is better and is what NTP does. NTP can't really know it's own absolute error from "true" UTC time. It if did it would be simple enough to make it zero. All it can do is report statistics about how in the recent past it differed along with an "error bar" on that.To really know how NTP is doing you need a second local clock that is about at least an order of magnitude better. Practically speaking for most of us that means a timing grade GPS receiver. But how do you know you have that set up right and you have accounted for the cable and internal delays. You get a second GPS, hopefully a different type. The good news is that these GPSes are dirt cheap. Under $20 if you hunt around. But it is absolutely true that unless you have access to "true UTC" and also have a way to double/tripple check then you can never know if your NTP server is accurate. You will need a way to measure phase differences in the PPS signals too.None of this stuff is very expensive and there seem to be endless supply from China. But it takes time to build up your ability to measure. So if you want an NTP server that works at the uS level, first assemble a timing lab that can measure nS. New modern timing grade receivers cost under $100 and are good for single digit nanoseconds. Finally, you are ready to start. Setting up an NTP server that does better then 1uS requires a very stable internal clock in side the PC. Most PCs are unsuitable and their internal clock frequency will track the building's air conditioning/heating cycles. If these cycles are faster than NTP's control loop there is not much the software can do. People have resorted to hardware mods like temperature control and upgrading the crystal clock on the computer main board. I found an Atom powered main board that does not need a heat sync, very stable. The bottom line is that getting NTP to sub uS is entering the realm of "rocket science". You can do it but not by tweaking software parameters and reading NTPs own status print outs. That method is fine for mS but not for uS BTW there're some guys in a lab at work (not me) who are measuring "femtoseconds". I'm impressed. -- Chris Albertson Redondo Beach, California ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On Fri, Dec 23, 2011 at 2:09 AM, Terje Mathisen <"terje.mathisen at tmsw.no"@ntp.org> wrote: > The best (and probably only possible) solution that does give you > single-digit us is to route a PPS signal to each and every server, then use > the network for approximate (~100 us) timing, with the PPS doing the last > two orders of magnitude. I think the above is true. Even then getting NTP to track the PPS at the uS level is not "plug and play". But there is no hop of doing uS level over the gigabit either net. Distributinng PPS is no simple task either. I'm experiment here with that. Im using the the "extra" wire pairs in the install cat-5 wire that s already in the walls. Of course PPS can't go through a router so you have to physically connect a twisted pair from the source (In my case this is a GPS) to every computer. This is very hard to do in a big campus, impossible really. So PPS only works within a building and only over about a "few" hundred meters of cable. I'm using a simple RS232 driver chip from Max as the transmuter. These are driven by a simple "hex inverter" TTL chip. The Max driver converts the signal to -9 and +9 volts. There is delay in this. Some in the driver chips and some in the cable. Cable is roughly about nanosecond per foot. You need to measure and remove this and it's different for every cable run. But most people with more than a few computers have wire closets and plenty of unused cat-5 wire pairs. This is a very cheap and good way to distribute PPS in a building. If you have two buildings buy a GPS for each building. Chris Albertson Redondo Beach, California ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On 12/22/2011 11:35 PM, unruh wrote: On 2011-12-23, Richard B. Gilbert wrote: On 12/22/2011 2:11 PM, Paul Sobey wrote: Dear All, I work for a firm which requires clocks to be synchronised to quite a high degree of accuracy. We have an existing ntp-based infrastructure but want to improve on it to the point where the bulk of our hosts are synchronised to single digit microseconds of each other if possible. We have about 400 hosts in production, spread across about 15 sites. I hear from many vendors and industry colleagues that 'ntp just isn't suitable for high precision work and anything less than 1-2ms precision requires ptp or direct connection to gps clock'. I find these numbers somewhat suspect, and wanted to ask the advice of you experts. In particular I've read several threads on this list and other sites which suggest that highly accurate synchronisations are possible, assuming OS and network jitter can be minimised. Our internal testing to this point is that a stock ntpd pointed against a stratum 1 clock on a low contention gigabit ethernet (stratum 1 source and client less than 1ms apart) reports its own accuracy at approx 200 microseconds. Further tuning the ntp config by adding the minpoll 4, maxpoll 6 and burst keywords result in ntpd reported accuracy dropping to within 10-20 microseconds (as reported by ntpq -p and borne out by loopstats). Further improvements can be made running ntpd in the RT priority class. My questions to you all, if you've read through the above waffle are: - what is a sensible expected accuracy of ntpd if pointed at several stratum 1 time sources across a low jitter gigabit network (we'd probably spread them over several UK and US sites for resiliency but all paths are low jitter and highly deterministic latency) - are there any obvious tunables to improve accuracy other than minpoll/burst and process scheduling class, and how agressive can the polling cycles be sensible made? - can ntpd's own reported offset (ntpq -p or loopstats) be trusted (assuming high priority means it gets scheduled as desired)? I've quoted our apparent numbers at several people and the response is always 'pfft you can't trust ntpd to know its own offset' - but nobody can ever tell me why I appreciate these may appear to be silly questions with obvious answers - I am grateful in advance for your patience, and any research sources you may direct me to. Many thanks, Paul If you can possibly site a GPS antenna and receiver at your location, you can get microsecond accuracy or better. The receiver will output a "tick" each second. One edge of the tick signal will be within about 50 nanoseconds of the "start" of a second. The receivers cost anywhere from $100 and up. Some people need, or just want this level of accuracy. You do need to be able to site an antenna with a clear view of the sky. The last time I heard, there were twenty-seven GPS satellites in service. There are usually anywhere from three to five or six above the horizon at any given time. Given at least three satellites in line-of-sight your GPS receiver can figure out the latitude, longitude, and elevation of your antenna. Once it has done this it only needs to see a single GPS satellite to get the time. This kind of accuracy is far more than most people really need. It's there if you need it even if you only need it for "bragging rights"! Or timing how long it takes neutrinos to get from Cern to Grand Sasso. How do you "tag" a neutrino so that you can say with assurance that the the neutrino that left Cern is the same neutrino that arrives at Sasso? ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
Richard B. Gilbert wrote: > How do you "tag" a neutrino so that you can say with assurance that the > the neutrino that left Cern is the same neutrino that arrives at Sasso? By sending them in a "pulse" of a known width. -- Jim Pennino Remove .spam.sux to reply. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
Hi Paul, This is probably going further up the track for you, however, with the HP "c class" blades you can use a PCI expansion blade with a serial IO card in it. The downside is that the expansion blade takes up one bay (to the left of a half-height blade, or bottom left slot next to a full height blade) If your enclosures are already quite full this maybe a problem for you? Another thought is the SUV Dongle that goes on the front, breaks out com1 to the DB9 on the Cable. However, loss of PPS due someone hijacking the cable for service reasons is probably quite high... I wouldn't recommend epoxying the cable in unless you are going to use the cheapest available blade and intend to chuck it if it fails. Another thought is to use the HP SL type blades, some of which have expansion slots natively. These are not so well known, however video rendering farms seem to love them.. I wish you luck on your exciting project, feel free to drop me a line if you have any HP blade or ISS technical related questions? Mark -Original Message- From: Paul Sobey [mailto:bud...@the-annexe.net] Sent: Saturday, 24 December 2011 1:47 AM To: questions@lists.ntp.org Subject: Re: [ntp:questions] Accuracy of NTP - Advice Needed Our problem will be that running coax around many sites to lots of machines, many of which don't have serial ports (think blades), is both highly time consuming and maintenance intensive. If we have to do it then we will but I'd like a clear idea as to the whys before I start down that particular path. In particular at this stage I'm trying to understand more about the theoretical accuracies obtainable under ideal conditions, and most important, how to independently verify the results of any tweaks we might apply. Say I have coalesence turned on a nic and I disable it - I'd like to be able to determine the effect, if any of that change. Is it possible for ntpd (or ptpd) to accurately determine its own accuracy, if that makes sense? If not what techniques might I use to independently measure? On a related note, I'm aware that there are various methods for querying the system time, some of which involve a context switch, and some of which can be done cheaply in userspace. I'm not sure whether the same is true for setting the time. Is anyone aware of how much ntpd operation involves context switches, which obviously would place quite a high ceiling on accuracy since we're at the mercy of the OS scheduler? Cheers, Paul ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
Richard B. Gilbert wrote: > How do you "tag" a neutrino so that you can say with assurance that the > the neutrino that left Cern is the same neutrino that arrives at Sasso? Jim Pennino writes: > By sending them in a "pulse" of a known width. It should be noted, however, that you cannot observe the same neutrino twice. In fact, no neutrinos at all are observed at the Cern end: just the protons that produce the pions that produce the neutrinos. An upcoming experiment at Fermilab will observe neutrinos at both ends (the far end will be in Minnesota). -- John Hasler jhas...@newsguy.com Dancing Horse Hill Elmwood, WI USA ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
John Hasler wrote: > Richard B. Gilbert wrote: >> How do you "tag" a neutrino so that you can say with assurance that the >> the neutrino that left Cern is the same neutrino that arrives at Sasso? > > Jim Pennino writes: >> By sending them in a "pulse" of a known width. > > It should be noted, however, that you cannot observe the same neutrino > twice. In fact, no neutrinos at all are observed at the Cern end: just > the protons that produce the pions that produce the neutrinos. An > upcoming experiment at Fermilab will observe neutrinos at both ends (the > far end will be in Minnesota). And your point would be? -- Jim Pennino Remove .spam.sux to reply. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
At 01:58 PM 12/23/2011, Chris Albertson wrote... But there is no hop of doing uS level over the gigabit either net. IEEE 1588. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On 2011-12-23, John Hasler wrote: > An > upcoming experiment at Fermilab will observe neutrinos at both ends (the > far end will be in Minnesota). But not the same neutrino, since you can only detect the neutrino after it has collided with something else. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On Fri, Dec 23, 2011 at 5:51 PM, Mike S wrote: > At 01:58 PM 12/23/2011, Chris Albertson wrote... > >> But there is no hop of doing uS >> level over the gigabit either net. > > > IEEE 1588. Seem 1588 can work at the sub uS level but actual real-work software on Linux works at the "tens of uS" level. The limiting factor is the hardware's ability to time stamp eithernet packets. If you have special hardware you can get to sub-uS but not using PTPd on Linux or BSD. Linux's and BSD's time stamper for the PPS interface works better (less jitter) then its time stamper for Ethernet. In a standard Cat-5 cable there are 8 wires in 4 pair. Ethernet only uses 2 pair. I simply used a "spare" pair for a pulse. The parts required cost under $5 assuming you have tools to build cables already or old cables you can hack. -- Chris Albertson Redondo Beach, California ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On 2011-12-23, John Hasler wrote: > Richard B. Gilbert wrote: >> How do you "tag" a neutrino so that you can say with assurance that the >> the neutrino that left Cern is the same neutrino that arrives at Sasso? > > Jim Pennino writes: >> By sending them in a "pulse" of a known width. > > It should be noted, however, that you cannot observe the same neutrino > twice. In fact, no neutrinos at all are observed at the Cern end: just > the protons that produce the pions that produce the neutrinos. An > upcoming experiment at Fermilab will observe neutrinos at both ends (the > far end will be in Minnesota). Well, no. At best the electrons or muons at one end. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On 2011-12-23, Richard B. Gilbert wrote: > On 12/22/2011 11:35 PM, unruh wrote: >> On 2011-12-23, Richard B. Gilbert wrote: >>> On 12/22/2011 2:11 PM, Paul Sobey wrote: Dear All, I work for a firm which requires clocks to be synchronised to quite a high degree of accuracy. We have an existing ntp-based infrastructure but want to improve on it to the point where the bulk of our hosts are synchronised to single digit microseconds of each other if possible. We have about 400 hosts in production, spread across about 15 sites. I hear from many vendors and industry colleagues that 'ntp just isn't suitable for high precision work and anything less than 1-2ms precision requires ptp or direct connection to gps clock'. I find these numbers somewhat suspect, and wanted to ask the advice of you experts. In particular I've read several threads on this list and other sites which suggest that highly accurate synchronisations are possible, assuming OS and network jitter can be minimised. Our internal testing to this point is that a stock ntpd pointed against a stratum 1 clock on a low contention gigabit ethernet (stratum 1 source and client less than 1ms apart) reports its own accuracy at approx 200 microseconds. Further tuning the ntp config by adding the minpoll 4, maxpoll 6 and burst keywords result in ntpd reported accuracy dropping to within 10-20 microseconds (as reported by ntpq -p and borne out by loopstats). Further improvements can be made running ntpd in the RT priority class. My questions to you all, if you've read through the above waffle are: - what is a sensible expected accuracy of ntpd if pointed at several stratum 1 time sources across a low jitter gigabit network (we'd probably spread them over several UK and US sites for resiliency but all paths are low jitter and highly deterministic latency) - are there any obvious tunables to improve accuracy other than minpoll/burst and process scheduling class, and how agressive can the polling cycles be sensible made? - can ntpd's own reported offset (ntpq -p or loopstats) be trusted (assuming high priority means it gets scheduled as desired)? I've quoted our apparent numbers at several people and the response is always 'pfft you can't trust ntpd to know its own offset' - but nobody can ever tell me why I appreciate these may appear to be silly questions with obvious answers - I am grateful in advance for your patience, and any research sources you may direct me to. Many thanks, Paul >>> >>> If you can possibly site a GPS antenna and receiver at your location, >>> you can get microsecond accuracy or better. The receiver will output a >>> "tick" each second. One edge of the tick signal will be within about >>> 50 nanoseconds of the "start" of a second. >>> >>> The receivers cost anywhere from $100 and up. Some people need, or just >>> want this level of accuracy. You do need to be able to site an antenna >>> with a clear view of the sky. >>> >>> The last time I heard, there were twenty-seven GPS satellites in >>> service. There are usually anywhere from three to five or six above the >>> horizon at any given time. Given at least three satellites in >>> line-of-sight your GPS receiver can figure out the latitude, longitude, >>> and elevation of your antenna. Once it has done this it only needs to >>> see a single GPS satellite to get the time. >>> >>> This kind of accuracy is far more than most people really need. It's >>> there if you need it even if you only need it for "bragging rights"! >> >> Or timing how long it takes neutrinos to get from Cern to Grand Sasso. >> >>> > > How do you "tag" a neutrino so that you can say with assurance that the > the neutrino that left Cern is the same neutrino that arrives at Sasso? The neutrinos are created by collisions of protons which produce pions which decal amonst other things to neutrinos. The protons come in bunches, so knowing when the protons raced by (at 99.9%c) the neutrinos are created in the same line and thus you know what the timing is between when the proton went by and the neutrino was absorbed. So you tag it by time. > > ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On 23 Dec, 2011, at 22:47 , Paul Sobey wrote: >>> I appreciate these may appear to be silly questions with obvious answers >>> - I am grateful in advance for your patience, and any research sources >>> you may direct me to. >> >> The best (and probably only possible) solution that does give you >> single-digit us is to route a PPS signal to each and every server, then use >> the network for approximate (~100 us) timing, with the PPS doing the last >> two orders of magnitude. > > Our problem will be that running coax around many sites to lots of machines, > many of which don't have serial ports (think blades), is both highly time > consuming and maintenance intensive. If we have to do it then we will but I'd > like a clear idea as to the whys before I start down that particular path. > > In particular at this stage I'm trying to understand more about the > theoretical accuracies obtainable under ideal conditions, and most important, > how to independently verify the results of any tweaks we might apply. Say I > have coalesence turned on a nic and I disable it - I'd like to be able to > determine the effect, if any of that change. Is it possible for ntpd (or > ptpd) to accurately determine its own accuracy, if that makes sense? If not > what techniques might I use to independently measure? If you really want to do this, with either NTP (the protocol, maybe not ntpd the implementation) or PTP, then I think the place you need to start is, unfortunately, with your operating system kernels. I have a board which implements a clock which can be synchronized to the 10 MHz and 1 PPS outputs from a GPS receiver. The board's clock resolution is about 3 ns (i.e. a 320 MHz internal clock) and the PIO interface to the board is designed so that it should be possible to transfer time from the board clock to the computer's clock with no more than +/- 10 ns or so of ambiguity (call it +/- 20 ns to be safe). When I first used this with a stock NetBSD kernel (whose clock code I think was copied from FreeBSD at some point) I was a little bit surprised to find that, despite the low tens of nanoseconds of accuracy the hardware was capable of, sampling the card against system timestamps gave me a result which jittered by on the order of several microseconds. After looking at why this was, I found that the jitter was in fact coming from the system clock itself and was caused by the way clock adjustments are applied at clock interrupt time (I believe some of the complaints about "interrupt latency" of the serial PPS driver are in fact seeing this system clock jitter and blaming it on something else; my very brief measurement of that driver found that while the fixed interrupt latency is 100's of nanoseconds it is also relatively constant, with outliers which are fairly easy to filter). Needless to say, if you can't get your system clock stable to better than microseconds you are unlikely to be able to synchronize it to a network source at that level. I fixed this by replacing the clock code, instead computing the time as a linear function of the value of the underlying counter, and getting rid of the clock interrupt discrete adjustments altogether (except when the NTP adjustment interface is in use, though that's a whole other story), so now my system clock doesn't jitter. The second operating system issue that's useful to address, whether the data is coming from NTP or PTP, is the clock adjustment system call interface. In particular, there are huge advantages to be gained by having a system call interface which allows you to make both clock frequency (i.e. rate of clock advance) and time offset adjustments, and which makes the adjustments you tell it to with great precision (or at least, tells you precisely what it did). The reason this is advantageous would require a long explanation, but the summary is that it allows you to treat the clock control process as solely a measurement process, rather than a feedback control process, and this makes it possible to begin to look at a broader variety of filtering procedures for incoming data to try to maximize the signal while minimizing the noise, without the additional burden of having to consider the stability (in the control system sense) of the adjustment process. The adjustments can be done open-loop. I believe that the operating system work described above, plus maybe some work on your ethernet card drivers, is necessary to achieve what you want with either with NTP or PTP. With my own implementation of the NTP daemon I can generally keep a client machine within 10 us of a server (measured with one of cards mentioned above in each machine) separated by (I think) 4 gigabit ethernet switches, I think with one 10 Gbps circuit in there, carrying company network traffic, with a 16 second polling interval. Note that I haven't tested this with ntpd yet, mostly because I don't like the way I had to jam support for the NTP system call inter
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On 2011-12-24, Chris Albertson wrote: > On Fri, Dec 23, 2011 at 5:51 PM, Mike S wrote: >> At 01:58 PM 12/23/2011, Chris Albertson wrote... >> >>> But there is no hop of doing uS >>> level over the gigabit either net. >> >> >> IEEE 1588. > > Seem 1588 can work at the sub uS level but actual real-work software > on Linux works at the "tens of uS" level. The limiting factor is the > hardware's ability to time stamp eithernet packets. If you have > special hardware you can get to sub-uS but not using PTPd on Linux or > BSD. If you really go to stamping the interrupt directly as it comes in on the kernel level, rather than waiting for a driver (eg the serial driver) to report that the itnerrupt has occured to userland, you can get it down to 1-2us. > > Linux's and BSD's time stamper for the PPS interface works better > (less jitter) then its time stamper for Ethernet. > > In a standard Cat-5 cable there are 8 wires in 4 pair. Ethernet only > uses 2 pair. I simply used a "spare" pair for a pulse. The parts > required cost under $5 assuming you have tools to build cables already > or old cables you can hack. > > ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Accuracy of NTP - Advice Needed
On Fri, Dec 23, 2011 at 10:06 PM, unruh wrote: > If you really go to stamping the interrupt directly as it comes in on > the kernel level, rather > than waiting for a driver (eg the serial driver) to report that the > itnerrupt has occured to userland, you can get it down to 1-2us. That is exactly how the Linux PPS system works. The kernel level driver is all of about a dozen lines of code. It is very simple. When the interrupt happens the driver copies the hardware nanosecond clock to some location in RAM and sets a flag that says "data available" A user land program waits for the flag. When it sees the flag set it can read the counter value that was saved by the driver and then the flag is re-set. It is unimportance how long it take the program to notice there is data and read it, so long as it does so before the next pulse. Linux does not queue the data. The next pulse over writes the time stamp. As you say 1 or 2 uS is about what it gets. On my system the nanosecond clock jumps in 1000 nS steps But notice this is not just a pulse per second interface. It is a pulse interface and can time stamp random pulses too. I know it works for 60Hz pulses. The interpretation of what to do with the data is a user land thing. NTP uses it to sync the clock. Other people use it for general-purpose pulse timing. Chris Albertson Redondo Beach, California ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
