Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On 21/10/2020 15:44, Sadique Urf Arbaz Sayyed wrote: The process will hold the 24Kb but actually the process is not visible in Task manager or anywhere but in RamMap we see that the executed process still occupied the Page table memory. Did you mean 24KB, or is that really 3KB? Any memory leakage that persists after the process is no longer visible in the process table is down to the OS. As already pointed out page table allocations are unlikely to be what that memory is. I'm not familiar with the details of Window NT family memory use reporting, but the only legitimate carry over would be for cached pages. Such pages are optimisations and don't deny memory to other processes. Both Linux and Windows try and allocate nearly all their memory to something. In the case of the Linux "free" command, it computes an available figure which is much larger than the free figure, by including memory that can safely be discarded. If there isn't an OS bug, I wonder if you are seeing that effect. One wild thought: is ntpq or one of its DLLs not position independent code? I could speculate as to why Windows might keep relocated pages around in case the code is reloaded at the same address before the page gets reused for other reasons. However, I don't actually know if Windows does something like that. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] How is the BIG guys (Google. FB etc) DNS and NTP architecture
On 2020-10-21, Miroslav Lichvar wrote: > On 2020-10-21, William Unruh wrote: >> On 2020-10-21, CRasch Net wrote: >>> Facebook is now using Chrony, you can read about their implementation: >>> >>> Building a more accurate time service at Facebook scale >>> https://engineering.fb.com/production-engineering/ntp-service/ >> >> Interesting. While I agree that chrony is more precise, I think that >> their results for ntpd are worse than they should be. ntpd can >> certainly do better than 1ms scatter/accuracy (and chrony can do better >> than 100us.There is something weird about their network paths.) About 10 >> years I ran a number of tests of chrony vs ntpd. and got about a fctor >> of 3-10 better, not 100. Interrupt latency/clock reading for chrony gave >> about 1us fluctuations. > > It's not clear how ntpd and chrony were configured in their tests. The > ntpq/chronyc outputs show a poll of 6, which is too long for a highly > stable synchronization in a local network. If they were using the > default minpoll 6 and maxpoll 10, a factor of 100 would not surprise me. > ntpd doesn't adjusts its polling very well when it has stable > measurements, so it would be effectively comparing ntpd polling at 10 vs > chrony polling at 6. > >> I find this whole thing about leap second smoothing to be a real farce. >> Just let the step occur instead of delivering the wrong time for hours. >> Or if you want, run your clocks on TIA not UTC and make the leapsecond >> conversion in the interpretation as is done for timezones. Would anyone >> advise leap day smoothing every 4 years so that we do not have trouble >> with our calenders? > > Well, yes. The trouble is that there are applications that break on > backward step, they need synchronized clocks, and not all NTP clients > can be configured to make a consistent slew on the leap second. So, the > easiest way to fix this is to make a slew on the server and hide the > leap second from the clients. When you internally do this everywhere and > you want to provide a public NTP service, it's easier to just serve your > internal leap-smeared time. Easier. It is probably even easier to forget about ntp and just free run your server. "Easy" is not the purpose of serving ntp time. > ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On 2020-10-21 15:16, Sadique Urf Arbaz Sayyed wrote: On Wednesday, October 21, 2020 at 6:19:21 PM UTC+5:30, Jakob Bohm wrote: On 2020-10-21 12:45, Sadique Urf Arbaz Sayyed wrote: On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz Sayyed wrote: We started with a brand new windows server 2019 datacenter edition and installed an infrastructure monitoring agent on it and strictly no other program. The machine had 8 GB of memory. As part of monitoring NTP offset from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, the memory utilisation had increased to significant level >80%. On analysis we found it was a gradual increase and using RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B in Private. Moreover this issue is specific to windows server 2019 we tried following same steps on windows server 2012 machine and it worked perfectly fine with no memory creeping issues. Any help or pointer are appreciated Full problem is mentioned here https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 From what you have posted above (I have not checked the MS forum) it sounds like the leak is of "PAGE"-d memory consumption, not page tables (also known in the NT world as hyperspace). Question is what is actually registered as owning that page-able memory. Maybe you have found a memory leak in the NTP server itself, maybe something in NT 10.??.2019?? (server 2019) leaks memory every time ntpq makes an UDP socket connection to the NTP service to query it. There are some advanced tools for checking what owns outstanding global (kernel) memory allocations on NT systems. However first you should check the "commit charge" of the NTP service process, as that is a direct measure of how much pageable virtual memory is allocated in the user mode part of that process. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded Ok so this NT server is used by our customers but in house I do not have any server. I'm simply running ntpq.exe -nc peers 127.0.0.1, here I am not expecting the output but why is it hogging onto memory. As you pointed out ntpq uses the UDP port 123 which is by default used by Windows Time service. And this issue does not occur if I have the windows time service disabled. So my question here is why does the process hogs onto memory if I have the Windows time service running (In windows 2019 it's running by default). ntpq.exe is designed to talk to ntpd.exe from the same source code bundle. Microsoft's NTP service (win32time) probably completely lacks the ability to correctly handle that query. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On 2020-10-21 12:45, Sadique Urf Arbaz Sayyed wrote: On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz Sayyed wrote: We started with a brand new windows server 2019 datacenter edition and installed an infrastructure monitoring agent on it and strictly no other program. The machine had 8 GB of memory. As part of monitoring NTP offset from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, the memory utilisation had increased to significant level >80%. On analysis we found it was a gradual increase and using RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B in Private. Moreover this issue is specific to windows server 2019 we tried following same steps on windows server 2012 machine and it worked perfectly fine with no memory creeping issues. Any help or pointer are appreciated Full problem is mentioned here https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 From what you have posted above (I have not checked the MS forum) it sounds like the leak is of "PAGE"-d memory consumption, not page tables (also known in the NT world as hyperspace). Question is what is actually registered as owning that page-able memory. Maybe you have found a memory leak in the NTP server itself, maybe something in NT 10.??.2019?? (server 2019) leaks memory every time ntpq makes an UDP socket connection to the NTP service to query it. There are some advanced tools for checking what owns outstanding global (kernel) memory allocations on NT systems. However first you should check the "commit charge" of the NTP service process, as that is a direct measure of how much pageable virtual memory is allocated in the user mode part of that process. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] How is the BIG guys (Google. FB etc) DNS and NTP architecture
On 2020-10-21 05:47, William Unruh wrote: On 2020-10-21, CRasch Net wrote: Facebook is now using Chrony, you can read about their implementation: Building a more accurate time service at Facebook scale https://engineering.fb.com/production-engineering/ntp-service/ Interesting. While I agree that chrony is more precise, I think that their results for ntpd are worse than they should be. ntpd can certainly do better than 1ms scatter/accuracy (and chrony can do better than 100us.There is something weird about their network paths.) About 10 years I ran a number of tests of chrony vs ntpd. and got about a fctor of 3-10 better, not 100. Interrupt latency/clock reading for chrony gave about 1us fluctuations. I find this whole thing about leap second smoothing to be a real farce. Just let the step occur instead of delivering the wrong time for hours. Or if you want, run your clocks on TIA not UTC and make the leapsecond conversion in the interpretation as is done for timezones. Would anyone advise leap day smoothing every 4 years so that we do not have trouble with our calenders? Originally, that is what they did, they smeared the leap day over 2 solar days. For a smeared timescale, perhaps something could be created that distributes UT instead of UTC, it would need an input file with the same data used to transmit UT-UTC in the standard radio protocol (see the ITU-R standard for UTC). Plus some kind of NTP protocol change to let clients know that the data incorporates the astronomical wobble and may need polling at the right time to pick up the wobble. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] How is the BIG guys (Google. FB etc) DNS and NTP architecture
On 2020-10-21, William Unruh wrote: > On 2020-10-21, CRasch Net wrote: >> Facebook is now using Chrony, you can read about their implementation: >> >> Building a more accurate time service at Facebook scale >> https://engineering.fb.com/production-engineering/ntp-service/ > > Interesting. While I agree that chrony is more precise, I think that > their results for ntpd are worse than they should be. ntpd can > certainly do better than 1ms scatter/accuracy (and chrony can do better > than 100us.There is something weird about their network paths.) About 10 > years I ran a number of tests of chrony vs ntpd. and got about a fctor > of 3-10 better, not 100. Interrupt latency/clock reading for chrony gave > about 1us fluctuations. It's not clear how ntpd and chrony were configured in their tests. The ntpq/chronyc outputs show a poll of 6, which is too long for a highly stable synchronization in a local network. If they were using the default minpoll 6 and maxpoll 10, a factor of 100 would not surprise me. ntpd doesn't adjusts its polling very well when it has stable measurements, so it would be effectively comparing ntpd polling at 10 vs chrony polling at 6. > I find this whole thing about leap second smoothing to be a real farce. > Just let the step occur instead of delivering the wrong time for hours. > Or if you want, run your clocks on TIA not UTC and make the leapsecond > conversion in the interpretation as is done for timezones. Would anyone > advise leap day smoothing every 4 years so that we do not have trouble > with our calenders? Well, yes. The trouble is that there are applications that break on backward step, they need synchronized clocks, and not all NTP clients can be configured to make a consistent slew on the leap second. So, the easiest way to fix this is to make a slew on the server and hide the leap second from the clients. When you internally do this everywhere and you want to provide a public NTP service, it's easier to just serve your internal leap-smeared time. -- Miroslav Lichvar ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
