Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
-Original Message- From: Sadique Urf Arbaz Sayyed We started with a brand new windows server 2019 datacenter edition and installed an infrastructure monitoring agent on it and strictly no other program. The machine had 8 GB of memory. As part of monitoring NTP offset from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, the memory utilisation had increased to significant level >80%. On analysis we found it was a gradual increase and using RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B in Private. Moreover this issue is specific to windows server 2019 we tried following same steps on windows server 2012 machine and it worked perfectly fine with no memory creeping issues. Any help or pointer are appreciated = Thanks for your report. On a newly installed PC running Windows-10 Pro, using standard NTP I see a loss of "Available" memory of about 16 kB per ntpq invocation. Numbers: running ntpq in a Perl script. This Perl script is called (with different parameters) four times for each node. I'm monitoring ~25 nodes, so that's about 100 calls for each MRTG check, which is every five minutes, making 28,800 calls per day. Loss (by visual inspection) is 8 GB in 17 days. So approximately 16 kB per ntpq invocation. From other monitoring, ntpd is completely constant in memory usage (35-40 MB). I'll see whether entering an NTP bug still worksit does so I'll add evidence to your 3695. Cheers, David -- SatSignal Software - Quality software for you Web: https://www.satsignal.eu Email: david-tay...@blueyonder.co.uk Twitter: @gm8arv ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On Wednesday, October 21, 2020 at 6:19:21 PM UTC+5:30, Jakob Bohm wrote: > On 2020-10-21 12:45, Sadique Urf Arbaz Sayyed wrote: > > On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz > > Sayyed wrote: > >> We started with a brand new windows server 2019 datacenter edition and > >> installed an infrastructure monitoring agent on it and strictly no other > >> program. The machine had 8 GB of memory. As part of monitoring NTP offset > >> from sync'd host we scheduled a ntpqexe. The problem started after 4-5 > >> days, the memory utilisation had increased to significant level >80%. On > >> analysis we found it was a gradual increase and using RAMMAP we saw every > >> time the ntpq.exe will run it will leave behind 24k of memory in PAGE > >> Table with 0 B in Private. Moreover this issue is specific to windows > >> server 2019 we tried following same steps on windows server 2012 machine > >> and it worked perfectly fine with no memory creeping issues. > >> > >> Any help or pointer are appreciated > > > > Full problem is mentioned here > > https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 > > > > > From what you have posted above (I have not checked the MS forum) it > sounds like the leak is of "PAGE"-d memory consumption, not page tables > (also known in the NT world as hyperspace). > > Question is what is actually registered as owning that page-able memory. > Maybe you have found a memory leak in the NTP server itself, maybe > something in NT 10.??.2019?? (server 2019) leaks memory every time ntpq > makes an UDP socket connection to the NTP service to query it. > > There are some advanced tools for checking what owns outstanding global > (kernel) memory allocations on NT systems. However first you should > check the "commit charge" of the NTP service process, as that is a > direct measure of how much pageable virtual memory is allocated in the > user mode part of that process. > > > Enjoy > > Jakob > -- > Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 > This public discussion message is non-binding and may contain errors. > WiseMo - Remote Service Management for PCs, Phones and Embedded Ok so this NT server is used by our customers but in house I do not have any server. I'm simply running ntpq.exe -nc peers 127.0.0.1, here I am not expecting the output but why is it hogging onto memory. As you pointed out ntpq uses the UDP port 123 which is by default used by Windows Time service. And this issue does not occur if I have the windows time service disabled. So my question here is why does the process hogs onto memory if I have the Windows time service running (In windows 2019 it's running by default). ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On Wednesday, October 21, 2020 at 8:02:33 PM UTC+5:30, Jakob Bohm wrote: > On 2020-10-21 15:16, Sadique Urf Arbaz Sayyed wrote: > > On Wednesday, October 21, 2020 at 6:19:21 PM UTC+5:30, Jakob Bohm wrote: > >> On 2020-10-21 12:45, Sadique Urf Arbaz Sayyed wrote: > >>> On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz > >>> Sayyed wrote: > We started with a brand new windows server 2019 datacenter edition and > installed an infrastructure monitoring agent on it and strictly no other > program. The machine had 8 GB of memory. As part of monitoring NTP > offset from sync'd host we scheduled a ntpqexe. The problem started > after 4-5 days, the memory utilisation had increased to significant > level >80%. On analysis we found it was a gradual increase and using > RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k > of memory in PAGE Table with 0 B in Private. Moreover this issue is > specific to windows server 2019 we tried following same steps on windows > server 2012 machine and it worked perfectly fine with no memory creeping > issues. > > Any help or pointer are appreciated > >>> > >>> Full problem is mentioned here > >>> https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 > >>> > >>> > >> From what you have posted above (I have not checked the MS forum) it > >> sounds like the leak is of "PAGE"-d memory consumption, not page tables > >> (also known in the NT world as hyperspace). > >> > >> Question is what is actually registered as owning that page-able memory. > >> Maybe you have found a memory leak in the NTP server itself, maybe > >> something in NT 10.??.2019?? (server 2019) leaks memory every time ntpq > >> makes an UDP socket connection to the NTP service to query it. > >> > >> There are some advanced tools for checking what owns outstanding global > >> (kernel) memory allocations on NT systems. However first you should > >> check the "commit charge" of the NTP service process, as that is a > >> direct measure of how much pageable virtual memory is allocated in the > >> user mode part of that process. > >> > >> > >> Enjoy > >> > >> Jakob > >> -- > >> Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > >> Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 > >> This public discussion message is non-binding and may contain errors. > >> WiseMo - Remote Service Management for PCs, Phones and Embedded > > > > Ok so this NT server is used by our customers but in house I do not have > > any server. I'm simply running ntpq.exe -nc peers 127.0.0.1, here I am not > > expecting the output but why is it hogging onto memory. As you pointed out > > ntpq uses the UDP port 123 which is by default used by Windows Time > > service. And this issue does not occur if I have the windows time service > > disabled. So my question here is why does the process hogs onto memory if I > > have the Windows time service running (In windows 2019 it's running by > > default). > > > ntpq.exe is designed to talk to ntpd.exe from the same source > code bundle. Microsoft's NTP service (win32time) probably > completely lacks the ability to correctly handle that query. > Enjoy > > Jakob > -- > Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 > This public discussion message is non-binding and may contain errors. > WiseMo - Remote Service Management for PCs, Phones and Embedded Yes Correct, But John here suppose my ntpd is running on a server and from the client side (Suppose windows time service is running) I run ntpq.exe with above command I do see the issue. The process will hold the 24Kb but actually the process is not visible in Task manager or anywhere but in RamMap we see that the executed process still occupied the Page table memory. So each query fired will consume 24KB and server will eventually see the 100% memory utilisation. I even had a call with Microsoft support they confirmed that this would be the application specific issue where caching is not cleared properly. I had the similar memory issue with IPMI utils as well and they provided me with the Hotfix. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
[ntp:questions] Ntpq.exe memory issue with windows 2019
We started with a brand new windows server 2019 datacenter edition and installed an infrastructure monitoring agent on it and strictly no other program. The machine had 8 GB of memory. As part of monitoring NTP offset from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, the memory utilisation had increased to significant level >80%. On analysis we found it was a gradual increase and using RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B in Private. Moreover this issue is specific to windows server 2019 we tried following same steps on windows server 2012 machine and it worked perfectly fine with no memory creeping issues. Any help or pointer are appreciated ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz Sayyed wrote: > We started with a brand new windows server 2019 datacenter edition and > installed an infrastructure monitoring agent on it and strictly no other > program. The machine had 8 GB of memory. As part of monitoring NTP offset > from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, > the memory utilisation had increased to significant level >80%. On analysis > we found it was a gradual increase and using RAMMAP we saw every time the > ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B > in Private. Moreover this issue is specific to windows server 2019 we tried > following same steps on windows server 2012 machine and it worked perfectly > fine with no memory creeping issues. > > Any help or pointer are appreciated Full problem is mentioned here https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] How is the BIG guys (Google. FB etc) DNS and NTP architecture
Facebook is now using Chrony, you can read about their implementation: Building a more accurate time service at Facebook scale https://engineering.fb.com/production-engineering/ntp-service/ ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
