[issue34060] regrtest: log "CPU usage" on Windows

STINNER Victor added the comment: New changeset 1069d38fa18f3a4f97c2e358bcb3b82cab1c051b by Victor Stinner in branch '3.7': [3.7] bpo-36719: sync regrtest with master branch (GH-12967) https://github.com/python/cpython/commit/1069d38fa18f3a4f97c2e358bcb3b82cab1c051b --

[issue34060] regrtest: log "CPU usage" on Windows

Change by STINNER Victor : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue34060] regrtest: log "CPU usage" on Windows

Cheryl Sabella added the comment: I've merged PR8357. I believe PR8287 can be closed now with PR8357 as the superseder? Thank you to everyone for your contributions to this discussion! -- ___ Python tracker

[issue34060] regrtest: log "CPU usage" on Windows

Cheryl Sabella added the comment: New changeset e16467af0bfcc9f399df251495ff2d2ad20a1669 by Cheryl Sabella (Ammar Askar) in branch 'master': bpo-34060: Report system load when running test suite for Windows (GH-8357)

[issue34060] regrtest: log "CPU usage" on Windows

Giampaolo Rodola' added the comment: > PR 8287 seems short to me and it seems like psutils doesn't expose Processor > Queue Length, so I'm not sure why we are talking about depending on psutils? I'm not sure if you're strictly interested in getting system load or if CPU utilization is also

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: > But then again, if it's solely for our tests, perhaps the best way to > approach this is to start a Python thread that periodically runs this command? I opened up https://github.com/python/cpython/pull/8357 with this strategy, in my opinion its a lot nicer

[issue34060] regrtest: log "CPU usage" on Windows

Change by Ammar Askar : -- pull_requests: +7892 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34060] regrtest: log "CPU usage" on Windows

STINNER Victor added the comment: PR 8287 seems short to me and it seems like psutils doesn't expose Processor Queue Length, so I'm not sure why we are talking about depending on psutils? -- ___ Python tracker

[issue34060] regrtest: log "CPU usage" on Windows

STINNER Victor added the comment: Processor Queue Length seems simpler and easier to read. I don't want to log 24 numbers per regrtest output line if a machine has 24 CPUs... The load average is a "raw" value to give the idea if the system is "loaded" or not. More precise metrics can be

[issue34060] regrtest: log "CPU usage" on Windows

Steve Dower added the comment: > It would probably be simpler to change the policy and allow (at least some) > some third party libs in cPython's test suite. =) I'm actually totally okay with this, as I'd really like to have JUnit XML output from the test suite, which is easiest to do with

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: > But then again, if it's solely for our tests, perhaps the best way to > approach this is to start a Python thread that periodically runs this command? This sounds like a very good solution to me, it avoids adding the complexity of the C code. We actually

[issue34060] regrtest: log "CPU usage" on Windows

Giampaolo Rodola' added the comment: psutil exposes this functionality as "psutil.cpu_percent()": https://github.com/giampaolo/psutil/blob/ac9dccab6b038835b5d612f92cf4804ec2662c2e/psutil/_psutil_windows.c#L992

[issue34060] regrtest: log "CPU usage" on Windows

STINNER Victor added the comment: > I also haven't seen it suggested, but perhaps GetProcessTimes > (https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-getprocesstimes) > (or GetThreadTimes) would provide enough information to detect the same >

[issue34060] regrtest: log "CPU usage" on Windows

Steve Dower added the comment: > wmic cpu ... This is the WMI solution we are trying to avoid. But then again, if it's solely for our tests, perhaps the best way to approach this is to start a Python thread that periodically runs this command? I also haven't seen it suggested, but perhaps

[issue34060] regrtest: log "CPU usage" on Windows

STINNER Victor added the comment: I found a command to get the CPU usage in percent *per* CPU. Here with 2 CPUs: vstinner@WIN C:\vstinner\python\master>wmic cpu get loadpercentage LoadPercentage 100

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: I don't think taking instantaneous values instead of averaging will work out too well. For reference I've attached a screenshot. It has sampled values at every second on an unloaded computer and then with running prime95 for cpu stress testing. The load tends

[issue34060] regrtest: log "CPU usage" on Windows

STINNER Victor added the comment: https://docs.microsoft.com/en-us/sql/relational-databases/performance-monitor/monitor-cpu-usage?view=sql-server-2017 Processor Queue Length: Corresponds to the number of threads waiting for processor time. A processor bottleneck develops when threads of a

[issue34060] regrtest: log "CPU usage" on Windows

STINNER Victor added the comment: Logging the current value can be an acceptable compromise. When we run tests in subprocesses (python3 -m test -jN), we can run a test every N milliseconds with no thread. It's ok if the average is not accurate. Most buildbots use -jN for performance but

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: Roger, personally I don't think its worth it to complicate the code in order to use the thread pool API. Especially considering this is just a private API and the only consumer will be the test suite runner. Let's see what the core devs think when this gets

[issue34060] regrtest: log "CPU usage" on Windows

Jeremy Kloth added the comment: The RegisterWaitForSingleObject() function does use the thread pool API: https://docs.microsoft.com/en-us/windows/desktop/ProcThread/thread-pool-api However, PdhCollectQueryDataEx() also creates a user-space thread to handle its work of setting the event

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: Is the function I used for the callback, RegisterWaitForSingleObject https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-registerwaitforsingleobject using the thread pooling system you're talking about underneath? The documentation seems to

[issue34060] regrtest: log "CPU usage" on Windows

Jeremy Kloth added the comment: Correct. Windows provides the building blocks for implementing getloadavg(), but does not provide an interface that does the averaging. That is deferred to a per application basis. The best that an application can do for that is to use thread pools. You

[issue34060] regrtest: log "CPU usage" on Windows

Giampaolo Rodola' added the comment: I suppose there is no way to emulate os.getloadavg() on Windows because that would necessarily imply using a thread to call the necessary routine (WMI, PDH, whatever...) every X secs or something, correct? --

[issue34060] regrtest: log "CPU usage" on Windows

Jeremy Kloth added the comment: Not that it matters all that much, but from a terminology standpoint, WMI != PDH != Performance Counters. Performance counters (the objects, not the topic) are provided by DLLs registered in the HKLM\SYSTEM\CurrentControlSet\Services key. Their data is

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: Aah, yeah I don't think there's a good way of doing it purely from the windows API. There might be a way to enumerate through all the processes and see if they're queued up but I didn't look into it. In this case it should be fine, we just pay a bit of WMI

[issue34060] regrtest: log "CPU usage" on Windows

Giampaolo Rodola' added the comment: > Giampaolo, on your psutil issue you specifically said, "(possibly without > using WMI)" Is there any particular problem with using WMI? Performance. In general WMI is (a lot) slower than the Windows API counterpart (psutil never uses WMI except in unit

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: Opened up a PR with a proof of concept to get feedback on if this approach is reasonable. Giampaolo, on your psutil issue you specifically said, "(possibly without using WMI)" Is there any particular problem with using WMI? --

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: Thanks a lot for that link Jeremy, it was really helpful. After reading up on it, my take is that winapi is the most appropriate place for this, it is a non public api that's used in the stdlib. I've used Windows APIs in a way that we don't need to manually

[issue34060] regrtest: log "CPU usage" on Windows

Change by Ammar Askar : -- keywords: +patch pull_requests: +7821 stage: -> patch review ___ Python tracker ___ ___

[issue34060] regrtest: log "CPU usage" on Windows

Jeremy Kloth added the comment: Also prior conversation: https://bugs.python.org/issue30263#msg296311 -- nosy: +jkloth ___ Python tracker ___

[issue34060] regrtest: log "CPU usage" on Windows

Giampaolo Rodola' added the comment: There is an old ticket for this in psutil with some (possible useful) references in it: https://github.com/giampaolo/psutil/issues/604 -- ___ Python tracker

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: Taking a shot at this, should take a day or so. -- ___ Python tracker ___ ___ Python-bugs-list

[issue34060] regrtest: log "CPU usage" on Windows

Ammar Askar added the comment: Annoyingly, it looks like Windows does not provide an API that gives an average value. There is a counter exposed called "System \ Processor Queue Length" which does what the equivalent of unix's load

[issue34060] regrtest: log "CPU usage" on Windows

Change by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34060] regrtest: log "CPU usage" on Windows

New submission from STINNER Victor : I would help to debug race conditions on Windows to log the "CPU usage" on regrtest, as we do on other platforms (using os.getloadavg()). Links: * https://stackoverflow.com/questions/13666458/python-collect-system-stats *