[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 1069d38fa18f3a4f97c2e358bcb3b82cab1c051b by Victor Stinner in 
branch '3.7':
[3.7] bpo-36719: sync regrtest with master branch (GH-12967)
https://github.com/python/cpython/commit/1069d38fa18f3a4f97c2e358bcb3b82cab1c051b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Cheryl Sabella]

Cheryl Sabella  added the comment:

I've merged PR8357.  I believe PR8287 can be closed now with PR8357 as the 
superseder?  

Thank you to everyone for your contributions to this discussion!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Cheryl Sabella]

Cheryl Sabella  added the comment:


New changeset e16467af0bfcc9f399df251495ff2d2ad20a1669 by Cheryl Sabella (Ammar 
Askar) in branch 'master':
bpo-34060: Report system load when running test suite for Windows (GH-8357)
https://github.com/python/cpython/commit/e16467af0bfcc9f399df251495ff2d2ad20a1669


--
nosy: +cheryl.sabella

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

> PR 8287 seems short to me and it seems like psutils doesn't expose Processor 
> Queue Length, so I'm not sure why we are talking about depending on psutils?

I'm not sure if you're strictly interested in getting system load or if CPU 
utilization is also fine. FWIW with psutil you would be able to get the 
system-wide CPU utilization occurred in a given period of time:

>>> import psutil, time
>>> psutil.cpu_percent(interval=None)  # non-blocking
0.0
>>> time.sleep(60)
>>> psutil.cpu_percent(interval=None)  # average of the last 60 secs
23.6
>>>

...and you can do the same for the current process too 
(psutil.Process().cpu_percent()).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

> But then again, if it's solely for our tests, perhaps the best way to 
> approach this is to start a Python thread that periodically runs this command?

I opened up https://github.com/python/cpython/pull/8357 with this strategy, in 
my opinion its a lot nicer and doesn't clutter up _winapi with half baked, 
extremely specialized functions. Its a bit more involved than running a thread, 
details are on the PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Change by Ammar Askar :


--
pull_requests: +7892

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

STINNER Victor  added the comment:

PR 8287 seems short to me and it seems like psutils doesn't expose Processor 
Queue Length, so I'm not sure why we are talking about depending on psutils?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

STINNER Victor  added the comment:

Processor Queue Length seems simpler and easier to read. I don't want to log 24 
numbers per regrtest output line if a machine has 24 CPUs... The load average 
is a "raw" value to give the idea if the system is "loaded" or not. More 
precise metrics can be used later to debug a test failure, but manually.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Steve Dower]

Steve Dower  added the comment:

> It would probably be simpler to change the policy and allow (at least some) 
> some third party libs in cPython's test suite. =)

I'm actually totally okay with this, as I'd really like to have JUnit XML 
output from the test suite, which is easiest to do with the existing 
third-party libraries.

Can we formalize a way by which optional third-party libraries are allowed? 
Provided they aren't critical for the overall pass/fail state of the test suite 
(or the more strict alternative: pass/fail state of *each* test), I don't see 
any particular harm in certain site packages being used.

(This is probably a discussion for python-dev, assuming the policy is written 
down somewhere.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

> But then again, if it's solely for our tests, perhaps the best way to 
> approach this is to start a Python thread that periodically runs this command?

This sounds like a very good solution to me, it avoids adding the complexity of 
the C code. We actually have two options here, to keep the results consistent 
with the unix load, we can use `typeperf "\System\Processor Queue Length"`

To get cpu usage, we can use the command Victor posted. I'll make an 
alternative PR with that today just so we can contrast the two approaches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

psutil exposes this functionality as "psutil.cpu_percent()":

https://github.com/giampaolo/psutil/blob/ac9dccab6b038835b5d612f92cf4804ec2662c2e/psutil/_psutil_windows.c#L992

https://github.com/giampaolo/psutil/blob/ac9dccab6b038835b5d612f92cf4804ec2662c2e/psutil/__init__.py#L1675

I'm not sure if it's worth it to copy all that stuff into Modules/_winapi.c and 
test/libregrtest/main.py though. It would probably be simpler to change the 
policy and allow (at least some) some third party libs in cPython's test suite. 
=)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

STINNER Victor  added the comment:

> I also haven't seen it suggested, but perhaps GetProcessTimes 
> (https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-getprocesstimes)
>  (or GetThreadTimes) would provide enough information to detect the same 
> information?

My intent is to get an idea if the whole system is busy. Not if the current 
Python process is busy. Most buildbots run tests with multiple worker processes 
(at least 2).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Steve Dower]

Steve Dower  added the comment:

> wmic cpu ...

This is the WMI solution we are trying to avoid.

But then again, if it's solely for our tests, perhaps the best way to approach 
this is to start a Python thread that periodically runs this command?

I also haven't seen it suggested, but perhaps GetProcessTimes 
(https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-getprocesstimes)
 (or GetThreadTimes) would provide enough information to detect the same 
information?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

STINNER Victor  added the comment:

I found a command to get the CPU usage in percent *per* CPU. Here with 2 CPUs:

vstinner@WIN C:\vstinner\python\master>wmic cpu get loadpercentage  
LoadPercentage  
100 
100

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

I don't think taking instantaneous values instead of averaging will work out 
too well. For reference I've attached a screenshot. It has sampled values at 
every second on an unloaded computer and then with running prime95 for cpu 
stress testing. The load tends to peak and fall.

>Is it exactly the same thing on Unix (load average)?

Indeed it is: 
https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation

"An idle computer has a load number of 0 (the idle process isn't counted). Each 
process using or waiting for CPU (the ready queue or run queue) increments the 
load number by 1."

>From what I can tell, the number of processors are dealt with the same way as 
>on Linux, that is, a single core processor is overloaded when the load is >1 
>and a quad core processor is overloaded when the load is >4

--
Added file: https://bugs.python.org/file47694/benchmark.PNG

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

STINNER Victor  added the comment:

https://docs.microsoft.com/en-us/sql/relational-databases/performance-monitor/monitor-cpu-usage?view=sql-server-2017

Processor Queue Length:

Corresponds to the number of threads waiting for processor time. A processor 
bottleneck develops when threads of a process require more processor cycles 
than are available. If more than a few processes attempt to utilize the 
processor's time, you might need to install a faster processor. Or, if you have 
a multiprocessor system, you could add a processor.

When you examine processor usage, consider the type of work that the instance 
of SQL Server performs. If SQL Server performs many calculations, such as 
queries involving aggregates or memory-bound queries that require no disk I/O, 
100 percent of the processor's time can be used. If this causes the performance 
of other applications to suffer, try changing the workload. For example, 
dedicate the computer to running the instance of SQL Server.

Usage rates around 100 percent, where many client requests are being processed, 
may indicate that processes are queuing up, waiting for processor time, and 
causing a bottleneck. Resolve the problem by adding faster processors.

--

Is it exactly the same thing on Unix (load average)? If not, I would prefer to 
use a different name in regrtest and "loadavg". Maybe "PQL avg"?

What is the impact of the number of CPUs on this value?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

STINNER Victor  added the comment:

Logging the current value can be an acceptable compromise.

When we run tests in subprocesses (python3 -m test -jN), we can run a test 
every N milliseconds with no thread. It's ok if the average is not accurate.

Most buildbots use -jN for performance but also to isolate the main processes 
from crashes and hard timeouts.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

Roger, personally I don't think its worth it to complicate the code in order to 
use the thread pool API. Especially considering this is just a private API and 
the only consumer will be the test suite runner. Let's see what the core devs 
think when this gets reviewed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Jeremy Kloth]

Jeremy Kloth  added the comment:

The RegisterWaitForSingleObject() function does use the thread pool API:

https://docs.microsoft.com/en-us/windows/desktop/ProcThread/thread-pool-api

However, PdhCollectQueryDataEx() also creates a user-space thread to handle its 
work of setting the event whenever the timeout elapses.

I would think that it should be preferable to use a single thread pool worker 
to retrieve the queue length and calculate the average.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

Is the function I used for the callback, RegisterWaitForSingleObject  
https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-registerwaitforsingleobject
using the thread pooling system you're talking about underneath? The 
documentation seems to suggest so, but I'll wait for your opinion on it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Jeremy Kloth]

Jeremy Kloth  added the comment:

Correct.  Windows provides the building blocks for implementing getloadavg(), 
but does not provide an interface that does the averaging.  That is deferred to 
a per application basis.  The best that an application can do for that is to 
use thread pools.  You can think of thread pools as kernel-managed threads 
(different from user-managed threads via CreateThread()).

As of Win10 1703, any process linked with DLLs automatically have thread pools 
created for them (to parallel-ize the loading of said DLLs).  Leveraging that 
feature would minimize the costs incurred to do the running average.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

I suppose there is no way to emulate os.getloadavg() on Windows because that 
would necessarily imply using a thread to call the necessary routine (WMI, PDH, 
whatever...) every X secs or something, correct?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Jeremy Kloth]

Jeremy Kloth  added the comment:

Not that it matters all that much, but from a terminology standpoint, WMI != 
PDH != Performance Counters.

Performance counters (the objects, not the topic) are provided by DLLs 
registered in the HKLM\SYSTEM\CurrentControlSet\Services key.  Their data is 
accessed via registry API functions using the HKEY_PERFORMANCE_DATA root key.

PDH (Performance Data Helper) provides an abstraction layer that can access 
those values among other things like a GUI or writing to log files.

WMI (Windows Management Instrumentation) is yet another layer on top of PDH and 
raw Performance Counters.

In this case of the "System" performance counter object, it is provided by a 
performance DLL (perfos.dll in the case of Win10 1803).  If overhead (memory 
and/or CPU) is a concern, then accessing the counter data via the registry is 
the way to go.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

Aah, yeah I don't think there's a good way of doing it purely from the windows 
API. There might be a way to enumerate through all the processes and see if 
they're queued up but I didn't look into it.

In this case it should be fine, we just pay a bit of WMI cost to initialize the 
query, all the updating and retrieval is done asynchronously to the python code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

> Giampaolo, on your psutil issue you specifically said, "(possibly without 
> using WMI)" Is there any particular problem with using WMI?

Performance. In general WMI is (a lot) slower than the Windows API counterpart 
(psutil never uses WMI except in unit tests). Don't know if this matters for 
this specific issue though, or whether a correspondent Windows API for doing 
the same thing exists.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

Opened up a PR with a proof of concept to get feedback on if this approach is 
reasonable. 

Giampaolo, on your psutil issue you specifically said, "(possibly without using 
WMI)"

Is there any particular problem with using WMI?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

Thanks a lot for that link Jeremy, it was really helpful. After reading up on 
it, my take is that winapi is the most appropriate place for this, it is a non 
public api that's used in the stdlib.

I've used Windows APIs in a way that we don't need to manually start up a 
thread and call a calc_load function, instead using a callback invoked by 
windows. Internally this uses a thread pool, but it means we don't have to 
worry about managing the thread ourselves.

The load is stored as a global but the winapi module is already marked as "-1" 
indicating it has global state, so that shouldn't be a problem. 
https://docs.python.org/3/c-api/module.html#c.PyModuleDef.m_size

Like Jeremy noted, using WMI does add a 5mb overhead or so to the calling 
process. One more caveat is that the PdhAddEnglishCounterW function is only 
available in Vista+. I'm not sure if we still support Windows XP, but the 
alternative is to use PdhAddCounter, which breaks if the system language is not 
english because the counter paths are localized.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Change by Ammar Askar :


--
keywords: +patch
pull_requests: +7821
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Jeremy Kloth]

Jeremy Kloth  added the comment:

Also prior conversation:

https://bugs.python.org/issue30263#msg296311

--
nosy: +jkloth

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Giampaolo Rodola']

Giampaolo Rodola'  added the comment:

There is an old ticket for this in psutil with some (possible useful) 
references in it: 
https://github.com/giampaolo/psutil/issues/604

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

Taking a shot at this, should take a day or so.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Ammar Askar]

Ammar Askar  added the comment:

Annoyingly, it looks like Windows does not provide an API that gives an average 
value. There is a counter exposed called "System \ Processor Queue Length" 
which does what the equivalent of unix's load

https://blogs.technet.microsoft.com/askperf/2008/01/15/an-overview-of-processor-bottlenecks/

But we're gonna have to average it ourselves if we want this information.

--
nosy: +ammar2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[Giampaolo Rodola']

Change by Giampaolo Rodola' :


--
nosy: +giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34060] regrtest: log "CPU usage" on Windows

[STINNER Victor]

New submission from STINNER Victor :

I would help to debug race conditions on Windows to log the "CPU usage" on 
regrtest, as we do on other platforms (using os.getloadavg()).

Links:

* https://stackoverflow.com/questions/13666458/python-collect-system-stats
* 
https://docs.microsoft.com/en-us/windows/desktop/api/pdh/nf-pdh-pdhmakecounterpatha
* GetSystemTimes()?
* 
https://stackoverflow.com/questions/23143693/retrieving-cpu-load-percent-total-in-windows-with-c

--
components: Tests, Windows
messages: 321177
nosy: paul.moore, steve.dower, tim.golden, vstinner, zach.ware
priority: normal
severity: normal
status: open
title: regrtest: log "CPU usage" on Windows
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com