Re: Counting Python threads vs C/C++ threads

[Peter J. Holzer]

On 2019-07-16 12:48:33 -0700, Dan Stromberg wrote:
> On Tue, Jul 16, 2019 at 11:13 AM Barry Scott  wrote:
> > Does top show the process using 100% CPU?
> >
> Nope.  CPU utilization and disk use are both low.
> 
> We've been going into top, and then hitting '1' to see things broken down
> by CPU core (there are 32 of them, probably counting hyperthreads as
> different cores), but the CPU use is in the teens or so.

If you had many CPU-bound Python threads, then with 32 cores each core
might show as 3 % busy (the sum of the threads can't use more then 100 %
because of the GIL and 100 / 32 = 3.125).

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Thomas Jollans]

On 17/07/2019 09.58, Barry Scott wrote:
>
>> On 16 Jul 2019, at 20:48, Dan Stromberg  wrote:
>>
>>
>>  
>> A question arises though: Does threading.active_count() only show Python 
>> threads created with the threading module?  What about threads created with 
>> the thread module?
> Only pythons threads, if you think about it why would python care about 
> threads it does not control?


As the docs say, this counts threading.Thread objects. It does not count
all threads started from Python: threads started with the _thread
module, for instance, are not included.

What is more, threads started in external libraries can acquire the GIL
and run Python code. A great example of this are QThreads in a PyQt5
application: QThreads are started by the Qt runtime, which calls a Qt
slot. This Qt slot then might happen to be implemented in Python. I'm
sure other libraries do similar things.

Example with _thread just to check active_count behaviour:

#!/usr/bin/env python3

import threading
import _thread
import time

def thread_func(i):
    print('Starting thread', i)
    time.sleep(0.5)
    print('Thread done', i)

print('Using threading.Thread')
t1 = threading.Thread(target=thread_func, args=(1,))
t1.start()
time.sleep(0.1)
print('active threads:', threading.active_count())
t1.join()


print('Using threading & _thread')
t1 = threading.Thread(target=thread_func, args=(1,))
t1.start()
t2_id = _thread.start_new_thread(thread_func, (2,))
time.sleep(0.1)
print('active threads:', threading.active_count())
time.sleep(0.6)
print('Done, hopefully')



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Barry Scott]

> On 16 Jul 2019, at 20:48, Dan Stromberg  wrote:
> 
> 
> 
> On Tue, Jul 16, 2019 at 11:13 AM Barry Scott  > wrote:
> I'm going to assume you are on linux.
> Yes, I am.  Ubuntu 16.04.6 LTS sometimes, Mint 19.1 other times.
> 
> On 16 Jul 2019, at 18:35, Dan Stromberg  > wrote:
> > 
> > I'm looking at a performance problem in a large CPython 2.x/3.x codebase
> > with quite a few dependencies.
> > 
> > I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
> > hard, and I/O on the system appears to be low - but throughput is poor.
> > I'm wondering if it could be CPU-bound Python threads causing the problem
> > (because of the threading+GIL thing).
> 
> Does top show the process using 100% CPU?
> Nope.  CPU utilization and disk use are both low.

Then your problem is latency. You need to find the slow operation.

> We've been going into top, and then hitting '1' to see things broken down by 
> CPU core (there are 32 of them, probably counting hyperthreads as different 
> cores), but the CPU use is in the teens or so.
> 
> I've also tried dstat and csysdig.  The hardware isn't breaking a sweat, but 
> throughput is poor.

> > The non-dependency Python portions don't Appear to have much in the way of
> > threading going on based on a quick grep, but csysdig says a process
> > running the code has around 32 threads running - the actual thread count
> > varies, but that's the ballpark.
> > 
> > I'm wondering if there's a good way to find two counts of those threads -
> > how many are from CPython code that could run afoul of the GIL, and how
> > many of them are from C/C++ extension modules that wouldn't be responsible
> > for a GIL issue.
> 
> >From the docs on threading:
> 
> threading.active_count()
>  
> 
> Return the number of Thread 
> 
>  objects currently alive. The returned count is equal to the length of the 
> list returned by enumerate() 
> .
> 
> Are you on a Mac?

Opss a file: link sorry should have search the online docs.

I use many operating systems: Fedora, macOS, Windows, NetBSD, CentOS and others 
in the past.

> 
> https://docs.python.org/2/library/threading.html 
>  appears to have some good 
> info. I'll probably try logging threading.active_count()
>  
> A question arises though: Does threading.active_count() only show Python 
> threads created with the threading module?  What about threads created with 
> the thread module?

Only pythons threads, if you think about it why would python care about threads 
it does not control?


> 
> Try running strace on the process to see what system calls its making.
> I've tried it, but thank you.  It's a good suggestion.
> 
> I often find that when strace'ing a program, there's a bunch of 
> mostly-irrelevant stuff at Initial Program Load (IPL), but then the main loop 
> fails into a small cycle of system calls.

And what are thoses sys calls and what is the timing of them?
If you are use select/poll how long before the call returns.
If you in a read how long before it returns.

> 
> Not with this program.  Its main loop is busy and large.

Does the code log any metrics or telemetry to help you?
I work on a product that produces time-series data to show key information 
about the service.
TPS, cache hit rates etc.

Should have mention before you can run the code under python's cprofile.

Do a test run against the process and then run analysis on the data that 
cprofile
produces to find out elapse times and cpu times of the code.

> 
> You could also connect gdb to the process and find out what code the threads 
> are running.
> 
> I used to use gdb, and wrappers for gdb, when I was doing C code, but I don't 
> have much experience using it on a CPython interrpreter.
> 
> Would I be doing a "thread apply all bt" or what?  I'm guessing those 
> backtraces could facilitate identifying the origin of a thread.

Yes  thread apply all bt works great on a python process. recent gdb releases 
knows how to format the stack and show you the python stack,
forgot the command, but its easy to google for.

Barry


> 
> Thanks a bunch.
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Dan Stromberg]

On Tue, Jul 16, 2019 at 11:13 AM Barry Scott  wrote:

> I'm going to assume you are on linux.
>
Yes, I am.  Ubuntu 16.04.6 LTS sometimes, Mint 19.1 other times.

On 16 Jul 2019, at 18:35, Dan Stromberg  wrote:
> >
> > I'm looking at a performance problem in a large CPython 2.x/3.x codebase
> > with quite a few dependencies.
> >
> > I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
> > hard, and I/O on the system appears to be low - but throughput is poor.
> > I'm wondering if it could be CPU-bound Python threads causing the problem
> > (because of the threading+GIL thing).
>
> Does top show the process using 100% CPU?
>
Nope.  CPU utilization and disk use are both low.

We've been going into top, and then hitting '1' to see things broken down
by CPU core (there are 32 of them, probably counting hyperthreads as
different cores), but the CPU use is in the teens or so.

I've also tried dstat and csysdig.  The hardware isn't breaking a sweat,
but throughput is poor.

> The non-dependency Python portions don't Appear to have much in the way of
> > threading going on based on a quick grep, but csysdig says a process
> > running the code has around 32 threads running - the actual thread count
> > varies, but that's the ballpark.
> >
> > I'm wondering if there's a good way to find two counts of those threads -
> > how many are from CPython code that could run afoul of the GIL, and how
> > many of them are from C/C++ extension modules that wouldn't be
> responsible
> > for a GIL issue.
>
> From the docs on threading:
>
> threading.active_count()
>
>  
> 
> Return the number of Thread
> 
> objects currently alive. The returned count is equal to the length of the
> list returned by enumerate()
> .
>

Are you on a Mac?

https://docs.python.org/2/library/threading.html appears to have some good
info. I'll probably try logging threading.active_count()

A question arises though: Does threading.active_count() only show Python
threads created with the threading module?  What about threads created with
the thread module?

Try running strace on the process to see what system calls its making.
>
I've tried it, but thank you.  It's a good suggestion.

I often find that when strace'ing a program, there's a bunch of
mostly-irrelevant stuff at Initial Program Load (IPL), but then the main
loop fails into a small cycle of system calls.

Not with this program.  Its main loop is busy and large.

You could also connect gdb to the process and find out what code the
> threads are running.
>

I used to use gdb, and wrappers for gdb, when I was doing C code, but I
don't have much experience using it on a CPython interrpreter.

Would I be doing a "thread apply all bt" or what?  I'm guessing those
backtraces could facilitate identifying the origin of a thread.

Thanks a bunch.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Barry Scott]

I'm going to assume you are on linux.

On 16 Jul 2019, at 18:35, Dan Stromberg  wrote:
> 
> I'm looking at a performance problem in a large CPython 2.x/3.x codebase
> with quite a few dependencies.
> 
> I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
> hard, and I/O on the system appears to be low - but throughput is poor.
> I'm wondering if it could be CPU-bound Python threads causing the problem
> (because of the threading+GIL thing).

Does top show the process using 100% CPU?

> 
> The non-dependency Python portions don't Appear to have much in the way of
> threading going on based on a quick grep, but csysdig says a process
> running the code has around 32 threads running - the actual thread count
> varies, but that's the ballpark.
> 
> I'm wondering if there's a good way to find two counts of those threads -
> how many are from CPython code that could run afoul of the GIL, and how
> many of them are from C/C++ extension modules that wouldn't be responsible
> for a GIL issue.

>From the docs on threading:

threading.active_count()
 

Return the number of Thread 

 objects currently alive. The returned count is equal to the length of the list 
returned by enumerate() 
.


Try running strace on the process to see what system calls its making.

You could also connect gdb to the process and find out what code the threads 
are running.

Barry

> 
> Does anyone know how to find out these different thread counts, without
> mounting a huge effort to scrutinize all the code and its dependencies?
> I'd prefer a tool that can scrutinize the process(es) from the outside, but
> modifying the code is not out of the question.
> 
> Please see
> https://stackoverflow.com/questions/56958009/how-can-i-tell-how-many-python-threads-a-process-has-from-the-outside
> for more information.  Or perhaps
> https://python-forum.io/Thread-How-to-get-a-count-of-Python-threads-from-the-outside-or-via-code-instrumentation
> 
> Thanks!
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Counting Python threads vs C/C++ threads

[Dan Stromberg]

I'm looking at a performance problem in a large CPython 2.x/3.x codebase
with quite a few dependencies.

I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
hard, and I/O on the system appears to be low - but throughput is poor.
I'm wondering if it could be CPU-bound Python threads causing the problem
(because of the threading+GIL thing).

The non-dependency Python portions don't Appear to have much in the way of
threading going on based on a quick grep, but csysdig says a process
running the code has around 32 threads running - the actual thread count
varies, but that's the ballpark.

I'm wondering if there's a good way to find two counts of those threads -
how many are from CPython code that could run afoul of the GIL, and how
many of them are from C/C++ extension modules that wouldn't be responsible
for a GIL issue.

Does anyone know how to find out these different thread counts, without
mounting a huge effort to scrutinize all the code and its dependencies?
I'd prefer a tool that can scrutinize the process(es) from the outside, but
modifying the code is not out of the question.

Please see
https://stackoverflow.com/questions/56958009/how-can-i-tell-how-many-python-threads-a-process-has-from-the-outside
for more information.  Or perhaps
https://python-forum.io/Thread-How-to-get-a-count-of-Python-threads-from-the-outside-or-via-code-instrumentation

Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list