Re: Counting Python threads vs C/C++ threads

[Peter J. Holzer]

On 2019-07-16 12:48:33 -0700, Dan Stromberg wrote:
> On Tue, Jul 16, 2019 at 11:13 AM Barry Scott  wrote:
> > Does top show the process using 100% CPU?
> >
> Nope.  CPU utilization and disk use are both low.
> 
> We've been going into top, and then hitting '1' to see things broken down
> by CPU core (there are 32 of them, probably counting hyperthreads as
> different cores), but the CPU use is in the teens or so.

If you had many CPU-bound Python threads, then with 32 cores each core
might show as 3 % busy (the sum of the threads can't use more then 100 %
because of the GIL and 100 / 32 = 3.125).

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson <https://www.edge.org/>


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Thomas Jollans]

On 17/07/2019 09.58, Barry Scott wrote:
>
>> On 16 Jul 2019, at 20:48, Dan Stromberg  wrote:
>>
>>
>>  
>> A question arises though: Does threading.active_count() only show Python 
>> threads created with the threading module?  What about threads created with 
>> the thread module?
> Only pythons threads, if you think about it why would python care about 
> threads it does not control?


As the docs say, this counts threading.Thread objects. It does not count
all threads started from Python: threads started with the _thread
module, for instance, are not included.

What is more, threads started in external libraries can acquire the GIL
and run Python code. A great example of this are QThreads in a PyQt5
application: QThreads are started by the Qt runtime, which calls a Qt
slot. This Qt slot then might happen to be implemented in Python. I'm
sure other libraries do similar things.

Example with _thread just to check active_count behaviour:

#!/usr/bin/env python3

import threading
import _thread
import time

def thread_func(i):
    print('Starting thread', i)
    time.sleep(0.5)
    print('Thread done', i)

print('Using threading.Thread')
t1 = threading.Thread(target=thread_func, args=(1,))
t1.start()
time.sleep(0.1)
print('active threads:', threading.active_count())
t1.join()


print('Using threading & _thread')
t1 = threading.Thread(target=thread_func, args=(1,))
t1.start()
t2_id = _thread.start_new_thread(thread_func, (2,))
time.sleep(0.1)
print('active threads:', threading.active_count())
time.sleep(0.6)
print('Done, hopefully')



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Barry Scott]

> On 16 Jul 2019, at 20:48, Dan Stromberg  wrote:
> 
> 
> 
> On Tue, Jul 16, 2019 at 11:13 AM Barry Scott  <mailto:ba...@barrys-emacs.org>> wrote:
> I'm going to assume you are on linux.
> Yes, I am.  Ubuntu 16.04.6 LTS sometimes, Mint 19.1 other times.
> 
> On 16 Jul 2019, at 18:35, Dan Stromberg  <mailto:drsali...@gmail.com>> wrote:
> > 
> > I'm looking at a performance problem in a large CPython 2.x/3.x codebase
> > with quite a few dependencies.
> > 
> > I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
> > hard, and I/O on the system appears to be low - but throughput is poor.
> > I'm wondering if it could be CPU-bound Python threads causing the problem
> > (because of the threading+GIL thing).
> 
> Does top show the process using 100% CPU?
> Nope.  CPU utilization and disk use are both low.

Then your problem is latency. You need to find the slow operation.

> We've been going into top, and then hitting '1' to see things broken down by 
> CPU core (there are 32 of them, probably counting hyperthreads as different 
> cores), but the CPU use is in the teens or so.
> 
> I've also tried dstat and csysdig.  The hardware isn't breaking a sweat, but 
> throughput is poor.

> > The non-dependency Python portions don't Appear to have much in the way of
> > threading going on based on a quick grep, but csysdig says a process
> > running the code has around 32 threads running - the actual thread count
> > varies, but that's the ballpark.
> > 
> > I'm wondering if there's a good way to find two counts of those threads -
> > how many are from CPython code that could run afoul of the GIL, and how
> > many of them are from C/C++ extension modules that wouldn't be responsible
> > for a GIL issue.
> 
> >From the docs on threading:
> 
> threading.active_count()
>  
> 
> Return the number of Thread 
> 
>  objects currently alive. The returned count is equal to the length of the 
> list returned by enumerate() 
> .
> 
> Are you on a Mac?

Opss a file: link sorry should have search the online docs.

I use many operating systems: Fedora, macOS, Windows, NetBSD, CentOS and others 
in the past.

> 
> https://docs.python.org/2/library/threading.html 
> <https://docs.python.org/2/library/threading.html> appears to have some good 
> info. I'll probably try logging threading.active_count()
>  
> A question arises though: Does threading.active_count() only show Python 
> threads created with the threading module?  What about threads created with 
> the thread module?

Only pythons threads, if you think about it why would python care about threads 
it does not control?


> 
> Try running strace on the process to see what system calls its making.
> I've tried it, but thank you.  It's a good suggestion.
> 
> I often find that when strace'ing a program, there's a bunch of 
> mostly-irrelevant stuff at Initial Program Load (IPL), but then the main loop 
> fails into a small cycle of system calls.

And what are thoses sys calls and what is the timing of them?
If you are use select/poll how long before the call returns.
If you in a read how long before it returns.

> 
> Not with this program.  Its main loop is busy and large.

Does the code log any metrics or telemetry to help you?
I work on a product that produces time-series data to show key information 
about the service.
TPS, cache hit rates etc.

Should have mention before you can run the code under python's cprofile.

Do a test run against the process and then run analysis on the data that 
cprofile
produces to find out elapse times and cpu times of the code.

> 
> You could also connect gdb to the process and find out what code the threads 
> are running.
> 
> I used to use gdb, and wrappers for gdb, when I was doing C code, but I don't 
> have much experience using it on a CPython interrpreter.
> 
> Would I be doing a "thread apply all bt" or what?  I'm guessing those 
> backtraces could facilitate identifying the origin of a thread.

Yes  thread apply all bt works great on a python process. recent gdb releases 
knows how to format the stack and show you the python stack,
forgot the command, but its easy to google for.

Barry


> 
> Thanks a bunch.
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Dan Stromberg]

On Tue, Jul 16, 2019 at 11:13 AM Barry Scott  wrote:

> I'm going to assume you are on linux.
>
Yes, I am.  Ubuntu 16.04.6 LTS sometimes, Mint 19.1 other times.

On 16 Jul 2019, at 18:35, Dan Stromberg  wrote:
> >
> > I'm looking at a performance problem in a large CPython 2.x/3.x codebase
> > with quite a few dependencies.
> >
> > I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
> > hard, and I/O on the system appears to be low - but throughput is poor.
> > I'm wondering if it could be CPU-bound Python threads causing the problem
> > (because of the threading+GIL thing).
>
> Does top show the process using 100% CPU?
>
Nope.  CPU utilization and disk use are both low.

We've been going into top, and then hitting '1' to see things broken down
by CPU core (there are 32 of them, probably counting hyperthreads as
different cores), but the CPU use is in the teens or so.

I've also tried dstat and csysdig.  The hardware isn't breaking a sweat,
but throughput is poor.

> The non-dependency Python portions don't Appear to have much in the way of
> > threading going on based on a quick grep, but csysdig says a process
> > running the code has around 32 threads running - the actual thread count
> > varies, but that's the ballpark.
> >
> > I'm wondering if there's a good way to find two counts of those threads -
> > how many are from CPython code that could run afoul of the GIL, and how
> > many of them are from C/C++ extension modules that wouldn't be
> responsible
> > for a GIL issue.
>
> From the docs on threading:
>
> threading.active_count()
>
>  
> 
> Return the number of Thread
> 
> objects currently alive. The returned count is equal to the length of the
> list returned by enumerate()
> .
>

Are you on a Mac?

https://docs.python.org/2/library/threading.html appears to have some good
info. I'll probably try logging threading.active_count()

A question arises though: Does threading.active_count() only show Python
threads created with the threading module?  What about threads created with
the thread module?

Try running strace on the process to see what system calls its making.
>
I've tried it, but thank you.  It's a good suggestion.

I often find that when strace'ing a program, there's a bunch of
mostly-irrelevant stuff at Initial Program Load (IPL), but then the main
loop fails into a small cycle of system calls.

Not with this program.  Its main loop is busy and large.

You could also connect gdb to the process and find out what code the
> threads are running.
>

I used to use gdb, and wrappers for gdb, when I was doing C code, but I
don't have much experience using it on a CPython interrpreter.

Would I be doing a "thread apply all bt" or what?  I'm guessing those
backtraces could facilitate identifying the origin of a thread.

Thanks a bunch.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Counting Python threads vs C/C++ threads

[Barry Scott]

I'm going to assume you are on linux.

On 16 Jul 2019, at 18:35, Dan Stromberg  wrote:
> 
> I'm looking at a performance problem in a large CPython 2.x/3.x codebase
> with quite a few dependencies.
> 
> I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
> hard, and I/O on the system appears to be low - but throughput is poor.
> I'm wondering if it could be CPU-bound Python threads causing the problem
> (because of the threading+GIL thing).

Does top show the process using 100% CPU?

> 
> The non-dependency Python portions don't Appear to have much in the way of
> threading going on based on a quick grep, but csysdig says a process
> running the code has around 32 threads running - the actual thread count
> varies, but that's the ballpark.
> 
> I'm wondering if there's a good way to find two counts of those threads -
> how many are from CPython code that could run afoul of the GIL, and how
> many of them are from C/C++ extension modules that wouldn't be responsible
> for a GIL issue.

>From the docs on threading:

threading.active_count()
 

Return the number of Thread 

 objects currently alive. The returned count is equal to the length of the list 
returned by enumerate() 
.


Try running strace on the process to see what system calls its making.

You could also connect gdb to the process and find out what code the threads 
are running.

Barry

> 
> Does anyone know how to find out these different thread counts, without
> mounting a huge effort to scrutinize all the code and its dependencies?
> I'd prefer a tool that can scrutinize the process(es) from the outside, but
> modifying the code is not out of the question.
> 
> Please see
> https://stackoverflow.com/questions/56958009/how-can-i-tell-how-many-python-threads-a-process-has-from-the-outside
> for more information.  Or perhaps
> https://python-forum.io/Thread-How-to-get-a-count-of-Python-threads-from-the-outside-or-via-code-instrumentation
> 
> Thanks!
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Counting Python threads vs C/C++ threads

[Dan Stromberg]

I'm looking at a performance problem in a large CPython 2.x/3.x codebase
with quite a few dependencies.

I'm not sure what's causing the slowness yet.  The CPU isn't getting hit
hard, and I/O on the system appears to be low - but throughput is poor.
I'm wondering if it could be CPU-bound Python threads causing the problem
(because of the threading+GIL thing).

The non-dependency Python portions don't Appear to have much in the way of
threading going on based on a quick grep, but csysdig says a process
running the code has around 32 threads running - the actual thread count
varies, but that's the ballpark.

I'm wondering if there's a good way to find two counts of those threads -
how many are from CPython code that could run afoul of the GIL, and how
many of them are from C/C++ extension modules that wouldn't be responsible
for a GIL issue.

Does anyone know how to find out these different thread counts, without
mounting a huge effort to scrutinize all the code and its dependencies?
I'd prefer a tool that can scrutinize the process(es) from the outside, but
modifying the code is not out of the question.

Please see
https://stackoverflow.com/questions/56958009/how-can-i-tell-how-many-python-threads-a-process-has-from-the-outside
for more information.  Or perhaps
https://python-forum.io/Thread-How-to-get-a-count-of-Python-threads-from-the-outside-or-via-code-instrumentation

Thanks!
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Emanuel Barry]

Emanuel Barry added the comment:

Let's just close this.

--
nosy: +ebarry
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Michael Hughes]

Michael Hughes added the comment:

Given that this is from five years ago, and I've moved on, I honestly can't say 
I care too deeply about this.

My use case was for handling threads:
* created by inexperienced python programmers that don't know about daemons
* using third party python scripts that it would be easier not to edit

I feel that my proposed change handles that in a reasonable way, and doesn't 
complicate the interface for threads terribly. Most users can completely ignore 
the new method I proposed, and it won't affect them. For those going to look 
for it, it'll be there.

But again, I'm not even working in Python and no one else has chimed in on this 
in five years. Does it matter anymore?

- Michael

> On Nov 30, 2016, at 1:58 PM, Julien Palard  wrote:
> 
> 
> Julien Palard added the comment:
> 
> If nobody has nothing to add on this issue, I think it just should be closed.
> 
> --
> 
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Julien Palard]

Julien Palard added the comment:

If nobody has nothing to add on this issue, I think it just should be closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Julien]

Julien added the comment:

`daemon` flag cannot be changed after thread is started, the documentation is 
right and the code of the daemon setter is actually:

if self._started.is_set():
raise RuntimeError("cannot set daemon status of active thread")

But still it looks like a code review problem: all your daemonic threads should 
be created as daemonic.

Breaking the `daemon` semantics looks like a bug magnet: any future uses of 
non-daemonic threads (by an "experienced" developer of your team, specifically 
needing a non-daemon thread for a specific task) will behave as a deamon thread 
and the specific task may not be correctly executed (like, logging an 
exception?).

--
nosy: +sizeof

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Berker Peksag]

Changes by Berker Peksag :


--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Akira Li]

Akira Li added the comment:

I've removed mentioning of GIL and uploaded a new patch.

--
Added file: 
http://bugs.python.org/file37850/docs-time.sleep-other-threads-are-not-blocked-2.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[R. David Murray]

R. David Murray added the comment:

Oops, typoed the issue number.

New changeset 3a9b1e5fe179 by R David Murray in branch '3.4':
#23215: reflow paragraph.
https://hg.python.org/cpython/rev/3a9b1e5fe179

New changeset 52a06812d5da by R David Murray in branch 'default':
Merge: #23215: note that time.sleep affects the current thread only.
https://hg.python.org/cpython/rev/52a06812d5da

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Roundup Robot]

Roundup Robot added the comment:

New changeset 55ad65c4f9e2 by R David Murray in branch '3.4':
#23251: Note that time.sleep affects the calling thread only.
https://hg.python.org/cpython/rev/55ad65c4f9e2

New changeset 5e01c68cabbf by R David Murray in branch '2.7':
#23251: note that time.sleep affects the current thread only.
https://hg.python.org/cpython/rev/5e01c68cabbf

New changeset 5db28a3199b2 by R David Murray in branch '2.7':
#23251: Reflow paragraph.
https://hg.python.org/cpython/rev/5db28a3199b2

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[R. David Murray]

R. David Murray added the comment:

Thanks, Akira, but I did not use your patch, since it still had the paragraph 
reflow in it.

--
resolution:  - fixed
stage: commit review - resolved

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Antoine Pitrou]

Antoine Pitrou added the comment:

Please let's stop it. Mentioning the GIL in every place in the documentation 
will only make people more worried and confused about something they shouldn't 
be confused or worried about.

If you think some docs should discuss the GIL and its effect on running threads 
then I suggest writing a separate document (a HOWTO for example, like the 
Unicode HOWTO).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Akira Li]

Akira Li added the comment:

 Only if the behaviour was unintuitive (i.e. if it *didn't* release the
 GIL) would it make sense to document it.

There is no intuitive interface, not even the nipple. It's all learned. [1]

 Yes, on consideration I agree with Antoine.  That last sentence should
 be deleted.  Otherwise we'd need to mention that the gil was released
 every place that the gil was released, which would be very redundant.

Whether or not other places mention it is unrelated to the current
issue.

Though the documentation should mention it more. Many programmers are
convinced that Python threads can't be executed in parallel.

 The general rule is that anything that blocks in python releases the
 GIL, therefore as Antoine says the only time we need to document GIL
 behavior is when that *doesn't* happen.

The reader of Python documentation is intelegent but not all-knowing.

Blocking is the first and only job for time.sleep() function.
GIL blocks Python code.
You can't understand what time.sleep does without knowing what happens
to GIL.

Ask yourself who and why reads the time.sleep() documentation (novice
and/or exprerienced Python user). Put yourself into the mind of the
reader.


[1] http://www.greenend.org.uk/rjk/misc/nipple.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Georg Brandl]

Georg Brandl added the comment:

Agreed. of the current thread is a good addition.  The sleep() docs are 
already longer than I would like.

--
nosy: +georg.brandl

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Andrew Svetlov]

Andrew Svetlov added the comment:

I'm with Antoine.

Yes, GIL is extremely important but please don't put GIL mentions everywhere.

On Sun, Jan 18, 2015 at 4:39 PM, Antoine Pitrou rep...@bugs.python.org wrote:

 Antoine Pitrou added the comment:

 Please let's stop it. Mentioning the GIL in every place in the documentation 
 will only make people more worried and confused about something they 
 shouldn't be confused or worried about.

 If you think some docs should discuss the GIL and its effect on running 
 threads then I suggest writing a separate document (a HOWTO for example, like 
 the Unicode HOWTO).

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue23251
 ___

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[R. David Murray]

R. David Murray added the comment:

Yes, on consideration I agree with Antoine.  That last sentence should be 
deleted.  Otherwise we'd need to mention that the gil was released every place 
that the gil was released, which would be very redundant.  The general rule is 
that anything that blocks in python releases the GIL, therefore as Antoine says 
the only time we need to document GIL behavior is when that *doesn't* happen.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Antoine Pitrou]

Antoine Pitrou added the comment:

I think it's superfluous to mention the GIL here, since it has no impact on the 
function.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Akira Li]

Akira Li added the comment:

 I think it's superfluous to mention the GIL here, since it has no impact on 
 the function.

If GIL is not released then all Python code in other threads is 
effectively blocked.
It is worth mentioning explicitly that it is guaranteed to be released
during the sleep.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Antoine Pitrou]

Antoine Pitrou added the comment:

 If GIL is not released then all Python code in other threads is 
 effectively blocked.

But that would be a stupid implementation of sleep(). It is not desirable to 
clutter the docs with such mentions: most calls to the OS in the stdlib release 
the GIL.

Only if the behaviour was unintuitive (i.e. if it *didn't* release the GIL) 
would it make sense to document it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Akira Li]

New submission from Akira Li:

There is the corresponding StackOverflow question with 60K view 
time.sleep — sleeps thread or process? [1]

The documentation patch is attached.

[1] http://stackoverflow.com/questions/92928/time-sleep-sleeps-thread-or-process

--
assignee: docs@python
components: Documentation
files: docs-time.sleep-other-threads-are-not-blocked.diff
keywords: patch
messages: 234135
nosy: akira, docs@python
priority: normal
severity: normal
status: open
title: mention in time.sleep() docs that it does not block other Python threads
type: enhancement
versions: Python 3.5
Added file: 
http://bugs.python.org/file37730/docs-time.sleep-other-threads-are-not-blocked.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[R. David Murray]

R. David Murray added the comment:

Can you re-upload the patch without reflowing the paragraph?  I think the only 
thing needed is the addition of the word thread, to mirror the equivalent unix 
man page phrasing, and I think that's what you've done, but I can't easily tell 
from the provided patch.

--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Martin Panter]

Martin Panter added the comment:

There is also a new sentence about the GIL at the end, but leaving the 
inbetween lines as they were would verify this

--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Akira Li]

Akira Li added the comment:

I do not understand. Have you tried to look at the patch in Rietveld?

The new content is highlighted in a darker green. It is clearly
visible. I've tested on Chromium, Firefox, Safari.

If I won't reflow then the first line will be longer than the
recommended 80 in devguide:

 The maximum line length is 80 characters for normal text, but
 tables, deeply indented code samples and long links may extend
 beyond that.

I've set *fill-column* to 80 in emacs. Do you suggest other settings?

Anyway, it doesn't affect how the final text is shown in a web
browser.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Andrew Svetlov]

Andrew Svetlov added the comment:

I guess R. David Murray asked you to make the least minimal change,
even it breaks the formatting rules.

Paragraph reflow is safe when it's done by the Core Developer but it
requires additional check (and probably mercurial conflict errors on
merging the change with default branch if the last has changes also).

In your case I see no problems though, but the final decision is on R.
David Murray

On Sat, Jan 17, 2015 at 12:56 AM, Akira Li rep...@bugs.python.org wrote:

 Akira Li added the comment:

 I do not understand. Have you tried to look at the patch in Rietveld?

 The new content is highlighted in a darker green. It is clearly
 visible. I've tested on Chromium, Firefox, Safari.

 If I won't reflow then the first line will be longer than the
 recommended 80 in devguide:

 The maximum line length is 80 characters for normal text, but
 tables, deeply indented code samples and long links may extend
 beyond that.

 I've set *fill-column* to 80 in emacs. Do you suggest other settings?

 Anyway, it doesn't affect how the final text is shown in a web
 browser.

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue23251
 ___
 ___
 docs mailing list
 d...@python.org
 https://mail.python.org/mailman/listinfo/docs

--
nosy: +asvetlov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[Martin Panter]

Martin Panter added the comment:

What I have sometimes done in this situation is just break the overly long line 
into two short lines

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23251] mention in time.sleep() docs that it does not block other Python threads

[R. David Murray]

R. David Murray added the comment:

I actually didn't know that reitveld was smart enough to highlight just the 
text changes in a reflowed paragraph.

Nevertheless, for ease of looking at diff in the repository using the hg 
command (which is not that smart), I prefer to commit doc changes without the 
reflow, then do the reflow in a separate commit.  I don't know if other 
developers do this or not.

I think the patch is fine.

--
stage:  - commit review
versions: +Python 2.7, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23251
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Florent Xicluna]

Florent Xicluna florent.xicl...@gmail.com added the comment:

Antoine wrote:
 You could instead enumerate() all threads and set their daemon flag
 to False, before shutting down the interpreter.

If it is intended to work this way, it should be mentioned in the documentation.

Currently the documentation for Thread.daemon says:
“This must be set before start() is called, otherwise RuntimeError is raised.”

--
nosy: +flox

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Load averages and Python threads on different Linux cloud provider setup

[Mohanaraj Gopala Krishnan]

Hello,

Apologies if this post is off topic.

We have a Python application that polls directories using threads and
inotify watchers. We have always run this application in a cloud
server provided by Voxel (http://www.voxel.net). We are currently
testing a different cloud server provider StormOnDemand
(http://stormondemand.com) and when we ran our application, our load
averages were a lot higher then they were when they were running on
the Voxel cloud server despite the specs being about the same (Refer
below for more details on setup). We have also ensured then when
testing the server was not handling any other loads.

I have written a simple test application (test_threads.py - attached,
or refer to http://pastebin.com/xGQU7JD0) that simulates the issues we
are seeing by starting up  threads that loops, sleeping for a user
defined time on each loop. It takes 2 parameters, the amount of
threads to start and the interval period.

When I run, python test_threads.py 50 0.1 for about 10 minutes

Load average results:

StormOnDemand:
$ uptime
 18:46:22 up  7:29,  6 users,  load average: 4.43, 4.16, 2.93

voxel.net
$ uptime
 18:48:14 up 9 days, 15:09,  9 users,  load average: 0.51, 0.47, 0.43

The load average on the StormOnDemand server is a lot higher.

Python version:
StormOnDemand - 2.6.5
Voxel - 2.6.5

Server spec:
StormOnDemand - 8 x Intel(R) Xeon(R) CPU E5506 @ 2.13GHz; 16GB RAM;
230GB HDD (Storm Bare Metal servers)
Voxel - 7 x Intel(R) Xeon(R) CPU L5640 @ 2.27GHz; 14GB RAM; 200GB HDD
(VoxCloud servers)

OS:
StormOnDemand - Ubuntu 10.04 - 2.6.36-rc8101910 #1 SMP Tue Oct 19
19:18:34 UTC 2010 x86_64 GNU/Linux
Voxel - Ubuntu 10.04 -  2.6.32-31-server #61-Ubuntu SMP Fri Apr 8
19:44:42 UTC 2011 x86_64 GNU/Linux

Virtualisation method:
StormOnDemand - Not 100% sure, but I think they use Xen
Voxel - Not sure, but the image that we are using looks to us like a
stock standard Ubuntu 10.04 server

Any suggestion on why the load would be a lot higher or how I could
debug this further is greatly appreciated.


-- 
Mohan
#Simple script to test load when running python threads
# usage:
# python test_threads.py number of threads thread loop interval
# e.g. :
# python test_threads.py 50 0.1
#

import sys
from threading import Lock, Thread
import logging
import time

logging.basicConfig()

log = logging.getLogger(test)
log.setLevel(logging.DEBUG) 

class MonitorDir(Thread):
def __init__(self, id, interval):
super(MonitorDir, self).__init__()
  	self.interval = interval 
self.id = id 
self._terminated = False

def finish(self):
if not self._terminated:
log.info('Terminating... %i'%self.id)
self._terminated = True

def run(self):
print 'Start threadi %i'%self.id

try:
while not self._terminated:
log.info('in id - %i \n'%self.id)
time.sleep(self.interval)
except:
log.exception('Unexpected error while monitoring')
finally:
log.info('Done monitoring')


if __name__ == '__main__':
mThreads = []
for i in range(int(sys.argv[1])):
print create thread %i%i
	mThread = MonitorDir(i, float(sys.argv[2]))
	mThreads.append(mThread)
	mThread.start()

while any(x.isAlive() for x in mThreads):
try:
	time.sleep(1)
except KeyboardInterrupt:
	log.debug('Exiting...')
	break
else:
   sys.exit(1) 

for mThread in mThreads:
mThread.finish()

for mThread in mThreads:
mThread.join()

sys.exit(0)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Load averages and Python threads on different Linux cloud provider setup

[Chris Angelico]

On Tue, May 31, 2011 at 9:25 PM, Mohanaraj Gopala Krishnan
moha...@gmail.com wrote:
 Any suggestion on why the load would be a lot higher or how I could
 debug this further is greatly appreciated.


First off, an easy question: Is system load low and comparable on both
systems when this script is _not_ running?

Chris Angelico
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Load averages and Python threads on different Linux cloud provider setup

[Mohan]

On May 31, 10:22 pm, Chris Angelico ros...@gmail.com wrote:
 On Tue, May 31, 2011 at 9:25 PM, Mohanaraj Gopala Krishnan

 moha...@gmail.com wrote:
  Any suggestion on why the load would be a lot higher or how I could
  debug this further is greatly appreciated.

 First off, an easy question: Is system load low and comparable on both
 systems when this script is _not_ running?

Yes, they both are are  0.5 (for all averages) and comparable.

 Chris Angelico

-- 
http://mail.python.org/mailman/listinfo/python-list

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Skip Montanaro]

Changes by Skip Montanaro s...@pobox.com:


--
nosy:  -skip.montanaro

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Michael Hughes]

Michael Hughes michaelahug...@gmail.com added the comment:

Hey guys
We don't always have control over all of the threads launched within our 
application. We might have inexperienced users writing basic scripts using 
threads, but they don't know enough about setting them to Daemon. Previous 
versions of the Python interpreter we were using in our app didn't block.
I'd like an option to override the blocking, because for our application, for 
the most part, we do want it to always quit and not wait for threads to quit 
properly.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

This looks like an ugly hack to solve a slightly exotic problem. You could 
instead enumerate() all threads and set their daemon flag to False, before 
shutting down the interpreter.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Éric Araujo]

Changes by Éric Araujo mer...@netwok.org:


--
nosy: +eric.araujo
versions:  -Python 2.6, Python 2.7, Python 3.1, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Michael Hughes]

New submission from Michael Hughes michaelahug...@gmail.com:

We use the Python interpreter embedded in our application, and would prefer to 
not block an exit of our application waiting for non-daemon threads to finish.

I would like a mechanism exposed that queries for whether or not to wait.

--
components: Library (Lib)
messages: 121354
nosy: michaelahughes
priority: normal
severity: normal
status: open
title: A mechanism is needed to override waiting for Python threads to finish
type: feature request
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Michael Hughes]

Michael Hughes michaelahug...@gmail.com added the comment:

I have a patch here.
It is to allow for a callback to be set on the main thread which gets called 
when there are non-daemon threads still alive on exit.
The callback returns True or False indicating whether or not it wants to block. 
By default, the whole thing blocks.

--
keywords: +patch
Added file: http://bugs.python.org/file19625/threadingchange.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Michael Hughes]

Changes by Michael Hughes michaelahug...@gmail.com:


Removed file: http://bugs.python.org/file19625/threadingchange.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Michael Hughes]

Michael Hughes michaelahug...@gmail.com added the comment:

Scratch that last patch. It was missing part of the fix.
I've removed the old patch now, and submitted the proper one.

--
Added file: http://bugs.python.org/file19626/real.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

How do you use it?

--
nosy: +amaury.forgeotdarc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Michael Hughes]

Michael Hughes michaelahug...@gmail.com added the comment:

To use the callback, you can do this:

import threading

def threadendcallback():
  # return False to indicate that we don't
  # want to wait for any threads before exiting
  return False

threading.currentThread().waitForThreadsOnExitFunc = threadendcallback

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10444] A mechanism is needed to override waiting for Python threads to finish

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Why is setting your threads as daemons not an option?

--
nosy: +skip.montanaro

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10444
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: ulimit stack size and python threads

[Martin v. Löwis]

 Always crashing because I asked the OS to please not allow a process
 to grow too big is what I call overloading the meaning of ulimit -s.

Please trust that there is no explicit code in the Python interpreter
that tests whether the stack size is 4GB, and then produces an explicit
crash.

 It's quite surprising. Not to mention the poor error message.

AFAICT, you didn't even *report* yet what the error message is,
so I don't know whether it is poor.

Regards,
Martin

--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[Martin v. Löwis]

 I see. I should be blaming the default behavior of pthreads. 

You shouldn't blame anybody. Instead, you should sit down and study
the problem in detail, until you fully understand it. Then you should
start contributing fixes. Never ever should you spread blame.

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

ulimit stack size and python threads

[Greg Lindahl]

I figure this is a FAQ, but I can't find it in any FAQs.

I want to limit the stacksize on my server.

If I set it to 8 megs, or unlimited, python is happy.

If I set it to 4 gigabytes, things like yum (which is a python
program) crash creating a thread. This is on an x86_64 linux kernel,
RHEL5, etc etc.

Why is Python overloading the meaning of the ulimit -s like this?
There are plenty of real non-python programs with huge stack usage,
and I'd like my system default stack limit to be less than unlimited
but much larger than Python will allow.

-- greg

--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[MRAB]

Greg Lindahl wrote:

I figure this is a FAQ, but I can't find it in any FAQs.

I want to limit the stacksize on my server.

If I set it to 8 megs, or unlimited, python is happy.

If I set it to 4 gigabytes, things like yum (which is a python
program) crash creating a thread. This is on an x86_64 linux kernel,
RHEL5, etc etc.

Why is Python overloading the meaning of the ulimit -s like this?
There are plenty of real non-python programs with huge stack usage,
and I'd like my system default stack limit to be less than unlimited
but much larger than Python will allow.

I'm only guessing, but could it be a 32-bit limit somewhere? Have you 
tried, say, 1GB, which would be within a 32-bit limit?

--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[Greg Lindahl]

 I'm only guessing, but could it be a 32-bit limit somewhere? Have you
 tried, say, 1GB, which would be within a 32-bit limit?

Indeed, ulimit -s 100 (a bit smaller than 1 GB) does work, but it
doesn't solve my problem, since I want to set the limit higher than 1
GB.

-- greg
--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[MRAB]

Greg Lindahl wrote:

I'm only guessing, but could it be a 32-bit limit somewhere? Have you
tried, say, 1GB, which would be within a 32-bit limit?


Indeed, ulimit -s 100 (a bit smaller than 1 GB) does work, but it
doesn't solve my problem, since I want to set the limit higher than 1
GB.

How much higher? You could try just under 4GB (unsigned 32-bit) and just 
under 2GB (signed 32-bit).

--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[Martin v. Löwis]

 Why is Python overloading the meaning of the ulimit -s like this?

Why do you think Python is overloading the meaning of that? I ensure
you it isn't - it doesn't actively care what the limits are.

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[Greg Lindahl]

 How much higher? You could try just under 4GB (unsigned 32-bit) and just
 under 2GB (signed 32-bit).

I'd like to set it to be about 1/2 the memory size of my server, which
happens to end up being 4 gbytes. And no, slightly less than 4 gb
doesn't work.

But even if that worked, I'd be worried that python is doing something
bad with the ulimit -s value under the covers.

-- greg
--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[Martin v. Löwis]

 But even if that worked, I'd be worried that python is doing something
 bad with the ulimit -s value under the covers.

Again: it definitely isn't.

Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[Greg Lindahl]

 Why do you think Python is overloading the meaning of that? I ensure
 you it isn't - it doesn't actively care what the limits are.

Always crashing because I asked the OS to please not allow a process
to grow too big is what I call overloading the meaning of ulimit -s.
It's quite surprising. Not to mention the poor error message.

-- greg

--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[Andrew MacIntyre]

Greg Lindahl wrote:

I figure this is a FAQ, but I can't find it in any FAQs.

I want to limit the stacksize on my server.

If I set it to 8 megs, or unlimited, python is happy.

If I set it to 4 gigabytes, things like yum (which is a python
program) crash creating a thread. This is on an x86_64 linux kernel,
RHEL5, etc etc.

Why is Python overloading the meaning of the ulimit -s like this?
There are plenty of real non-python programs with huge stack usage,
and I'd like my system default stack limit to be less than unlimited
but much larger than Python will allow.


The Python interpreter itself (absent a call to resource.setrlimit())
does nothing with resource limits - it just uses the environment it is
loaded into.

In the absence of effective alternative solutions, it may be possible to
achieve the effect you desire by calling resource.setrlimit() in your
Python installation's site.py, effectively over-riding the system
default.

--
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: andy...@bullseye.apana.org.au  (pref) | Snail: PO Box 370
   andy...@pcug.org.au (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/   |Australia
--
http://mail.python.org/mailman/listinfo/python-list

Re: ulimit stack size and python threads

[Greg Lindahl]

I see. I should be blaming the default behavior of pthreads. I did
work on a OpenMP library once, and we worked around this problem, plus
we gave good error messages. Given the number of HPC sites which use
Python, I'd think that Python would have grown similar features. (HPC
sites are more likely to have intermediate-sized stack limits due to
use of Fortran.)

-- greg
--
http://mail.python.org/mailman/listinfo/python-list

Re: How do i : Python Threads + KeyboardInterrupt exception

[Rhamphoryncus]

On Jun 19, 11:09 pm, Brendon Costa [EMAIL PROTECTED] wrote:
  If only the main thread can receive KeyboardInterrupt, is there any
  reason why you couldn't move the functionality of the Read thread into
  the main thread? It looks like it's not doing any work, just waiting
  for the Proc thread to finish.

  You could start the Proc thread, do the current Read thread
  functionality until the interrupt occurs, put the apporpriate message
  in the queue, and then wait for the Proc thread to finish.

 It is already doing that. You will notice that the Proc() function is
 called by a threading.Thread instance so Proc() is running in a
 thread, but the Read() function is being called by the main thread
 right after this. It DOES work with the Ctrl + C, but i can find no
 way at all of closing down the script from within the Proc() thread.

 The relevant bit of code is:
 t = MyThread(Proc, queue, sys.stderr, None)
 Read(queue, sys.stdin, sys.stderr)

 In the end, the problem is that i am trying to multiplex IO and other
 operations. In UNIX i would use select with the input file descriptor
 and an anonymous pipe for the extra commands to achieve this without
 any threads, but this script needs to run primarily on a windows box
 and i would like to use it on UNIX too. I thought i could use threads
 to achieve the IO Multiplexing in python, but it seems not or at least
 not simply.

 How do people usually manage IO multiplexing (not just using sockets)
 cross platform in python?

 I only need to multiplex two sources really:
 * Input file or stdin
 * A input event queue
    This will have messages injected from various locations: timers,
 the processing thread itself, and possibly from a single GUI thread at
 a later point in time.

 Though i can foresee that at some point in the future i may also need
 to multiplex those two above and some sockets (For a server with a few
 clients).

 I was thinking of looking at some asynchronous IO libraries for python
 on Windows + UNIX, any suggestions (Must include more than just
 sockets)?

They either use an event-driven library.. or they use a timeout of
around 1 second.  1 second will definitely waste power on laptops (and
desktops), but it *works*.

python-safethread has this fixed - any lowlevel trickery needed is
done for you - but it's not ported to windows yet.
--
http://mail.python.org/mailman/listinfo/python-list

Re: How do i : Python Threads + KeyboardInterrupt exception

[Brendon Costa]

I tested this a bit more. My windows example was incorrect. It should
have used CTRL_C_EVENT. But even then, the problem is that the process
will also close the console window from which it was called because of
the 0. Also this requires that the process actually have a console and
is not a GUI application.

Is there some other method rather than a blocking for line in fin:
that i can use for reading from a file like device that plays nicely
with other threads asynchronously waking it up?

Sorry for all the posts. I am giving updates as i look further into
the problem.
--
http://mail.python.org/mailman/listinfo/python-list

Re: How do i : Python Threads + KeyboardInterrupt exception

[Brendon Costa]

I tested this a bit more. My windows example was incorrect. It should
have used CTRL_C_EVENT. But even then, the problem is that the process
will also close the console window from which it was called because of
the 0. Also this requires that the process actually have a console and
is not a GUI application.

Is there some other method rather than a blocking for line in fin:
that i can use for reading from a file like device that plays nicely
with other threads asynchronously waking it up?

Sorry for all the posts. I am giving updates as i look further into
the problem.
--
http://mail.python.org/mailman/listinfo/python-list

Re: How do i : Python Threads + KeyboardInterrupt exception

[MRAB]

On Jun 19, 7:54 am, Dennis Lee Bieber [EMAIL PROTECTED] wrote:
 On Wed, 18 Jun 2008 21:33:42 -0700 (PDT), Brendon Costa
 [EMAIL PROTECTED] declaimed the following in comp.lang.python:

  Unfortunately that is the problem. It is blocking in a IO system call
  and i want to force it to exit that with an error, hopefully causing a
  Python exception. I looked at what you mentioned and it is described a
  bit here too:http://sebulba.wikispaces.com/recipe+thread2

         There is no safe means of forcing a Python thread to exit... Heck,
 I believe the MSDN is full of warnings that killing a thread at the
 Windows OS level is fraught with danger.

         Unfortunately, Windows doesn't allow for use of select() on anything
 other than sockets -- otherwise you could code a wait that included a
 die object and write to that object to break the blocking wait.

         If your blocking call has no time-out variant, you may be stuck...

If only the main thread can receive KeyboardInterrupt, is there any
reason why you couldn't move the functionality of the Read thread into
the main thread? It looks like it's not doing any work, just waiting
for the Proc thread to finish.

You could start the Proc thread, do the current Read thread
functionality until the interrupt occurs, put the apporpriate message
in the queue, and then wait for the Proc thread to finish.
--
http://mail.python.org/mailman/listinfo/python-list

Re: How do i : Python Threads + KeyboardInterrupt exception

[Brendon Costa]

 If only the main thread can receive KeyboardInterrupt, is there any
 reason why you couldn't move the functionality of the Read thread into
 the main thread? It looks like it's not doing any work, just waiting
 for the Proc thread to finish.

 You could start the Proc thread, do the current Read thread
 functionality until the interrupt occurs, put the apporpriate message
 in the queue, and then wait for the Proc thread to finish.

It is already doing that. You will notice that the Proc() function is
called by a threading.Thread instance so Proc() is running in a
thread, but the Read() function is being called by the main thread
right after this. It DOES work with the Ctrl + C, but i can find no
way at all of closing down the script from within the Proc() thread.

The relevant bit of code is:
t = MyThread(Proc, queue, sys.stderr, None)
Read(queue, sys.stdin, sys.stderr)

In the end, the problem is that i am trying to multiplex IO and other
operations. In UNIX i would use select with the input file descriptor
and an anonymous pipe for the extra commands to achieve this without
any threads, but this script needs to run primarily on a windows box
and i would like to use it on UNIX too. I thought i could use threads
to achieve the IO Multiplexing in python, but it seems not or at least
not simply.

How do people usually manage IO multiplexing (not just using sockets)
cross platform in python?

I only need to multiplex two sources really:
* Input file or stdin
* A input event queue
   This will have messages injected from various locations: timers,
the processing thread itself, and possibly from a single GUI thread at
a later point in time.

Though i can foresee that at some point in the future i may also need
to multiplex those two above and some sockets (For a server with a few
clients).

I was thinking of looking at some asynchronous IO libraries for python
on Windows + UNIX, any suggestions (Must include more than just
sockets)?
--
http://mail.python.org/mailman/listinfo/python-list

How do i : Python Threads + KeyboardInterrupt exception

[Brendon Costa]

Hi all,

I have a small python project i am working on. Basically i always have
two threads. A Read thread that sits in a loop reading a line at a
time from some input (Usually stdin) and then generating events to be
processed and a Proc thread that processes incoming events from a
queue. There will be additional threads as well that asynchronously
insert events into the queue to be processed, but they are not a part
of this so i have omitted them.

What i want to know is: What is the standard/best way of implementing
such a pattern that works in the presence of errors and particularly
with the KeyboardInterrupt exception?

Some sample code is shown below. This code works as is, except in the
case where the Proc thread wants to initiate the exit of the
application.

For example:
* running the code below and pressing Ctrl + C works fine as the Read
thread is initiating the shutdown.
* running the code below and entering:
   pquit
   some other data
   Ctrl + D

will cause oytput:

Processing: pquit
Proc: Initiating quit

and then it HANGS waiting for the Read thread to exit.

Some questions i have that are:
* KeyboardInterrupt exception seems to only be recieved by the main
thread. Is this ALWAYS the case across all UNIX + windows platforms
(not so worried about others)?
* Can i somehow get the Proc thread to force the Read thread to
generate a KeyboardInterrupt or somehow exit its blocking for line in
fin: call?


Thanks,
Brendon


--- SNIP ---
# Two or more threads
#
# proc : Is a processing thread that basically reads events from a
event queue and processes them
# read : Is a thread reading in a loop from stdin and generating
events for proc
# * : Other additional threads that may asynchronously add events to
the queue to be processed

import Queue
import threading
import sys

def Read(queue, fin, fout):
   ret = (1, 'Normal Exit')
   try:
  for line in fin:
 queue.put((0, line))
 #raise Exception(Blah)
 #raise Blah
   except KeyboardInterrupt:  ret = (1, 'KeyboardInterrupt')
   except Exception, e:   ret = (1, 'ERROR: ' + str(e))
   except:ret = (1, 'UNKNOWN-ERROR')

   # Notify Proc thread that we are exiting.
   queue.put(ret)
   print fout, 'Read: Initiating quit'


def Proc(queue, fout, ignore):
   quit = False
   while not quit:
  (evt_type, evt_data) = queue.get()

  if   evt_type == 0:
 print fout, 'Processing: ' + str(evt_data)
 if evt_data.startswith('pquit'):
print fout, 'Proc: Initiating quit'
quit = True

  elif evt_type == 1:
 print fout, 'Quit: ' + str(evt_data)
 quit = True

class MyThread(threading.Thread):
   def __init__(self, func, queue, file1, file2, *args, **kwds):
  threading.Thread.__init__(self, *args, **kwds)
  self.func = func
  self.queue = queue
  self.file1 = file1
  self.file2 = file2
  self.start()

   def run(self):
  return self.func(self.queue, self.file1, self.file2)


if __name__ == '__main__':
   queue = Queue.Queue()

   # Read thread is the main thread and seems to get the
KeyboardInterrupt exception.
   t = MyThread(Proc, queue, sys.stderr, None)
   Read(queue, sys.stdin, sys.stderr)

   # Read thread is NOT the main thread and never seems to get the
KeyboardInterrupt exception.
   # This doesnt work for that reason.
   #t = MyThread(Read, queue, sys.stdin, sys.stderr)
   #Proc(queue, sys.stderr, None)


   # @@@Brendon How do we notify the Read thread that they should
exit?
   # If the Read thread initiated the quit then all is fine.
   # If the Proc thread initiated the quit then i need to get the
Read
   # thread to exit too somehow. But it is currently blocking in a
read
   # on an input file.
   print sys.stderr, 'Joining thread.'
   t.join()

--
http://mail.python.org/mailman/listinfo/python-list

Re: How do i : Python Threads + KeyboardInterrupt exception

[Gabriel Genellina]

En Wed, 18 Jun 2008 21:39:41 -0300, Brendon Costa [EMAIL PROTECTED]  
escribió:



I have a small python project i am working on. Basically i always have
two threads. A Read thread that sits in a loop reading a line at a
time from some input (Usually stdin) and then generating events to be
processed and a Proc thread that processes incoming events from a
queue. There will be additional threads as well that asynchronously
insert events into the queue to be processed, but they are not a part
of this so i have omitted them.

What i want to know is: What is the standard/best way of implementing
such a pattern that works in the presence of errors and particularly
with the KeyboardInterrupt exception?


I don't know the standard way, but perhaps you can get some ideas from  
this recent thread:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/82636f1bdd1d2d83/


Some sample code is shown below. This code works as is, except in the
case where the Proc thread wants to initiate the exit of the
application.


You might try using the PyThreadState_SetAsyncExc function (from the  
Python C API) to inject a KeyboardInterrupt exception into the Read thread  
- but I don't know if it will work at all, the execution might be blocked  
waiting for an I/O call to complete, and never return to Python code...


--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

Re: How do i : Python Threads + KeyboardInterrupt exception

[Brendon Costa]

 I don't know the standard way, but perhaps you can get some ideas from
 this recent 
 thread:http://groups.google.com/group/comp.lang.python/browse_thread/thread/...


I had a quick read through that thread. I think i will need some more
time to think about what they are saying in there though. They seem to
talking about killing a python thread kind of similar to the C
functions TerminateThread() in windows or pthread_cancel() on UNIX
which are not suitable for my purpose.


 You might try using the PyThreadState_SetAsyncExc function (from the
 Python C API) to inject a KeyboardInterrupt exception into the Read thread
 - but I don't know if it will work at all, the execution might be blocked
 waiting for an I/O call to complete, and never return to Python code...


Unfortunately that is the problem. It is blocking in a IO system call
and i want to force it to exit that with an error, hopefully causing a
Python exception. I looked at what you mentioned and it is described a
bit here too: http://sebulba.wikispaces.com/recipe+thread2

I really need a python mechanism for interrupting blocking system
calls.  have dealt with this sort of thing before in C/C++ on windows
and UNIX. For UNIX it is really a matter of sending a signal to the
process (SIGINTR), the main thread which is the only one in Python to
accept signals (others as i understand are masked) will get the signal
and and return with an EINTR breaking out of the blocking read
hopefully a python exception of Interrupted IO or KeyboardInterrupt.
Note that this only works for the one thread , but that is all i need.

For windows, it is possible to do a similar thing using:
GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, 0) with which behaves a bit
like a UNIX signal and i assume is what causes the KeyboardInterrupt
in the first place.

The problem i have is how do I achieve this in python?
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python threads and memory usage

[Mike]

On May 30, 9:16 am, Dennis Lee Bieber [EMAIL PROTECTED] wrote:
 On Thu, 29 May 2008 12:01:30 -0700 (PDT), Mike [EMAIL PROTECTED]
 declaimed the following in comp.lang.python:

  I observed, that every thread reserved some memory, and after exit
  thread doesn't freed it. When i leaved my server working for 3 days,
  then it takes 15% of 512MB memory (during that time about 15000
  threads were created and stopped). When server starts it only takes
  about 1% of memory.

         Do you have any outstanding references to the threads? If so, have
 you performed a .join() with the thread? Until you join it, the thread
 state (thread local objects/variables) are probably being held for
 access from outside the thread.
 --
         Wulfraed        Dennis Lee Bieber               KD6MOG
         [EMAIL PROTECTED]             [EMAIL PROTECTED]
                 HTTP://wlfraed.home.netcom.com/
         (Bestiaria Support Staff:               [EMAIL PROTECTED])
                 HTTP://www.bestiaria.com/

I'm joining threads only during my program exit. I'll try to do what
You suggest.

THX
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python threads and memory usage

[Mike]

On May 30, 9:42 am, Mike [EMAIL PROTECTED] wrote:
 On May 30, 9:16 am, Dennis Lee Bieber [EMAIL PROTECTED] wrote:



  On Thu, 29 May 2008 12:01:30 -0700 (PDT), Mike [EMAIL PROTECTED]
  declaimed the following in comp.lang.python:

   I observed, that every thread reserved some memory, and after exit
   thread doesn't freed it. When i leaved my server working for 3 days,
   then it takes 15% of 512MB memory (during that time about 15000
   threads were created and stopped). When server starts it only takes
   about 1% of memory.

          Do you have any outstanding references to the threads? If so, have
  you performed a .join() with the thread? Until you join it, the thread
  state (thread local objects/variables) are probably being held for
  access from outside the thread.
  --
          Wulfraed        Dennis Lee Bieber               KD6MOG
          [EMAIL PROTECTED]             [EMAIL PROTECTED]
                  HTTP://wlfraed.home.netcom.com/
          (Bestiaria Support Staff:               [EMAIL PROTECTED])
                  HTTP://www.bestiaria.com/

 I'm joining threads only during my program exit. I'll try to do what
 You suggest.

 THX

It helped. Now all threads are added to thread list and every some
period of time I'm checking which threads are alive (enumerate), and
joining all which aren't. Now [memory usage is still on 1% :D:D:D

Thanks
--
http://mail.python.org/mailman/listinfo/python-list

Python threads and memory usage

[Mike]

Hi,

I'm writing client-server application in Python. It's monitoring
system, where server listen and waits for TCP connections, and every
connection takes own thread. Every thread puts data from clients to
Queue and exits. Then there is one DB loader thread, which loads all
data from Queue to MySQL DB.

I observed, that every thread reserved some memory, and after exit
thread doesn't freed it. When i leaved my server working for 3 days,
then it takes 15% of 512MB memory (during that time about 15000
threads were created and stopped). When server starts it only takes
about 1% of memory.

I know that I can made client which connects once and keep this
session with server thread all time (it should resolve my problems),
but I'm curious if this is normal or it is something like memory leak.
Or maybe I'm doing something wrong :)

If You need I can paste all my code.

Regards
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads - stopped vs. gone vs. Alive

[RossGK]

On May 28, 12:01 pm, RossGK [EMAIL PROTECTED] wrote:
 I'm a newbie to python threads, and playing with some simple client
 server stuff and lots of print statements.

 My server thread launched with
  self.worker = WorkerThread(self)
 completes an interaction and then if I check on it's status with
  print Status:, self.workerI getStatus none

 A client thread behaves differently. Launched as
   self.clientWorker( = ClientThreadself)
 when it finishes it's work, I instead get:
  Status: ClientThread(Thread-2, stopped)

 If I check the isAlive status on each of those, self.worker.isAlive
 throws an exception, 'cause there is nothing there anymore to check
 isAlive on.  But self.clientWorker comes back as False, since it is a
 stopped thread and hasn't gone away (which I'd like it to after it
 finishes its work).

 So my question is when a thread finishes its work how do I more
 predictably control whether it is just stopped, or goes away all
 together?   I don't want to do a double nested 'if' statement to check
 if it exists before I check if it's alive.



Pls ignore the obvious typos, this isn't a syntax question and google
groups seems to be messing with my typing and spacing(!)  e.g.
   self.clientWorker( = ClientThreadself)
should have read
   self.clientWorker = ClientThread(self)

As I swear I had typed it...


--
http://mail.python.org/mailman/listinfo/python-list

Python Threads - stopped vs. gone vs. Alive

[RossGK]

I'm a newbie to python threads, and playing with some simple client
server stuff and lots of print statements.

My server thread launched with
 self.worker = WorkerThread(self)
completes an interaction and then if I check on it's status with
 print Status:, self.workerI getStatus none


A client thread behaves differently. Launched as
  self.clientWorker( = ClientThreadself)
when it finishes it's work, I instead get:
 Status: ClientThread(Thread-2, stopped)

If I check the isAlive status on each of those, self.worker.isAlive
throws an exception, 'cause there is nothing there anymore to check
isAlive on.  But self.clientWorker comes back as False, since it is a
stopped thread and hasn't gone away (which I'd like it to after it
finishes its work).

So my question is when a thread finishes its work how do I more
predictably control whether it is just stopped, or goes away all
together?   I don't want to do a double nested 'if' statement to check
if it exists before I check if it's alive.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads - stopped vs. gone vs. Alive

[RossGK]

I've answered my own question about the None state - an event was
setting the thread to None where I didn't expect it.

However, my question slightly repositioned is if a Thread is Stopped
it still seems to exist. Is there someway to make it go away, send it
to garbage collection etc?

Other part of the original Q - I assume a Stopped thread gets a false
from isAlive and a Running thread gets a true?

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads - stopped vs. gone vs. Alive

[Francesco Bochicchio]

On Wed, 28 May 2008 11:38:53 -0700, RossGK wrote:

 
 I've answered my own question about the None state - an event was
 setting the thread to None where I didn't expect it.
 
 However, my question slightly repositioned is if a Thread is Stopped
 it still seems to exist. Is there someway to make it go away, send it
 to garbage collection etc?
 
You have to call the join() method of the thread object from another
thread. This will terminate the thread and free its resources. This is
usually the task of the parent thread which usually does something
like:
 t = MyTread(...)
 t.start()
 # do your own stuff, then before quitting
 t.join() # Note that this waits until t is terminated



 Other part of the original Q - I assume a Stopped thread gets a false
 from isAlive and a Running thread gets a true?

Yes. You can try yourself:

 import threading
 class MyThread(threading.Thread):
def __init__(self): threading.Thread.__init__(self)
def stop(self): self._stop = True
def run(self):
self._stop= False
while self._stop == False:
import time
time.sleep(0.1)


 t = MyThread()
 t.start()
 t.isAlive()
True
 t.isAlive()
False
 t.join()
 t.isAlive()
False


Ciao

FB
--
http://mail.python.org/mailman/listinfo/python-list

Re: advanced usage of python threads

[hyperboreean]

Chris, I have already made my choice, I am asking just for a little help 
with some documentation.
I know about twisted.enterprise.adbapi, but the company is working with 
sqlalchemy at the time.
So please, I know you have good intentions but you're kind of not 
helping me :)
Well, probably instead of asking you people I should just start 
searching the net, thing that I am already doing. But if any of you has 
some info, please share.

Thanks.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: advanced usage of python threads

[Steve Holden]

hyperboreean wrote:
 Chris, I have already made my choice, I am asking just for a little help 
 with some documentation.
 I know about twisted.enterprise.adbapi, but the company is working with 
 sqlalchemy at the time.
 So please, I know you have good intentions but you're kind of not 
 helping me :)
 Well, probably instead of asking you people I should just start 
 searching the net, thing that I am already doing. But if any of you has 
 some info, please share.
 
 Thanks.
 
Well, if you plan to use threading with Twisted you would probably be a 
lot better off asking questions on the Twisted mailing list.

   [EMAIL PROTECTED]

regards
  Steve
-- 
Steve Holden+1 571 484 6266   +1 800 494 3119
Holden Web LLC  http://www.holdenweb.com/

-- 
http://mail.python.org/mailman/listinfo/python-list

advanced usage of python threads

[hyperboreean]

Hi,
Is there a document where I can find some advanced information about 
python threads? I know the basic things about them and did some 
practice, but when I try to advance I don't know where to go or how to go.
Thanks.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: advanced usage of python threads

[Christian Heimes]

hyperboreean wrote:
 Hi,
 Is there a document where I can find some advanced information about 
 python threads? I know the basic things about them and did some 
 practice, but when I try to advance I don't know where to go or how to go.

What's your application doing? Most people are not aware that threading
isn't the best solution for concurrency. For IO bound applications (like
network applications) async event IO is faster and costs less resources.
For other problems you are better off with forking processes.

This is Python and not Java. This is the 21st century and not the early
90ties of the last century. Java favors threads because back in the
90ties it was design for dumb set-top boxes which neither supported
separate process spaces nor threads.

Christian

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: advanced usage of python threads

[hyperboreean]

Well, I will be writing the application server of a three-tier 
architecture system. I will be using Twisted for the communication with 
the client but from there I have to make several calls to a database and 
this asks threading. The tables will be filled by another system that 
gathers some data. I want to implement several threads (workers) that 
takes care of each client and a coordinator thread that takes care of 
all the threads. But I have never worked with threads at this level and 
I feel like I need some coordination. I will be thankful with 
documentation on advanced threads, I probably can apply that in practice.

Thanks to all.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: advanced usage of python threads

[Chris Mellon]

On Fri, Feb 22, 2008 at 12:32 PM, hyperboreean
[EMAIL PROTECTED] wrote:
 Well, I will be writing the application server of a three-tier
  architecture system. I will be using Twisted for the communication with
  the client but from there I have to make several calls to a database and
  this asks threading. The tables will be filled by another system that
  gathers some data. I want to implement several threads (workers) that
  takes care of each client and a coordinator thread that takes care of
  all the threads. But I have never worked with threads at this level and
  I feel like I need some coordination. I will be thankful with
  documentation on advanced threads, I probably can apply that in practice.

  Thanks to all.


Twisted has an asynchronous database API (built on top of threads)
already. Look at twisted.enterprise.adbapi.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can python threads take advantage of use dual core ?

[Diez B. Roggisch]

nikhilketkar schrieb:
 What are the implications of the Global Interpreter Lock in Python ?
 Does this mean that Python threads cannot exploit a dual core
 processor and the only advantage of using threads is in that
 computation and IO-bound operations can continue in parallel ?


Essentially, yes. That is unless the computation is done in C-code which 
released the GIL beforehand. But a certain tradeoff is to expected 
nontheless.

Diez
-- 
http://mail.python.org/mailman/listinfo/python-list

Can python threads take advantage of use dual core ?

[nikhilketkar]

What are the implications of the Global Interpreter Lock in Python ?
Does this mean that Python threads cannot exploit a dual core
processor and the only advantage of using threads is in that
computation and IO-bound operations can continue in parallel ?

Thanks,
Nikhil

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can python threads take advantage of use dual core ?

[Stefan Behnel]

Diez B. Roggisch wrote:
 nikhilketkar schrieb:
 What are the implications of the Global Interpreter Lock in Python ?
 Does this mean that Python threads cannot exploit a dual core
 processor and the only advantage of using threads is in that
 computation and IO-bound operations can continue in parallel ?
 
 
 Essentially, yes. That is unless the computation is done in C-code which
 released the GIL beforehand.

Which virtually all computation-intensive extensions do. Also, note the
processing package, which allows you to use a separate process more or less
like a thread, thus avoiding GIL issues completely.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can python threads take advantage of use dual core ?

[Bjoern Schliessmann]

nikhilketkar wrote:

 What are the implications of the Global Interpreter Lock in Python
 ? 

Please have a look at the archives. This topic is brought up anew
every few weeks.

 Does this mean that Python threads cannot exploit a dual core 
 processor and the only advantage of using threads is in that
 computation and IO-bound operations can continue in parallel ?

Not generally, no.

Regards,


Björn

-- 
BOFH excuse #93:

Feature not yet implemented

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can python threads take advantage of use dual core ?

[sturlamolden]

On Aug 17, 6:00 pm, nikhilketkar [EMAIL PROTECTED] wrote:


 What are the implications of the Global Interpreter Lock in Python ?

This is asked every second week or so.

The GIL is similar to the Big Kernel Lock (BKL) use to support SMPs in
older versions of the Linux kernel. The GIL prevents the Python
interpreter to be active in more than one thread at a time.

If you have a single CPU with a single core, the GIL has no
consequence what so ever.

If you have multiple CPUs and/or multiple cores, the GIL has the
consequence that you can forget about SMP scalability using Python
threads. SMP scalability require more fine grained object locking,
which is what the Java and .NET runtimes do, as well as current
versions of the Linux kernel. Note that IronPython and Jython use fine
grained locking instead of a GIL.

 Does this mean that Python threads cannot exploit a dual core
 processor and the only advantage of using threads is in that
 computation and IO-bound operations can continue in parallel ?


The Python standard library releases the GIL in read/write operations
for e.g. files and sockets. This can be combined with threads to allow
multiple IO operations to continue in parallel. The GIL has no or very
little significance for IO-bound scalability in Python.

CPU-bound tasks are a different game. This is where the GIL matter.
You must either release the GIL or use multiple processes for
exploiting multiple processors in an SMP computer with Python. The GIL
can be released explicitely in C extension code. f2py and ctypes can
also release the GIL.

Note that you would NOT use threads for speeding up CPU-bound
operations, even when programming in C or Fortran. Threads are neither
the only nor the preferred way to exploit multicore CPUs for CPU-bound
tasks. Instead of threads, use either an MPI library or OpenMP
compiler pragmas. You can use MPI directly from Python (e.g. mpi4py),
or you can use OpenMP pragmas in C or Fortran code which you call
using ctypes or f2py.

Summary:

Use Python threads if you need to run IO operations in parallel.
Do not use Python threads if you need to run computations in parallel.



Regards,
Sturla Molden



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can python threads take advantage of use dual core ?

[sturlamolden]

On Aug 17, 6:00 pm, nikhilketkar [EMAIL PROTECTED] wrote:


 What are the implications of the Global Interpreter Lock in Python ?

This is asked every second week or so.

The GIL is similar to the Big Kernel Lock (BKL) use to support SMPs in
older versions of the Linux kernel. The GIL prevents the Python
interpreter to be active in more than one thread at a time.

If you have a single CPU with a single core, the GIL has no
consequence what so ever.

If you have multiple CPUs and/or multiple cores, the GIL has the
consequence that you can forget about SMP scalability using Python
threads. SMP scalability require more fine grained object locking,
which is what the Java and .NET runtimes do, as well as current
versions of the Linux kernel. Note that IronPython and Jython use fine
grained locking instead of a GIL.

 Does this mean that Python threads cannot exploit a dual core
 processor and the only advantage of using threads is in that
 computation and IO-bound operations can continue in parallel ?


The Python standard library releases the GIL in read/write operations
for e.g. files and sockets. This can be combined with threads to allow
multiple IO operations to continue in parallel. The GIL has no or very
little significance for IO-bound scalability in Python.

CPU-bound tasks are a different game. This is where the GIL matter.
You must either release the GIL or use multiple processes for
exploiting multiple processors in an SMP computer with Python. The GIL
can be released explicitely in C extension code. f2py and ctypes can
also release the GIL.

Note that you would NOT use threads for speeding up CPU-bound
operations, even when programming in C or Fortran. Threads are neither
the only nor the preferred way to exploit multicore CPUs for CPU-bound
tasks. Instead of threads, use either an MPI library or OpenMP
compiler pragmas. You can use MPI directly from Python (e.g. mpi4py),
or you can use OpenMP pragmas in C or Fortran code which you call
using ctypes or f2py.

Summary:

Use Python threads if you need to run IO operations in parallel.
Do not use Python threads if you need to run computations in parallel.



Regards,
Sturla Molden



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Can python threads take advantage of use dual core ?

[Alex Martelli]

Stefan Behnel [EMAIL PROTECTED] wrote:
   ...
 Which virtually all computation-intensive extensions do. Also, note the

gmpy doesn't (release the GIL), even though it IS computationally
intensive -- I tried, but it slows things down horribly even on an Intel
Core Duo.  I suspect that may partly be due to the locking strategy of
the underlying GMP 4.2 library (which I haven't analyzed in depth).  In
practice, when I want to exploit both cores to the hilt with gmpy-based
computations, I run multiple processes.


Alex
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: M2Crypto-0.17 blocks python threads?

[John Nagle]

Heikki Toivonen wrote:
 [EMAIL PROTECTED] wrote:
 
I am having a problem with python threads and M2Crypto.  It appears
the M2Crypto used in multi-thread application blocks other threads
from running:
 
 
 This turned into https://bugzilla.osafoundation.org/show_bug.cgi?id=9401
 
 Thanks for the report!

That helps.  I've been getting occasional stalls and timeouts in
a multithreaded program that can have several M2Crypto operations
going, and this is probably the explaination.

John Nagle
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: M2Crypto-0.17 blocks python threads?

[Heikki Toivonen]

[EMAIL PROTECTED] wrote:
 I am having a problem with python threads and M2Crypto.  It appears
 the M2Crypto used in multi-thread application blocks other threads
 from running:

This turned into https://bugzilla.osafoundation.org/show_bug.cgi?id=9401

Thanks for the report!

-- 
  Heikki Toivonen
-- 
http://mail.python.org/mailman/listinfo/python-list

M2Crypto-0.17 blocks python threads?

[reizes]

I am having a problem with python threads and M2Crypto.  It appears
the M2Crypto used in multi-thread application blocks other threads
from running:

Environment: Linux 2.6 (centos 5.0), OpenSSL 0.9.8b, M2Crypto-0.17

I am using echod-thread.py and echo.py as test vehicles.

Start up echod-thread.py
Connect with echo.py - everything looks ok, but connect with second
echo.py and notice that the server does not respond until after the
first echo session is terminated. And yes, echod-thread.py does call
M2Crypto.threading.init()

So my questions are has anyone seen this kind of threading behavior?
If so how did you fix it?

(NOTE:  This used to work with old M2Crytpo-0.13)

I edited a version of the server to print some debug messages.  Around
the main server loop:

while 1:
print  waiting for connection on port 
conn, addr = sock.accept()
thread.start_new_thread(echo_handler, (ctx, conn, addr))
print  started thread, main thread sleeping for 2
seconds
time.sleep(2)   # give first session time to start

[EMAIL PROTECTED] ssl]$ python echod-thread.py
 waiting for connection on port 
 started thread, main thread sleeping for 2 seconds
NOTE: main thread does not continue until thread started with first
echo session terminates!

 first echo session thread -- works ok
[EMAIL PROTECTED] ssl]$ python echo.py -h belgrade.tablus.com
LOOP: SSL connect: before/connect initialization
LOOP: SSL connect: SSLv3 write client hello A
LOOP: SSL connect: SSLv3 read server hello A
LOOP: SSL connect: SSLv3 read server certificate A
LOOP: SSL connect: SSLv3 read server key exchange A
LOOP: SSL connect: SSLv3 read server done A
LOOP: SSL connect: SSLv3 write client key exchange A
LOOP: SSL connect: SSLv3 write change cipher spec A
LOOP: SSL connect: SSLv3 write finished A
LOOP: SSL connect: SSLv3 flush data
LOOP: SSL connect: SSLv3 read finished A
INFO: SSL connect: SSL negotiation finished successfully
Host = belgrade.tablus.com
Cipher = DHE-RSA-AES256-SHA
Server = /CN=belgrade.tablus.com/ST=CA/C=US/
[EMAIL PROTECTED]/O=Root Certification Authority
Ye Newe Threading Echo Servre
Echo this
Echo this

 second echo session thread -- hangs waiting for server to respond
until first session exits
[EMAIL PROTECTED] ssl]$ python echo.py -h belgrade.tablus.com
LOOP: SSL connect: before/connect initialization
LOOP: SSL connect: SSLv3 write client hello A

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads -

[Daniel Nogradi]

 Can you please suggest a technique in Python where we can spawn few number
 of worker threads and later map them to a function/s to execute individual
 Jobs.

Have a look at the threading module:
http://docs.python.org/lib/module-threading.html

HTH,
Daniel
-- 
http://mail.python.org/mailman/listinfo/python-list

Python Threads -

[S.Mohideen]

Hi All,

Can you please suggest a technique in Python where we can spawn few number 
of worker threads and later map them to a function/s to execute individual 
Jobs.

Any references would be helpful..

Thanks
Moin
 

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads -

[Aahz]

In article [EMAIL PROTECTED],
S.Mohideen [EMAIL PROTECTED] wrote:

Can you please suggest a technique in Python where we can spawn few number 
of worker threads and later map them to a function/s to execute individual 
Jobs.

You can see an example for a web spider on my Python website.
-- 
Aahz ([EMAIL PROTECTED])   * http://www.pythoncraft.com/

Help a hearing-impaired person: http://rule6.info/hearing.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads -

[Alex Martelli]

S.Mohideen [EMAIL PROTECTED] wrote:

 Hi All,
 
 Can you please suggest a technique in Python where we can spawn few number
 of worker threads and later map them to a function/s to execute individual
 Jobs.
 
 Any references would be helpful..

I believe I give some examples in the Nutshell, but the basic idea is
very simple: start the N worker threads you want in your pool -- they
all use .get on the same Queue.Queue instance qI which contains work
requests (WRs).  A WR could e.g. be a tuple with a Queue qO (or None to
indicate that the result is not interesting), a callable, and 0+
arguments (or an empty tuple to request thread termination); the worker
thread just calls the callable on the arguments and puts the result to
qO (if qO is not None).  Many other similar arrangements are possible,
depending on your application's exact needs, but I hope the general idea
is clear.


Alex
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads

[Sick Monkey]

I think that I found a solution to my thread issues, however I know it is
not the most efficient method possible.
Just to give you a little information on what this project is all about
  I have 3 lists of email addresses.
 (1)  host email address =  contains a list of all of my emails
address (around 150,000 users)
 (2)  email addresses  =  contains a list of email addresses that I
have to match with the file host email address.  If there are any matches,
then I have to print them out to a file. (this could be up to 8 million
users)
 (3)  domain addresses =  contains a list of email domains that I
have to match with the host email address file.  If there are any matched,
then I have to print them out to  a file. (could be 2000 or more domains)

 When running the application, you will have the host email address and
can have either one or both of the other files running at the same time.

My problem was that when the application ran, it appeared to stall.  I
decided to use threads for (1) the processing of data and (2) the progress
bar.  The solution I found that enabled the two threads to communicate was
the use of global variables.

I know this is this is not the most efficient method but, using this
solution, I do not see the stalling issue that I found before (which is a
good thing).  I am still not happy with it, because I know it is not
efficient, but I found this to be the best solution for my needs.

Thoughts?

The code is below.  Before you see the code, I must thank everyone who
helped me with this project (including the open source coders).
===
#! /usr/bin/env python
import difflib, sys, thread, re, os, time import Tkinter from Tkinter import
* from sets import Set import tkFileDialog, tkMessageBox  from tkFileDialog
import *
listName = ['','','']
threadStat = 0
mailsrch = re.compile(r'[EMAIL PROTECTED],4}')
domsrch = re.compile(r@(\S+))
statusVar = 0.0  # for the progress bar
startProgress = 0

class Meter(Tkinter.Frame):
   def __init__(self, master, width=300, height=20, bg='black',
fillcolor='cyan',\
value=0.0, text=None, font=None, textcolor='white', *args,
**kw):
   Tkinter.Frame.__init__(self, master, bg=bg, width=width,
height=height, *args, **kw)
   self._value = value

   self._canv = Tkinter.Canvas(self, bg=self['bg'],
width=self['width'], height=self['height'],\
   highlightthickness=0, relief='flat',
bd=0)
   self._canv.pack(fill='both', expand=1)
   self._rect = self._canv.create_rectangle(0, 0, 0,
self._canv.winfo_reqheight(), fill=fillcolor,\
width=0)
   self._text = self._canv.create_text(self._canv.winfo_reqwidth()/2,
self._canv.winfo_reqheight()/2,\
   text='', fill=textcolor)
   if font:
   self._canv.itemconfigure(self._text, font=font)

   self.set(value, text)
   self.bind('Configure', self._update_coords)

   def _update_coords(self, event):
   '''Updates the position of the text and rectangle inside the canvas
when the size of
   the widget gets changed.'''
   self._canv.update_idletasks()
   self._canv.coords(self._text, self._canv.winfo_width()/2,
self._canv.winfo_height()/2)
   self._canv.coords(self._rect, 0, 0,
self._canv.winfo_width()*self._value, self._canv.winfo_height())
   self._canv.update_idletasks()

   def get(self):
   return self._value, self._canv.itemcget(self._text, 'text')

   def set(self, value=0.0, text=None):
   #make the value failsafe:
   if value  0.0:
   value = 0.0
   elif value  1.0:
   value = 1.0
   self._value = value
   if text == None:
   #if no text is specified use the default percentage string:
   text = str(int(round(100 * value))) + ' %'
   self._canv.coords(self._rect, 0, 0, self._canv.winfo_width()*value,
self._canv.winfo_height())
   self._canv.itemconfigure(self._text, text=text)
   self._canv.update_idletasks()


##
def fail(msg):
   out = sys.stderr.write
   out(msg + \n\n)
   out(__doc__)
   return 0

def fopen(fname):
   try:
   return open(fname, 'U')
   except IOError, detail:
   return fail(couldn't open  + fname + :  + str(detail))

def fetchFiles(file1,file2,file3): #file1: host list file2 = email list;
file3=domain; method=method = ''
  print file1
  print file2
  print file3
  f1 = fopen(file1)
  a = f1.readlines(); f1.close()
  d1 = {}
  for c in a:
 for m in mailsrch.findall(c):
d1[m.lower()] = None
  print starting list 2
  thread.start_new_thread(showProcessing, ())
  #DOMAIN COMPARISONif file2 == '':
 domain(d1,file3)
  #EMAIL COMPARISON 

Re: Python Threads

[Sick Monkey]

Great, thanks for the tip Gabriel!

On 2/18/07, Gabriel Genellina [EMAIL PROTECTED] wrote:


En Sun, 18 Feb 2007 23:37:02 -0300, Sick Monkey [EMAIL PROTECTED]
escribió:

 Well if this cannot be done, can a thread call a function in the main
 method?
 I have been trying and have not been successive.  Perhaps I am using
 thread
 incorrectly.

The safe way to pass information between threads is to use Queue. From
inside the working thread, you put() an item with enough state
information. On the main (GUI) thread, you use after() to check for any
data in the queue, and then update the interfase accordingly.
I think there is a recipe in the Python Cookbook
http://aspn.activestate.com/ASPN/Cookbook/Python

--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

Python Threads

[Sick Monkey]

Is there anyway to get 2 python threads to talk to one another?

I have a GUI which is spawning a thread to make a large calculation (that
way the GUI does not appear to be non responsive).  I am trying to attach a
progress bar to the threaded action.  As the thread is calculating data, I
would like it to communicate with the other (progress) thread to update it.

On windows, when I spawn the progress bar without using threads, the
progress bar stalls  this does not happen on linux.  So I thought by
spawning the progress bar by using a thread, it would not stall...  If
anyone can think of another technique I am all ears.

~~START SAMPLE
def progressBar(status):
   mroot = Tkinter.Tk(className='Worker Bee')
   metric = Meter(mroot, relief='ridge', bd=3)
   metric.pack(fill='x')
   metric.set(status, 'Starting ...')

def fetchFiles(file1,file2,file3):
  method = ''
  print file1
  print file2
  print file3
  f1 = fopen(file1)
  a = f1.readlines(); f1.close()
  d1 = {}
  for c in a:
 for m in mailsrch.findall(c):
d1[m.lower()] = None

  I Would like to  Update the progress bar running in the other thread
here.
  ## set status = .33 and update progress bar.
  if file2 == '':
 domain(d1,file3)
#...

def startProc():
 status = 0
 thread.start_new_thread(fetchFiles, (f1name,f2name,f3name,))
 thread.start_new_thread(progressBar, (status,))

~~END SAMPLE
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads

[Sick Monkey]

Well if this cannot be done, can a thread call a function in the main
method?
I have been trying and have not been successive.  Perhaps I am using thread
incorrectly.

On 2/18/07, Sick Monkey [EMAIL PROTECTED] wrote:


Is there anyway to get 2 python threads to talk to one another?

I have a GUI which is spawning a thread to make a large calculation (that
way the GUI does not appear to be non responsive).  I am trying to attach a
progress bar to the threaded action.  As the thread is calculating data, I
would like it to communicate with the other (progress) thread to update it.

On windows, when I spawn the progress bar without using threads, the
progress bar stalls  this does not happen on linux.  So I thought by
spawning the progress bar by using a thread, it would not stall...  If
anyone can think of another technique I am all ears.

~~START SAMPLE
def progressBar(status):
mroot = Tkinter.Tk(className='Worker Bee')
metric = Meter(mroot, relief='ridge', bd=3)
metric.pack(fill='x')
metric.set(status, 'Starting ...')

def fetchFiles(file1,file2,file3):
   method = ''
   print file1
   print file2
   print file3
   f1 = fopen(file1)
   a = f1.readlines(); f1.close()
   d1 = {}
   for c in a:
  for m in mailsrch.findall(c):
 d1[m.lower()] = None

   I Would like to  Update the progress bar running in the other
thread here.
   ## set status = .33 and update progress bar.
   if file2 == '':
  domain(d1,file3)
#...

def startProc():
  status = 0
  thread.start_new_thread(fetchFiles, (f1name,f2name,f3name,))
  thread.start_new_thread(progressBar, (status,))

~~END SAMPLE

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Threads

[Gabriel Genellina]

En Sun, 18 Feb 2007 23:37:02 -0300, Sick Monkey [EMAIL PROTECTED]  
escribió:

 Well if this cannot be done, can a thread call a function in the main
 method?
 I have been trying and have not been successive.  Perhaps I am using  
 thread
 incorrectly.

The safe way to pass information between threads is to use Queue. From  
inside the working thread, you put() an item with enough state  
information. On the main (GUI) thread, you use after() to check for any  
data in the queue, and then update the interfase accordingly.
I think there is a recipe in the Python Cookbook  
http://aspn.activestate.com/ASPN/Cookbook/Python

-- 
Gabriel Genellina

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The reliability of python threads

[Steve Holden]

Carl J. Van Arsdall wrote:
 Steve Holden wrote:
 [snip]

 Are you using memory with built-in error detection and correction?

   
 You mean in the hardware?  I'm not really sure, I'd assume so but is 
 there any way I can check on this?  If the hardware isn't doing that, is 
 there anything I can do with my software to offer more stability?
 
You might be able to check using the OS features (have you said what OS 
you are using?) - alternatively Google for information from the system 
supplier.

If you don't have that feature in hardware you are up sh*t creek without 
a paddle, as it can't be emulated.

regards
  Steve
-- 
Steve Holden   +44 150 684 7255  +1 800 494 3119
Holden Web LLC/Ltd  http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Blog of Note:  http://holdenweb.blogspot.com
See you at PyCon? http://us.pycon.org/TX2007

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The reliability of python threads

[John Nagle]

Aahz wrote:
 In article [EMAIL PROTECTED],
 Carl J. Van Arsdall [EMAIL PROTECTED] wrote:
 My point is that an app that dies only once every few months under load
 is actually pretty damn stable!  That is not the kind of problem that
 you are likely to stimulate.

 This has all been so vague.  How does it die?

 It would be useful if Python detected obvious deadlock.  If all threads
are blocked on mutexes, you're stuck, and at that point, it's time
to abort and do tracebacks on all threads.   You shouldn't have to
run under a debugger to detect that.

 Then a timer, so that if the Global Python Lock
stays locked for more than N seconds, you get an abort and a traceback.
That way, if you get stuck in some C library, it gets noticed.

 Those would be some good basic facilities to have in thread support.

 In real-time work, you usually have a high-priority thread which
wakes up periodically and checks that a few flags have been set
indicating progress of the real time work, then clears the flags.
Throughout the real time code, flags are set indicating progress
for the checking thread to notice.  All serious real time systems
have some form of stall timer like that; there's often a stall
timer in hardware.

John Nagle
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The reliability of python threads

[Carl J. Van Arsdall]

Steve Holden wrote:
 [snip]

 Are you using memory with built-in error detection and correction?

   
You mean in the hardware?  I'm not really sure, I'd assume so but is 
there any way I can check on this?  If the hardware isn't doing that, is 
there anything I can do with my software to offer more stability?





-- 

Carl J. Van Arsdall
[EMAIL PROTECTED]
Build and Release
MontaVista Software

-- 
http://mail.python.org/mailman/listinfo/python-list