Re: Multi-Threading and KeyboardInterrupt

[Matt]

I'm going to use the multipocessing library from here forward so I can take advantage of multiple cores and clusters. Either one should work for my use, since in my non-demonstration code each thread spends most of it's time waiting for a separate non-Python subprocess (created with subprocess.Popen) to finish anyway. (I guess Python would see this as IO-blocking) Therefore, if you can fix my toy example with threading, that's fine.

~Matt

On Jun 15, 2009, at 11:53 AM, Mike Kazantsev wrote:

DB.py, followed by a KeyboardInterrupt yields the output in a.out. I want roughly the output in desired.out. What do I need to do to modify this code to get my desired output and corresponding functionality? It would be a shame if this wasn't possible in any pure-Python way.

On Mon, 15 Jun 2009 05:37:14 -0700 (PDT)
OdarR <olivier.da...@gmail.com> wrote:

On 13 juin, 07:25, Mike Kazantsev <mk.frag...@gmail.com> wrote:

There was quite interesting explaination of what happens when you send

^C with threads, posted on concurrency-sig list recently:

 http://blip.tv/file/2232410
 http://www.dabeaz.com/python/GIL.pdf

Can be quite shocking, but my experience w/ threads only confirms that.

Hi there,
please read this package page (in 2.6), this is very interesting.
http://docs.python.org/library/multiprocessing.html

I tested it : it works. Multi-core cpu's are happy :-)

I'd certainly prefer using processes because they indeed work
flawlessly in that respect, but threads are way simplier and much more
integrated into the language, so I can avoid re-imlementing tons of
shared stuff, IPC and locking by using threads which bassically run in
the same context.
That might mean 90% of code for trivial but parallel task.

Alas, they don't work flawlessly in all cases, but there is still
million and one use for them.

--
Mike Kazantsev // fraggod.net
--
http://mail.python.org/mailman/listinfo/python-list

I'm going to use the multipocessing library from here forward so I can take advantage of multiple cores and clusters. Either one should work for my use, since in my non-demonstration code each thread spends most of it's time waiting for a separate non-Python subprocess (created with subprocess.Popen) to finish anyway. (I guess Python would see this as IO-blocking) DB.py, followed by a KeyboardInterrupt yields the output in a.out. I want roughly the output in desired.out.

~Matt

On Jun 15, 2009, at 11:53 AM, Mike Kazantsev wrote:

What do I need to do to modify this code to get my desired output and corresponding functionality? It would be a shame if this wasn't possible in any pure-Python way.

On Mon, 15 Jun 2009 05:37:14 -0700 (PDT)
OdarR <olivier.da...@gmail.com> wrote:

On 13 juin, 07:25, Mike Kazantsev <mk.frag...@gmail.com> wrote:

There was quite interesting explaination of what happens when you send

^C with threads, posted on concurrency-sig list recently:

http://blip.tv/file/2232410
http://www.dabeaz.com/python/GIL.pdf

Can be quite shocking, but my experience w/ threads only confirms that.

Hi there,
please read this package page (in 2.6), this is very interesting.
http://docs.python.org/library/multiprocessing.html

I tested it : it works. Multi-core cpu's are happy :-)

I'd certainly prefer using processes because they indeed work
flawlessly in that respect, but threads are way simplier and much more
integrated into the language, so I can avoid re-imlementing tons of
shared stuff, IPC and locking by using threads which bassically run in
the same context.
That might mean 90% of code for trivial but parallel task.

Alas, they don't work flawlessly in all cases, but there is still
million and one use for them.

--
Mike Kazantsev // fraggod.net
--
http://mail.python.org/mailman/listinfo/python-list

a.out
Description: Binary data

desired.out
Description: Binary data

#!/usr/bin/env python

import sys, time

try:
	while True:
		pass
except KeyboardInterrupt:
	print '********* keyboard int %s ********' % sys.argv[1]
	raise
#!/usr/bin/env python

from subprocess			import Popen
from multiprocessing	import Process, Semaphore

MAX_SUBPROCS	= 3
RUN_PERMISSION	= Semaphore(MAX_SUBPROCS)

def runSingle(i):
	with RUN_PERMISSION:
		Popen(['./Sub.py', str(i)]).wait()
		
def runAll():
	workers = [ Process(target = runSingle, name = str(i), args = [i])
			for i in xrange(MAX_SUBPROCS + 1) ]
	try:
		for w in workers:
			w.start()
		for w in workers:
			w.join()
	except KeyboardInterrupt:
		## I want this to be all that is shown on a KeyboardInterrupt
		print '********* stopped midway ********'
			
runAll()
-- 
http://mail.python.org/mailman/listinfo/python-list