[issue22013] Add at least minimal support for thread groups

[Irit Katriel]

Irit Katriel  added the comment:

And now we have TaskGroups in asyncio as well.

I'm closing this as it seems to have been abandoned. Feel free to reopen if 
this is still needed.

--
nosy: +iritkatriel
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Tshepang Lekhonkhobe]

Changes by Tshepang Lekhonkhobe :


--
nosy: +tshepang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Charles-François Natali]

Charles-François Natali added the comment:

> Tin Tvrtković added the comment:
>
> For your examples, my first instinct would be to use a thread pool executor. 
> It's a nice high level API and can already do the aggregate join.

Indeed, the examples posted don't make much sense: thread/process
pools are the way to go.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Claudiu Popa]

Claudiu Popa added the comment:

This seems indeed like a weaker version of ThreadPoolExecutor. Here's how your 
example looks with it, not very different and still easy to understand and 
grasp:


from concurrent.futures import ThreadPoolExecutor
from urllib.request import urlretrieve

with ThreadPoolExecutor(max_workers=3) as executor:
url = 'http://www.{site}.org/'
for site in ('perl', 'python', 'jython', 'pypy'):
future = executor.submit(urlretrieve, url.format(site=site), site)



3 lines without imports and the initialisation of the pool.

--
nosy: +Claudiu.Popa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Tin Tvrtković]

Tin Tvrtković added the comment:

For your examples, my first instinct would be to use a thread pool executor. 
It's a nice high level API and can already do the aggregate join.

--
nosy: +tinchester

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Raymond Hettinger added the comment:

> Even the Java API doesn't look very useful.

This isn't a Java-only concept.  It is widely available.  Here's a sample:

c++
"thread_group provides for a collection of threads that are related in some 
fashion."
http://www.boost.org/doc/libs/1_35_0/doc/html/thread/thread_management.html#thread.thread_management.threadgroup

perl
"Thread::Pool - group of threads for performing similar jobs"
http://search.cpan.org/dist/Thread-Pool/lib/Thread/Pool.pm

haskell
"This module extends Control.Concurrent.Thread with the ability to wait for a 
group of threads to terminate."
http://hackage.haskell.org/package/threads-0.5.0.2/docs/Control-Concurrent-Thread-Group.html

ruby
"ThreadGroup provides a means of keeping track of a number of threads as a 
group."
http://www.ruby-doc.org/core-2.1.2/ThreadGroup.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
Removed message: http://bugs.python.org/msg223544

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Raymond Hettinger added the comment:

c++
"thread_group provides for a collection of threads that are related in some 
fashion."
http://www.boost.org/doc/libs/1_35_0/doc/html/thread/thread_management.html#thread.thread_management.threadgroup

perl
"Thread::Pool - group of threads for performing similar jobs"
http://search.cpan.org/dist/Thread-Pool/lib/Thread/Pool.pm

> Even the Java API doesn't look very useful.

This isn't a Java-only concept.  It is widely available.

haskell
"This module extends Control.Concurrent.Thread with the ability to wait for a 
group of threads to terminate."
http://hackage.haskell.org/package/threads-0.5.0.2/docs/Control-Concurrent-Thread-Group.html

ruby
"ThreadGroup provides a means of keeping track of a number of threads as a 
group."
http://www.ruby-doc.org/core-2.1.2/ThreadGroup.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
nosy: +Christian.Tismer, aleax, kristjan.jonsson, lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Raymond Hettinger added the comment:

> The example looks like something you could use concurrent.futures for

The example was minimal to show how it works.  The concept of having groups of 
related threads is perfectly general.  It is a tool for organizing and 
reasoning about code more complex than this.  

The main virtue is in the aggregate join operation (wait for all workers doing 
a given kind of task to finish).  I believe this is reasonably common (I've 
seen aggregate joins at more than one client). 

The goal is to simplify common patterns of parallel work in phases (each phase 
must be complete before the next phase starts).

Old way:

phase1_workers = []
for data in pool:
t = threading.Thread(target=phase1, args=(data,))
t.start()
phase1_workers.append(t)
for t in phase1_workers:
t.join()

phase2_workers = []
for data in phase1_pool:
t = threading.Thread(target=phase2, args=(data,))
t.start()
phase2_workers.append(t)
for t in phase2_workers:
t.join()

phase3_workers = []
for data in phase2_pool:
t = threading.Thread(target=phase3, args=(data,))
t.start()
phase3_workers.append(t)
for t in phase3_workers:
t.join()

print('Done')

New way with cleaner code:

phase1 = SimpleThreadGroup('phase1')
phase2 = SimpleThreadGroup('phase2')
phase3 = SimpleThreadGroup('phase3')

for data in pool:
t = threading.Thread(phase1, phase1_task, args=(data,)).start()
phase1.join()

for data in phase1_pool:
t = threading.Thread(phase2, phase2_task, args=(data,)).start()
phase2.join()

for data in phase2_pool:
t = threading.Thread(phase3, phase3_task, args=(data,)).start()
phase3.join()

print('Done')

The new code is easier to write, to reason about, and to maintain because the 
thread group takes care of building the aggregate collection and applying the 
aggregate join operation to make sure each phase is complete before going on to 
the next phase (i.e. all sprites moved, all user inputs processed, all monsters 
generated, all conflicts resolved, all points accumulated, all bonuses applied, 
...)

As discussed in 
http://journals.ecs.soton.ac.uk/java/tutorial/java/threads/threadgroup.html , 
the feature would be more useful if we had the ability to suspend, resume, or 
stop collections of threads, but our threading have more limited controls 
(check name, get identifier, check whether the thread is alive, and most 
usefully wait for the thread with a join).

For people who write complex multi-threaded code (i.e. my clients), this would 
offer a nice simplification.  I don't see any reason to leave this feature left 
as a stub in perpetuity.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Changes by Raymond Hettinger :


Removed file: http://bugs.python.org/file35998/parallel_download_application.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Changes by Raymond Hettinger :


Added file: http://bugs.python.org/file36007/parallel_download_application.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[R. David Murray]

R. David Murray added the comment:

The example looks like something you could use concurrent.futures for and get 
more features besides?

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Antoine Pitrou]

Antoine Pitrou added the comment:

I have a hard time understanding what it would bring in Python's context. Even 
the Java API doesn't look very useful.

--
nosy: +neologix, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
keywords: +patch
Added file: http://bugs.python.org/file35999/threadgroup.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

Changes by Raymond Hettinger :


Added file: http://bugs.python.org/file35998/parallel_download_application.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22013] Add at least minimal support for thread groups

[Raymond Hettinger]

New submission from Raymond Hettinger:

Currently, threading.Thread requires that group=None pending implementation of 
a ThreadGroup class such as that described in 
http://www.javaworld.com/article/2074481

This has been an open todo for very long time, possibly because it may be too 
big of a task to implement all the features including subgroups.

I think we can implement a small but useful set of features without too much 
difficultly:

path_explorers = threading.ThreadGroup('path_explorers')
for path in paths:
threading.Thread(path_explorers, explore, (path,))
for thread in path_explorers: # enumerate unfinished explorers
print(thread)
path_explorers.start()# begin parallel search
path_explorers.join() # wait for group to finish
print("Result:", best_path)

--
components: Library (Lib)
messages: 223498
nosy: rhettinger
priority: normal
severity: normal
stage: needs patch
status: open
title: Add at least minimal support for thread groups
type: enhancement
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com