[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Thank you Antoine!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Change by Antoine Pitrou :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset 0a2ff23fe6efb1653d655ac19d0a4e1629fd8d95 by Antoine Pitrou in 
branch 'master':
Silence error output in test_concurrent_futures (bpo-21423) (#4347)
https://github.com/python/cpython/commit/0a2ff23fe6efb1653d655ac19d0a4e1629fd8d95


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Change by Antoine Pitrou :


--
pull_requests: +4304
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Serhiy, I think I have found a way to deal with the logging output:
https://github.com/python/cpython/pull/4347

--
stage: patch review -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I'm not well experienced with logging, but if we can change the logging in 
subprocesses, I think we could change it to not output messages at all. It 
would be better to save logging messages in subprocesses and check that 
expected logging messages are emitted in the main process. There is 
assertLogs(), but it works only with logging in the same process.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Vinay Sajip]

Vinay Sajip  added the comment:

> What is the best way to silence logging in subprocesses?

Are you referring to the output shown in msg305601? If it's caused by logging 
statements, the best way would be either to pipe stderr to /dev/null or to 
change the logging to use sys.stdout (sys.stderr is just the default) and then 
pipe stdout to /dev/null.

I hope I haven't misunderstood your question, but I fear I may have.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

What is the best way to silence logging in subprocesses?

--
nosy: +vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Unfortunately, there is no obvious way to capture the output of the child 
process here, short of running the entire test under a subprocess.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

And please don't forget to edit a commit message when merge a PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

test_concurrent_futures now produces too much output on stderr.

$ ./python -m test test_concurrent_futures >/dev/null
Exception in initializer:
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/concurrent/futures/process.py", line 170, 
in _process_worker
initializer(*initargs)
  File "/home/serhiy/py/cpython/Lib/test/test_concurrent_futures.py", line 66, 
in init_fail
raise ValueError('error in initializer')
ValueError: error in initializer
Exception in initializer:
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/concurrent/futures/process.py", line 170, 
in _process_worker
initializer(*initargs)
  File "/home/serhiy/py/cpython/Lib/test/test_concurrent_futures.py", line 66, 
in init_fail
raise ValueError('error in initializer')
ValueError: error in initializer
...

What is worse, this output looks as errors report.

--
nosy: +serhiy.storchaka
resolution: fixed -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Change by Antoine Pitrou :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset 63ff4131af86e8a48cbedb9fbba95bd65ca90061 by Antoine Pitrou in 
branch 'master':
bpo-21423: Add an initializer argument to {Process,Thread}PoolExecutor (#4241)
https://github.com/python/cpython/commit/63ff4131af86e8a48cbedb9fbba95bd65ca90061


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Antoine Pitrou  added the comment:

By the way, the submitted PR follows Dan's argument about the initializer's 
argument: the actual call is `initializer(*initargs)`. If someone wants to know 
about the current thread or process, it's trivial to call 
`thread.current_thread()` or `multiprocessing.current_process()` (and I don't 
it's a bad idiom in itself :-)).

If you want to comment on the PR, I would recommend doing it quick, as I plan 
to merge in a day or two :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I've opened a PR for this at https://github.com/python/cpython/pull/4241, 
loosely based on Dan's last patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antony Lee]

Change by Antony Lee :


--
nosy:  -Antony.Lee

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Change by Antoine Pitrou :


--
pull_requests: +4206

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Change by Antoine Pitrou :


--
versions: +Python 3.7 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antony Lee]

Antony Lee added the comment:

For cross-referencing purposes: I have proposed in 
http://bugs.python.org/issue25293 to allow passing a Thread/Process subclass as 
argument instead of an initializer function, which would both handle Mark 
Dickinson's comment (http://bugs.python.org/issue21423#msg218040) about passing 
the thread object as argument, and also allow for finalization.

--
nosy: +Antony.Lee

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Pedro Algarvio]

Pedro Algarvio added the comment:

Would also love to see this in the stdlib soon. My use case is logging 
setup(forward logs to the main process).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Pedro Algarvio]

Changes by Pedro Algarvio :


--
nosy: +Pedro.Algarvio

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[John O'Connor]

Changes by John O'Connor :


--
nosy: +jcon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Alexander Mohr]

Alexander Mohr added the comment:

any chance if this getting into 3.5.2? I have some gross code to get around it 
(setting global properties)

--
nosy: +thehesiod

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Shiming He]

Changes by Shiming He heshim...@gmail.com:


--
nosy: +Shiming He

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Claudiu Popa]

Changes by Claudiu Popa pcmantic...@gmail.com:


--
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Claudiu Popa]

Changes by Claudiu Popa pcmantic...@gmail.com:


--
nosy: +Claudiu.Popa

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Dan O'Reilly]

Changes by Dan O'Reilly oreil...@gmail.com:


--
type:  - enhancement

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Mark Dickinson]

Mark Dickinson added the comment:

 I'm also not sure how useful it would be for ProcessPoolExecutor to receive 
 an instance of itself.

Agreed that this doesn't make much sense; I hadn't really thought about the 
ProcessPoolExecutor case.  I withdraw my comments!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Nicholas Chammas]

Changes by Nicholas Chammas nicholas.cham...@gmail.com:


--
nosy: +Nicholas Chammas

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Dan O'Reilly]

Changes by Dan O'Reilly oreil...@gmail.com:


--
title: concurrent.futures.ThreadPoolExecutor should accept an initializer 
argument - concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should 
accept an initializer argument

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Dan O'Reilly]

Dan O'Reilly added the comment:

Here's an updated patch. Changes:

* Fixed what appears to have been a find/replace typo I made prior to uploading 
the previous patch.

* The tracebacks from the negative unit tests are now suppressed. 

* Fixed a race condition in the initializer failure handling for 
ThreadPoolExecutor. Futures that were submitted before the initializer actually 
failed will now raise a RuntimeError indicating that initializer failed.

* Suppressed an occasional queue.Full exception that would pop up while 
shutting down a ProcessPoolExecutor that was broken due to an initializer fail. 
As best as I can tell the exception is harmless, so suppressing it should be ok.

*Updated the docs.

--
nosy: +pitrou
Added file: http://bugs.python.org/file36335/init_patch_updated.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Antoine Pitrou]

Antoine Pitrou added the comment:

It seems the addition of the initargs argument doesn't tackle Mark's objection 
here:

 the initialiser and uninitialiser for the EnhancedThreadPoolExecutor
 accept no arguments. In retrospect, it would have been better to have 
 them take the thread itself as a single argument.

Regardless, I'm going to review the patch soon.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Dan O'Reilly]

Dan O'Reilly added the comment:

 It seems the addition of the initargs argument doesn't tackle Mark's 
 objection here:

  the initialiser and uninitialiser for the EnhancedThreadPoolExecutor
  accept no arguments. In retrospect, it would have been better to have 
  them take the thread itself as a single argument.

This would be inconsistent with multiprocessing.Pool's initializer/initargs 
behavior. I'm not sure if consistency is important there or not, but its worth 
pointing out.

I'm also not sure how useful it would be for ProcessPoolExecutor to receive an 
instance of itself. With multiprocessing.Pool,  initializer is a commonly used 
to pass objects that can only be inherited to workers (Queues, Locks, etc.). 
The same pattern would be useful for ProcessPoolExecutor, which means 
initializer needs to be able to take an arbitrary number of arguments, rather 
than just an instance to the Process object itself.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21423] concurrent.futures.ThreadPoolExecutor/ProcessPoolExecutor should accept an initializer argument

[Dan O'Reilly]

Dan O'Reilly added the comment:

Another updated patch. This one changes ProcessPoolExecutor behavior so that 
RuntimeErrors are raised in any active Futures, and on subsequent calls to 
submit after the initializer fails. This makes its behavior consistent with 
ThreadPoolExecutor.

--
Added file: http://bugs.python.org/file36339/init_patch_updated2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21423
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com