Brian Quinlan added the comment:
I'll try to take a look at this before the end of the week, but I'm currently
swamped with other life stuff :-(
--
___
Python tracker
<https://bugs.python.org/issue39
Brian Quinlan added the comment:
New changeset 884eb89d4a5cc8e023deaa65001dfa74a436694c by Brian Quinlan in
branch 'master':
bpo-39205: Tests that highlight a hang on ProcessPoolExecutor shutdown (#18221)
https://github.com/python/cpython/commit/884eb89d4a5cc8e023deaa65001dfa74a436694c
Change by Brian Quinlan :
--
keywords: +patch
pull_requests: +17601
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/18221
___
Python tracker
<https://bugs.python.org/issu
New submission from Brian Quinlan :
```
from concurrent.futures import ProcessPoolExecutor
import time
t = ProcessPoolExecutor(max_workers=3)
t.map(time.sleep, [1,2,3])
t.shutdown(wait=False)
```
Results in this exception and then a hang (i.e. Python doesn't terminate):
```
Exception
Brian Quinlan added the comment:
Can I add "needs backport to 3.8" and "needs backport to 3.7" labels now or
do I have to use cherry_picker at this point?
On Mon, Jul 1, 2019 at 3:55 PM Ned Deily wrote:
>
> Ned Deily added the comment:
>
> > I
Brian Quinlan added the comment:
I don't know what the backport policy is. The bug is only theoretical AFAIK
i.e. someone noticed it through code observation but it has not appeared in
the wild.
On Mon, Jul 1, 2019 at 3:25 PM Ned Deily wrote:
>
> Ned Deily added the comment:
>
Change by Brian Quinlan :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Brian Quinlan added the comment:
New changeset 242c26f53edb965e9808dd918089e664c0223407 by Brian Quinlan in
branch 'master':
bpo-31783: Fix a race condition creating workers during shutdown (#13171)
https://github.com/python/cpython/commit/242c26f53edb965e9808dd918089e664c0223407
Change by Brian Quinlan :
--
resolution: -> duplicate
stage: -> resolved
status: open -> closed
superseder: -> function changed when pickle bound method object
___
Python tracker
<https://bugs.python
Change by Brian Quinlan :
--
stage: -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue37208>
___
___
Python-bugs-list
Change by Brian Quinlan :
--
resolution: -> duplicate
superseder: -> picke cannot dump Exception subclasses with different super()
args
___
Python tracker
<https://bugs.python.org/i
Change by Brian Quinlan :
--
title: picke cannot dump exceptions subclasses with different super() args ->
picke cannot dump Exception subclasses with different super() args
___
Python tracker
<https://bugs.python.org/issu
New submission from Brian Quinlan :
$ ./python.exe nopickle.py
TypeError: __init__() missing 1 required positional argument: 'num'
The issue is that the arguments passed to Exception.__init__ (via `super()`)
are collected into `args` and then serialized by pickle e.g.
>>> Pickle
Brian Quinlan added the comment:
That's a super interesting bug! It looks like this issue is that your exception
can't be round-tripped using pickle i.e.
>>> class PoolBreaker(Exception):
... def __init__(self, num):
... super().__init__()
... self.
Brian Quinlan added the comment:
Joshua, I'm closing this since I haven't heard from you in a month. Please
re-open if you use case isn't handled by `initializer` and `initargs`.
--
assignee: -> bquinlan
resolution: -> out of date
stage: -> resolved
status: open
Brian Quinlan added the comment:
We can bike shed over the name in the PR ;-)
--
___
Python tracker
<https://bugs.python.org/issue36780>
___
___
Python-bug
Change by Brian Quinlan :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Brian Quinlan added the comment:
My understanding is that tracebacks have a pretty large memory profile so I'd
rather not keep them alive. Correct me if I'm wrong about that.
--
___
Python tracker
<https://bugs.python.org/issue32
Brian Quinlan added the comment:
After playing with it for a while, https://github.com/python/cpython/pull/6375
seems reasonable to me.
It needs tests and some documentation.
Antoine, are you still -1 because of the complexity increase
Brian Quinlan added the comment:
When I first wrote and started using ThreadPoolExecutor, I had a lot of code
like this:
with ThreadPoolExecutor(max_workers=500) as e:
e.map(download, images)
I didn't expect that `images` would be a large list but, if it was, I wanted
all
Brian Quinlan added the comment:
Was this fixed by https://github.com/python/cpython/pull/3895 ?
--
___
Python tracker
<https://bugs.python.org/issue30
Brian Quinlan added the comment:
Brian, I was looking for an example where the current executor isn't sufficient
for testing i.e. a useful test that would be difficult to write with a real
executor but would be easier with a fake.
Maybe you have such an example from your tests
Change by Brian Quinlan :
--
pull_requests: +13087
stage: -> patch review
___
Python tracker
<https://bugs.python.org/issue31783>
___
___
Python-bugs-list mai
Brian Quinlan added the comment:
I think that ProcessPoolExecutor might have a similar race condition - but not
in exactly this code path since it would only be with the queue management
thread (which is only started once).
--
___
Python tracker
Brian Quinlan added the comment:
Great report Steven!
I was able to reproduce this with the attached patch (just adds some sleeps and
prints) and this script:
from threading import current_thread
from concurrent.futures import ThreadPoolExecutor
from time import sleep
pool
Change by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<https://bugs.python.org/issue31783>
___
___
Python-bugs-list mailing list
Unsubscrib
Brian Quinlan added the comment:
How did the experiment go? Are people still interested in this?
--
___
Python tracker
<https://bugs.python.org/issue22
Brian Quinlan added the comment:
Ben, do you still think that your patch is relevant or shall we close this bug?
--
___
Python tracker
<https://bugs.python.org/issue22
Brian Quinlan added the comment:
Do the `initializer` and `initargs` parameters deal with this use case for you?
https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
--
___
Python tracker
<ht
Brian Quinlan added the comment:
If we supported this, aren't we promising that we will always materialize the
iterator passed to map?
I think that we'd need a really strong use-case for this to be worth-while.
--
___
Python tracker
<ht
Brian Quinlan added the comment:
Hey Brian,
I understand the non-determinism. I was wondering if you had a non-theoretical
example i.e. some case where the non-determinism had impacted a real test that
you wrote?
--
___
Python tracker
<ht
Brian Quinlan added the comment:
Hey Hrvoje,
I agree that #1 is the correct approach. `disown` might not be the best name -
maybe `allow_shutdown` or something. But we can bike shed about that later.
Are you interested in writing a patch
Brian Quinlan added the comment:
Hey Ethan, I'm really sorry about dropping the ball on this. I've been burnt
out on Python stuff for the last couple of years.
When we left this, it looked like the -1s were in the majority and no one new
has jumped on to support `filter`.
If you wanted
Brian Quinlan added the comment:
Using a default executor could be dangerous because it could lead to deadlocks.
For example:
mylib.py
def my_func():
tsubmit(...)
tsubmit(...)
tsubmit(somelib.some_func, ...)
somelib.py
--
def some_func():
tsubmit(...) # Potential
Brian Quinlan added the comment:
So you actually use the result of ex.submit i.e. use the resulting future?
If you don't then it might be easier to just create your own thread.
--
___
Python tracker
<https://bugs.python.org/issue22
Brian Quinlan added the comment:
Do you have a example that you could share?
I can't think of any other fakes in the standard library and I'm hesitant to be
the person who adds the first one ;-)
--
___
Python tracker
<https://bugs.python.
Change by Brian Quinlan :
--
stage: -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue26374>
___
___
Python-bugs-list
Brian Quinlan added the comment:
Can we close this bug then?
--
___
Python tracker
<https://bugs.python.org/issue26374>
___
___
Python-bugs-list mailin
Brian Quinlan added the comment:
Hey Brian, why can't you use threads in your unit tests? Are you worried about
non-determinism or resource usage? Could you make a ThreadPoolExecutor with a
single worker?
--
___
Python tracker
<ht
Change by Brian Quinlan :
--
keywords: +patch
pull_requests: +13045
stage: needs patch -> patch review
___
Python tracker
<https://bugs.python.org/issu
Brian Quinlan added the comment:
OK, I completely disagree with my statement:
"""If you added this as an argument to shutdown() then you'd probably also have
to add it as an option to the constructors (for people using Executors as
context managers). But, if
Brian Quinlan added the comment:
BTW, the 61 process limit comes from:
63 - -
--
___
Python tracker
<https://bugs.python.org/issue26903>
___
___
Python-bug
Change by Brian Quinlan :
--
assignee: -> bquinlan
___
Python tracker
<https://bugs.python.org/issue26903>
___
___
Python-bugs-list mailing list
Unsubscrib
Brian Quinlan added the comment:
If no one has short-term plans to improve multiprocessing.connection.wait, then
I'll update the docs to list this limitation, ensure that ProcessPoolExecutor
never defaults to >60 processes on windows and raises a ValueError if the user
explicitly pas
Brian Quinlan added the comment:
>> The current behavior is explicitly documented, so presumably
>> it can't be (easily) changed
And it isn't clear that it would be desirable to change this even if it were
possible - doing structured resource clean-up seems consistent w
Brian Quinlan added the comment:
Actually it immediately converts the iterable into a list. Recall:
def filter(self, fn, iterable, timeout=None):
l = list(iterable) # iterable = list
return (item for (item, keep) in zip(l, self.map(fn, l, timeout)) if keep
Brian Quinlan added the comment:
Ethan: I'm -0 so I'd happily go with the consensus. Does anyone have a strong
opinion?
Ram: What did you mean by Possible issues? Did you mean that to be an example
using the executor.map() technique?
--
___
Python
Brian Quinlan added the comment:
This feature seems unnecessary to me but couldn't the implementation be
simplified to work in terms of map? i.e. (pseudocode):
def filter(self, fn, iterable, timeout=None):
l = list(iterable)
return (item for (item, keep) in zip(l, self.map(fn, l, timeout
Brian Quinlan added the comment:
Committed, thanks for the fix!
--
resolution: - fixed
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21362
Brian Quinlan added the comment:
The patch looks good to me too. I'll commit it soon.
--
assignee: - bquinlan
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21362
Brian Quinlan added the comment:
Thanks very much for the patch Glenn!
--
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20319
Brian Quinlan added the comment:
Oops, no. That was junk due to my sloppiness. I’ll fix it in a minute.
On Jan 31, 2014, at 5:03 PM, STINNER Victor rep...@bugs.python.org wrote:
STINNER Victor added the comment:
New changeset 0bcf23a52d55 by Brian Quinlan in branch 'default':
Issue
Changes by Brian Quinlan br...@sweetapp.com:
--
assignee: - bquinlan
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20319
___
___
Python-bugs-list
Brian Quinlan added the comment:
I'm looking at futures.patch.
I don't understand why these blocks are helpful -_create_and_install_waiters
has two call sites and both (as_completed and wait) call
_create_and_install_waiters from within an _AcquireFutures context manager
Brian Quinlan added the comment:
I'm not currently working on concurrent.futures but I do look at patches and
bug reports. I'll take a look at this and Issue20297 sometime this week.
--
___
Python tracker rep...@bugs.python.org
http
Brian Quinlan added the comment:
Can't you accomplish what you want using add_done_callback?
e.g.
# Pseudocode
class MyExecutor(ThreadPoolExecutor):
def __init__(self):
self._count = 0
def _decrement(self):
with self._some_lock:
self._count -= 1
def submit(self, fn, *args
Brian Quinlan added the comment:
Hi Victor,
I don't understand your problem. Could you be very specific in your description?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
Changes by Brian Quinlan br...@sweetapp.com:
--
resolution: - fixed
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11161
Brian Quinlan added the comment:
Any progress on this or can I close the bug?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13785
Brian Quinlan added the comment:
The only thing that I've used .running() for is for UI reasons. But it is
easier to just iterative over your futures and call a method than to install a
callback and handle the state yourself.
--
nosy: +bquinlan
Brian Quinlan added the comment:
Take a look at http://bugs.python.org/issue11161 and its fix. Do you think that
just saying that ProcessPoolExecutor will not work from the interactive shell
is sufficient?
--
nosy: +bquinlan
___
Python tracker rep
Brian Quinlan added the comment:
OK, working as intended.
--
resolution: - invalid
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7200
Brian Quinlan added the comment:
I think that we are happy to not fix this.
--
resolution: accepted - wont fix
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10918
Brian Quinlan added the comment:
The concurrent.futures stuff looks good to me.
Could you add a comment explaining why the delete is necessary? And, as Antoine
said, the test should be CPython only.
--
___
Python tracker rep...@bugs.python.org
http
Brian Quinlan added the comment:
This has come up before. Did you actually bang into this? Because the fix seems
pretty ugly to me and the work-around (using functools.partial) is pretty easy.
But, if people are actually hitting this, then your is probably the best that
we can do
Changes by Brian Quinlan br...@sweetapp.com:
--
stage: - needs patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13785
___
___
Python-bugs-list
Brian Quinlan added the comment:
I'm closing this since the filer hasn't specified exactly what they want.
--
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8792
Changes by Brian Quinlan br...@sweetapp.com:
--
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11271
___
___
Python-bugs-list
Brian Quinlan br...@sweetapp.com added the comment:
My current thinking is that adding a state property probably isn't worth it
given the fact that you can construct this yourself and the requirement for a
history property is too specialized.
The callback idea seams reasonable and sufficient
Brian Quinlan br...@sweetapp.com added the comment:
I've had people request that they be able control the order of processed work
submissions. So a more general way to solve your problem might be to make the
two executors take an optional Queue argument in their constructors.
You'd have
Brian Quinlan br...@sweetapp.com added the comment:
Hey Nam,
I'm not sure that I understand. You want ThreadPoolExecutor.submit to block if
there are too many work items in the queue? Are you sure that this happens
currently with ProcessPoolExecutor? I can't see why it would
Brian Quinlan br...@sweetapp.com added the comment:
Thanks for the patch!
--
resolution: - fixed
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15015
Brian Quinlan br...@sweetapp.com added the comment:
The queue that you identified i.e.
self._call_queue = multiprocessing.Queue(self._max_workers +
EXTRA_QUEUED_CALLS)
does not get considered during submit() - are you sure that it somehow causes
submit
Changes by Brian Quinlan br...@sweetapp.com:
--
assignee: - bquinlan
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15015
___
___
Python-bugs-list
Brian Quinlan br...@sweetapp.com added the comment:
Could you run just the test_concurrent_futures test, hit ctrl-C at the point
where it hangs, and send the traceback here?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Changes by Brian Quinlan br...@sweetapp.com:
--
assignee: - bquinlan
nosy: +bquinlan
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14119
Brian Quinlan br...@sweetapp.com added the comment:
I'm closing this since tbrink didn't respond to pitrou's comments.
--
resolution: - out of date
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11271
Changes by Brian Quinlan br...@sweetapp.com:
--
nosy: +bquinlan
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14148
___
___
Python-bugs-list
Brian Quinlan br...@sweetapp.com added the comment:
I guess the question is: why do you need to know the state in that form?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13785
Brian Quinlan br...@sweetapp.com added the comment:
Thanks for the patch!
1. The fetching the state feature seems reasonable but I think that explaining
the difference between CANCELLED and CANCELLED_AND_NOTIFIED is going to be
hard. Maybe you could look at how Doc/library
Brian Quinlan br...@sweetapp.com added the comment:
You'll probably get more traction if you file a new bug.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5788
Brian Quinlan br...@sweetapp.com added the comment:
On my crappy computer, ProcessPoolExecutor.map adds 3ms of added execution
time per call using your test framework.
What is your use case where that is too much?
That being said, I don't have any objections to making improvements.
If you
Changes by Brian Quinlan br...@sweetapp.com:
--
resolution: - fixed
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11864
Brian Quinlan br...@sweetapp.com added the comment:
Under what circumstances do we expect a ProcessPoolExecutor child process to be
killed outside of the control of the ProcessPoolExecutor?
If the user kills a child then maybe all we want to do is raise an exception
rather than deadlock
Brian Quinlan br...@sweetapp.com added the comment:
Killed by the user, or by an automatic device (such as the Linux OOM
killer), or crashed.
Crashed would be bad - it would indicate a bug in the
ProcessPoolExecutor code.
If the user kills a child then maybe all we want to do is raise
Brian Quinlan br...@sweetapp.com added the comment:
It looks like the timing is too sensitive.
--
assignee: - bquinlan
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11864
Brian Quinlan br...@sweetapp.com added the comment:
Nice catch. I hadn't noticed that the docs are lying :-)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11777
Changes by Brian Quinlan br...@sweetapp.com:
--
resolution: - fixed
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11777
Brian Quinlan br...@sweetapp.com added the comment:
I think that it surprising behavior, especially considering that asking for the
*first* element in the iterator causes *all* of the futures to be created.
--
___
Python tracker rep
New submission from Brian Quinlan br...@sweetapp.com:
from concurrent import futures
with futures.ThreadPoolExecutor(max_workers=5) as e:
e.map(print, range(10))
# No output
--
assignee: bquinlan
components: Library (Lib)
messages: 133104
nosy: bquinlan
priority: normal
severity
On 3 Apr 2011, at 16:22, geremy condra wrote:
I think we're talking at cross purposes. The point I'm making is that
there are lots of issues where popularity as a third party module
isn't really a viable test for whether a feature is sufficiently
awesome to be in core python. As part of
On 3 Apr 2011, at 15:30, harrismh777 wrote:
Brian Quinlan wrote:
I suspect that this debate is a sink hole that I won't be able to
escape
from alive but...
... live long and prosper my friend.
Something to consider is that OOP philosophy is technically one of
the most aesthetic
I suspect that this debate is a sink hole that I won't be able to
escape from alive but...
On 2 Apr 2011, at 19:29, harrismh777 wrote:
In other words, does the PSF have a responsibility to maintain the
L.sort(cmp= key= reverse=) interface for strictly *philosophical*
principle based on
Brian Quinlan br...@sweetapp.com added the comment:
Filing a new bug might have been a mistake. Once the investigation in issue
10517 isolated the failure as being in a different module, I thought it best to
file a new bug with a minimal repro case.
Fill free to cleanup
Brian Quinlan br...@sweetapp.com added the comment:
This is the expected behavior.
Future.cancel() returns a bool indicating if the future was cancelled or not.
If it returns False then there is no way to stop it.
The behavior that you are seeing for shutdown is documented as:
shutdown(wait
Brian Quinlan br...@sweetapp.com added the comment:
No, I wasn't able to replicate.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10632
Brian Quinlan br...@sweetapp.com added the comment:
Your approach seems workable but your patch allows the interpreter to exit
while work items are still being processed. See the comment at the top of
concurrent/futures/thread.py.
--
___
Python
Brian Quinlan br...@sweetapp.com added the comment:
Sorry, I didn't read an error message very carefully. When I apply your patch I
see:
from concurrent.futures import *
from time import *
t = ThreadPoolExecutor(5)
t.submit(sleep, 100)
Future at 0x8acc94c state=running
ctrl-D
Error
Brian Quinlan br...@sweetapp.com added the comment:
I would suggest that you base your patch on 3.3/default.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11635
Brian Quinlan br...@sweetapp.com added the comment:
ProcessPoolExecutor is not expected to work with any interactive shell on
Windows because it uses the multiprocessing module behind the scenes, which has
that limitation.
I'm reclassifying this as a documentation bug since either a reference
1 - 100 of 261 matches
Mail list logo
