Richard Oudkerk added the comment:
So there are really two situations:
1) The __main__ module *should not* be imported. This is the case if you use
__main__.py in a package or if you use nose to call test_main().
This should really be detected in get_preparation_data() in the parent process
Richard Oudkerk added the comment:
> I appear to be somehow getting child processes where __main__.__file__ is
> set, but __main__.__spec__ is not.
That seems to be true for the __main__ module even when multiprocessing is not
involved. Running a file /tmp/foo.py containing
impo
Richard Oudkerk added the comment:
Thanks for your hard work Nick!
--
___
Python tracker
<http://bugs.python.org/issue19946>
___
___
Python-bugs-list mailin
Richard Oudkerk added the comment:
On 19/12/2013 10:00 pm, Nick Coghlan wrote:
> I think that needs to be fixed on the multiprocessing side rather than just
> in the tests - we shouldn't create a concrete context for a start method
> that isn't going to work on that platform
Richard Oudkerk added the comment:
How often has this happened?
If the machine was very loaded then maybe the timeout was not enough time for
the semaphore to be cleaned up by the tracker process. But I would expect 1
second to be more than ample
Richard Oudkerk added the comment:
It is probably harmless then.
I don't think increasing the timeout is necessary -- the multiprocessing tests
already take a long time.
--
___
Python tracker
<http://bugs.python.org/is
Richard Oudkerk added the comment:
The following from the docs is wrong:
> ... module globals are no longer forced to None during interpreter
> shutdown.
Actually, in 3.4 module globals *sometimes* get forced to None during
interpreter shutdown, so the version the __del__ method can
Richard Oudkerk added the comment:
_overlapped is linked against the socket library whereas _winapi is not so
it can be bundled in with python3.dll.
I did intend to switch multiprocessing over to using _overlapped but I did
not get round to it.
Since this is a private module the names of
Richard Oudkerk added the comment:
This is expected. Killing processes which use shared locks is never going to
end well. Even without the lock deadlock, the data in the pipe would be liable
to be corrupted if a processes is killed while putting or getting from a queue.
If you want to be
Richard Oudkerk added the comment:
LGTM
--
___
Python tracker
<http://bugs.python.org/issue20540>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
BTW, I see little difference between 3.2 and the unpatched default branch on
MacOSX:
$ py-32/release/python.exe ~/Downloads/test_manager.py
0.0007331371307373047
8.20159912109375e-05
9.417533874511719e-05
8.082389831542969e-05
7.796287536621094e-05
Richard Oudkerk added the comment:
On Unix, using the fork start method (which was the only option till 3.4),
every sub process will incref every shared object for which its parent has a
reference.
This is deliberate because there is not really any way to know which shared
objects a
Richard Oudkerk added the comment:
> Thanks Richard. The set_start_method() call will affect any process
> started from that time on? Is it possible to change idea at some point in
> the future?
You can use different start methods in the same program by creating different
Changes by Richard Oudkerk :
--
assignee: -> sbt
___
Python tracker
<http://bugs.python.org/issue7503>
___
___
Python-bugs-list mailing list
Unsubscrib
Changes by Richard Oudkerk :
--
assignee: -> sbt
___
Python tracker
<http://bugs.python.org/issue20633>
___
___
Python-bugs-list mailing list
Unsubscrib
Richard Oudkerk added the comment:
I am not sure method_to_typeid and create_method were really intended to be
public -- they are only used by Pool proxies.
You can maybe work around the problem by registering a second typeid without
specifying callable. That can be used in method_to_typeid
Richard Oudkerk added the comment:
> Perhaps NEWS item needed for this change.
Done.
--
___
Python tracker
<http://bugs.python.org/issue16743>
___
___
Python-
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: patch review -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
> Richard, do you still want to push this forward? Otherwise I'd like to
> finalize the patch (in the other sense ;-).
I started to worry a bit about daemon threads. I think they can still run
while atexit functions are being run.* So if a da
Richard Oudkerk added the comment:
On 14/02/2013 3:16pm, Antoine Pitrou wrote:
> Mmmh... thread switching is already blocked at shutdown:
> http://hg.python.org/cpython/file/0f827775f7b7/Python/ceval.c#l434
But in Py_Finalize(), call_py_exitfuncs() is called *before* _Py_Finalizing is
se
Changes by Richard Oudkerk :
--
status: pending -> closed
___
Python tracker
<http://bugs.python.org/issue16246>
___
___
Python-bugs-list mailing list
Unsubscri
Richard Oudkerk added the comment:
> In any case, I think it's just something we'll have to live with. Daemon
> threads are not a terrific idea in general.
I agree.
--
___
Python tracker
<http://bugs.py
Richard Oudkerk added the comment:
I did not realize there was a 'Extension Modules' section. I have been putting
changes to C extensions in the 'Library' section instead. It looks like most
people do the same as me.
--
nosy: +sbt
__
Richard Oudkerk added the comment:
> Was not it be yanked in 1fabff717ef4?
Looks like it was reintroduced by this merge changeset:
http://hg.python.org/cpython/rev/30fc620e240e
--
___
Python tracker
<http://bugs.python.org/issu
Richard Oudkerk added the comment:
> Good, except that you have to add a gc.collect() call for the
> non-refcounted implementations.
Better to use test.support.gc_collect().
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/i
Richard Oudkerk added the comment:
Banning md5 as a matter of policy may be perfectly sensible.
However, I think the way multiprocessing uses hmac authentication is *not*
affected by the collision attacks the advisory talks about. These depend on
the attacker being able to determine for
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17261>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
A pool should only be used by the process that created it (unless you use a
managed pool).
If you are creating long lived processes then you could create a new pool on
demand. For example (untested)
pool_pid = (None, None)
def get_pool
Richard Oudkerk added the comment:
> Richard, are you suggesting that we close this, or do you see an
> actionable issue? (a plausible patch to the repository?)
I skimmed the documentation and could not see that this restriction has been
documented.
So I think a documentation patch wo
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: commit review -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
The new test seems to be reliably failing on Windows:
==
FAIL: test_issue17223 (__main__.UnicodeTest)
--
Traceback (most
Richard Oudkerk added the comment:
I think this change will potentially make the main module get imported twice
under different names when we transfer pickled data between processes. The
current code (which is rather a mess) goes out of its way to avoid that.
Basically the main process makes
Richard Oudkerk added the comment:
It looks like queues_contention.diff has the line
obj = pickle.dumps(obj)
in both _feed() and put(). Might that be why the third set of benchmarks was
slower than the second?
--
___
Python tracker
<h
Richard Oudkerk added the comment:
On 04/03/2013 8:01pm, Charles-François Natali wrote:
>> It looks like queues_contention.diff has the line
>>
>> obj = pickle.dumps(obj)
>>
>> in both _feed() and put(). Might that be why the third set of benchmarks
>> w
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17364>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
The change in your patch is in a Windows-only section -- a few lines before the
chunk you can see _winapi.GetExitCodeProcess().
Since read() on Windows never fails with EINTR there is no need for
_eintr_retry_call().
If you are using Linux then there must
Richard Oudkerk added the comment:
BTW, on threads are only used on Windows. On Unix select() or poll() is used.
--
___
Python tracker
<http://bugs.python.org/issue17
Richard Oudkerk added the comment:
I will close the issue then.
If you track the problem down to a bug in Python then you can open a new one.
--
resolution: -> invalid
stage: -> committed/rejected
status: open -> closed
___
Python track
Richard Oudkerk added the comment:
+1
To use Tools/builbot/*.bat doesn't the current directory have to be the main
directory of the repository? Then I see no point in the "-C" argument: just
set the correct directory automatically.
I think make.bat should also support creati
Richard Oudkerk added the comment:
> What does running 'kill-python before re-building python do? I have not
> seen it mentioned in the in the devguide or pcbuild/readme.
It kills any currently running python(_d).exe processes. This is because
Windows does not allow program or li
Richard Oudkerk added the comment:
LGTM (although the warning is actually harmless).
--
___
Python tracker
<http://bugs.python.org/issue17395>
___
___
Python-bug
Richard Oudkerk added the comment:
> Which does give me a thought - perhaps lru_cache in 3.4 could accept a
> "key" argument that is called as "key(*args, **kwds)" to derive the cache
> key? (that would be a separate issue, of course)
Agreed. I suggeste
Richard Oudkerk added the comment:
> Why > 1? This should be commented.
The manager process will always be included in active_children().
--
___
Python tracker
<http://bugs.python.org/i
Richard Oudkerk added the comment:
Does this happen every time you run the tests? (I don't see these errors.)
--
___
Python tracker
<http://bugs.python.org/is
Richard Oudkerk added the comment:
Could you try the following program:
import socket
import multiprocessing
import multiprocessing.reduction
import multiprocessing.connection
def socketpair():
with socket.socket() as l:
l.bind(('localhost', 0))
l.listen(1)
Richard Oudkerk added the comment:
Now could you try the attached file? (It will not work on 2.7 because a
missing socket.fromfd().)
P.S. It looks like the error for 3.3 is associated with a file
f:\python\mypy\traceback.py which presumably clashes with the one in the
standard library
Richard Oudkerk added the comment:
> Both 3.2 and 3.3 give essentially the same traceback as 3.2 did before,
> both with installed python and yesterdays debug builds.
It looks like on your machine socket handles are not correctly inherited by
child processes -- I had assumed that they
Richard Oudkerk added the comment:
> My original report was for 32 bit debug build on 64 bit Win 7 machine.
> I just re-ran test_multiprocessing with installed 64 bit python with same
> result. Was "I don't see these errors." on different Windows or non-Windows.
On 64
Richard Oudkerk added the comment:
Sorry, I was not very clear.
If you use the O_TEMPORARY flag with open() to get a file handle, then the
share mode used with the underlying CreateFile() function is
X = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE
whereas, if you don'
Richard Oudkerk added the comment:
Actually, it is not quite the same semantics as Unix.
After you delete the the file you cannot create a file of the same name or
delete the directory which contains it until the handle has been closed.
However, one can work around that by moving the file
Richard Oudkerk added the comment:
On 14/03/2013 1:00pm, Martin v. Löwis wrote:
> That's why I was asking for an actual patch. The proposed change may
> well not be implementable. If os.open continues to create CRT handles,
> a way needs to be found to get a CRT hand
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17444>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
On 22/03/2013 3:19pm, Charles-François Natali wrote:
> The _flag is checked without any lock held: although it won't be a
> problem with CPython, a standard memory model (e.g. Java's one)
> doesn't guarantee that reading _flag outside
Richard Oudkerk added the comment:
On 22/03/2013 3:31pm, Charles-François Natali wrote:
>> I was under the impression that dict access (and therefore attribute
>> access for "simple" objects) was guaranteed to be atomic even in
>> alternative implementations like Jyt
Richard Oudkerk added the comment:
The old code deleted the obj in the feeder thread as soon as it was sent at
lines 247 and 253 -- see Issue #16284. I think that should be retained.
Apart from that LGTM.
--
___
Python tracker
<h
Richard Oudkerk added the comment:
On 24/03/2013 12:16pm, Charles-François Natali wrote:
> The object is overwritten by the pickled data, so it's not necessary
> anymore, no?
Yes, you are right.
--
___
Python tracker
<http://b
Richard Oudkerk added the comment:
It seems to be a problem with ForkAwareThreadLock. Could you try the attached
patch?
--
Added file: http://bugs.python.org/file29593/forkawarethreadlock.patch
___
Python tracker
<http://bugs.python.org/issue17
Richard Oudkerk added the comment:
_afterfork_registry is not supposed to be cleared. But the problem with
ForkAwareThreadLocal meant that the size of the registry at generation n is
2**n!
--
___
Python tracker
<http://bugs.python.org/issue17
Richard Oudkerk added the comment:
> I *think* we need to keep compatibility with the wire format, but perhaps
> we could use a special length value (-1?) to introduce a longer (64-bit)
> length value.
Yes we could, although that would not help on Windows pipe connections (wh
Richard Oudkerk added the comment:
On 27/03/2013 5:13pm, mrjbq7 wrote:
> On a machine with 256GB of RAM, it makes more sense to send arrays
> of this size than say on a laptop...
I was thinking more of speed than memory consumption.
--
___
Richard Oudkerk added the comment:
On 27/03/2013 5:47pm, Charles-François Natali wrote:
>> multiprocessing currently only allows sharing of such shared arrays
>> using inheritance.
>
> You mean through fork() COW?
Through fork, yes, but "shared" rather than "
Richard Oudkerk added the comment:
On 27/03/2013 7:27pm, Charles-François Natali wrote:
>
> Charles-François Natali added the comment:
>
>> Through fork, yes, but "shared" rather than "copy-on-write".
>
> There's a subtlety: because of refcounting,
Richard Oudkerk added the comment:
On 27/03/2013 8:14pm, Charles-François Natali wrote:
>
> Charles-François Natali added the comment:
>
>> Apart from creating, unlinking and resizing the file I don't think there
>> should be any disk I/O.
>>
>> On Linu
Richard Oudkerk added the comment:
On 27/03/13 21:09, Charles-François Natali wrote:
> I could, but I don't have to: a shared memory won't incur any I/O or
> copy (except if it is swapped). A file-backed mmap will incur a *lot*
> of I/O: really, just try writting a 1GB f
Richard Oudkerk added the comment:
I don't think this is a bug -- processes started with fork() should nearly
always be exited with _exit(). And anyway, using sys.exit() does *not*
guarantee that all deallocators will be called. To be sure of cleanup at exit
you could use (the undocum
Richard Oudkerk added the comment:
Maybe this is related to
http://bugs.python.org/issue13673
which causes PyTraceback_Print() to fail if a signal is received but
PyErr_CheckSignals() has not been called.
Note that wrapping in "try: ... except: raise" makes a traceback appear
Richard Oudkerk added the comment:
Adding PyErr_CheckSignals() after PyOS_Readline() in builtin_input() seems to
fix the problem.
--
Added file: http://bugs.python.org/file29665/input-ctrlc.patch
___
Python tracker
<http://bugs.python.
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file29665/input-ctrlc.patch
___
Python tracker
<http://bugs.python.org/issue13673>
___
___
Python-bug
Richard Oudkerk added the comment:
Sorry, last message was for another issue.
--
___
Python tracker
<http://bugs.python.org/issue13673>
___
___
Python-bugs-list m
Richard Oudkerk added the comment:
Adding PyErr_CheckSignals() after PyOS_Readline() in builtin_input() seems to
fix the problem.
--
keywords: +patch
Added file: http://bugs.python.org/file29666/input-ctrlc.patch
___
Python tracker
<h
Richard Oudkerk added the comment:
> Richard - are you in a position to commit / push?
Done.
--
___
Python tracker
<http://bugs.python.org/issue17619>
___
_
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
This problem also affects processes started by multiprocessing (although it is
fixed in http://hg.python.org/sandbox/sbt#spawn).
As far as I am concerned O_NOINHERIT should really have been applied by default
in Python 3. You only get inheritable fds on
Richard Oudkerk added the comment:
You seem to end your subroutines (or whatever they are called) using "goto end"
rather than "exit /b". Since popd follows the "end" label, does this mean that
you get a popd after calling each subroutine? Is this intended and
Richard Oudkerk added the comment:
I don't see how this is a subprocess problem, or could be fixed in subprocess.
IIUC, SetConsoleTextAttribute() only has an effect if the output is connected
to a console. But that is not the case if you redirect the output to a pipe
(which is presu
Richard Oudkerk added the comment:
On 07/04/2013 7:21pm, R. David Murray wrote:
> Certainly on unix if you write ANSI color codes to stdout and the reader
> doesn't strip them, they will be preserved and can be redisplayed, so
> being able to do something similar on Windows
Richard Oudkerk added the comment:
On 07/04/2013 9:02pm, Caitlin Potter wrote:
> I'm not entirely positive that it would be doable, but looking at the
> subprocess code, it looks like we do have an open handle to the windows
> stdout buffer, including buffer attributes, so it shou
Richard Oudkerk added the comment:
@gjwebber: How exactly are you running the program to get that traceback?
The following lines make it look like you are doing something non-standard (as
opposed to just saving the file and running it from the command line):
File "", li
Richard Oudkerk added the comment:
On 10/04/2013 6:32am, Charles-François Natali wrote:
> Richard, IIRC, you said somewhere that FD passing failures on OS X
> could be made to work by passing a FD at a time, or something like
> that. What do you think of those faiilures?
I think se
Richard Oudkerk added the comment:
> I could mark the new test as cpython only, or add a gc.collect (but I'm
> not sure if the latter is enough, either).
test.support.gc_collect() should work on non-refcounted implementations.
--
Richard Oudkerk added the comment:
Note that in Python 3 you can also do __class__.f() in a staticmethod. Not
sure if that is encouraged though.
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue10
Changes by Richard Oudkerk :
--
title: Creating new processes after importing multiprocessing.managers consumes
more and more memory -> ForkAwareThreadLock leak after fork
versions: +Python 3.3, Python 3.4
___
Python tracker
<http://bugs.pyth
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
Pickling an exception (assuming it works) does not capture the traceback.
Doing so would be difficult/impossible since the traceback refers to a linked
list of frames, and each frame has references to lots of other stuff like the
code object, the global
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17778>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
Would this mean that the destructor could be run more than once (or
prematurely)?
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17
Richard Oudkerk added the comment:
In Python 3 "/" gives float division, whereas in Python 2 it is integer
division, the same as "//".
--
nosy: +sbt
___
Python tracker
<http://bug
Richard Oudkerk added the comment:
Just to clarify, if you use float division then you get rounding errors.
309657313492949847071 is a rounding error:
>>> x = 284397269195572115652769428988866694680//17
>>> x - int(float(x))
3096
Richard Oudkerk added the comment:
Yes.
--
resolution: -> invalid
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
Duplicate of #13831.
--
resolution: -> duplicate
___
Python tracker
<http://bugs.python.org/issue17836>
___
___
Python-
Richard Oudkerk added the comment:
It might be possible to come up with a hack so that when the exception is
unpickled in the main process it gets a secondary exception chained to it using
__cause__ or __context__ whose stringification contains the stringification of
the original traceback
Changes by Richard Oudkerk :
--
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue17836>
___
___
Pyth
Richard Oudkerk added the comment:
Some sort of error is expected.
On Windows any functions or classes used in a task sent the executor must be
picklable/unpicklable. This means that they must be defined in an importable
module rather than being defined in the interactive shell.
On Unix
Richard Oudkerk added the comment:
> Ah. Then, a documentation error. The error message ("queue.Full"?) and
> the documentation are totally not clear about that.
Once something goes wrong you are likely to get a cascade of errors, and the
first one reported is not necessar
Richard Oudkerk added the comment:
It might be simplest to make the implementation match the docs by making
AsyncResult an alias for ApplyResult.
--
___
Python tracker
<http://bugs.python.org/issue17
Richard Oudkerk added the comment:
Attached is a patch for 3.4 which uses the __cause__ hack to embed the remote
traceback in the local traceback. It will not work for 2.x though.
>>> import multiprocessing, subprocess
>>> with multiprocessing.Pool() as p: p.apply(su
Richard Oudkerk added the comment:
Here is an updated patch. It is only really the example in the docs which is
different, plus a note about daemon threads.
Antoine, do think this is ready to be committed?
--
Added file: http://bugs.python.org/file30140/finalize.patch
Richard Oudkerk added the comment:
The only (non-doc, non-comment) changes were the two one-liners you suggested
in msg172077. So I will commit.
--
___
Python tracker
<http://bugs.python.org/issue15
Richard Oudkerk added the comment:
I don't see any difference in meaning:
http://idioms.thefreedictionary.com/as+far+as+possible
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/is
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: patch review -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
601 - 700 of 737 matches
Mail list logo
