Richard Oudkerk added the comment:
I can't remember why I did not use fstat() -- probably it did not occur to me.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18174
Richard Oudkerk added the comment:
Updated version of the patch. Still needs docs.
--
Added file: http://bugs.python.org/file35902/memoryview-array-value.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14953
Changes by Richard Oudkerk shibt...@gmail.com:
--
assignee: - sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10850
___
___
Python-bugs-list
Richard Oudkerk added the comment:
Since there are no new features added to Python 2, this would be a Python 3
only feature.
I think for Python 3 it is better to concentrate on developing
concurrent.futures rather than multiprocessing.Pool
Changes by Richard Oudkerk shibt...@gmail.com:
--
assignee: - sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21779
___
___
Python-bugs-list
Changes by Richard Oudkerk shibt...@gmail.com:
--
assignee: - sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21664
___
___
Python-bugs-list
Changes by Richard Oudkerk shibt...@gmail.com:
--
assignee: - sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20147
___
___
Python-bugs-list
Richard Oudkerk added the comment:
register_after_fork() is intentionally undocumented and for internal use.
It is only run when starting a new process using the fork start method
whether on Windows or not -- the fork in its name is a hint.
--
resolution: - not a bug
stage
Richard Oudkerk added the comment:
If you use the short timeouts to make the wait interruptible then you can
use waitformultipleobjects (which automatically waits on an extra event
object) instead of waitforsingleobject.
--
___
Python tracker rep
Richard Oudkerk added the comment:
I added some comments.
Your problem with lost data may be caused by the fact you call ov.cancel() and
expect ov.pending to tell you whether the write has/will succeed. Instead you
should use ov.getresult() and expect either success or an aborted error
Richard Oudkerk added the comment:
Can you explain why you write in 512 byte chunks. Writing in one chunk should
not cause a deadlock.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1191964
Richard Oudkerk added the comment:
I would guess that the problem is simply that LogisticRegression objects are
not picklable. Does the problem still occur if you do not use freeze?
--
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
Ah, I misunderstood: you meant that it freezes/hangs, not that you used a
freeze tool.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21162
Richard Oudkerk added the comment:
Could you try pickling and unpickling the result of func():
import cPickle
data = cPickle.dumps(func([1,2,3]), -1)
print cPickle.loads(data)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Richard Oudkerk added the comment:
I would recommended using _overlapped instead of _winapi.
I intend to move multiprocessing over in future.
Also note that you can do nonblocking reads by starting an overlapped read
then cancelling it immediately if it fails with incomplete. You will
need
Richard Oudkerk added the comment:
Using truncate() to zero extend is not really portable: it is only guaranteed
on XSI-compliant POSIX systems.
Also, the FreeBSD man page for mmap() has the following warning:
WARNING! Extending a file with ftruncate(2), thus creating a big
hole
Richard Oudkerk added the comment:
Using asyncio and the IOCP eventloop it is not necessary to use threads.
(Windows may use worker threads for overlapped IO, but that is hidden from
Python.) See
https://code.google.com/p/tulip/source/browse/examples/child_process.py
for vaguely expect
Richard Oudkerk added the comment:
No, the argument will not go away now.
However, I don't much like the API which is perhaps why I did not get round to
documenting it.
It does have tests. Currently 'xmlrpclib' is the only supported alternative,
but JSON support could be added quite easily
Richard Oudkerk added the comment:
Testing the is_forking() requires cx_freeze or something similar, so it really
cannot go in the test suite.
I have tested it manually (after spending too long trying to get cx_freeze to
work with a source build).
It should be noted that on Unix freezing
Richard Oudkerk added the comment:
For reasons we all know unpickling unauthenticated data received over TCP is
very risky. Sending an unencrypted authentication key (as part of a pickle)
over TCP would make the authentication useless.
When a proxy is pickled the authkey is deliberately
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20633
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: test needed - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20980
Changes by Richard Oudkerk shibt...@gmail.com:
--
assignee: - sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20990
___
___
Python-bugs-list
Richard Oudkerk added the comment:
We should only wrap the exception with ExceptionWithTraceback in the process
case where it will be pickled and then unpickled.
--
assignee: - sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Richard Oudkerk added the comment:
I am not sure method_to_typeid and create_method were really intended to be
public -- they are only used by Pool proxies.
You can maybe work around the problem by registering a second typeid without
specifying callable. That can be used in method_to_typeid
Changes by Richard Oudkerk shibt...@gmail.com:
--
assignee: - sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20633
___
___
Python-bugs-list
Changes by Richard Oudkerk shibt...@gmail.com:
--
assignee: - sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7503
___
___
Python-bugs-list
Richard Oudkerk added the comment:
Thanks Richard. The set_start_method() call will affect any process
started from that time on? Is it possible to change idea at some point in
the future?
You can use different start methods in the same program by creating different
contexts:
spawn_ctx
Richard Oudkerk added the comment:
On Unix, using the fork start method (which was the only option till 3.4),
every sub process will incref every shared object for which its parent has a
reference.
This is deliberate because there is not really any way to know which shared
objects
Richard Oudkerk added the comment:
LGTM
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20540
___
___
Python-bugs-list mailing list
Unsubscribe
Richard Oudkerk added the comment:
BTW, I see little difference between 3.2 and the unpatched default branch on
MacOSX:
$ py-32/release/python.exe ~/Downloads/test_manager.py
0.0007331371307373047
8.20159912109375e-05
9.417533874511719e-05
8.082389831542969e-05
7.796287536621094e-05
Richard Oudkerk added the comment:
This is expected. Killing processes which use shared locks is never going to
end well. Even without the lock deadlock, the data in the pipe would be liable
to be corrupted if a processes is killed while putting or getting from a queue.
If you want
Richard Oudkerk added the comment:
_overlapped is linked against the socket library whereas _winapi is not so
it can be bundled in with python3.dll.
I did intend to switch multiprocessing over to using _overlapped but I did
not get round to it.
Since this is a private module the names
Richard Oudkerk added the comment:
The following from the docs is wrong:
... module globals are no longer forced to None during interpreter
shutdown.
Actually, in 3.4 module globals *sometimes* get forced to None during
interpreter shutdown, so the version the __del__ method can still
Richard Oudkerk added the comment:
It is probably harmless then.
I don't think increasing the timeout is necessary -- the multiprocessing tests
already take a long time.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20114
Richard Oudkerk added the comment:
How often has this happened?
If the machine was very loaded then maybe the timeout was not enough time for
the semaphore to be cleaned up by the tracker process. But I would expect 1
second to be more than ample
Richard Oudkerk added the comment:
On 19/12/2013 10:00 pm, Nick Coghlan wrote:
I think that needs to be fixed on the multiprocessing side rather than just
in the tests - we shouldn't create a concrete context for a start method
that isn't going to work on that platform. Finding that kind
Richard Oudkerk added the comment:
Thanks for your hard work Nick!
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19946
___
___
Python-bugs-list
Richard Oudkerk added the comment:
So there are really two situations:
1) The __main__ module *should not* be imported. This is the case if you use
__main__.py in a package or if you use nose to call test_main().
This should really be detected in get_preparation_data() in the parent process
Richard Oudkerk added the comment:
I appear to be somehow getting child processes where __main__.__file__ is
set, but __main__.__spec__ is not.
That seems to be true for the __main__ module even when multiprocessing is not
involved. Running a file /tmp/foo.py containing
import sys
Richard Oudkerk added the comment:
I guess this is a case where we should not be trying to import the main module.
The code for determining the path of the main module (if any) is rather crufty.
What is sys.modules['__main__'] and sys.modules['__main__'].__file__ if you run
under nose
Richard Oudkerk added the comment:
From what I remember a proxy method will be thread/process-safe if the
referent's corresponding method is thread safe.
It should certainly be documented that the exposed methods of a proxied object
should be thread-safe
Richard Oudkerk added the comment:
I've always had an implicit understanding that calls with timeouts may,
for whatever reason, return sooner than requested (or later!), and the
most careful approach is to re-check the clock again.
I've always had the implicit understanding that if I use
Richard Oudkerk added the comment:
Could you try this patch?
--
keywords: +patch
Added file: http://bugs.python.org/file32822/wait-for-handle.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19740
Richard Oudkerk added the comment:
Possibly related: ...
That looks unrelated since it does not involve wait_for_handle().
Unfortunately test_utils.run_briefly() offers few guarantees when using the
IOCP event loop.
--
___
Python tracker rep
Richard Oudkerk added the comment:
It would be nice to try this on another Vista machine - the WinXP, Win7,
Windows Server 2003 and Windows Server 2008 buildbots don't seem to show this
failure.
It looks as though the TimerOrWaitFired argument passed to the callback
registered
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19599
Richard Oudkerk added the comment:
I don't think the patch to the _test_multiprocessing will work. It defines
cls._Popen but I don't see how that would be used by cls.Pool to start the
processes.
I will have a think about a fix.
--
___
Python
Richard Oudkerk added the comment:
If the result of os.read() was stored in a Python daemon thread, the
memory should be released since the following changeset. Can someone
check if this issue still exist?
If a daemon thread is killed while it is blocking on os.read() then the bytes
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
type: behavior -
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16998
Richard Oudkerk added the comment:
Thanks for the patches.
Fixed in 7aabbe919f55, 11cafbe6519f.
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19338
Richard Oudkerk added the comment:
Hopefully the applied change will fix failure (or at least make this much less
likey).
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
type: - behavior
___
Python tracker rep
Richard Oudkerk added the comment:
Note that on Windows if you redirect the standard streams then *all*
inheritable handles are inherited by the child process.
Presumably the handle for f_w file object (and/or a duplicate of it) created in
one thread is accidentally leaked to the other child
Richard Oudkerk added the comment:
If you have a pending overlapped operation then the associated buffer should
not be deallocated until that operation is complete, or else you are liable to
get a crash or memory corruption.
Unfortunately WinXP provides no reliable way to cancel a pending
Richard Oudkerk added the comment:
As close() on regular files, I would prefer to call explicitly cancel()
to control exactly when the overlapped operation is cancelled.
If you use daemon threads then you have no guarantee that the thread will ever
get a chance to explicitly call cancel
Richard Oudkerk added the comment:
I think the attached patch should fix it. Note that with the patch the
RuntimeError can probably only occur on Windows XP.
Shall I apply it?
--
keywords: +patch
Added file: http://bugs.python.org/file32597/dealloc-runtimeerror.patch
Richard Oudkerk added the comment:
On 13/11/2013 3:07pm, STINNER Victor wrote:
On Vista and later, yes, this is done in the deallocator using
CancelIoEx(), although there is still a warning.
I don't understand. The warning is emitted because an operating is not done
nor cancelled. Why
Richard Oudkerk added the comment:
This was fixed for 3.3 in #1692335.
The issue of backporting to 2.7 is discussed in #17296.
--
resolution: - duplicate
status: open - closed
superseder: - Cannot unpickle classes derived from 'Exception'
type: crash - behavior
Richard Oudkerk added the comment:
So hopefully the bug should disappear entirely in future releases of tcl,
but for now you can work around it by building tcl without threads,
calling exec in between the fork and any use of tkinter in the child
process, or not importing tkinter until
Richard Oudkerk added the comment:
Fixed by #11161.
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
superseder: - futures.ProcessPoolExecutor hangs
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Richard Oudkerk added the comment:
Given PEP 446 (fds are now CLOEXEC by default) I prepared an updated patch
where the fork lock is undocumented and subprocess no longer uses the fork
lock. (I did not want to encourage the mixing of threads with fork() without
exec() by exposing the fork
Richard Oudkerk added the comment:
It is a recent kernel and does support pipe2().
After some debugging it appears that a pipe handle created in Popen.__init__()
was being leaked to a forked process, preventing Popen.__init__() from
completing before the forked process did.
Previously
Richard Oudkerk added the comment:
Although it is undocumented, in python 3.4 you can control the prefix used by
doing
multiprocessing.current_process()._config['semprefix'] = 'myprefix'
in the main process at the beginning of the program.
Unfortunately, this will make the full prefix
Richard Oudkerk added the comment:
This is a test of threading.Barrier rather than anything implemented directly
by multiprocessing.
Tests which involve timeouts tend to be a bit flaky. Increasing the length of
timeouts usually helps, but makes the tests take even longer.
How often have you
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: needs patch - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19425
Richard Oudkerk added the comment:
Won't using a prepare handler mean that the parent and child processes will use
the same seed until one or other of them forks again?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19227
Richard Oudkerk added the comment:
Would it make sense to use socketpair() instead of pipe() on AIX? We could
check for the bug directly rather than checking specifically for AIX.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Richard Oudkerk added the comment:
Is this patch still of relevance for asyncio?
No, the _overlapped extension contains the IOCP stuff.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16175
Richard Oudkerk added the comment:
Richard, do you have time to get your patch ready for 3.4?
Yes. But we don't seem to have concensus on how to handle exceptions. The
main question is whether a failed prepare callback should prevent the fork from
happenning, or just be printed
Richard Oudkerk added the comment:
- now that FDs are non-inheritable by default, fork locks around
subprocess and multiprocessing shouldn't be necessary anymore? What
other use cases does the fork-lock have?
CLOEXEC fds will still be inherited by forked children.
- the current
Richard Oudkerk added the comment:
The following uses socketpair() instead of pipe() for stdin, and works for me
on Linux:
diff -r 7d94e4a68b91 asyncio/unix_events.py
--- a/asyncio/unix_events.pySun Oct 20 20:25:04 2013 -0700
+++ b/asyncio/unix_events.pyMon Oct 21 17:15:19 2013 +0100
Richard Oudkerk added the comment:
I guess we'll have to write platform-dependent code and make this an
optional feature. (Essentially, on platforms like AIX, for a
write-pipe, connection_lost() won't be called unless you try to write
some more bytes to it.)
If we are not capturing stdout
Richard Oudkerk added the comment:
I guess this should be clarified in the docs, but multiprocessing.pool.Pool is
a *class* whose constructor takes a context argument, where as
multiprocessing.Pool() is a *bound method* of the default context. (In
previous versions multiprocessing.Pool
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10015
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
I can reproduce the problem on the Non-Debug Gentoo buildbot using only
os.fork() and os.kill(pid, signal.SIGTERM). See
http://hg.python.org/cpython/file/9853d3a20849/Lib/test/_test_multiprocessing.py#l339
To investigate further I think strace
Richard Oudkerk added the comment:
I fixed the out of space last night. (Someday I'll get around to figuring
out which test it is that is leaving a bunch of data around when it fails,
but I haven't yet).
It looks like on the Debug Gentoo buildbot configure and clean are failing.
http
Richard Oudkerk added the comment:
I finally have a gdb backtrace of a stuck child (started using os.fork() not
multiprocessing):
#1 0xb76194da in ?? () from /lib/libc.so.6
#2 0xb6d59755 in ?? ()
from
/var/lib/buildslave/custom.murray-gentoo/build/build/lib.linux-i686-3.4-pydebug
Richard Oudkerk added the comment:
Actually, according to strace the call which blocks is
futex(0xb7839454, FUTEX_WAIT_PRIVATE, 1, NULL
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19227
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: - committed/rejected
status: open - pending
title: Robustness issues in multiprocessing.{get,set}_start_method - Support
different contexts in multiprocessing
type: behavior - enhancement
Changes by Richard Oudkerk shibt...@gmail.com:
--
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18999
___
___
Python-bugs
Richard Oudkerk added the comment:
On 16/10/2013 8:14pm, Guido van Rossum wrote:
(2) I get this message -- what does it mean and should I care?
2 tests altered the execution environment:
test_asyncio.test_base_events test_asyncio.test_futures
Perhaps threads from the ThreadExecutor
Richard Oudkerk added the comment:
I think at module level you can do
if sys.platform != 'win32':
raise unittest.SkipTest('Windows only')
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19262
Richard Oudkerk added the comment:
After running ugly_hack(), trying to malloc a largeish block (1MB) fails:
int main(void)
{
int first;
void *ptr;
ptr = malloc(1024*1024);
assert(ptr != NULL);/* succeeds */
free(ptr);
first = ugly_hack();
ptr = malloc
Richard Oudkerk added the comment:
I haven't read all of your patch yet, but does this mean a forkserver
will be started regardless of whether it is later used?
No, it is started on demand. But since it is started using
_posixsbuprocess.fork_exec(), nothing is inherited from the main
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12413
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
I'm already confused by the fact that the test is named
test_multiprocessing_spawn and the error is coming from a module named
popen_fork...)
popen_spawn_posix.Popen is a subclass of popen_fork.Popen
Richard Oudkerk added the comment:
BTW, the context objects are singletons.
I could not see a sensible way to make ctx.Process be a picklable class (rather
than a method) if there can be multiple instances of a context type. This
means that the helper processes survive until the program
Richard Oudkerk added the comment:
Attached is a patch which allows the use of separate contexts. For example
try:
ctx = multiprocessing.get_context('forkserver')
except ValueError:
ctx = multiprocessing.get_context('spawn')
q = ctx.Queue()
p = ctx.Process
Richard Oudkerk added the comment:
These functions are compliant with POSIX standards and the return values
are actually useful, they return the previously set masks and handlers,
often are ignored but in complex cases it's good to know their previous
state.
Yes. But my point
Richard Oudkerk added the comment:
I think misuse is an exageration. Various functions change some state and
return a value that is usually ignored, e.g. os.umask(), signal.signal().
Global variables usage is a pattern which might lead to code errors and many
developers discourage from
Richard Oudkerk added the comment:
the previous initializers were not supposed to return any value
Previously, any returned value would have been ignored. But the documentation
does not say that the function has to return None. So I don't think we can
assume there is no compatibility issue
Richard Oudkerk added the comment:
Is BoundedSemaphore really supposed to be robust in the face of too many
releases, or does it just provide a sanity check?
I think that releasing a bounded semaphore too many times is a programmer
error, and the exception is just a debugging aid
Richard Oudkerk added the comment:
Well, perhaps we can special-case builtins not to be wiped at shutdown.
However, there is another problem here in that the Popen object survives
until the builtins module is wiped. This should be investigated too.
Maybe it is because it uses the evil
Richard Oudkerk added the comment:
See
http://bugs.python.org/issue436259
This is a problem with Window's implementation of spawn*() and exec*(). Just
use subprocess instead which gets this stuff right.
Note that on Windows exec*() is useless: it just starts a subprocess and exits
Richard Oudkerk added the comment:
I am not sure that I should see there. There is discussion of DOS,
which is not supported, also some complain about Windows execv
function, which deprecated since VC++ 2005 (which I hope also not
supported). Can you be more specific?
_spawn*() and _exec
Richard Oudkerk added the comment:
As I wrote in http://bugs.python.org/issue19066, on Windows execv() is
equivalent to
os.spawnv(os.P_NOWAIT, ...)
os._exit(0)
This means that control is returned to cmd when the child process *starts* (and
afterwards you have cmd and the child
Richard Oudkerk added the comment:
It is said that execv() is deprecated, but it is not said that it is
alias of _execv(). It is only said that _execv() is C++ compliant.
http://msdn.microsoft.com/en-us/library/ms235416(v=vs.90).aspx
Microsoft seems to have decided that all functions
Richard Oudkerk added the comment:
Where did you get that info? MSDN is silent about that.
http://msdn.microsoft.com/en-us/library/886kc0as(v=vs.90).aspx
Reading the source code for the C runtime included with Visual Studio.
The problem is not in what I should or should not use. The problem
Richard Oudkerk added the comment:
Hey. This ticket is about os.execv failing on spaced paths on Windows. It
is not a duplicate of issue19124.
It is a duplicate of #436259 [Windows] exec*/spawn* problem with spaces in
args.
--
___
Python tracker
Richard Oudkerk added the comment:
Visual Studio 10+ ? Is it available somewhere for a reference?
Old versions of the relevant files are here:
http://www.controllogics.com/software/VB6/VC98/CRT/SRC/EXECVE.C
http://www.controllogics.com/software/VB6/VC98/CRT/SRC/SPAWNVE.C
http
1 - 100 of 736 matches
Mail list logo
