Richard Oudkerk added the comment:
> If someone used regular sockets deliberately, they could crash
> multiprocessing server code deliberately. Any chance of doing a real message
> length check against the embedded message length check?
You can do
message = conn.recv_bytes(maxle
Richard Oudkerk added the comment:
Thanks for the report.
It seems to only affect Windows, and only when using sockets rather than pipes.
Till this is fixed you could use
temp = bool(multiprocessing.connection.wait([cl], 1))
instead of
temp = cl.poll(1)
As I mentioned on the other
Richard Oudkerk added the comment:
The commits did not have the intended effect.
They just define a _poll() function (and only on Windows) and it is not
referenced anywhere else.
I will look in to fixing this -- on 2.7 and 3.2 this will need to be done in
the C code.
--
resolution
Richard Oudkerk added the comment:
This should be fixed now.
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
type: -> behavior
___
Python tracker
<http://bugs.python
Richard Oudkerk added the comment:
> What do you mean? The intent was to use poll() instead of select()
> anywhere available in order to avoid running out of fds.
> The change didn't affect Windows because as of right now select() is
> the only thing available.
The chan
Richard Oudkerk added the comment:
It looks like the change to multiprocessing/connection.py committed does not
match the one uploaded as issue10527-3.patch
changeset 81174:e971a70984b8
1.1 --- a/Lib/multiprocessing/connection.py
1.2 +++ b/Lib/multiprocessing/connection.py
1.3
Richard Oudkerk added the comment:
> Added some comments on Rietveld.
> The .fileno() method is missing. Can this cause a problem when the file
> is passed to stdlib functions? subprocess for example?
Thanks. An older version of the patch had a fileno() method which returned th
Richard Oudkerk added the comment:
> What does this proposal bring exactly?
Unless we are willing to completely replace fds with handles on Windows,
perhaps not too much. (At one point I had assumed that that was the plan for
py3k.)
Although not advertised, openhandle() does hav
Richard Oudkerk added the comment:
If you want to communicate between processes of the same progam, you are best
off calling multiprocessing.Pipe() or multiprocessing.Queue() in the main
process. Queues or connections can then be inherited by the child processes.
Usually all communication
Richard Oudkerk added the comment:
For the reasons I wrote in the other issue, I don't think this an approach to
encourage.
There was no need to create a new issue: if you post to a closed issue then
people on the nosy list will still see your message.
So I will close this issue.
(
Richard Oudkerk added the comment:
New patch reflecting Amaury's comments.
--
Added file: http://bugs.python.org/file28737/winfileio.patch
___
Python tracker
<http://bugs.python.org/is
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file28707/winfileio.patch
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bug
Richard Oudkerk added the comment:
> I thought that access to the value field of Value instances was
> protected by locks to avoid lost updates.
Loads and stores are both atomic. But "+=" is made up of two operations, a
load followed by a store, and the lock is dropped betwe
Richard Oudkerk added the comment:
> I see. Then this is a documentation bug. The examples in the
> documentation use such non-thread-safe assignments (combined with the
> statement "These shared objects will be process and thread safe.").
Are you talking about this d
Richard Oudkerk added the comment:
> That compiles (after hacking the line endings). One Tulip test fails,
> PollEventLooptests.testSockClientFail. But that's probably because the
> PollSelector class hasn't been adjusted for Windows yet (need to dig this
> out of the
Richard Oudkerk added the comment:
On 21/01/2013 5:38pm, Guido van Rossum wrote:
> This is a very good question to which I have no good answer.
> If it weren't for this, we could probably do away with the distinction
> between add_writer and add_connector, and a lot of code coul
Richard Oudkerk added the comment:
On 21/01/2013 7:00pm, Guido van Rossum wrote:
> Regarding your IOCP changes, that sounds pretty exciting. Richard,
> could you check those into the Tulip as a branch? (Maybe a new branch
> named 'iocp'.)
OK. It may take me a w
Richard Oudkerk added the comment:
I have created an iocp branch.
--
___
Python tracker
<http://bugs.python.org/issue16507>
___
___
Python-bugs-list mailin
Richard Oudkerk added the comment:
It appears that Linux's "spurious readiness notifications" are a deliberate
deviation from the POSIX standard. (They are mentioned in the BUGS section of
the man page for select.)
Should I just apply the following patch to the default b
Richard Oudkerk added the comment:
According to Alan Cox
It's a design decision and a huge performance win. It's one of the areas
where POSIX read in its strictest form cripples your performance.
See https://lkml.org/lkml/2011/6/18/103
> (For write ready, you can o
Richard Oudkerk added the comment:
> For SOCK_STREAM, yes, not for SOCK_DGRAM
I thought SOCK_DGRAM messages just got truncated at the receiving end.
--
___
Python tracker
<http://bugs.python.org/issu
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17018>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
On 27/01/2013 8:27pm, Terry J. Reedy wrote:
> I agree we do not need to retain unpredictable 'dumb luck' -- in
> future versions. But in the absence of a clear discrepancy
> between doc and behavior (the definition of a bug) I believe
> b
Richard Oudkerk added the comment:
On 27/01/2013 9:06pm, Serhiy Storchaka wrote:
> Every bugfix breaks some code. As a compromise I propose set
> m_obj->size = PY_SSIZE_T_MAX in case of overflow and emit a warning.
Trying to allocate PY_SSIZE_T_MAX bytes always seems to
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17097>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
There are lots of things that behave differently depending on the currently set
start method: Lock(), Semaphore(), Queue(), Value(), ... It is not just when
creating a Process or Pool that you need to know the start method.
Passing a context or start_method
Richard Oudkerk added the comment:
I'll review the patch. (According to http://www.python.org/dev/peps/pep-0429/
feature freeze is expected in late November, so there is not too much of rush.)
--
___
Python tracker
<http://bugs.py
Richard Oudkerk added the comment:
An alternative would be to have separate files NEWS-3.2, NEWS-3.3, NEWS-3.4
etc. If a fix is added to 3.2 and will be merged to 3.3 and 3.4 then you add
an entry to NEWS-3.2 and append some sort of tags to indicate merges:
- Issue #1234: Fix something
Richard Oudkerk added the comment:
Thanks for the doc cleanup -- I am rather busy right now.
Note that stuff does still get replaced by None at shutdown, and this can still
produce errors, even if they are much harder to trigger. If I run the
following program
import _weakref
import
Richard Oudkerk added the comment:
The clearing of modules at shutdown has been substantially changed in 3.4. Now
a best effort is made to let the module go away purely by gc. Those modules
which survive get purged in random order.
In 3.3 all modules were purged, but builtins was special
Richard Oudkerk added the comment:
An alternative would be to use weakref.finalize() which would guarantee that
cleanup happens before any purging occurs. That would allow the use of shutil:
class TemporaryDirectory(object):
def __init__(self, suffix="", prefix=template
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue19112>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
See
http://bugs.python.org/issue436259
This is a problem with Window's implementation of spawn*() and exec*(). Just
use subprocess instead which gets this stuff right.
Note that on Windows exec*() is useless: it just starts a subprocess and exits
Richard Oudkerk added the comment:
> I am not sure that I should see there. There is discussion of DOS,
> which is not supported, also some complain about Windows execv
> function, which deprecated since VC++ 2005 (which I hope also not
> supported). Can you be more specific?
_
Richard Oudkerk added the comment:
As I wrote in http://bugs.python.org/issue19066, on Windows execv() is
equivalent to
os.spawnv(os.P_NOWAIT, ...)
os._exit(0)
This means that control is returned to cmd when the child process *starts* (and
afterwards you have cmd and the child
Richard Oudkerk added the comment:
> It is said that execv() is deprecated, but it is not said that it is
> alias of _execv(). It is only said that _execv() is C++ compliant.
> http://msdn.microsoft.com/en-us/library/ms235416(v=vs.90).aspx
Microsoft seems to have decided that all fun
Richard Oudkerk added the comment:
> Where did you get that info? MSDN is silent about that.
> http://msdn.microsoft.com/en-us/library/886kc0as(v=vs.90).aspx
Reading the source code for the C runtime included with Visual Studio.
> The problem is not in what I should or should not
Richard Oudkerk added the comment:
> Hey. This ticket is about os.execv failing on spaced paths on Windows. It
> is not a duplicate of issue19124.
It is a duplicate of #436259 "[Windows] exec*/spawn* problem with spaces in
args".
--
__
Richard Oudkerk added the comment:
> Visual Studio 10+ ? Is it available somewhere for a reference?
Old versions of the relevant files are here:
http://www.controllogics.com/software/VB6/VC98/CRT/SRC/EXECVE.C
http://www.controllogics.com/software/VB6/VC98/CRT/SRC/SPAWNVE.C
h
Changes by Richard Oudkerk :
--
nosy: -sbt
___
Python tracker
<http://bugs.python.org/issue19066>
___
___
Python-bugs-list mailing list
Unsubscribe:
Changes by Richard Oudkerk :
--
nosy: -sbt
___
Python tracker
<http://bugs.python.org/issue19124>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
> Well, perhaps we can special-case builtins not to be "wiped" at shutdown.
> However, there is another problem here in that the Popen object survives
> until the builtins module is wiped. This should be investigated too.
Maybe it is becau
Richard Oudkerk added the comment:
Is BoundedSemaphore really supposed to be "robust" in the face of too many
releases, or does it just provide a sanity check?
I think that releasing a bounded semaphore too many times is a programmer
error, and the exception is just a debugging a
Richard Oudkerk added the comment:
> the previous initializers were not supposed to return any value
Previously, any returned value would have been ignored. But the documentation
does not say that the function has to return None. So I don't think we can
assume there is no compa
Richard Oudkerk added the comment:
I think "misuse" is an exageration. Various functions change some state and
return a value that is usually ignored, e.g. os.umask(), signal.signal().
> Global variables usage is a pattern which might lead to code errors and many
> developers
Richard Oudkerk added the comment:
> These functions are compliant with POSIX standards and the return values
> are actually useful, they return the previously set masks and handlers,
> often are ignored but in complex cases it's good to know their previous
> state.
Yes.
Richard Oudkerk added the comment:
BTW, the context objects are singletons.
I could not see a sensible way to make ctx.Process be a picklable class (rather
than a method) if there can be multiple instances of a context type. This
means that the helper processes survive until the program
Richard Oudkerk added the comment:
Attached is a patch which allows the use of separate contexts. For example
try:
ctx = multiprocessing.get_context('forkserver')
except ValueError:
ctx = multiprocessing.get_context('spawn')
q = ctx.Queue()
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue12413>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
> I'm already confused by the fact that the test is named
> test_multiprocessing_spawn and the error is coming from a module named
> popen_fork...)
popen_spawn_posix.Popen is a subclass of po
Richard Oudkerk added the comment:
> I haven't read all of your patch yet, but does this mean a forkserver
> will be started regardless of whether it is later used?
No, it is started on demand. But since it is started using
_posixsbuprocess.fork_exec(), nothing is inherited fr
Richard Oudkerk added the comment:
After running ugly_hack(), trying to malloc a largeish block (1MB) fails:
int main(void)
{
int first;
void *ptr;
ptr = malloc(1024*1024);
assert(ptr != NULL);/* succeeds */
free(ptr);
first = ugly_hack();
ptr = malloc
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> pending
title: Robustness issues in multiprocessing.{get,set}_start_method -> Support
different contexts in multiprocessing
type: behavior -&
Changes by Richard Oudkerk :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue18999>
___
___
Python-bugs-list mailing list
Unsubscrib
Richard Oudkerk added the comment:
On 16/10/2013 8:14pm, Guido van Rossum wrote:
> (2) I get this message -- what does it mean and should I care?
> 2 tests altered the execution environment:
> test_asyncio.test_base_events test_asyncio.test_futures
Perhaps threads from the Threa
Richard Oudkerk added the comment:
I think at module level you can do
if sys.platform != 'win32':
raise unittest.SkipTest('Windows only')
--
___
Python tracker
<http://bug
Richard Oudkerk added the comment:
I can reproduce the problem on the Non-Debug Gentoo buildbot using only
os.fork() and os.kill(pid, signal.SIGTERM). See
http://hg.python.org/cpython/file/9853d3a20849/Lib/test/_test_multiprocessing.py#l339
To investigate further I think strace and/or
Richard Oudkerk added the comment:
> I fixed the out of space last night. (Someday I'll get around to figuring
> out which test it is that is leaving a bunch of data around when it fails,
> but I haven't yet).
It looks like on the Debug Gentoo buildbot configure an
Richard Oudkerk added the comment:
I finally have a gdb backtrace of a stuck child (started using os.fork() not
multiprocessing):
#1 0xb76194da in ?? () from /lib/libc.so.6
#2 0xb6d59755 in ?? ()
from
/var/lib/buildslave/custom.murray-gentoo/build/build/lib.linux-i686-3.4-pydebug
Richard Oudkerk added the comment:
Actually, according to strace the call which blocks is
futex(0xb7839454, FUTEX_WAIT_PRIVATE, 1, NULL
--
___
Python tracker
<http://bugs.python.org/issue19
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue10015>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
I guess this should be clarified in the docs, but multiprocessing.pool.Pool is
a *class* whose constructor takes a context argument, where as
multiprocessing.Pool() is a *bound method* of the default context. (In
previous versions multiprocessing.Pool was a
Richard Oudkerk added the comment:
> I guess we'll have to write platform-dependent code and make this an
> optional feature. (Essentially, on platforms like AIX, for a
> write-pipe, connection_lost() won't be called unless you try to write
> some more bytes to it.)
I
Richard Oudkerk added the comment:
Would it make sense to use socketpair() instead of pipe() on AIX? We could
check for the "bug" directly rather than checking specifically for AIX.
--
___
Python tracker
<http://bugs.python.o
Richard Oudkerk added the comment:
> Is this patch still of relevance for asyncio?
No, the _overlapped extension contains the IOCP stuff.
--
___
Python tracker
<http://bugs.python.org/issu
Richard Oudkerk added the comment:
> Richard, do you have time to get your patch ready for 3.4?
Yes. But we don't seem to have concensus on how to handle exceptions. The
main question is whether a failed prepare callback should prevent the fork from
happenning, or just be
Richard Oudkerk added the comment:
> - now that FDs are non-inheritable by default, fork locks around
> subprocess and multiprocessing shouldn't be necessary anymore? What
> other use cases does the fork-lock have?
CLOEXEC fds will still be inherited by forked children.
&
Richard Oudkerk added the comment:
The following uses socketpair() instead of pipe() for stdin, and works for me
on Linux:
diff -r 7d94e4a68b91 asyncio/unix_events.py
--- a/asyncio/unix_events.pySun Oct 20 20:25:04 2013 -0700
+++ b/asyncio/unix_events.pyMon Oct 21 17:15:19 2013 +0100
Richard Oudkerk added the comment:
Won't using a prepare handler mean that the parent and child processes will use
the same seed until one or other of them forks again?
--
___
Python tracker
<http://bugs.python.org/is
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
This is a test of threading.Barrier rather than anything implemented directly
by multiprocessing.
Tests which involve timeouts tend to be a bit flaky. Increasing the length of
timeouts usually helps, but makes the tests take even longer.
How often have you
Richard Oudkerk added the comment:
Given PEP 446 (fds are now CLOEXEC by default) I prepared an updated patch
where the fork lock is undocumented and subprocess no longer uses the fork
lock. (I did not want to encourage the mixing of threads with fork() without
exec() by exposing the fork
Richard Oudkerk added the comment:
It is a recent kernel and does support pipe2().
After some debugging it appears that a pipe handle created in Popen.__init__()
was being leaked to a forked process, preventing Popen.__init__() from
completing before the forked process did.
Previously the
Richard Oudkerk added the comment:
Although it is undocumented, in python 3.4 you can control the prefix used by
doing
multiprocessing.current_process()._config['semprefix'] = 'myprefix'
in the main process at the beginning of the program.
Unfortunately, this will
Richard Oudkerk added the comment:
This was fixed for 3.3 in #1692335.
The issue of backporting to 2.7 is discussed in #17296.
--
resolution: -> duplicate
status: open -> closed
superseder: -> Cannot unpickle classes derived from 'Exception'
type
Richard Oudkerk added the comment:
> So hopefully the bug should disappear entirely in future releases of tcl,
> but for now you can work around it by building tcl without threads,
> calling exec in between the fork and any use of tkinter in the child
> process, or not importing t
Richard Oudkerk added the comment:
Fixed by #11161.
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
superseder: -> futures.ProcessPoolExecutor hangs
___
Python tracker
<http://bugs.python
Richard Oudkerk added the comment:
If you have a pending overlapped operation then the associated buffer should
not be deallocated until that operation is complete, or else you are liable to
get a crash or memory corruption.
Unfortunately WinXP provides no reliable way to cancel a pending
Richard Oudkerk added the comment:
> As close() on regular files, I would prefer to call explicitly cancel()
> to control exactly when the overlapped operation is cancelled.
If you use daemon threads then you have no guarantee that the thread will ever
get a chance to explicitly call
Richard Oudkerk added the comment:
I think the attached patch should fix it. Note that with the patch the
RuntimeError can probably only occur on Windows XP.
Shall I apply it?
--
keywords: +patch
Added file: http://bugs.python.org/file32597/dealloc-runtimeerror.patch
Richard Oudkerk added the comment:
On 13/11/2013 3:07pm, STINNER Victor wrote:
>> On Vista and later, yes, this is done in the deallocator using
>> CancelIoEx(), although there is still a warning.
>
> I don't understand. The warning is emitted because an operating is no
Richard Oudkerk added the comment:
Note that on Windows if you redirect the standard streams then *all*
inheritable handles are inherited by the child process.
Presumably the handle for f_w file object (and/or a duplicate of it) created in
one thread is accidentally "leaked" to
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
type: behavior ->
___
Python tracker
<http://bugs.python
Richard Oudkerk added the comment:
Thanks for the patches.
Fixed in 7aabbe919f55, 11cafbe6519f.
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
Hopefully the applied change will fix failure (or at least make this much less
likey).
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
type: -> behavior
___
Python tr
Richard Oudkerk added the comment:
I don't think the patch to the _test_multiprocessing will work. It defines
cls._Popen but I don't see how that would be used by cls.Pool to start the
processes.
I will have a think about a fix.
--
Richard Oudkerk added the comment:
> If the result of os.read() was stored in a Python daemon thread, the
> memory should be released since the following changeset. Can someone
> check if this issue still exist?
If a daemon thread is killed while it is blocking on os.read() then
Changes by Richard Oudkerk :
--
resolution: -> fixed
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue19599>
___
___
Python-bugs-list
Richard Oudkerk added the comment:
It would be nice to try this on another Vista machine - the WinXP, Win7,
Windows Server 2003 and Windows Server 2008 buildbots don't seem to show this
failure.
It looks as though the TimerOrWaitFired argument passed to the callback
registered
Richard Oudkerk added the comment:
Could you try this patch?
--
keywords: +patch
Added file: http://bugs.python.org/file32822/wait-for-handle.patch
___
Python tracker
<http://bugs.python.org/issue19
Richard Oudkerk added the comment:
> Possibly related: ...
That looks unrelated since it does not involve wait_for_handle().
Unfortunately test_utils.run_briefly() offers few guarantees when using the
IOCP event loop.
--
___
Python tracker
&l
Richard Oudkerk added the comment:
> I've always had an implicit understanding that calls with timeouts may,
> for whatever reason, return sooner than requested (or later!), and the
> most careful approach is to re-check the clock again.
I've always had the implicit understa
Richard Oudkerk added the comment:
>From what I remember a proxy method will be thread/process-safe if the
>referent's corresponding method is thread safe.
It should certainly be documented that the exposed methods of a proxied object
should be
Changes by Richard Oudkerk :
--
assignee: -> sbt
___
Python tracker
<http://bugs.python.org/issue10850>
___
___
Python-bugs-list mailing list
Unsubscrib
Richard Oudkerk added the comment:
Since there are no new features added to Python 2, this would be a Python 3
only feature.
I think for Python 3 it is better to concentrate on developing
concurrent.futures rather than multiprocessing.Pool
Changes by Richard Oudkerk :
--
assignee: -> sbt
___
Python tracker
<http://bugs.python.org/issue21779>
___
___
Python-bugs-list mailing list
Unsubscrib
Changes by Richard Oudkerk :
--
assignee: -> sbt
___
Python tracker
<http://bugs.python.org/issue21664>
___
___
Python-bugs-list mailing list
Unsubscrib
Richard Oudkerk added the comment:
Updated version of the patch. Still needs docs.
--
Added file: http://bugs.python.org/file35902/memoryview-array-value.patch
___
Python tracker
<http://bugs.python.org/issue14
Richard Oudkerk added the comment:
I can't remember why I did not use fstat() -- probably it did not occur to me.
--
___
Python tracker
<http://bugs.python.org/is
Richard Oudkerk added the comment:
I guess this is a case where we should not be trying to import the main module.
The code for determining the path of the main module (if any) is rather crufty.
What is sys.modules['__main__'] and sys.modules['__main__'].__file__ if
501 - 600 of 737 matches
Mail list logo
