Richard Oudkerk added the comment:
Richard - are you in a position to commit / push?
Done.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17619
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17619
Richard Oudkerk added the comment:
I don't think this is a bug -- processes started with fork() should nearly
always be exited with _exit(). And anyway, using sys.exit() does *not*
guarantee that all deallocators will be called. To be sure of cleanup at exit
you could use (the undocumented
Richard Oudkerk added the comment:
It seems to be a problem with ForkAwareThreadLock. Could you try the attached
patch?
--
Added file: http://bugs.python.org/file29593/forkawarethreadlock.patch
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
_afterfork_registry is not supposed to be cleared. But the problem with
ForkAwareThreadLocal meant that the size of the registry at generation n is
2**n!
--
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
I *think* we need to keep compatibility with the wire format, but perhaps
we could use a special length value (-1?) to introduce a longer (64-bit)
length value.
Yes we could, although that would not help on Windows pipe connections (where
byte oriented
Richard Oudkerk added the comment:
On 27/03/2013 5:13pm, mrjbq7 wrote:
On a machine with 256GB of RAM, it makes more sense to send arrays
of this size than say on a laptop...
I was thinking more of speed than memory consumption.
--
___
Python
Richard Oudkerk added the comment:
On 27/03/2013 5:47pm, Charles-François Natali wrote:
multiprocessing currently only allows sharing of such shared arrays
using inheritance.
You mean through fork() COW?
Through fork, yes, but shared rather than copy-on-write.
Perhaps we need
Richard Oudkerk added the comment:
On 27/03/2013 7:27pm, Charles-François Natali wrote:
Charles-François Natali added the comment:
Through fork, yes, but shared rather than copy-on-write.
There's a subtlety: because of refcounting, just treating a COW object
as read-only (e.g. iteratin
Richard Oudkerk added the comment:
On 27/03/2013 8:14pm, Charles-François Natali wrote:
Charles-François Natali added the comment:
Apart from creating, unlinking and resizing the file I don't think there
should be any disk I/O.
On Linux disk I/O only occurs when fsync() or close
Richard Oudkerk added the comment:
On 27/03/13 21:09, Charles-François Natali wrote:
I could, but I don't have to: a shared memory won't incur any I/O or
copy (except if it is swapped). A file-backed mmap will incur a *lot*
of I/O: really, just try writting a 1GB file, and you'll see your
Richard Oudkerk added the comment:
The old code deleted the obj in the feeder thread as soon as it was sent at
lines 247 and 253 -- see Issue #16284. I think that should be retained.
Apart from that LGTM.
--
___
Python tracker rep
Richard Oudkerk added the comment:
On 24/03/2013 12:16pm, Charles-François Natali wrote:
The object is overwritten by the pickled data, so it's not necessary
anymore, no?
Yes, you are right.
--
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
On 22/03/2013 3:19pm, Charles-François Natali wrote:
The _flag is checked without any lock held: although it won't be a
problem with CPython, a standard memory model (e.g. Java's one)
doesn't guarantee that reading _flag outside of the lock will return
Richard Oudkerk added the comment:
On 22/03/2013 3:31pm, Charles-François Natali wrote:
I was under the impression that dict access (and therefore attribute
access for simple objects) was guaranteed to be atomic even in
alternative implementations like Jython and IronPython
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17444
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
On 14/03/2013 1:00pm, Martin v. Löwis wrote:
That's why I was asking for an actual patch. The proposed change may
well not be implementable. If os.open continues to create CRT handles,
a way needs to be found to get a CRT handle
Richard Oudkerk added the comment:
Actually, it is not quite the same semantics as Unix.
After you delete the the file you cannot create a file of the same name or
delete the directory which contains it until the handle has been closed.
However, one can work around that by moving the file
Richard Oudkerk added the comment:
Does this happen every time you run the tests? (I don't see these errors.)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17399
Richard Oudkerk added the comment:
Could you try the following program:
import socket
import multiprocessing
import multiprocessing.reduction
import multiprocessing.connection
def socketpair():
with socket.socket() as l:
l.bind(('localhost', 0))
l.listen(1)
s
Richard Oudkerk added the comment:
Now could you try the attached file? (It will not work on 2.7 because a
missing socket.fromfd().)
P.S. It looks like the error for 3.3 is associated with a file
f:\python\mypy\traceback.py which presumably clashes with the one in the
standard library
Richard Oudkerk added the comment:
Both 3.2 and 3.3 give essentially the same traceback as 3.2 did before,
both with installed python and yesterdays debug builds.
It looks like on your machine socket handles are not correctly inherited by
child processes -- I had assumed that they always
Richard Oudkerk added the comment:
My original report was for 32 bit debug build on 64 bit Win 7 machine.
I just re-ran test_multiprocessing with installed 64 bit python with same
result. Was I don't see these errors. on different Windows or non-Windows.
On 64-bit Windows 7 with both 32
Richard Oudkerk added the comment:
Sorry, I was not very clear.
If you use the O_TEMPORARY flag with open() to get a file handle, then the
share mode used with the underlying CreateFile() function is
X = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE
whereas, if you don't use
Richard Oudkerk added the comment:
LGTM (although the warning is actually harmless).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17395
Richard Oudkerk added the comment:
Which does give me a thought - perhaps lru_cache in 3.4 could accept a
key argument that is called as key(*args, **kwds) to derive the cache
key? (that would be a separate issue, of course)
Agreed. I suggested the same in an earlier post
Richard Oudkerk added the comment:
Why 1? This should be commented.
The manager process will always be included in active_children().
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17395
Richard Oudkerk added the comment:
+1
To use Tools/builbot/*.bat doesn't the current directory have to be the main
directory of the repository? Then I see no point in the -C argument: just
set the correct directory automatically.
I think make.bat should also support creation of non-debug
Richard Oudkerk added the comment:
What does running 'kill-python before re-building python do? I have not
seen it mentioned in the in the devguide or pcbuild/readme.
It kills any currently running python(_d).exe processes. This is because
Windows does not allow program or library files
Richard Oudkerk added the comment:
The change in your patch is in a Windows-only section -- a few lines before the
chunk you can see _winapi.GetExitCodeProcess().
Since read() on Windows never fails with EINTR there is no need for
_eintr_retry_call().
If you are using Linux then there must
Richard Oudkerk added the comment:
BTW, on threads are only used on Windows. On Unix select() or poll() is used.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17367
Richard Oudkerk added the comment:
I will close the issue then.
If you track the problem down to a bug in Python then you can open a new one.
--
resolution: - invalid
stage: - committed/rejected
status: open - closed
___
Python tracker rep
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17364
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
It looks like queues_contention.diff has the line
obj = pickle.dumps(obj)
in both _feed() and put(). Might that be why the third set of benchmarks was
slower than the second?
--
___
Python tracker rep
Richard Oudkerk added the comment:
On 04/03/2013 8:01pm, Charles-François Natali wrote:
It looks like queues_contention.diff has the line
obj = pickle.dumps(obj)
in both _feed() and put(). Might that be why the third set of benchmarks
was slower than the second?
_feed() is a Queue
Richard Oudkerk added the comment:
I think this change will potentially make the main module get imported twice
under different names when we transfer pickled data between processes. The
current code (which is rather a mess) goes out of its way to avoid that.
Basically the main process makes
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: commit review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10527
Richard Oudkerk added the comment:
The new test seems to be reliably failing on Windows:
==
FAIL: test_issue17223 (__main__.UnicodeTest)
--
Traceback (most
Richard Oudkerk added the comment:
A pool should only be used by the process that created it (unless you use a
managed pool).
If you are creating long lived processes then you could create a new pool on
demand. For example (untested)
pool_pid = (None, None)
def get_pool
Richard Oudkerk added the comment:
Richard, are you suggesting that we close this, or do you see an
actionable issue? (a plausible patch to the repository?)
I skimmed the documentation and could not see that this restriction has been
documented.
So I think a documentation patch would
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17261
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
Good, except that you have to add a gc.collect() call for the
non-refcounted implementations.
Better to use test.support.gc_collect().
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
Banning md5 as a matter of policy may be perfectly sensible.
However, I think the way multiprocessing uses hmac authentication is *not*
affected by the collision attacks the advisory talks about. These depend on
the attacker being able to determine
Richard Oudkerk added the comment:
I did not realize there was a 'Extension Modules' section. I have been putting
changes to C extensions in the 'Library' section instead. It looks like most
people do the same as me.
--
nosy: +sbt
___
Python
Richard Oudkerk added the comment:
Was not it be yanked in 1fabff717ef4?
Looks like it was reintroduced by this merge changeset:
http://hg.python.org/cpython/rev/30fc620e240e
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Richard Oudkerk added the comment:
Richard, do you still want to push this forward? Otherwise I'd like to
finalize the patch (in the other sense ;-).
I started to worry a bit about daemon threads. I think they can still run
while atexit functions are being run.* So if a daemon thread
Richard Oudkerk added the comment:
On 14/02/2013 3:16pm, Antoine Pitrou wrote:
Mmmh... thread switching is already blocked at shutdown:
http://hg.python.org/cpython/file/0f827775f7b7/Python/ceval.c#l434
But in Py_Finalize(), call_py_exitfuncs() is called *before* _Py_Finalizing is
set
Changes by Richard Oudkerk shibt...@gmail.com:
--
status: pending - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16246
___
___
Python-bugs
Richard Oudkerk added the comment:
In any case, I think it's just something we'll have to live with. Daemon
threads are not a terrific idea in general.
I agree.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15528
Richard Oudkerk added the comment:
Perhaps NEWS item needed for this change.
Done.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16743
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16743
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17097
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
On 27/01/2013 8:27pm, Terry J. Reedy wrote:
I agree we do not need to retain unpredictable 'dumb luck' -- in
future versions. But in the absence of a clear discrepancy
between doc and behavior (the definition of a bug) I believe
breaking such code
Richard Oudkerk added the comment:
On 27/01/2013 9:06pm, Serhiy Storchaka wrote:
Every bugfix breaks some code. As a compromise I propose set
m_obj-size = PY_SSIZE_T_MAX in case of overflow and emit a warning.
Trying to allocate PY_SSIZE_T_MAX bytes always seems to fail
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17018
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
It appears that Linux's spurious readiness notifications are a deliberate
deviation from the POSIX standard. (They are mentioned in the BUGS section of
the man page for select.)
Should I just apply the following patch to the default branch?
diff -r
Richard Oudkerk added the comment:
According to Alan Cox
It's a design decision and a huge performance win. It's one of the areas
where POSIX read in its strictest form cripples your performance.
See https://lkml.org/lkml/2011/6/18/103
(For write ready, you can obviously have
Richard Oudkerk added the comment:
For SOCK_STREAM, yes, not for SOCK_DGRAM
I thought SOCK_DGRAM messages just got truncated at the receiving end.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16507
Richard Oudkerk added the comment:
On 21/01/2013 5:38pm, Guido van Rossum wrote:
This is a very good question to which I have no good answer.
If it weren't for this, we could probably do away with the distinction
between add_writer and add_connector, and a lot of code could be
simpler
Richard Oudkerk added the comment:
On 21/01/2013 7:00pm, Guido van Rossum wrote:
Regarding your IOCP changes, that sounds pretty exciting. Richard,
could you check those into the Tulip as a branch? (Maybe a new branch
named 'iocp'.)
OK. It may take me a while to rebase them
Richard Oudkerk added the comment:
I have created an iocp branch.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16507
___
___
Python-bugs-list
Richard Oudkerk added the comment:
I see. Then this is a documentation bug. The examples in the
documentation use such non-thread-safe assignments (combined with the
statement These shared objects will be process and thread safe.).
Are you talking about this documentation:
If lock
Richard Oudkerk added the comment:
That compiles (after hacking the line endings). One Tulip test fails,
PollEventLooptests.testSockClientFail. But that's probably because the
PollSelector class hasn't been adjusted for Windows yet (need to dig this
out of the Pollster code
Richard Oudkerk added the comment:
I thought that access to the value field of Value instances was
protected by locks to avoid lost updates.
Loads and stores are both atomic. But += is made up of two operations, a
load followed by a store, and the lock is dropped between the two.
The same
Richard Oudkerk added the comment:
New patch reflecting Amaury's comments.
--
Added file: http://bugs.python.org/file28737/winfileio.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Changes by Richard Oudkerk shibt...@gmail.com:
Removed file: http://bugs.python.org/file28707/winfileio.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Richard Oudkerk added the comment:
Added some comments on Rietveld.
The .fileno() method is missing. Can this cause a problem when the file
is passed to stdlib functions? subprocess for example?
Thanks. An older version of the patch had a fileno() method which returned the
handle
Richard Oudkerk added the comment:
What does this proposal bring exactly?
Unless we are willing to completely replace fds with handles on Windows,
perhaps not too much. (At one point I had assumed that that was the plan for
py3k.)
Although not advertised, openhandle() does have
Richard Oudkerk added the comment:
If you want to communicate between processes of the same progam, you are best
off calling multiprocessing.Pipe() or multiprocessing.Queue() in the main
process. Queues or connections can then be inherited by the child processes.
Usually all communication
Richard Oudkerk added the comment:
For the reasons I wrote in the other issue, I don't think this an approach to
encourage.
There was no need to create a new issue: if you post to a closed issue then
people on the nosy list will still see your message.
So I will close this issue.
(Maybe
Richard Oudkerk added the comment:
If someone used regular sockets deliberately, they could crash
multiprocessing server code deliberately. Any chance of doing a real message
length check against the embedded message length check?
You can do
message = conn.recv_bytes(maxlength)
if you
Richard Oudkerk added the comment:
Thanks for the report.
It seems to only affect Windows, and only when using sockets rather than pipes.
Till this is fixed you could use
temp = bool(multiprocessing.connection.wait([cl], 1))
instead of
temp = cl.poll(1)
As I mentioned on the other
Richard Oudkerk added the comment:
The commits did not have the intended effect.
They just define a _poll() function (and only on Windows) and it is not
referenced anywhere else.
I will look in to fixing this -- on 2.7 and 3.2 this will need to be done in
the C code.
--
resolution
Richard Oudkerk added the comment:
This should be fixed now.
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
type: - behavior
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16955
Richard Oudkerk added the comment:
What do you mean? The intent was to use poll() instead of select()
anywhere available in order to avoid running out of fds.
The change didn't affect Windows because as of right now select() is
the only thing available.
The change *only* effects Windows
Richard Oudkerk added the comment:
It looks like the change to multiprocessing/connection.py committed does not
match the one uploaded as issue10527-3.patch
changeset 81174:e971a70984b8
1.1 --- a/Lib/multiprocessing/connection.py
1.2 +++ b/Lib/multiprocessing/connection.py
1.3
Richard Oudkerk added the comment:
Attached is a new patch which is implemented completely in C.
It adds a WinFileIO class to the io module, which has the same API
as FileIO except that:
* It has a handle attribute instead of a fileno() method.
* It has staticmethods openhandle
Changes by Richard Oudkerk shibt...@gmail.com:
Removed file: http://bugs.python.org/file28544/winfileio.c
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Changes by Richard Oudkerk shibt...@gmail.com:
Removed file: http://bugs.python.org/file28545/test_winfileio.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Changes by Richard Oudkerk shibt...@gmail.com:
Removed file: http://bugs.python.org/file28590/winfileio.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Richard Oudkerk added the comment:
Forgot to mention, the handles are non-inheritable.
You can use _winapi.DuplicateHandle() to create an inheritable duplicate handle
if you really need to.
--
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
Why are you connecting to a multiprocessing listener with a raw socket? You
should be using multiprocessing.connection.Client to create a client connection.
Connection.send(obj) writes a 32 bit unsigned int (in network order) to the
socket representing
Richard Oudkerk added the comment:
try:
_MAXFD = os.sysconf(SC_OPEN_MAX)
-except:
+except ValueError:
_MAXFD = 256
os.sysconf() might raise OSError. I think ValueError is only raised if
_SC_OPEN_MAX was undefined when the module was compiled.
--
nosy: +sbt
Richard Oudkerk added the comment:
Richard, in Tulip's WSAPoll code, it reads:
class WindowsPollPollster(PollPollster):
Pollster implementation using WSAPoll.
WSAPoll is only available on Windows Vista and later. Python
does not currently support WSAPoll
Richard Oudkerk added the comment:
Is this actually a problem?
If events are arranged in a queue and epoll_wait() just removes the oldest
events (up to maxevents) from that queue then there would be no problem with
using a small value for maxevents.
I don't *know* if that is the case, but I
Richard Oudkerk added the comment:
Attached is a patch which adds a winio module which is a replacement for io,
but uses windows handles instead of fds.
It reimplements FileIO and open(), and provides openhandle() and closehandle()
as replacements for os.open() and os.close().
test_io has
Richard Oudkerk added the comment:
I actually wrote a script to reproduce this issue:
The program does *not* demonstrate starvation because you are servicing the
resource represented by the starved duplicate fds before calling poll() again.
You are creating thousands of duplicate handles
Richard Oudkerk added the comment:
The fact that that the FDs are duped shouldn't change anything to the
events reported: it works while the number of FDs is less than
FD_SETSIZE (epoll_wait() maxevents argument).
That assumes that epoll_wait() is supposed to return *all* ready fds
Richard Oudkerk added the comment:
Here is a version which uses epoll to service a number of pipes which is larger
than maxevents. (If NUM_WRITERS is too large then I get OSError: [Errno 24]
Too many open files.)
All pipes get serviced and the output is:
Working with 20 FDs, 5 maxevents
[5
Richard Oudkerk added the comment:
Yes, but the problem is that between two epoll_wait() calls, the
readiness of the FDs can have changed: and if that happens, you'll get
the same list over and over.
If an fd *was* ready but isn't anymore then why would you want to know about
it? Trying
Richard Oudkerk added the comment:
I don't like the idea of a specific I/O module for an OS. Is the public API
different?
It was partly to make integration with the existing tests easier: _io, _pyio
and winio are tested in parallel.
Can't you reuse the io module?
In what sense?
I don't
Richard Oudkerk added the comment:
Note that on Windows there is an O_NOINHERIT flag which almost corresponds to
O_CLOEXEC on Linux.
I don't think there is a need to use the win32 api.
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
A while ago I did write a PipeIO class which subclasses io.RawIOBase and works
for overlapped pipe handles. (It was intended for multiprocessing and doing
asynchronous IO with subprocess.)
As it is it would not work with normal files because when you do
Richard Oudkerk added the comment:
Attached is a module for Python 3.3+ which subclasses io.RawIOBase. The
constructor signature is
WinFileIO(handle, mode=r, closehandle=True)
where mode is r, w, r+ or w+. Handles can be created using
_winapi.CreateFile().
Issues:
- No support
Changes by Richard Oudkerk shibt...@gmail.com:
Added file: http://bugs.python.org/file28545/test_winfileio.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9586
Richard Oudkerk added the comment:
The fileno argument looks like an implementation detail to me.
It has at least one potential use. On Windows socket.detach() returns a socket
handle but there is no documented way to close it -- os.close() will not work.
The only way to close it that I
Richard Oudkerk added the comment:
There is an alternative (documented) interface:
socket.fromfd(handle, socket.AF_INET, socket.SOCK_STREAM).close()
socket.fromfd() duplicates the handle, so that does not close the original
handle
New submission from Richard Oudkerk:
The actual signature is
socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None)
but the documented signature is
socket.socket([family[, type[, proto]]])
Should the fileno argument be documented or is it considered an implementation
Richard Oudkerk added the comment:
Richard, apart from performance, what's the advantage of this approach over
the
fork+exec version?
It is really just performance. For context running the unittests in a 1 cpu
linux VM gives me
fork:
real0m53.868s
user0m1.496s
sys 0m9.757s
301 - 400 of 736 matches
Mail list logo