Richard Oudkerk added the comment:
> 6cf6b8265e57 and 8172cc8bfa6d have fixed the issue on my VM. I didn't
> noticed the issue as I only tested hashlib with the release builds, not
> the debug builds. Sorry for that.
Ah. I did not even notice there was _sha3.vcxproj.
Is there
New submission from Richard Oudkerk:
ctypes.WinError() is defined as
def WinError(code=None, descr=None):
if code is None:
code = GetLastError()
if descr is None:
descr = FormatError(code).strip()
return WindowsError(code, descr)
Since
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
> Cogen [4] uses ctypes wrapper.
In the code for the IOCP reactor only ctypes.FormatError() is used from ctypes.
It uses pywin32 instead.
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issu
Richard Oudkerk added the comment:
Note that since Python 3.3, multiprocessing and _winapi make some use of
overlapped IO.
One can use _winapi.ReadFile() and _winapi.WriteFile() to do overlapped IO on
normal socket handles created using socket.socket
Richard Oudkerk added the comment:
Adding the IOCP functions to _winapi is straightforward -- see patch. Note
that there seems to be no way to unregister a handle from an IOCP.
Creating overlapped equivalents of socket.accept() and socket.connect() looks
more complicated. Perhaps that
Changes by Richard Oudkerk :
Added file: http://bugs.python.org/file27516/iocp_example.py
___
Python tracker
<http://bugs.python.org/issue16175>
___
___
Python-bugs-list m
Richard Oudkerk added the comment:
I think this is a duplicate of Issue #15646 which has been fixed in the 2.7 and
3.x branches.
If you run Lib/test/mp_fork_bomb.py you should get a RuntimeError with a
helpful message telling you to use the 'if __name__ == "__main
Richard Oudkerk added the comment:
> select() other than being supported on all platforms has the advantage of
> being simple and quick to use (you just call it once by passing a set of fds
> and then you're done).
Do you mean at the C level? Wouldn't you just do
stru
New submission from Richard Oudkerk:
Using VS2010 _socket links against ws2_32.lib but select links against
wsock32.lib.
Using VS2008 both extensions link against ws2_32.lib. It appears that the
conversion to VS2010 caused the regression.
(Compare #10295 and #11750.)
--
messages
Richard Oudkerk added the comment:
> A preliminary patch is in attachment.
> By default it uses select() but looks for ValueError (raised in case
> FD_SETSIZE gets hit) and falls back on using poll().
>
> This is the failure I get when running tests on Linux.
> It is related
Richard Oudkerk added the comment:
> Using poll() by default is controversial for 2 reasons, I think:
>
> #1 - a certain slowdown is likely to be introduced (I'll measure it)
With a single fd poll is a bit faster than select:
$ python -m timeit -s 'from select import
Richard Oudkerk added the comment:
> Still not getting what you refer to when you talk about > 512 fds
> problem.
Whether you get back the original objects or only their fds will depend on
whether some fd was larger than FD_SETSIZE.
--
_
Richard Oudkerk added the comment:
> This problem affects any single use of select(): instead of using an
> ad-hoc wrapper in each module, it would probably make sense to add a
> higher level selector class to the select module which would fallback on
> the right syscall (i.
Richard Oudkerk added the comment:
> A use case for not using fork() is when your parent process opens some
> system resources of some sort (for example a listening TCP socket). The
> child will then inherit those resources, which can have all kinds of
> unforeseen and
Richard Oudkerk added the comment:
LGTM
--
___
Python tracker
<http://bugs.python.org/issue16284>
___
___
Python-bugs-list mailing list
Unsubscribe:
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue16307>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
For updated code see http://hg.python.org/sandbox/sbt#spawn
This uses _posixsubprocess and closefds=True.
--
hgrepos: +157
___
Python tracker
<http://bugs.python.org/issue8
Changes by Richard Oudkerk :
--
assignee: -> sbt
___
Python tracker
<http://bugs.python.org/issue20990>
___
___
Python-bugs-list mailing list
Unsubscrib
Richard Oudkerk added the comment:
We should only wrap the exception with ExceptionWithTraceback in the process
case where it will be pickled and then unpickled.
--
assignee: -> sbt
___
Python tracker
<http://bugs.python.org/issu
Richard Oudkerk added the comment:
For reasons we all know unpickling unauthenticated data received over TCP is
very risky. Sending an unencrypted authentication key (as part of a pickle)
over TCP would make the authentication useless.
When a proxy is pickled the authkey is deliberately
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: test needed -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
Testing the is_forking() requires cx_freeze or something similar, so it really
cannot go in the test suite.
I have tested it manually (after spending too long trying to get cx_freeze to
work with a source build).
It should be noted that on Unix freezing is
Richard Oudkerk added the comment:
No, the argument will not go away now.
However, I don't much like the API which is perhaps why I did not get round to
documenting it.
It does have tests. Currently 'xmlrpclib' is the only supported alternative,
but JSON support could be add
Richard Oudkerk added the comment:
Using asyncio and the IOCP eventloop it is not necessary to use threads.
(Windows may use worker threads for overlapped IO, but that is hidden from
Python.) See
https://code.google.com/p/tulip/source/browse/examples/child_process.py
for vaguely "e
Richard Oudkerk added the comment:
Using truncate() to zero extend is not really portable: it is only guaranteed
on XSI-compliant POSIX systems.
Also, the FreeBSD man page for mmap() has the following warning:
WARNING! Extending a file with ftruncate(2), thus creating a big
hole, and then
Richard Oudkerk added the comment:
I would recommended using _overlapped instead of _winapi.
I intend to move multiprocessing over in future.
Also note that you can do nonblocking reads by starting an overlapped read
then cancelling it immediately if it fails with "incomplete". You
Richard Oudkerk added the comment:
I would guess that the problem is simply that LogisticRegression objects are
not picklable. Does the problem still occur if you do not use freeze?
--
___
Python tracker
<http://bugs.python.org/issue21
Richard Oudkerk added the comment:
Ah, I misunderstood: you meant that it freezes/hangs, not that you used a
freeze tool.
--
___
Python tracker
<http://bugs.python.org/issue21
Richard Oudkerk added the comment:
Could you try pickling and unpickling the result of func():
import cPickle
data = cPickle.dumps(func([1,2,3]), -1)
print cPickle.loads(data)
--
___
Python tracker
<http://bugs.python.org/issue21
Richard Oudkerk added the comment:
Can you explain why you write in 512 byte chunks. Writing in one chunk should
not cause a deadlock.
--
___
Python tracker
<http://bugs.python.org/issue1191
Richard Oudkerk added the comment:
I added some comments.
Your problem with lost data may be caused by the fact you call ov.cancel() and
expect ov.pending to tell you whether the write has/will succeed. Instead you
should use ov.getresult() and expect either success or an "aborted&q
Richard Oudkerk added the comment:
If you use the short timeouts to make the wait interruptible then you can
use waitformultipleobjects (which automatically waits on an extra event
object) instead of waitforsingleobject.
--
___
Python tracker
<h
Changes by Richard Oudkerk :
--
assignee: -> sbt
___
Python tracker
<http://bugs.python.org/issue20147>
___
___
Python-bugs-list mailing list
Unsubscrib
Richard Oudkerk added the comment:
register_after_fork() is intentionally undocumented and for internal use.
It is only run when starting a new process using the "fork" start method
whether on Windows or not -- the "fork" in its name is a hint.
--
resolution:
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue7292>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
Maybe lru_cache() should have a key argument so you can specify a specialized
key function. So you might have
def _compile_key(args, kwds, typed):
return args
@functools.lru_cache(maxsize=500, key=_compile_key)
def _compile(pattern
Richard Oudkerk added the comment:
The patch does not apply correctly against vanilla Python 3.3. I would guess
that you are using a version of Python which has been patched to add mingw
support. Where did you get it from?
(In vanilla Python 3.3, setup.py does not contain any mention of
Changes by Richard Oudkerk :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue16454>
___
___
Python-bugs-list mailing list
Unsubscri
Richard Oudkerk added the comment:
In Torsten's example
from . import moduleX
can be replaced with
moduleX = importlib.import_module('.moduleX', __package__) (*)
or
moduleX = importlib.import_module('package.moduleX')
If that is not pretty en
New submission from Richard Oudkerk:
On Windows the handle for a child process is not closed when the popen/process
object is garbage collected.
--
messages: 175629
nosy: sbt
priority: normal
severity: normal
stage: needs patch
status: open
title: process handle leak on windows in
Richard Oudkerk added the comment:
Fixed in c574ce78cd61 and cb612c5f30cb.
--
resolution: -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
pthread_atfork() allows the registering of three types of callbacks:
1) prepare callbacks which are called before the fork,
2) parent callbacks which are called in the parent after the fork
3) child callbacks which are called in the child after the fork.
I
Richard Oudkerk added the comment:
Note that Gregory P. Smith has written
http://code.google.com/p/python-atfork/
I also started a pure python patch but did not get round it posting it. (It
also implements the fork lock idea.) I'll attach it here.
How do you intend to handl
Richard Oudkerk added the comment:
> IFF we are going to walk the hard and rocky road of exception handling,
> then we are going to need at least four hooks and a register function that
> takres four callables as arguments: register(prepare, error, parent,
> child). Each prepare() ca
Richard Oudkerk added the comment:
http://hg.python.org/sandbox/sbt#spawn now contains support for starting
processes via a separate server process. This depends on fd passing support.
This also solves the problem of mixing threads and processes, but is much
faster than using fork+exec. It
Richard Oudkerk added the comment:
I am reopening this issue because 26bbff4562a7 only dealt with objects which
cannot be pickled. But CalledProcessError instances *can* be pickled: the
problem is that the resulting data cannot be unpickled.
Note that in Python 3.3 CalledProcessError can be
Richard Oudkerk added the comment:
The example works correctly on 3.3 because of #1692335. I am not sure if it is
appropriate to backport it though.
This is a duplicate of #9400 which I have assigned to myself. (I had thought
it was already fixed.)
--
resolution: -> duplic
Richard Oudkerk added the comment:
The patch is liable to break programs which explicitly call base constructors
because the constructor will be called more than once.
It also assumes that the __init__() method of all base classes should be called
with no arguments (other than self). That is
Richard Oudkerk added the comment:
> But I think the problem remains: do you agree that Classes should include
> a super() call in their __init__ ?
No, I don't.
I think super() is an attractive nuisance which is difficult to use correctly
in an __init__() method, except in the t
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue16579>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
Attached is an alternative patch which only touches selectmodule.c. It still
does not support WinXP.
Note that in this version register() and modify() do not ignore the POLLPRI
flag if it was *explicitly* passed. But I am not sure how best to deal with
Richard Oudkerk added the comment:
Here is a version which loads WSAPoll at runtime. Still no tests or docs.
--
Added file: http://bugs.python.org/file28207/runtime_wsapoll.patch
___
Python tracker
<http://bugs.python.org/issue16
New submission from Richard Oudkerk:
PollTests.poll_unit_tests() is not run because its method name does not begin
with "test". It looks it was accidentally disabled when test_poll() was
converted to unittest in f56b25168142.
Renaming it test_poll_unit_tests() makes it run succes
Richard Oudkerk added the comment:
It seems that the return code of WSAPoll() does not include the count of array
items with revents == POLLNVAL. In the case where all of them are POLLNVAL,
instead of returning 0 (which usually indicates a timeout) it returns -1 and
WSAGetLastError
Changes by Richard Oudkerk :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue16616>
___
___
Python-bugs-list mailing list
Unsubscri
Changes by Richard Oudkerk :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue15526>
___
___
Python-bugs-list mailing list
Unsubscri
Richard Oudkerk added the comment:
Here is a new version with tests and docs.
Note that the docs do not mention the bug mentioned in
http://daniel.haxx.se/blog/2012/10/10/wsapoll-is-broken/
Maybe they should?
Note that that bug makes it a bit difficult to use poll with tulip on Windows
Richard Oudkerk added the comment:
When you run wy.py the wow module gets partially imported, and then garbage
collected because it fails to import successfully. The destructor for the
module replaces values in the module's __dict__ with None. So when the cleanup
function runs you ge
Richard Oudkerk added the comment:
> Things should be better in the future, when modules are cleared with true
> garbage collection.
When is this future of which you speak?
I am not sure whether it would affect performance, but a weakrefable subclass
of dict could be used for module
Richard Oudkerk added the comment:
> See issue812369 for the shutdown procedure and modules cleanup.
I am aware of that issue, but the original patch is 9 years old. Which is why
I ask if/when it will actually happen.
--
___
Python tracker
&l
New submission from Richard Oudkerk:
Relevant code:
int timeout = 0, poll_result, i, j;
...
tout = PyNumber_Long(tout);
if (!tout)
return NULL;
timeout = PyLong_AsLong(tout); <-- implicit cast to int
--
messages: 177811
nosy: sbt
prior
Richard Oudkerk added the comment:
Thanks. I will close.
--
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.org/i
Richard Oudkerk added the comment:
Perhaps the simplest thing would be to stop doing anything special when a
module is garbage collected: the garbage collector can take care of any
orphaned ref-cycles involving the module dict. Then at shutdown the remaining
modules in sys.modules could have
Richard Oudkerk added the comment:
I suspect that the size of the 5GB file is originally a 64 bit quantity, but
gets cast unsafely to a 32 bit size_t to give 1GB. This is causing the
miscalculations.
There is no way to map all of a 5GB file in a 32 bit process -- 4GB is the
maximum -- so
Richard Oudkerk added the comment:
This bit looks wrong to me:
if (offset - size > PY_SSIZE_T_MAX)
/* Map area too large to fit in memory */
m_obj->size = (Py_ssize_t) -1;
Should it not be "size - offset" instead of "offset - siz
Richard Oudkerk added the comment:
On 32 bit Unix mmap() will raise ValueError("mmap length is too large") in
Marc's example. This is correct since Python's sequence protocol does not
support indexes larger than sys.maxsize.
But on 32 bit Windows, if length == 0 then t
Richard Oudkerk added the comment:
> This change is not backward compatible. Now user can mmap a larger file
> and safely access lower 2 GiB. With the patch it will fail.
They should specify length=2GiB-1 if that is what they want.
With length=0 you can only access the lower 2
Richard Oudkerk added the comment:
New patch with same check for Unix.
--
Added file: http://bugs.python.org/file28446/mmap.patch
___
Python tracker
<http://bugs.python.org/issue16
Richard Oudkerk added the comment:
> Isn't 2 GiB + 1 bytes mmap file enough for testing?
Yes.
But creating multigigabyte files is very slow on Windows. On Linux/FreeBSD
test_mmap takes a fraction of a second, whereas on Windows it takes over 2
minutes. (Presumably Linux/Fr
Changes by Richard Oudkerk :
--
title: mmap accepts files > 1 GB, but processes only 1 GB -> mmap on Windows
can mishandle files larger than sys.maxsize
type: enhancement -> behavior
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
> Richard, apart from performance, what's the advantage of this approach over
> the
> fork+exec version?
It is really just performance. For context running the unittests in a 1 cpu
linux VM gives me
fork:
real0m53.868s
user
Richard Oudkerk added the comment:
Numbers when running on Linux on a laptop with 2 cores + hyperthreading.
RUNNING UNITTESTS:
fork:
real0m50.687s
user0m9.213s
sys 0m4.012s
fork+exec:
real1m9.062s
user0m48.579s
sys 0m6.648s
forkserver
Changes by Richard Oudkerk :
Added file: http://bugs.python.org/file28461/8f08d83264a0.diff
___
Python tracker
<http://bugs.python.org/issue8713>
___
___
Python-bug
Richard Oudkerk added the comment:
> The safest default would be fork+exec though we need to implement the
> fork+exec code as a C extension module or have it use subprocess (as I
> noted in the mb_fork_exec.patch review).
That was an old version of the patch.
In the branch
New submission from Richard Oudkerk:
The actual signature is
socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None)
but the documented signature is
socket.socket([family[, type[, proto]]])
Should the fileno argument be documented or is it considered an implementation
Richard Oudkerk added the comment:
> The "fileno" argument looks like an implementation detail to me.
It has at least one potential use. On Windows socket.detach() returns a socket
handle but there is no documented way to close it -- os.close() will not work.
The only way to cl
Richard Oudkerk added the comment:
> There is an alternative (documented) interface:
>
> socket.fromfd(handle, socket.AF_INET, socket.SOCK_STREAM).close()
socket.fromfd() duplicates the handle, so that does not close the origina
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.o
Richard Oudkerk added the comment:
Note that on Windows there is an O_NOINHERIT flag which almost corresponds to
O_CLOEXEC on Linux.
I don't think there is a need to use the win32 api.
--
nosy: +sbt
___
Python tracker
<http://bugs.py
Richard Oudkerk added the comment:
A while ago I did write a PipeIO class which subclasses io.RawIOBase and works
for overlapped pipe handles. (It was intended for multiprocessing and doing
asynchronous IO with subprocess.)
As it is it would not work with normal files because when you do
Richard Oudkerk added the comment:
Attached is a module for Python 3.3+ which subclasses io.RawIOBase. The
constructor signature is
WinFileIO(handle, mode="r", closehandle=True)
where mode is "r", "w", "r+" or "w+". Handles can be
Changes by Richard Oudkerk :
Added file: http://bugs.python.org/file28545/test_winfileio.py
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bug
Richard Oudkerk added the comment:
Is this actually a problem?
If events are arranged in a queue and epoll_wait() just removes the oldest
events (up to maxevents) from that queue then there would be no problem with
using a small value for maxevents.
I don't *know* if that is the case,
Richard Oudkerk added the comment:
Attached is a patch which adds a winio module which is a replacement for io,
but uses windows handles instead of fds.
It reimplements FileIO and open(), and provides openhandle() and closehandle()
as replacements for os.open() and os.close().
test_io has
Richard Oudkerk added the comment:
> I actually wrote a script to reproduce this issue:
The program does *not* demonstrate starvation because you are servicing the
resource represented by the "starved" duplicate fds before calling poll() again.
You are creating thousands of dupl
Richard Oudkerk added the comment:
> The fact that that the FDs are duped shouldn't change anything to the
> events reported: it works while the number of FDs is less than
> FD_SETSIZE (epoll_wait() maxevents argument).
That assumes that epoll_wait() is supposed to return *all* r
Richard Oudkerk added the comment:
Here is a version which uses epoll to service a number of pipes which is larger
than maxevents. (If NUM_WRITERS is too large then I get "OSError: [Errno 24]
Too many open files".)
All pipes get serviced and the output is:
Working with 20 FDs, 5
Richard Oudkerk added the comment:
> Yes, but the problem is that between two epoll_wait() calls, the
> readiness of the FDs can have changed: and if that happens, you'll get
> the same list over and over.
If an fd *was* ready but isn't anymore then why would you want to know
Richard Oudkerk added the comment:
> I don't like the idea of a specific I/O module for an OS. Is the public API
> different?
It was partly to make integration with the existing tests easier: _io, _pyio
and winio are tested in parallel.
> Can't you reuse the io module?
Richard Oudkerk added the comment:
> Richard, in Tulip's WSAPoll code, it reads:
>
> class WindowsPollPollster(PollPollster):
> """Pollster implementation using WSAPoll.
>
> WSAPoll is only available on Windows Vista and later. Pytho
Richard Oudkerk added the comment:
> try:
> _MAXFD = os.sysconf("SC_OPEN_MAX")
> -except:
> +except ValueError:
> _MAXFD = 256
os.sysconf() might raise OSError. I think ValueError is only raised if
_SC_OPEN_MAX was undefined when the module was compile
Richard Oudkerk added the comment:
Why are you connecting to a multiprocessing listener with a raw socket? You
should be using multiprocessing.connection.Client to create a client connection.
Connection.send(obj) writes a 32 bit unsigned int (in network order) to the
socket representing the
Richard Oudkerk added the comment:
Attached is a new patch which is implemented completely in C.
It adds a WinFileIO class to the io module, which has the same API
as FileIO except that:
* It has a handle attribute instead of a fileno() method.
* It has staticmethods openhandle() and
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file28544/winfileio.c
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bugs-list m
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file28545/test_winfileio.py
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bug
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file28590/winfileio.patch
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bug
Richard Oudkerk added the comment:
Forgot to mention, the handles are non-inheritable.
You can use _winapi.DuplicateHandle() to create an inheritable duplicate handle
if you really need to.
--
___
Python tracker
<http://bugs.python.org/issue12
401 - 500 of 737 matches
Mail list logo
