[issue16113] Add SHA-3 (Keccak) support

2012-10-07 Thread Richard Oudkerk
Richard Oudkerk added the comment: > 6cf6b8265e57 and 8172cc8bfa6d have fixed the issue on my VM. I didn't > noticed the issue as I only tested hashlib with the release builds, not > the debug builds. Sorry for that. Ah. I did not even notice there was _sha3.vcxproj. Is there

[issue16169] ctypes.WinError() confuses errno with winerror

2012-10-08 Thread Richard Oudkerk
New submission from Richard Oudkerk: ctypes.WinError() is defined as def WinError(code=None, descr=None): if code is None: code = GetLastError() if descr is None: descr = FormatError(code).strip() return WindowsError(code, descr) Since

[issue16169] ctypes.WinError() confuses errno with winerror

2012-10-09 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- resolution: -> fixed stage: needs patch -> committed/rejected status: open -> closed ___ Python tracker <http://bugs.python.or

[issue16175] Add I/O Completion Ports wrapper

2012-10-09 Thread Richard Oudkerk
Richard Oudkerk added the comment: > Cogen [4] uses ctypes wrapper. In the code for the IOCP reactor only ctypes.FormatError() is used from ctypes. It uses pywin32 instead. -- nosy: +sbt ___ Python tracker <http://bugs.python.org/issu

[issue16175] Add I/O Completion Ports wrapper

2012-10-09 Thread Richard Oudkerk
Richard Oudkerk added the comment: Note that since Python 3.3, multiprocessing and _winapi make some use of overlapped IO. One can use _winapi.ReadFile() and _winapi.WriteFile() to do overlapped IO on normal socket handles created using socket.socket

[issue16175] Add I/O Completion Ports wrapper

2012-10-10 Thread Richard Oudkerk
Richard Oudkerk added the comment: Adding the IOCP functions to _winapi is straightforward -- see patch. Note that there seems to be no way to unregister a handle from an IOCP. Creating overlapped equivalents of socket.accept() and socket.connect() looks more complicated. Perhaps that

[issue16175] Add I/O Completion Ports wrapper

2012-10-10 Thread Richard Oudkerk
Changes by Richard Oudkerk : Added file: http://bugs.python.org/file27516/iocp_example.py ___ Python tracker <http://bugs.python.org/issue16175> ___ ___ Python-bugs-list m

[issue16246] Multiprocessing infinite loop on Windows

2012-10-16 Thread Richard Oudkerk
Richard Oudkerk added the comment: I think this is a duplicate of Issue #15646 which has been fixed in the 2.7 and 3.x branches. If you run Lib/test/mp_fork_bomb.py you should get a RuntimeError with a helpful message telling you to use the 'if __name__ == "__main

[issue16269] multiprocessing.Queue uses select()

2012-10-17 Thread Richard Oudkerk
Richard Oudkerk added the comment: > select() other than being supported on all platforms has the advantage of > being simple and quick to use (you just call it once by passing a set of fds > and then you're done). Do you mean at the C level? Wouldn't you just do stru

[issue16295] With VS2010 the select extension links against wsock32 instead of ws2_32

2012-10-21 Thread Richard Oudkerk
New submission from Richard Oudkerk: Using VS2010 _socket links against ws2_32.lib but select links against wsock32.lib. Using VS2008 both extensions link against ws2_32.lib. It appears that the conversion to VS2010 caused the regression. (Compare #10295 and #11750.) -- messages

[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"

2012-10-22 Thread Richard Oudkerk
Richard Oudkerk added the comment: > A preliminary patch is in attachment. > By default it uses select() but looks for ValueError (raised in case > FD_SETSIZE gets hit) and falls back on using poll(). > > This is the failure I get when running tests on Linux. > It is related

[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"

2012-10-22 Thread Richard Oudkerk
Richard Oudkerk added the comment: > Using poll() by default is controversial for 2 reasons, I think: > > #1 - a certain slowdown is likely to be introduced (I'll measure it) With a single fd poll is a bit faster than select: $ python -m timeit -s 'from select import

[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"

2012-10-22 Thread Richard Oudkerk
Richard Oudkerk added the comment: > Still not getting what you refer to when you talk about > 512 fds > problem. Whether you get back the original objects or only their fds will depend on whether some fd was larger than FD_SETSIZE. -- _

[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"

2012-10-22 Thread Richard Oudkerk
Richard Oudkerk added the comment: > This problem affects any single use of select(): instead of using an > ad-hoc wrapper in each module, it would probably make sense to add a > higher level selector class to the select module which would fallback on > the right syscall (i.

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-10-24 Thread Richard Oudkerk
Richard Oudkerk added the comment: > A use case for not using fork() is when your parent process opens some > system resources of some sort (for example a listening TCP socket). The > child will then inherit those resources, which can have all kinds of > unforeseen and

[issue16284] concurrent.futures ThreadPoolExecutor keeps unnecessary references to worker functions.

2012-10-24 Thread Richard Oudkerk
Richard Oudkerk added the comment: LGTM -- ___ Python tracker <http://bugs.python.org/issue16284> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue16295] With VS2010 the select extension links against wsock32 instead of ws2_32

2012-10-24 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- resolution: -> fixed stage: needs patch -> committed/rejected status: open -> closed ___ Python tracker <http://bugs.python.or

[issue16307] multiprocess.pool.map_async callables not working

2012-10-24 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- nosy: +sbt ___ Python tracker <http://bugs.python.org/issue16307> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-10-26 Thread Richard Oudkerk
Richard Oudkerk added the comment: For updated code see http://hg.python.org/sandbox/sbt#spawn This uses _posixsubprocess and closefds=True. -- hgrepos: +157 ___ Python tracker <http://bugs.python.org/issue8

[issue20990] pyflakes: undefined names, get_context() and main(), in multiprocessing

2014-03-20 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- assignee: -> sbt ___ Python tracker <http://bugs.python.org/issue20990> ___ ___ Python-bugs-list mailing list Unsubscrib

[issue20980] In multiprocessing.pool, ExceptionWithTraceback should derive from Exception

2014-03-20 Thread Richard Oudkerk
Richard Oudkerk added the comment: We should only wrap the exception with ExceptionWithTraceback in the process case where it will be pickled and then unpickled. -- assignee: -> sbt ___ Python tracker <http://bugs.python.org/issu

[issue7503] multiprocessing AuthenticationError "digest sent was rejected" when pickling proxy

2014-03-23 Thread Richard Oudkerk
Richard Oudkerk added the comment: For reasons we all know unpickling unauthenticated data received over TCP is very risky. Sending an unencrypted authentication key (as part of a pickle) over TCP would make the authentication useless. When a proxy is pickled the authkey is deliberately

[issue20633] SystemError: Parent module 'multiprocessing' not loaded, cannot perform relative import

2014-03-23 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- resolution: -> fixed stage: -> committed/rejected status: open -> closed ___ Python tracker <http://bugs.python.or

[issue20980] In multiprocessing.pool, ExceptionWithTraceback should derive from Exception

2014-03-23 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- resolution: -> fixed stage: test needed -> committed/rejected status: open -> closed ___ Python tracker <http://bugs.python.or

[issue20990] pyflakes: undefined names, get_context() and main(), in multiprocessing

2014-03-25 Thread Richard Oudkerk
Richard Oudkerk added the comment: Testing the is_forking() requires cx_freeze or something similar, so it really cannot go in the test suite. I have tested it manually (after spending too long trying to get cx_freeze to work with a source build). It should be noted that on Unix freezing is

[issue21078] multiprocessing.managers.BaseManager.__init__'s "serializer" argument is not documented

2014-03-29 Thread Richard Oudkerk
Richard Oudkerk added the comment: No, the argument will not go away now. However, I don't much like the API which is perhaps why I did not get round to documenting it. It does have tests. Currently 'xmlrpclib' is the only supported alternative, but JSON support could be add

[issue1191964] asynchronous Subprocess

2014-03-30 Thread Richard Oudkerk
Richard Oudkerk added the comment: Using asyncio and the IOCP eventloop it is not necessary to use threads. (Windows may use worker threads for overlapped IO, but that is hidden from Python.) See https://code.google.com/p/tulip/source/browse/examples/child_process.py for vaguely "e

[issue21116] Failure to create multiprocessing shared arrays larger than 50% of memory size under linux

2014-04-02 Thread Richard Oudkerk
Richard Oudkerk added the comment: Using truncate() to zero extend is not really portable: it is only guaranteed on XSI-compliant POSIX systems. Also, the FreeBSD man page for mmap() has the following warning: WARNING! Extending a file with ftruncate(2), thus creating a big hole, and then

[issue1191964] asynchronous Subprocess

2014-04-03 Thread Richard Oudkerk
Richard Oudkerk added the comment: I would recommended using _overlapped instead of _winapi. I intend to move multiprocessing over in future. Also note that you can do nonblocking reads by starting an overlapped read then cancelling it immediately if it fails with "incomplete". You

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit

2014-04-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: I would guess that the problem is simply that LogisticRegression objects are not picklable. Does the problem still occur if you do not use freeze? -- ___ Python tracker <http://bugs.python.org/issue21

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit

2014-04-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: Ah, I misunderstood: you meant that it freezes/hangs, not that you used a freeze tool. -- ___ Python tracker <http://bugs.python.org/issue21

[issue21162] code in multiprocessing.pool freeze if inside some code from scikit-learn (and probably liblinear) executed on ubuntu 12.04 64 Bit

2014-04-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: Could you try pickling and unpickling the result of func(): import cPickle data = cPickle.dumps(func([1,2,3]), -1) print cPickle.loads(data) -- ___ Python tracker <http://bugs.python.org/issue21

[issue1191964] asynchronous Subprocess

2014-04-09 Thread Richard Oudkerk
Richard Oudkerk added the comment: Can you explain why you write in 512 byte chunks. Writing in one chunk should not cause a deadlock. -- ___ Python tracker <http://bugs.python.org/issue1191

[issue1191964] asynchronous Subprocess

2014-04-12 Thread Richard Oudkerk
Richard Oudkerk added the comment: I added some comments. Your problem with lost data may be caused by the fact you call ov.cancel() and expect ov.pending to tell you whether the write has/will succeed. Instead you should use ov.getresult() and expect either success or an "aborted&q

[issue1191964] asynchronous Subprocess

2014-04-17 Thread Richard Oudkerk
Richard Oudkerk added the comment: If you use the short timeouts to make the wait interruptible then you can use waitformultipleobjects (which automatically waits on an extra event object) instead of waitforsingleobject. -- ___ Python tracker <h

[issue20147] multiprocessing.Queue.get() raises queue.Empty exception if even if an item is available

2014-06-08 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- assignee: -> sbt ___ Python tracker <http://bugs.python.org/issue20147> ___ ___ Python-bugs-list mailing list Unsubscrib

[issue21372] multiprocessing.util.register_after_fork inconsistency

2014-06-08 Thread Richard Oudkerk
Richard Oudkerk added the comment: register_after_fork() is intentionally undocumented and for internal use. It is only run when starting a new process using the "fork" start method whether on Windows or not -- the "fork" in its name is a hint. -- resolution:

[issue7292] Multiprocessing Joinable race condition?

2012-11-05 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- nosy: +sbt ___ Python tracker <http://bugs.python.org/issue7292> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue16389] re._compiled_typed's lru_cache causes significant degradation of the mako_v2 bench

2012-11-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: Maybe lru_cache() should have a key argument so you can specify a specialized key function. So you might have def _compile_key(args, kwds, typed): return args @functools.lru_cache(maxsize=500, key=_compile_key) def _compile(pattern

[issue16454] Mostly for discussion: _winapi as builtin for bootstrapping distutils.

2012-11-11 Thread Richard Oudkerk
Richard Oudkerk added the comment: The patch does not apply correctly against vanilla Python 3.3. I would guess that you are using a version of Python which has been patched to add mingw support. Where did you get it from? (In vanilla Python 3.3, setup.py does not contain any mention of

[issue16454] Mostly for discussion: _winapi as builtin for bootstrapping distutils.

2012-11-11 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue16454> ___ ___ Python-bugs-list mailing list Unsubscri

[issue992389] attribute error due to circular import

2012-11-14 Thread Richard Oudkerk
Richard Oudkerk added the comment: In Torsten's example from . import moduleX can be replaced with moduleX = importlib.import_module('.moduleX', __package__) (*) or moduleX = importlib.import_module('package.moduleX') If that is not pretty en

[issue16481] process handle leak on windows in multiprocessing

2012-11-15 Thread Richard Oudkerk
New submission from Richard Oudkerk: On Windows the handle for a child process is not closed when the popen/process object is garbage collected. -- messages: 175629 nosy: sbt priority: normal severity: normal stage: needs patch status: open title: process handle leak on windows in

[issue16481] process handle leak on windows in multiprocessing

2012-11-16 Thread Richard Oudkerk
Richard Oudkerk added the comment: Fixed in c574ce78cd61 and cb612c5f30cb. -- resolution: -> fixed stage: needs patch -> committed/rejected status: open -> closed ___ Python tracker <http://bugs.python.or

[issue16500] Add an 'afterfork' module

2012-11-18 Thread Richard Oudkerk
Richard Oudkerk added the comment: pthread_atfork() allows the registering of three types of callbacks: 1) prepare callbacks which are called before the fork, 2) parent callbacks which are called in the parent after the fork 3) child callbacks which are called in the child after the fork. I

[issue16500] Add an 'afterfork' module

2012-11-19 Thread Richard Oudkerk
Richard Oudkerk added the comment: Note that Gregory P. Smith has written http://code.google.com/p/python-atfork/ I also started a pure python patch but did not get round it posting it. (It also implements the fork lock idea.) I'll attach it here. How do you intend to handl

[issue16500] Add an 'afterfork' module

2012-11-20 Thread Richard Oudkerk
Richard Oudkerk added the comment: > IFF we are going to walk the hard and rocky road of exception handling, > then we are going to need at least four hooks and a register function that > takres four callables as arguments: register(prepare, error, parent, > child). Each prepare() ca

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-11-20 Thread Richard Oudkerk
Richard Oudkerk added the comment: http://hg.python.org/sandbox/sbt#spawn now contains support for starting processes via a separate server process. This depends on fd passing support. This also solves the problem of mixing threads and processes, but is much faster than using fork+exec. It

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

2012-11-26 Thread Richard Oudkerk
Richard Oudkerk added the comment: I am reopening this issue because 26bbff4562a7 only dealt with objects which cannot be pickled. But CalledProcessError instances *can* be pickled: the problem is that the resulting data cannot be unpickled. Note that in Python 3.3 CalledProcessError can be

[issue16558] multiprocessing fails to raise exception with parameters

2012-11-26 Thread Richard Oudkerk
Richard Oudkerk added the comment: The example works correctly on 3.3 because of #1692335. I am not sure if it is appropriate to backport it though. This is a duplicate of #9400 which I have assigned to myself. (I had thought it was already fixed.) -- resolution: -> duplic

[issue16572] Bad multi-inheritance support in some libs like threading or multiprocessing

2012-11-28 Thread Richard Oudkerk
Richard Oudkerk added the comment: The patch is liable to break programs which explicitly call base constructors because the constructor will be called more than once. It also assumes that the __init__() method of all base classes should be called with no arguments (other than self). That is

[issue16572] Bad multi-inheritance support in some libs like threading or multiprocessing

2012-11-28 Thread Richard Oudkerk
Richard Oudkerk added the comment: > But I think the problem remains: do you agree that Classes should include > a super() call in their __init__ ? No, I don't. I think super() is an attractive nuisance which is difficult to use correctly in an __init__() method, except in the t

[issue16579] .pyw disturb multiprocessing behavior

2012-12-02 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- nosy: +sbt ___ Python tracker <http://bugs.python.org/issue16579> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue16507] Patch selectmodule.c to support WSAPoll on Windows

2012-12-03 Thread Richard Oudkerk
Richard Oudkerk added the comment: Attached is an alternative patch which only touches selectmodule.c. It still does not support WinXP. Note that in this version register() and modify() do not ignore the POLLPRI flag if it was *explicitly* passed. But I am not sure how best to deal with

[issue16507] Patch selectmodule.c to support WSAPoll on Windows

2012-12-04 Thread Richard Oudkerk
Richard Oudkerk added the comment: Here is a version which loads WSAPoll at runtime. Still no tests or docs. -- Added file: http://bugs.python.org/file28207/runtime_wsapoll.patch ___ Python tracker <http://bugs.python.org/issue16

[issue16616] test_poll.PollTests.poll_unit_tests() is dead code

2012-12-05 Thread Richard Oudkerk
New submission from Richard Oudkerk: PollTests.poll_unit_tests() is not run because its method name does not begin with "test". It looks it was accidentally disabled when test_poll() was converted to unittest in f56b25168142. Renaming it test_poll_unit_tests() makes it run succes

[issue16507] Patch selectmodule.c to support WSAPoll on Windows

2012-12-07 Thread Richard Oudkerk
Richard Oudkerk added the comment: It seems that the return code of WSAPoll() does not include the count of array items with revents == POLLNVAL. In the case where all of them are POLLNVAL, instead of returning 0 (which usually indicates a timeout) it returns -1 and WSAGetLastError

[issue16616] test_poll.PollTests.poll_unit_tests() is dead code

2012-12-10 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue16616> ___ ___ Python-bugs-list mailing list Unsubscri

[issue15526] test_startfile crash on Windows 7 AMD64

2012-12-10 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue15526> ___ ___ Python-bugs-list mailing list Unsubscri

[issue16507] Patch selectmodule.c to support WSAPoll on Windows

2012-12-16 Thread Richard Oudkerk
Richard Oudkerk added the comment: Here is a new version with tests and docs. Note that the docs do not mention the bug mentioned in http://daniel.haxx.se/blog/2012/10/10/wsapoll-is-broken/ Maybe they should? Note that that bug makes it a bit difficult to use poll with tulip on Windows

[issue16718] Mysterious atexit fail

2012-12-18 Thread Richard Oudkerk
Richard Oudkerk added the comment: When you run wy.py the wow module gets partially imported, and then garbage collected because it fails to import successfully. The destructor for the module replaces values in the module's __dict__ with None. So when the cleanup function runs you ge

[issue16718] Mysterious atexit fail

2012-12-19 Thread Richard Oudkerk
Richard Oudkerk added the comment: > Things should be better in the future, when modules are cleared with true > garbage collection. When is this future of which you speak? I am not sure whether it would affect performance, but a weakrefable subclass of dict could be used for module

[issue16718] Mysterious atexit fail

2012-12-19 Thread Richard Oudkerk
Richard Oudkerk added the comment: > See issue812369 for the shutdown procedure and modules cleanup. I am aware of that issue, but the original patch is 9 years old. Which is why I ask if/when it will actually happen. -- ___ Python tracker &l

[issue16736] select.poll() converts long to int without checking for overflow

2012-12-20 Thread Richard Oudkerk
New submission from Richard Oudkerk: Relevant code: int timeout = 0, poll_result, i, j; ... tout = PyNumber_Long(tout); if (!tout) return NULL; timeout = PyLong_AsLong(tout); <-- implicit cast to int -- messages: 177811 nosy: sbt prior

[issue16736] select.poll() converts long to int without checking for overflow

2012-12-20 Thread Richard Oudkerk
Richard Oudkerk added the comment: Thanks. I will close. -- stage: -> committed/rejected status: open -> closed ___ Python tracker <http://bugs.python.org/i

[issue16718] Mysterious atexit fail

2012-12-20 Thread Richard Oudkerk
Richard Oudkerk added the comment: Perhaps the simplest thing would be to stop doing anything special when a module is garbage collected: the garbage collector can take care of any orphaned ref-cycles involving the module dict. Then at shutdown the remaining modules in sys.modules could have

[issue16743] mmap accepts files > 1 GB, but processes only 1 GB

2012-12-24 Thread Richard Oudkerk
Richard Oudkerk added the comment: I suspect that the size of the 5GB file is originally a 64 bit quantity, but gets cast unsafely to a 32 bit size_t to give 1GB. This is causing the miscalculations. There is no way to map all of a 5GB file in a 32 bit process -- 4GB is the maximum -- so

[issue16743] mmap accepts files > 1 GB, but processes only 1 GB

2012-12-24 Thread Richard Oudkerk
Richard Oudkerk added the comment: This bit looks wrong to me: if (offset - size > PY_SSIZE_T_MAX) /* Map area too large to fit in memory */ m_obj->size = (Py_ssize_t) -1; Should it not be "size - offset" instead of "offset - siz

[issue16743] mmap accepts files > 1 GB, but processes only 1 GB

2012-12-26 Thread Richard Oudkerk
Richard Oudkerk added the comment: On 32 bit Unix mmap() will raise ValueError("mmap length is too large") in Marc's example. This is correct since Python's sequence protocol does not support indexes larger than sys.maxsize. But on 32 bit Windows, if length == 0 then t

[issue16743] mmap accepts files > 1 GB, but processes only 1 GB

2012-12-26 Thread Richard Oudkerk
Richard Oudkerk added the comment: > This change is not backward compatible. Now user can mmap a larger file > and safely access lower 2 GiB. With the patch it will fail. They should specify length=2GiB-1 if that is what they want. With length=0 you can only access the lower 2

[issue16743] mmap accepts files > 1 GB, but processes only 1 GB

2012-12-26 Thread Richard Oudkerk
Richard Oudkerk added the comment: New patch with same check for Unix. -- Added file: http://bugs.python.org/file28446/mmap.patch ___ Python tracker <http://bugs.python.org/issue16

[issue16743] mmap accepts files > 1 GB, but processes only 1 GB

2012-12-26 Thread Richard Oudkerk
Richard Oudkerk added the comment: > Isn't 2 GiB + 1 bytes mmap file enough for testing? Yes. But creating multigigabyte files is very slow on Windows. On Linux/FreeBSD test_mmap takes a fraction of a second, whereas on Windows it takes over 2 minutes. (Presumably Linux/Fr

[issue16743] mmap on Windows can mishandle files larger than sys.maxsize

2012-12-26 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- title: mmap accepts files > 1 GB, but processes only 1 GB -> mmap on Windows can mishandle files larger than sys.maxsize type: enhancement -> behavior ___ Python tracker <http://bugs.python.or

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Richard Oudkerk
Richard Oudkerk added the comment: > Richard, apart from performance, what's the advantage of this approach over > the > fork+exec version? It is really just performance. For context running the unittests in a 1 cpu linux VM gives me fork: real0m53.868s user

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Richard Oudkerk
Richard Oudkerk added the comment: Numbers when running on Linux on a laptop with 2 cores + hyperthreading. RUNNING UNITTESTS: fork: real0m50.687s user0m9.213s sys 0m4.012s fork+exec: real1m9.062s user0m48.579s sys 0m6.648s forkserver

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Richard Oudkerk
Changes by Richard Oudkerk : Added file: http://bugs.python.org/file28461/8f08d83264a0.diff ___ Python tracker <http://bugs.python.org/issue8713> ___ ___ Python-bug

[issue8713] multiprocessing needs option to eschew fork() under Linux

2012-12-27 Thread Richard Oudkerk
Richard Oudkerk added the comment: > The safest default would be fork+exec though we need to implement the > fork+exec code as a C extension module or have it use subprocess (as I > noted in the mb_fork_exec.patch review). That was an old version of the patch. In the branch

[issue16802] fileno argument to socket.socket() undocumented

2012-12-28 Thread Richard Oudkerk
New submission from Richard Oudkerk: The actual signature is socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None) but the documented signature is socket.socket([family[, type[, proto]]]) Should the fileno argument be documented or is it considered an implementation

[issue16802] fileno argument to socket.socket() undocumented

2012-12-31 Thread Richard Oudkerk
Richard Oudkerk added the comment: > The "fileno" argument looks like an implementation detail to me. It has at least one potential use. On Windows socket.detach() returns a socket handle but there is no documented way to close it -- os.close() will not work. The only way to cl

[issue16802] fileno argument to socket.socket() undocumented

2012-12-31 Thread Richard Oudkerk
Richard Oudkerk added the comment: > There is an alternative (documented) interface: > > socket.fromfd(handle, socket.AF_INET, socket.SOCK_STREAM).close() socket.fromfd() duplicates the handle, so that does not close the origina

[issue9586] "warning: comparison between pointer and integer" in multiprocessing build on Tiger

2013-01-01 Thread Richard Oudkerk
Changes by Richard Oudkerk : -- resolution: -> fixed stage: -> committed/rejected status: open -> closed ___ Python tracker <http://bugs.python.o

[issue12105] open() does not able to set flags, such as O_CLOEXEC

2013-01-03 Thread Richard Oudkerk
Richard Oudkerk added the comment: Note that on Windows there is an O_NOINHERIT flag which almost corresponds to O_CLOEXEC on Linux. I don't think there is a need to use the win32 api. -- nosy: +sbt ___ Python tracker <http://bugs.py

[issue12939] Add new io.FileIO using the native Windows API

2013-01-03 Thread Richard Oudkerk
Richard Oudkerk added the comment: A while ago I did write a PipeIO class which subclasses io.RawIOBase and works for overlapped pipe handles. (It was intended for multiprocessing and doing asynchronous IO with subprocess.) As it is it would not work with normal files because when you do

[issue12939] Add new io.FileIO using the native Windows API

2013-01-03 Thread Richard Oudkerk
Richard Oudkerk added the comment: Attached is a module for Python 3.3+ which subclasses io.RawIOBase. The constructor signature is WinFileIO(handle, mode="r", closehandle=True) where mode is "r", "w", "r+" or "w+". Handles can be

[issue12939] Add new io.FileIO using the native Windows API

2013-01-03 Thread Richard Oudkerk
Changes by Richard Oudkerk : Added file: http://bugs.python.org/file28545/test_winfileio.py ___ Python tracker <http://bugs.python.org/issue12939> ___ ___ Python-bug

[issue16873] increase epoll.poll() maxevents default value, and improve documentation

2013-01-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: Is this actually a problem? If events are arranged in a queue and epoll_wait() just removes the oldest events (up to maxevents) from that queue then there would be no problem with using a small value for maxevents. I don't *know* if that is the case,

[issue12939] Add new io.FileIO using the native Windows API

2013-01-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: Attached is a patch which adds a winio module which is a replacement for io, but uses windows handles instead of fds. It reimplements FileIO and open(), and provides openhandle() and closehandle() as replacements for os.open() and os.close(). test_io has

[issue16873] increase epoll.poll() maxevents default value, and improve documentation

2013-01-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: > I actually wrote a script to reproduce this issue: The program does *not* demonstrate starvation because you are servicing the resource represented by the "starved" duplicate fds before calling poll() again. You are creating thousands of dupl

[issue16873] increase epoll.poll() maxevents default value, and improve documentation

2013-01-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: > The fact that that the FDs are duped shouldn't change anything to the > events reported: it works while the number of FDs is less than > FD_SETSIZE (epoll_wait() maxevents argument). That assumes that epoll_wait() is supposed to return *all* r

[issue16873] increase epoll.poll() maxevents default value, and improve documentation

2013-01-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: Here is a version which uses epoll to service a number of pipes which is larger than maxevents. (If NUM_WRITERS is too large then I get "OSError: [Errno 24] Too many open files".) All pipes get serviced and the output is: Working with 20 FDs, 5

[issue16873] increase epoll.poll() maxevents default value, and improve documentation

2013-01-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: > Yes, but the problem is that between two epoll_wait() calls, the > readiness of the FDs can have changed: and if that happens, you'll get > the same list over and over. If an fd *was* ready but isn't anymore then why would you want to know

[issue12939] Add new io.FileIO using the native Windows API

2013-01-06 Thread Richard Oudkerk
Richard Oudkerk added the comment: > I don't like the idea of a specific I/O module for an OS. Is the public API > different? It was partly to make integration with the existing tests easier: _io, _pyio and winio are tested in parallel. > Can't you reuse the io module?

[issue16853] add a Selector to the select module

2013-01-08 Thread Richard Oudkerk
Richard Oudkerk added the comment: > Richard, in Tulip's WSAPoll code, it reads: > > class WindowsPollPollster(PollPollster): > """Pollster implementation using WSAPoll. > > WSAPoll is only available on Windows Vista and later. Pytho

[issue16261] Fix bare excepts in various places in std lib

2013-01-09 Thread Richard Oudkerk
Richard Oudkerk added the comment: > try: > _MAXFD = os.sysconf("SC_OPEN_MAX") > -except: > +except ValueError: > _MAXFD = 256 os.sysconf() might raise OSError. I think ValueError is only raised if _SC_OPEN_MAX was undefined when the module was compile

[issue16920] multiprocessing.connection listener gets MemoryError on recv

2013-01-10 Thread Richard Oudkerk
Richard Oudkerk added the comment: Why are you connecting to a multiprocessing listener with a raw socket? You should be using multiprocessing.connection.Client to create a client connection. Connection.send(obj) writes a 32 bit unsigned int (in network order) to the socket representing the

[issue12939] Add new io.FileIO using the native Windows API

2013-01-12 Thread Richard Oudkerk
Richard Oudkerk added the comment: Attached is a new patch which is implemented completely in C. It adds a WinFileIO class to the io module, which has the same API as FileIO except that: * It has a handle attribute instead of a fileno() method. * It has staticmethods openhandle() and

[issue12939] Add new io.FileIO using the native Windows API

2013-01-12 Thread Richard Oudkerk
Changes by Richard Oudkerk : Removed file: http://bugs.python.org/file28544/winfileio.c ___ Python tracker <http://bugs.python.org/issue12939> ___ ___ Python-bugs-list m

[issue12939] Add new io.FileIO using the native Windows API

2013-01-12 Thread Richard Oudkerk
Changes by Richard Oudkerk : Removed file: http://bugs.python.org/file28545/test_winfileio.py ___ Python tracker <http://bugs.python.org/issue12939> ___ ___ Python-bug

[issue12939] Add new io.FileIO using the native Windows API

2013-01-12 Thread Richard Oudkerk
Changes by Richard Oudkerk : Removed file: http://bugs.python.org/file28590/winfileio.patch ___ Python tracker <http://bugs.python.org/issue12939> ___ ___ Python-bug

[issue12939] Add new io.FileIO using the native Windows API

2013-01-12 Thread Richard Oudkerk
Richard Oudkerk added the comment: Forgot to mention, the handles are non-inheritable. You can use _winapi.DuplicateHandle() to create an inheritable duplicate handle if you really need to. -- ___ Python tracker <http://bugs.python.org/issue12

<    1   2   3   4   5   6   7   8   >