Re: [Python-Dev] Socket timeout and completion based sockets
This is getting off-topic, but: CancelIoEx() is a new api that I wasn't aware of (my IOCP solution dates back to 2005). It appears that this can be used, although the documentation is sketchy. This worries me: If the file handle is associated with a completion port, an I/O completion packet is not queued to the port if a synchronous operation is successfully canceled. For asynchronous operations still pending, the cancel operation will queue an I/O completion packet. This _might_ mean that synchronizing the cancel request with a callback that expects to be called, but then, may not be called, can be difficult. It is possible that MS didn't think their API completely through (nothing new here.) Anyway, that is somewhat beside the point Even if we can cancel an ongoing operation there needs to be synchronization, so that any data that is received(or sent) is correctly communicated ot the app. If there isn't a cancel option in the API (not talking about windows here) then this would mean queueing up data for recv(), and no simple solution for send(). So, basically, what I'm saying is, that enabling re-startable socket timeout semantics for sockets implemented with completion semantics, rather than ready semantics _can_ be difficult, hence my question. -Original Message- From: Python-Dev [mailto:python-dev- bounces+kristjan=ccpgames@python.org] On Behalf Of Richard Oudkerk Sent: 26. nóvember 2012 16:05 To: python-dev@python.org Subject: Re: [Python-Dev] Socket timeout and completion based sockets On 26/11/2012 11:49am, Kristján Valur Jónsson wrote: However, other implementations of python sockets, e.g. ones that rely on IO completion, may not have the luxury of using select. For example, on Windows, there is no way to abort an IOCP socket call, so a timeout must be implemented by aborting the wait. Dealing with the resulting race can be an interesting challenge. I am not quite sure what you mean by aborting the wait. But you can abort an overlapped operation using CancelIo()/CancelIoEx(). I have just done some experimenting. Using CancelIo()/CancelIoEx() to abort an operation started with WSARecv() does not seem to cause a problem -- you just call GetOverlappedResult() afterwards to check whether the operation completed before it could be aborted. But aborting an operation started with WSASend() sometimes seems to break the connection: a subsequent WSARecv()/WSASend() will fail with WSAECONNABORTED or WSAECONNRESET depending on which end of the connection you are on. So, as you say, if you abort a send then you cannot expect to successfully resend the data later. -- Richard ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python- dev/kristjan%40ccpgames.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket timeout and completion based sockets
Yes, well, as a matter of fact, I do have an IOCP based socket implementation with stackless python. From the programmer's perspective, operations appear blocking while IOCP is used to switch tasklets in the background. But my socket timeout implementation does not guarantee that the socket is left in a valid state for retrying a recv() operation. I was under the (perhaps mistaken) impression that the current async work _could_ result in a standardized way to create such alternative socket implementations, ones that might do their magic using greenlets, tasklets, or generators. But if that were the case, such loosely defined features of the socket API would need clearer definitions. K -Original Message- From: gvanros...@gmail.com [mailto:gvanros...@gmail.com] On Behalf Of Guido van Rossum Sent: 26. nóvember 2012 15:59 To: Kristján Valur Jónsson Cc: Python-Dev (python-dev@python.org) Subject: Re: [Python-Dev] Socket timeout and completion based sockets If you're talking about the standard socket module, I'm not aware that it uses IOCP on Windows. Are you asking this just in the abstract, or do you know of a Python implementation that uses IOCP to implement the standard socket type? As to the design of the async I/O library (which I am still working on!), I cannot guarantee anything, and the issue will probably be moot -- the API won't have the same kind of timeout as the current socket object (it will have other ways to set deadlines though). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tru64 support
On Tue, Sep 04, 2012 at 02:24:10AM -0700, Andrew Svetlov wrote: Is it still up to date? Bug has been created in 2004. I don't see Tru64 in list of available buildbots. Do we need to care about this platform? And how to make sure what existing code works fine for that? There's a Snakebite Tru64 box now, accessible via t5 from the sb menu. 2.7 compiles fine but hangs on some of the subprocess epipe tests, I'll set up a slave once I've addressed that. One nice thing about Tru64 is that everything is implicitly 64-bit. There's no notion of dual 32-bit/64-bit libs/binaries or a 32-bit memory architecture. The 64-bit `python` on Tru64 is in a much better state than said binaries on AIX and HP-UX (which still need a lot of work). As for VMS/OpenVMS, still working with HP to get access to media. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket timeout and completion based sockets
On 27/11/2012 9:35am, Kristján Valur Jónsson wrote: This worries me: If the file handle is associated with a completion port, an I/O completion packet is not queued to the port if a synchronous operation is successfully canceled... I think you can only abort a synchronous operation if you use CancelIoEx() from a different thread. That won't be an issue if you do all your IO stuff in one thread. -- Richard ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket timeout and completion based sockets
On Tue, Nov 27, 2012 at 1:23 AM, Kristján Valur Jónsson krist...@ccpgames.com wrote: Yes, well, as a matter of fact, I do have an IOCP based socket implementation with stackless python. It would have been nice if you had given more context and stated your objective upfront instead of asking what appeared to be an obscure question about a technical detail. I have been distracted by other stuff temporarily, but I do plan to continue working on tulip more, and I want to assure you that EVERYTHING IS STILL OPEN. I.e. don't take the current tulip code base as a draft PEP -- it is just my way of learning about the issues. Also note that Richard Oudkerk has developed an alternative, which you can find in the proactor branch of the tulip repository. From the programmer's perspective, operations appear blocking while IOCP is used to switch tasklets in the background. But my socket timeout implementation does not guarantee that the socket is left in a valid state for retrying a recv() operation. I was under the (perhaps mistaken) impression that the current async work _could_ result in a standardized way to create such alternative socket implementations, ones that might do their magic using greenlets, tasklets, or generators. But if that were the case, such loosely defined features of the socket API would need clearer definitions. If you have a specific requirement, please state it explicitly, rather than just hoping tulip will be your solution. FWIW, it is likely that tulip will hide the actual socket object completely inside a higher-level abstraction, so that the actual semantics of sockets (which are extremely platform-dependent) cannot affect the specification of the API. There will be platform-specific ways to reach inside, but they will be of limited use -- it's possible that on Windows (at least when using IOCP) there won't be a socket object at all. Finally, I am not at all interested in greenlets(*). Tulip will support two API surfaces: a low-level one, suitable for interfacing with e.g. Twisted or Tornado, that uses callbacks to signal ready-ness or completion. And a high-level one, useful for e.g. writing protocol handling libraries and applications, that will somehow use yield and/or yield-from. The details of each layer are very much unspecified at this point. NOW WOULD BE A GOOD TIME TO WRITE DOWN YOUR REQUIREMENTS. (*) Greenlets are a fine mechanism for some application areas, but ultimately not fit for the standard library, and they have some significant downsides. --Guido K -Original Message- From: gvanros...@gmail.com [mailto:gvanros...@gmail.com] On Behalf Of Guido van Rossum Sent: 26. nóvember 2012 15:59 To: Kristján Valur Jónsson Cc: Python-Dev (python-dev@python.org) Subject: Re: [Python-Dev] Socket timeout and completion based sockets If you're talking about the standard socket module, I'm not aware that it uses IOCP on Windows. Are you asking this just in the abstract, or do you know of a Python implementation that uses IOCP to implement the standard socket type? As to the design of the async I/O library (which I am still working on!), I cannot guarantee anything, and the issue will probably be moot -- the API won't have the same kind of timeout as the current socket object (it will have other ways to set deadlines though). -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New Contributor Experience in Python and other FOSS Communities - A Survey
On Mon, Nov 19, 2012 at 9:01 AM, Brian Curtin br...@python.org wrote: Hi all, Along with a number of other free and open communities, Python is being included in a survey of new contributors since January 2010. The survey is being done by Kevin Carillo, a PhD student at Victoria University of Wellington who is studying various approaches that projects use to gain and work with new contributors. If you have written patches, reviewed issues, or done anything to contribute to the development of Python and you started this after January 2010, I hope you can spare around 20 minutes to complete this survey: https://limesurvey.sim.vuw.ac.nz/index.php?sid=65151lang=en There's a longer blog post up at http://blog.python.org/2012/11/new-contributor-experience-in-python.html if you would like a bit more information. On behalf of Kevin Carillo, I thank you for your time and consideration of this survey. Brian Curtin Just a quick reminder - Kevin could really use the help if any of you recent newcomers can spare the time. I took the survey myself since I became a committer right after the survey range begins, and I think it took me 15 minutes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Handling support for newer OS features at run time
The hackiest part of my WSAPoll patch is this: --- a/PC/pyconfig.h Sun Nov 18 10:42:42 2012 + +++ b/PC/pyconfig.h Sun Nov 18 17:27:29 2012 -0500 @@ -158,12 +158,12 @@ /* set the version macros for the windows headers */ #ifdef MS_WINX64 /* 64 bit only runs on XP or greater */ -#define Py_WINVER 0x0501 /* _WIN32_WINNT_WINXP */ -#define Py_NTDDI NTDDI_WINXP +#define Py_WINVER 0x0600 /* _WIN32_WINNT_WINXP */ +#define Py_NTDDI NTDDI_WIN6 #else /* Python 2.6+ requires Windows 2000 or greater */ -#define Py_WINVER 0x0500 /* _WIN32_WINNT_WIN2K */ -#define Py_NTDDI NTDDI_WIN2KSP4 +#define Py_WINVER 0x0600 /* _WIN32_WINNT_WIN2K */ +#define Py_NTDDI NTDDI_WIN6 #endif Basically, because of the way our build is currently configured, I had to bump the minimum supported Windows version from XP to Vista just to get access to the WSAPoll headers. (Issue: http://bugs.python.org/issue16507 Patch: http://bugs.python.org/file28038/wsapoll.patch) Ideally, a Windows binary should make WSAPoll/select.poll() available if running on Vista or above, without impacting the ability to run on XP. I don't think we've currently got the ability to do this, right? Is there a precedent set anywhere else? I suspect it's not as much of an issue on *NIX platforms as you'll typically compile from source. Windows, not so much. Thoughts? A single binary that dynamically loads applicable modules seems cleaner but will obviously take more work. Other approach would be to start distributing multiple versions of Python based on the underlying Windows version. Not the nicest approach. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Handling support for newer OS features at run time
Am 27.11.2012 23:49, schrieb Trent Nelson: I don't think we've currently got the ability to do this, right? Is there a precedent set anywhere else? I suspect it's not as much of an issue on *NIX platforms as you'll typically compile from source. Windows, not so much. Thoughts? A single binary that dynamically loads applicable modules seems cleaner but will obviously take more work. Other approach would be to start distributing multiple versions of Python based on the underlying Windows version. Not the nicest approach. Usually I have to build a python package on a build daemon running the kernel of the latest stable (or latest stable long term support) release. Thus if a configure test checks the current kernel, then you may get an unexpected answer and a missing feature. It is somehow different that I already build different binaries (per release), but the hard-coding of kernel features during configure time looks like the same issue. Currently working around it by patching configure to remove the test and hard-code it for a particular build. Another solution maybe would to have something like --enable-kernel=version (as found in glibc), and hope that you can deduce the features from the kernel version. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Handling support for newer OS features at run time
On 27/11/2012 10:49pm, Trent Nelson wrote: Ideally, a Windows binary should make WSAPoll/select.poll() available if running on Vista or above, without impacting the ability to run on XP. I assume you can do something like int WSAAPI (*pWSAPoll)(WSAPOLLFD *, ULONG, INT); HINSTANCE hWinsock2 = GetModuleHandle(WS2_32); *(FARPROC *)pWSAPoll = GetProcAddress(hWinsock2, WSAPoll); if (pWSAPoll == NULL) ... to get a function pointer for WSAPoll on versions which support it. Modules/_winapi.c does something similar to get CancelIoEx. -- Richard ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Handling support for newer OS features at run time
On Tue, Nov 27, 2012 at 03:09:12PM -0800, Matthias Klose wrote: Am 27.11.2012 23:49, schrieb Trent Nelson: I don't think we've currently got the ability to do this, right? Is there a precedent set anywhere else? I suspect it's not as much of an issue on *NIX platforms as you'll typically compile from source. Windows, not so much. Thoughts? A single binary that dynamically loads applicable modules seems cleaner but will obviously take more work. Other approach would be to start distributing multiple versions of Python based on the underlying Windows version. Not the nicest approach. Usually I have to build a python package on a build daemon running the kernel of the latest stable (or latest stable long term support) release. Thus if a configure test checks the current kernel, then you may get an unexpected answer and a missing feature. It is somehow different that I already build different binaries (per release), but the hard-coding of kernel features during configure time looks like the same issue. Currently working around it by patching configure to remove the test and hard-code it for a particular build. Another solution maybe would to have something like --enable-kernel=version (as found in glibc), and hope that you can deduce the features from the kernel version. Hmmm. How often do Linux kernel versions expose new features that we can make use of in Python? i.e. do you run into this problem often? Can you cite some recent examples? I'm not sure how much could be shared between Windows and Linux with what you've just described. With Windows, specifically with regards to providing dynamic select.poll() support, I was thinking of having multiple modules: Python33\ DLLs\ select.pyd select_win6.pyd select_win7.pyd select_win8.pyd And Python would automatically import the appropriate one. I don't think this would be that useful on Linux -- as you say, the decision is typically made at ./configure time. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Handling support for newer OS features at run time
On Tue, Nov 27, 2012 at 03:14:00PM -0800, Richard Oudkerk wrote: On 27/11/2012 10:49pm, Trent Nelson wrote: Ideally, a Windows binary should make WSAPoll/select.poll() available if running on Vista or above, without impacting the ability to run on XP. I assume you can do something like int WSAAPI (*pWSAPoll)(WSAPOLLFD *, ULONG, INT); HINSTANCE hWinsock2 = GetModuleHandle(WS2_32); *(FARPROC *)pWSAPoll = GetProcAddress(hWinsock2, WSAPoll); if (pWSAPoll == NULL) ... to get a function pointer for WSAPoll on versions which support it. Ah! Good point, that's probably a much better approach in this instance, given I only need one or two more symbols. Thanks! Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way)
On Thu, Nov 22, 2012 at 5:09 AM, Nick Coghlan ncogh...@gmail.com wrote: On Thu, Nov 22, 2012 at 9:44 PM, Kristján Valur Jónsson krist...@ccpgames.com wrote: Where in the tracker? I tried searching but didn’t find it. This one: http://bugs.python.org/issue13475 This and the issue about being able to configure coverage.py cleanly in subprocesses (http://bugs.python.org/issue14803) are the two issues which have got me thinking about the interpreter startup and configuration process in general lately. Both would add *more* complexity to an already complicated and hard to follow initialisation sequence, so I now think we should be look at opportunities for rationalisation *first*, before we make things even messier. There's also http://bugs.python.org/issue15716 (Ability to specify the PYTHONPATH via a command line flag). -eric ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way)
On Tue, Nov 27, 2012 at 9:19 PM, Eric Snow ericsnowcurren...@gmail.com wrote: On Thu, Nov 22, 2012 at 5:09 AM, Nick Coghlan ncogh...@gmail.com wrote: On Thu, Nov 22, 2012 at 9:44 PM, Kristján Valur Jónsson krist...@ccpgames.com wrote: Where in the tracker? I tried searching but didn’t find it. This one: http://bugs.python.org/issue13475 This and the issue about being able to configure coverage.py cleanly in subprocesses (http://bugs.python.org/issue14803) are the two issues which have got me thinking about the interpreter startup and configuration process in general lately. Both would add *more* complexity to an already complicated and hard to follow initialisation sequence, so I now think we should be look at opportunities for rationalisation *first*, before we make things even messier. There's also http://bugs.python.org/issue15716 (Ability to specify the PYTHONPATH via a command line flag). -eric I knew there was one more: http://bugs.python.org/issue16499 (CLI option for isolated mode). -eric ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Handling support for newer OS features at run time
On Tue, Nov 27, 2012 at 3:19 PM, Trent Nelson tr...@snakebite.org wrote: On Tue, Nov 27, 2012 at 03:09:12PM -0800, Matthias Klose wrote: Am 27.11.2012 23:49, schrieb Trent Nelson: I don't think we've currently got the ability to do this, right? Is there a precedent set anywhere else? I suspect it's not as much of an issue on *NIX platforms as you'll typically compile from source. Windows, not so much. Thoughts? A single binary that dynamically loads applicable modules seems cleaner but will obviously take more work. Other approach would be to start distributing multiple versions of Python based on the underlying Windows version. Not the nicest approach. Usually I have to build a python package on a build daemon running the kernel of the latest stable (or latest stable long term support) release. Thus if a configure test checks the current kernel, then you may get an unexpected answer and a missing feature. It is somehow different that I already build different binaries (per release), but the hard-coding of kernel features during configure time looks like the same issue. Currently working around it by patching configure to remove the test and hard-code it for a particular build. Another solution maybe would to have something like --enable-kernel=version (as found in glibc), and hope that you can deduce the features from the kernel version. Hmmm. How often do Linux kernel versions expose new features that we can make use of in Python? i.e. do you run into this problem often? Can you cite some recent examples? Here's an example of using the pipe2() system call. The code is only compiled in if the C library supports it. If the C library supports it, the system call can still return an error because the running kernel doesn't support it (ENOSYS). In that case it falls back to the legacy code: http://hg.python.org/cpython/file/618ea5612e83/Modules/_posixsubprocess.c#l738 -gps I'm not sure how much could be shared between Windows and Linux with what you've just described. With Windows, specifically with regards to providing dynamic select.poll() support, I was thinking of having multiple modules: Python33\ DLLs\ select.pyd select_win6.pyd select_win7.pyd select_win8.pyd And Python would automatically import the appropriate one. I don't think this would be that useful on Linux -- as you say, the decision is typically made at ./configure time. Trent. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Handling support for newer OS features at run time
Am 28.11.2012 06:37, schrieb Gregory P. Smith: On Tue, Nov 27, 2012 at 3:19 PM, Trent Nelson tr...@snakebite.org wrote: On Tue, Nov 27, 2012 at 03:09:12PM -0800, Matthias Klose wrote: Am 27.11.2012 23:49, schrieb Trent Nelson: I don't think we've currently got the ability to do this, right? Is there a precedent set anywhere else? I suspect it's not as much of an issue on *NIX platforms as you'll typically compile from source. Windows, not so much. Thoughts? A single binary that dynamically loads applicable modules seems cleaner but will obviously take more work. Other approach would be to start distributing multiple versions of Python based on the underlying Windows version. Not the nicest approach. Usually I have to build a python package on a build daemon running the kernel of the latest stable (or latest stable long term support) release. Thus if a configure test checks the current kernel, then you may get an unexpected answer and a missing feature. It is somehow different that I already build different binaries (per release), but the hard-coding of kernel features during configure time looks like the same issue. Currently working around it by patching configure to remove the test and hard-code it for a particular build. Another solution maybe would to have something like --enable-kernel=version (as found in glibc), and hope that you can deduce the features from the kernel version. Hmmm. How often do Linux kernel versions expose new features that we can make use of in Python? i.e. do you run into this problem often? Can you cite some recent examples? Here's an example of using the pipe2() system call. The code is only compiled in if the C library supports it. If the C library supports it, the system call can still return an error because the running kernel doesn't support it (ENOSYS). In that case it falls back to the legacy code: http://hg.python.org/cpython/file/618ea5612e83/Modules/_posixsubprocess.c#l738 another one is the check for working semaphores for the multiprocessing module, seen at https://launchpad.net/bugs/630511 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
