[issue6560] socket sendmsg(), recvmsg() methods
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset e64bec91ac91 by Richard Oudkerk in branch 'default': Issue #14669: Skip multiprocessing connection pickling test on MacOSX http://hg.python.org/cpython/rev/e64bec91ac91 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Andrew Grover andy.gro...@gmail.com: -- nosy: -Andrew.Grover ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Bill Janssen bill.jans...@gmail.com added the comment: I'm guessing these things are due to interaction with some Apple security update, as the buildbots were working well 8 months ago. Bill On Wed, Sep 7, 2011 at 4:01 PM, Nick Coghlan rep...@bugs.python.org wrote: Nick Coghlan ncogh...@gmail.com added the comment: If Bill gets a chance to investigate this before the weekend, great, otherwise my plan to stop making noise in the buildbot results will be to: 1. Create a separate issue specifically for the errors reported by the Mac OS X buildbots (allowing the problem to be spelled out more clearly for readers, and also allowing the feature request itself to be closed) 2. Flag the offending tests as expected failures on Mac OS X, with a pointer back to the new tracker issue. That way, if these failures are due to underlying OS bugs or limitations (as they appear to be), we'll get a clear indication in the buildbots when Apple have fixed the relevant problems. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: The feature patch for sendmsg/recvmsg support came with a swathe of new tests, and the failures are in those new tests rather than anything breaking in the old ones. As Charles-François noted though, it doesn't look like the feature implementation itself is doing anything wrong, just that there are limits to what Mac OS X allows us to do with it (hence why I closed this feature request and opened issue #12958 to cover the task of updating the test suite to accurately reflect that situation). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Michael Schurter m...@schmichael.com: -- nosy: +schmichael ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: Closing the feature request as complete. The remaining Mac OS X buildbot issues now have their own tracker item: #12958 -- resolution: - fixed stage: needs patch - committed/rejected status: open - closed superseder: - test_socket failures on Mac OS X ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: If Bill gets a chance to investigate this before the weekend, great, otherwise my plan to stop making noise in the buildbot results will be to: 1. Create a separate issue specifically for the errors reported by the Mac OS X buildbots (allowing the problem to be spelled out more clearly for readers, and also allowing the feature request itself to be closed) 2. Flag the offending tests as expected failures on Mac OS X, with a pointer back to the new tracker issue. That way, if these failures are due to underlying OS bugs or limitations (as they appear to be), we'll get a clear indication in the buildbots when Apple have fixed the relevant problems. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Bill Janssen bill.jans...@gmail.com added the comment: I'll take a look at this next week, when I'm more on-line again. Bill 2011/8/25 Charles-François Natali rep...@bugs.python.org: Charles-François Natali neolo...@free.fr added the comment: The OS X buildbots show some failures: It seems to fail consistently on every OS X version. I've had another look both at the code and the test, and couldn't find anything wrong with it. Since there are a number of known bugs pertaining to FD passing on OS X - even on recent versions - I'd suggest to skip those tests on this platform. As for sendto/sendmsg not being interrupted by the signal, I'd be curious to see if running test_socket alone solves the problem (just to make sure no other thread is running, which might receive the signal). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by STINNER Victor victor.stin...@haypocalc.com: -- nosy: -haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Brett Cannon br...@python.org: -- nosy: +brett.cannon resolution: fixed - stage: committed/rejected - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: Putting this back to open until we decide what to do about the OS X test failures. It sounds like it could really do with some more poking and prodding to figure out whether or not it poses a potential security risk or is just a relatively cosmetic problem with the API, so I'm reluctant to just skip the failing tests at this point. -- status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Charles-François Natali neolo...@free.fr added the comment: The OS X buildbots show some failures: It seems to fail consistently on every OS X version. I've had another look both at the code and the test, and couldn't find anything wrong with it. Since there are a number of known bugs pertaining to FD passing on OS X - even on recent versions - I'd suggest to skip those tests on this platform. As for sendto/sendmsg not being interrupted by the signal, I'd be curious to see if running test_socket alone solves the problem (just to make sure no other thread is running, which might receive the signal). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Charles-François Natali neolo...@free.fr added the comment: Here's a patch skipping testFDPassSeparate and testFDPassSeparateMinSpace on OS X 10.5, due to known kernel bugs (see http://developer.apple.com/library/mac/#qa/qa1541/_index.html). For InterruptedSendTimeoutTest and testInterruptedSendmsgTimeout, it also looks like a kernel bug. There could be another explanation, though: if, for some reason, other threads are running at that time, the signal might be delivered to another thread, and our main thread remains stuck on sendto/sendmsg once the socket buffer is full. I'm however not sure why this would only affect OS X (since FreeBSD behaves in the same way when it comes to signals, contrarily to Linux). Also, I'm not sure why this would not affect recv/recvmsg. -- Added file: http://bugs.python.org/file23027/pass_fds_osx.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___diff -r 1696e2789d91 Lib/test/test_socket.py --- a/Lib/test/test_socket.py Wed Aug 24 02:15:25 2011 +0200 +++ b/Lib/test/test_socket.py Wed Aug 24 18:36:22 2011 +0200 @@ -2090,7 +2090,9 @@ def _testFDPassCMSG_LEN(self): self.createAndSendFDs(1) +# Passing multiple FDs at once isn't reliable on OS X 10.5. @requireAttrs(socket, CMSG_SPACE) +@support.requires_mac_ver(10, 5) def testFDPassSeparate(self): # Pass two FDs in two separate arrays. Arrays may be combined # into a single control message by the OS. @@ -2111,6 +2113,7 @@ len(MSG)) @requireAttrs(socket, CMSG_SPACE) +@support.requires_mac_ver(10, 5) def testFDPassSeparateMinSpace(self): # Pass two FDs in two separate arrays, receiving them into the # minimum space for two arrays. ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Charles-François Natali neolo...@free.fr added the comment: As noted by Antoine, the OS X 10.5 buildbots are also failing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: On Tue 23 Aug 2011, Nick Coghlan wrote: As you can see, I just pushed a change that removed the new methods from SSLSocket objects. If anyone wants to step up with a valid use case (not already covered by wrap_socket), preferably with a patch to add them back that includes proper tests and documentation changes, please open a new feature request and attach the new patch to that issue. Hi, sorry about the trouble caused by the broken tests, but SSLSocket should at least override sendmsg() to stop misguided programs sending data in the clear: http://bugs.python.org/issue12835 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Charles-François Natali neolo...@free.fr added the comment: However, in doing this, I noticed that these methods will, at best, work during the time between connection and the socket going secure and were not added to the list of methods that the SSL is documented as exposing. Perhaps we should just ditch them entirely? +1 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Antoine Pitrou pit...@free.fr added the comment: However, in doing this, I noticed that these methods will, at best, work during the time between connection and the socket going secure and were not added to the list of methods that the SSL is documented as exposing. Perhaps we should just ditch them entirely? Well, apparently the SSLSocket object is meant to be usable in clear text mode. I'm not sure why it's like that, but your patch looks ok to me (except that it has a debug print() call). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: That's the part I'm questioning though. I'm not clear why you'd ever do that instead of doing everything on the original socket before invoking ssl.wrap_socket. What I missed on the original patch before committing it (mea culpa) is that the SSL part is neither documented nor tested properly (the tests only check that it is disallowed on a secured SSLSocket, not that it works on a connected-but-not-secured-yet SSLSocket object). The absence of proper tests and documentation is the main reason I'm tempted to just revert those parts of the patch that touch the ssl module and its tests. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Antoine Pitrou pit...@free.fr added the comment: That's the part I'm questioning though. I'm not clear why you'd ever do that instead of doing everything on the original socket before invoking ssl.wrap_socket. What I missed on the original patch before committing it (mea culpa) is that the SSL part is neither documented nor tested properly (the tests only check that it is disallowed on a secured SSLSocket, not that it works on a connected-but-not-secured-yet SSLSocket object). Bill, do you know? The absence of proper tests and documentation is the main reason I'm tempted to just revert those parts of the patch that touch the ssl module and its tests. Then perhaps raise NotImplementedError, so that people know it's deliberate and not an oversight. -- nosy: +janssen ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset fd10d042b41d by Nick Coghlan in branch 'default': Remove the SSLSocket versions of sendmsg/recvmsg due to lack of proper tests and documentation in conjunction with lack of any known use cases (see issue #6560 for details) http://hg.python.org/cpython/rev/fd10d042b41d -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: As you can see, I just pushed a change that removed the new methods from SSLSocket objects. If anyone wants to step up with a valid use case (not already covered by wrap_socket), preferably with a patch to add them back that includes proper tests and documentation changes, please open a new feature request and attach the new patch to that issue. -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: Regarding the 'missing methods' aspect, the SSL docs are already pretty clear that SSLSocket objects don't expose the full socket API: http://docs.python.org/dev/library/ssl#ssl-sockets Those docs are actually what really got me started down the path of wondering whether or not these new methods even belong on SSLSocket in the first place. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: And the Windows buildbots are now happy (at least with respect to this change, anyway - they're still griping about a few other issues). I don't know if it's feasible to support these new APIs at the socket module level on Windows, but any patches along those lines should also be placed in a new issue rather than being added to this one. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Antoine Pitrou pit...@free.fr added the comment: The OS X buildbots show some failures: http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%202%203.x -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 37721ee145a2 by Nick Coghlan in branch 'default': Credit patch authors in NEWS for #6560 http://hg.python.org/cpython/rev/37721ee145a2 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Charles-François Natali neolo...@free.fr added the comment: New changeset c64216addd7f by Nick Coghlan in branch 'default': Add support for the send/recvmsg API to the socket module. Patch by David Watson and Heiko Wundram. (Closes #6560) http://hg.python.org/cpython/rev/c64216addd7f I had actually started doing a review, I should have said so... There was nothing spectacular though, so I'll just make some updates later. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: Someone here at the sprints pointed out a redundant unsigned comparison to 0 that I missed, so a second set of eyes to double-check things like that would be good. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: Reopening to track buildbot failures (at least on Windows) - at a glance, it looks like the SSL socket wrapper is still adding (and trying to test) the send/receivemsg methods even when they're missing from the socket object on the current platform. -- assignee: - ncoghlan status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Nick Coghlan ncogh...@gmail.com added the comment: Attached patch (ssl_fixes_v1) makes the presence of the sendmsg/recvmsg methods in the ssl module conditional on their being present in the underlying socket module. However, in doing this, I noticed that these methods will, at best, work during the time between connection and the socket going secure and were not added to the list of methods that the SSL is documented as exposing. Perhaps we should just ditch them entirely? -- Added file: http://bugs.python.org/file23011/issue6560_ssl_fixes_v1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Brian May br...@microcomaustralia.com.au: -- nosy: +ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset c64216addd7f by Nick Coghlan in branch 'default': Add support for the send/recvmsg API to the socket module. Patch by David Watson and Heiko Wundram. (Closes #6560) http://hg.python.org/cpython/rev/c64216addd7f -- nosy: +python-dev resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Antoine Pitrou pit...@free.fr added the comment: What needs to happen to get recvmsg() supported in Python? Well, I guess that the only reason is that no committer is motivated enough to bring this into Python: it's a rather large patch, and honestly, I'm not sure that many people are going to use it. The feature I personally like the most about sendmsg/recvmsg is the ability to do scatter-gather I/O, but if the performance is critical, then I won't be using Python. I know that sendmsg also has some other advantages (passing FDs, ancillary data...). Modules/_multiprocessing already has code using sendmsg/recvmsg, precisely to pass FDs IIRC. Generic support for sendmsg() and recvmsg() in the socket module would allow to rewrite that code in pure Python. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Brian May br...@microcomaustralia.com.au added the comment: To address some of the comments above: * Yes, the patch is large. However, most of this is in the tests. Only A relatively small part is in the code that implements the required functionality. I don't care much myself about the test cases, however would have assumed these would help get it into upstream. (baikie-hwundram-v5-hg.diff) Doc/library/socket.rst | 176 +++ Lib/ssl.py | 24 Lib/test/test_socket.py | 2120 Lib/test/test_ssl.py| 13 Modules/socketmodule.c | 809 ++ 5 files changed, 3142 insertions(+) * raw sockets are inappropriate for using UDP TPROXY support. I don't want to bypass the kernel code and have to pass headers etc myself, I just want to get the destination address so I know where the packet was original destined for. recvmsg() is the correct solution. * Thanks for the 2.x patch. Will try that out now. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Charles-François Natali neolo...@free.fr added the comment: What needs to happen to get recvmsg() supported in Python? Well, I guess that the only reason is that no committer is motivated enough to bring this into Python: it's a rather large patch, and honestly, I'm not sure that many people are going to use it. The feature I personally like the most about sendmsg/recvmsg is the ability to do scatter-gather I/O, but if the performance is critical, then I won't be using Python. I know that sendmsg also has some other advantages (passing FDs, ancillary data...). recvmsg() is required to get get transparent UDP proxies working under Linux using tproxy, the code needs to run recvmsg() to be able to find out what the original destination address was for the the packet. Sounds like a job for raw sockets, no? (well, you need CAP_NET_RAW) In short, I think that you just need to find a core developer interested, I personally am not (but I'm not opposed to it either :-). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Brian May br...@microcomaustralia.com.au added the comment: Have tested my code with this patch, the recvmsg(...) call seems to work fine. Also had a half-hearted attempt at porting to Python 2.7, but didn't get past compiling, the code requires BEGIN_SELECT_LOOP and END_SELECT_LOOP macros that aren't defined in Python 2.7 - I tried copying the definitions from Python 3.3, but that didn't work either. Not sure if it is worth the effort if Python 2.7 is closed to new features. Brian May -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Gergely Kálmán kalman.gerg...@duodecad.hu added the comment: No, indeed this is a lot better. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Gergely Kálmán kalman.gerg...@duodecad.hu added the comment: On 05/22/11 03:14, Brian May wrote: Brian Maybr...@microcomaustralia.com.au added the comment: What needs to happen to get recvmsg() supported in Python? recvmsg() is required to get get transparent UDP proxies working under Linux using tproxy, the code needs to run recvmsg() to be able to find out what the original destination address was for the the packet. Thanks Brian May -- nosy: +brian ___ Python trackerrep...@bugs.python.org http://bugs.python.org/issue6560 ___ Hello Brian! It's been a while I had a look at that code. As far as I remember though the code is fairly decent not taking the missing unit tests into account. There are a few todos, and also a pretty bad bug that I've fixed but not committed. The TODOs include better parsing of auxiliary data, support for scatter-gather, addressed messages. If you wish I can send you the latest patch that has the bug fixed and applies to 3.2. Do you want this to be pushed to 3.3? Gergely Kalman -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: On Mon 23 May 2011, Gergely Kálmán wrote: It's been a while I had a look at that code. As far as I remember though the code is fairly decent not taking the missing unit tests into account. There are a few todos, and also a pretty bad bug that I've fixed but not committed. The TODOs include better parsing of auxiliary data, support for scatter-gather, addressed messages. If you wish I can send you the latest patch that has the bug fixed and applies to 3.2. Erm, have you seen the separately-implemented patch I posted at http://bugs.python.org/file19962/baikie-hwundram-v5.diff ? It's basically complete IIRC. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Brian May br...@microcomaustralia.com.au added the comment: Hello, Are there any problems applying the v5 version of the patch to 3.3? Also is there any remote chance for a backport to 2.7? Thanks -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +charles-francois.natali, rosslagerwall versions: +Python 3.3 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Brian May br...@microcomaustralia.com.au added the comment: What needs to happen to get recvmsg() supported in Python? recvmsg() is required to get get transparent UDP proxies working under Linux using tproxy, the code needs to run recvmsg() to be able to find out what the original destination address was for the the packet. Thanks Brian May -- nosy: +brian ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: New version with minor changes. Will also upload at http://codereview.appspot.com/1487041/show -- Added file: http://bugs.python.org/file17659/baikie-hwundram-v4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: Optional patch to replace SocketTCPTest, etc. with the classes from the sendmsg patch. -- Added file: http://bugs.python.org/file17660/v4-replace-existing-classes.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: OK. I don't like creating/using a Google account, but here it is: http://codereview.appspot.com/1487041/show -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Antoine Pitrou pit...@free.fr added the comment: Would you like to upload your patch to http://codereview.appspot.com/? It would make reviewing easier. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Andrew Grover andy.gro...@gmail.com: -- nosy: +Andrew.Grover ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: Here is a new version of the patch; I've added some tests which use the RFC 3542 interface (IPv6 advanced API) and am now quite happy with it generally. As well as Linux, I've tested it on an old (unsupported) FreeBSD 5.3 installation, which required a few changes in the tests and the code, but is probably representative of many socket implementations. Testing on other systems would be appreciated! The main issue was that when truncating ancillary data, FreeBSD seemed to just copy as much into the buffer as would fit and did not adjust the last item's cmsg_len member to reflect the amount of data that was actually present, so the item would appear to extend past the end of the buffer. The last version of the patch detected this and raised RuntimeError, which prevented any truncated receives from succeeding. The new version instead issues a warning and returns as much of the last item as is in the buffer. The warning could perhaps be disabled for systems like this, given that it happens every time ancillary data is truncated, but truncation generally shouldn't happen in a program's normal operation, and on other platforms a bad cmsg_len value might indicate that the returned data is actually incorrect in some way. After some investigation, I've stuck with using CMSG_FIRSTHDR() and CMSG_NXTHDR() to step through the headers when assembling the ancillary data in sendmsg(). The KAME IPv6 userspace utilities at [1] include several programs which send multiple control messages at once, and these always use CMSG_NXTHDR() to advance to the next uninitialized header, while some (but not all) of them zero-fill the buffer beforehand, suggesting they ran into the issue with glibc's macros returning NULL (KAME developed the BSD IPv6 stack, and the zero-filling isn't necessary with the BSD macros). The alternative would be to add CMSG_SPACE(size) to the pointer to get to the next header. Going by the diagram in RFC 3542, that should be equivalent, but if some system defined CMSG_SPACE(len) as, say, CMSG_LEN(len) + 3, instead of (CMSG_LEN(len) + 3) ~3, it would probably go unnoticed until someone tried to use CMSG_SPACE() that way. So given the KAME example, I think using CMSG_NXTHDR() with a zero-filled buffer is the way to go. [1] http://www.kame.net/dev/cvsweb2.cgi/kame/kame/kame/ -- Added file: http://bugs.python.org/file17483/baikie-hwundram-v3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +haypo stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Giampaolo Rodola' g.rod...@gmail.com: -- nosy: +giampaolo.rodola ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +exarkun ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: I just found that the IPv6 tests don't get skipped when IPv6 is available but disabled in the build - you can create IPv6 sockets, but not use them :/ This version fixes the problem. -- Added file: http://bugs.python.org/file16422/baikie-hwundram-v2.1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: OK, here's a new version as a work in progress. A lot of the new stuff is uncommented (particularly the support code for the tests), but there are proper docs this time and a fairly complete test suite (but see below). There are a couple of changes to the interface (hopefully the last). The recvmsg() methods no longer receive ancillary data by default, since calling them on an AF_UNIX socket with the old default buffer could allow a malicious sender to send unwanted file descriptors up to receiver's resource limit, and in a multi-threaded program, another thread could then be prevented from opening new file descriptors before the receiving thread had a chance to close the unwanted ones. Since the ancillary buffer size argument is now more likely to need a value, I've moved it to second place; the basic argument order is now the same as in Kalman Gergely's patch. CMSG_LEN() and CMSG_SPACE() are now provided. I've also used socket.error instead of ValueError when rejecting some buffer object/array for being too big to handle, since the system call itself might cause socket.error to be raised for a smaller (oversized) object, failing with EMSGSIZE or whatever. The code is now much more paranoid about checking the results of the CMSG_*() macros, and will raise RuntimeError if it finds its assumptions are not met. I'd still like to add tests for receiving some of the RFC 3542 ancillary data items, especially since the SCM_RIGHTS tests can't always (ever?) test recvmsg() with multiple items (if you send two FD arrays, the OS can coalesce them into a single array before delivering them). -- Added file: http://bugs.python.org/file16417/baikie-hwundram-v2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: Thanks for your interest! I'm actually still working on the patch I posted, docs and a test suite, and I'll post something soon. Yes, you could just use b.join() with sendmsg() (and get slightly annoyed because it doesn't accept buffers ;) ). I made sendmsg() take multiple buffers because that's the way the system call works, but also to match recvmsg_into(), which gives you the convenience of being able to receive part of the message into a bytearray and part into an array.array(i), say, if that's how the data is formatted. As you might know, gather-write with sendmsg() can give a performance benefit by letting the kernel assemble the message while copying the data from userspace rather than having userspace copy the data once to form the message and then having the kernel copy it again when the system call is made. I suppose with Python you just need a larger message to see the benefit :) Since it can read from buffers, though, socket.sendmsg() can pull a large chunk of data straight out of an mmap object, say, and attach headers from a bytes object without the mmapped data being touched by Python at all (or even entering userspace, in this case). The patch is for 3.x, BTW - y* is valid there (and does take a buffer). As for a good reference, I haven't personally seen one. There's POSIX and RFC 3542, but they don't provide a huge amount of detail. Perhaps the (updated) W. Richard Stevens networking books? I've got the Stevens/Rago second edition of Advanced Programming in the Unix Environment, which discusses FD and credential passing with sendmsg/recvmsg, but not very well (it misuses CMSG_LEN, for one thing). The networking books were updated by different people though, so perhaps they do better. The question of whether to use CMSG_NXTHDR() to step to the next header when constructing the buffer for sendmsg() is a bit murky, in particular. I've assumed that this is the way to do it since the examples in RFC 3542 (and most of the code I've seen generally) use CMSG_FIRSTHDR() to get the initial pointer, but I've found that glibc's CMSG_NXTHDR() can (wrongly, I think) return NULL if the buffer hasn't been zero-filled beforehand (this causes segfaults with the patch I initially posted). @Wim: Yes, the rfc3542 module from that package looks as if it would be usable with these patches - although it's Python 2-only, GPL-only and looks unmaintained. Those kind of ancillary data constructors will actually be needed to make full portable use of sendmsg() and recvmsg() for things like IPv6, SCTP, Linux's socket error queues, etc. The same goes for data for the existing get/setsockopt() methods, in fact - the present suggestion to use the struct module is pretty inadequate when there are typedefs involved and implementations might add and reorder fields, etc. The objects in that package seem a bit overcomplicated, though, messing about with setter methods instead of just subclassing bytes and having different constructors to create the object from individual arguments or received bytes (say, ucred(1, 2, 3) or ucred.from_bytes(...)). Maybe the problem of testing patches well has been putting people off so far? Really exercising the system's CMSG_*HDR() macros in particular isn't entirely straightforward. I suppose there's also a reluctance to write tests while still uncertain about how to present the interface - that's another reason why I went for the most general multiple-buffer form of sendmsg()! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Wim w...@.org added the comment: I just ran across yet another implementation of sendmsg support for python sockets, whose feature set seems to complement Kalman Gergely's implementation: http://www.pps.jussieu.fr/~ylg/PyXAPI by Yves Legrandgerard (Out of curiosity, people have been submitting requests for, and patches with implementations of, this feature for five or more years now and they're never accepted. Is there some opposition to sendmsg support in Python? It's a missing feature that bites me pretty regularly.) -- nosy: +wiml ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Jack Diederich jackd...@gmail.com added the comment: I've been digging into the patch. Is there a reason sendmsg() wants an iterable of buffers instead of just accepting a str? The list-of-buffers more closely matches the underlying syscall but I'm not sure what the python benefit is, especially when recvmsg() only returns a single value (it only creates 1 iovec under the covers). Python doesn't have readv like methods so making sendmsg/recvmsg work like recv/send (straight strings) seems like the way to go. Also, the y* format character for packing/unpacking tuples is no longer supported - I'm assuming it used to mean buffers. Does anyone have a good reference for using recvmsg/sendmsg? I read the man pages and googled around but couldn't find anything. I have no experience with using the calls in-the-wild. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Jack Diederich jackd...@gmail.com added the comment: one of the other sprinters just pointed out that Modules/_multiprocessing.c (py3k branch) uses sendmsg/recvmsg internally to pass file descriptors back and forth. The code is very short and readable. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
David Watson bai...@users.sourceforge.net added the comment: Hi, I'm afraid there may have been some duplication of effort here - I set about reworking Heiko Wundram's original patch (issue #1194378) without knowing about this one. I don't have unit tests yet either, but I saw this and thought I'd better post what I had. It includes the items on Kalman's to-do list. This patch is based on Heiko's original code, but changes the interface, dropping the special processing of SCM_RIGHTS, etc. (I think the encoding/decoding would be better handled with separate functions/classes), and adds source address (msg_name) support and scatter/gather I/O via sendmsg() and a new recvmsg_into() method, as well as fixing various bugs and limitations. Comments/flames welcome too. You'll see a few XXX comments in the code. One in particular refers to the msg_name value from recvmsg() on a connected socket; I've said in the docstring that it is unspecified in this case, but it might or might not contain a valid address, depending on the OS. These methods may also need to be conditionally compiled if, say, CMSG_SPACE (or sendmsg/recvmsg?) isn't available somewhere. I didn't add a facility to receive into only part of a buffer in recvmsg_into(), like the one in recv_into(), since memoryview objects make it redundant AFAICT. -- nosy: +baikie Added file: http://bugs.python.org/file15561/baikie-hwundram.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
New submission from Kálmán Gergely kalman.gerg...@duodecad.hu: This is the rewritten-from-scratch implementation of the sendmsg()/recvmsg() methods. Any comments / suggestions / flames are very welcome. Currently it supports what I need and I'm only releasing it because I don't have much time to develop it further in the forseeable future (1-2 months). It is rewritten from scratch, using the python c-api documents. I've tried my best, but I wouldn't bet that it works error-free. I'd be glad if someone could give me a review on what I've might done wrong. The features that are missing: - using scatter/gather - using it with non-stream oriented sockets (doesn't support addresses /msg_name/) These should be very easy to implement. If no one takes up the task of implementing the missing features than I'll do it of course, you just have to wait a little while. Of course any errors present in the code right now will be fixed. Thanks Kalman Gergely -- components: Library (Lib) files: srmsg.patch keywords: patch messages: 90875 nosy: synapse severity: normal status: open title: socket sendmsg(), recvmsg() methods type: feature request versions: Python 3.2 Added file: http://bugs.python.org/file14556/srmsg.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Kálmán Gergely kalman.gerg...@duodecad.hu added the comment: the tester application -- Added file: http://bugs.python.org/file14557/sendrecvmsgtest.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6560] socket sendmsg(), recvmsg() methods
Thomas Herve the...@free.fr added the comment: This is a duplicate (although updated patch) from bug #1194378. It would still need unit tests... -- nosy: +therve ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6560 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
