Bugs item #1494314, was opened at 2006-05-24 06:51 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1494314&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Extension Modules Group: Python 2.4 Status: Closed Resolution: Accepted Priority: 5 Submitted By: Michael Smith (mlrsmith) Assigned to: Anthony Baxter (anthonybaxter) Summary: Cannot use high-numbered sockets in 2.4.3 Initial Comment: Python 2.4.3 introduced (in Modules/socketmodule.c) the IS_SELECTABLE() macro, to check whether calling select() on a given file descriptor is permissible. However, it executes this check even when it won't actually call select(). Specifically, select() is called ONLY when s->sock_timeout > 0 (in internal_select mostly, but also internal_connect). I have a twisted application which uses many FDs (several thousand), and upgrading python to 2.4.3 makes it hit this limit (at 1024 file descriptors), regardless of ulimit settings. Twisted itself always uses non-blocking I/O on sockets, so with older versions of python this worked fine. A simple solution relies on the fact that select is only ever actually called, and changes the IS_SELECTABLE macro as in the attached fairly trivial patch. This is sufficient to restore my application to its previous state (where it works fine). This doesn't solve the more general problem in socketmodule - that we simply can't do all operations properly with the current reliance on select(), though, and it seems like a bit of a hack... If I wrote up a patch to use poll() on systems where it's available, throughout socketmodule.c, in preference to select(), would this be accepted? Mike ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2006-10-10 19:40 Message: Logged In: YES user_id=33168 I assume 2.3.3c1 means 2.4.4c1. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2006-10-10 18:46 Message: Logged In: YES user_id=29957 This is applied and will be in 2.3.3c1 ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2006-07-11 10:21 Message: Logged In: YES user_id=29957 Re-opening to remind myself to apply this to release24-maint for the eventual 2.4.4 release. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2006-07-10 20:52 Message: Logged In: YES user_id=29957 Applied. Patch will be in 2.5b2 (to be released shortly). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-07-10 19:53 Message: Logged In: YES user_id=33168 Anthony checked this in to 2.5 as 50567. I will backport at least part of it later. ---------------------------------------------------------------------- Comment By: Martin v. Löwis (loewis) Date: 2006-07-09 22:47 Message: Logged In: YES user_id=21627 The patch is fine, please apply. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-07-09 16:57 Message: Logged In: YES user_id=33168 I meant I don't think we *care* in this case (not can). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-07-09 16:55 Message: Logged In: YES user_id=33168 I think you're right Martin. Looking at what it means to have a broken poll, I don't think we can in this instance. So I removed all refs to HAVE_BROKEN_POLL. What do you think of the new version? ---------------------------------------------------------------------- Comment By: Martin v. Löwis (loewis) Date: 2006-07-07 08:50 Message: Logged In: YES user_id=21627 The __APPLE__ stuff looks wrong in file 184131. You would have to use selectmodule.c:select_have_broken_poll at run-time to be sure you can use poll(2) on OS X (you can use it on 10.4, but can't on 10.3, IIRC). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-07-06 23:10 Message: Logged In: YES user_id=33168 I've added a more complete patch (against 2.5, hopefully applies to 2.4 too). It cleans up some things and adds support for SSL sockets too. Can people please review/test this? I manually tested this with regular sockets and it seemed to work. All the tests pass, but this is somewhat tricky. I hate the duplicated code between socket and ssl modules, but added to it. It would be great to clean this up for 2.6. If you are forced to use select with a high socket, the exception on connect is operation in progress rather than can't select on socket. I guess that's ok, it doesn't actually change the existing behaviour. That would have been more involved and I'm not sure it's worth it. ---------------------------------------------------------------------- Comment By: Michael Smith (mlrsmith) Date: 2006-06-09 03:31 Message: Logged In: YES user_id=1488997 Ok, I'll attach a patch that uses poll when available (HAVE_POLL is already being set by the configure stuff appropriately). It replaces one of the two uses of select() (specifically, the internal_select() function) in socketmodule.c. The other is win32-specific, so replacing it with poll() wouldn't make sense. greg: epoll/kevent don't make sense for replacing the use of select/poll in this particular case - socketmodule.c always selects/polls precisely one file descriptor. I've tested this locally, and it fixes the problem (linux/x86). I don't have a windows system to test it on, but it shouldn't change behaviour in any way for windows. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2006-06-07 22:51 Message: Logged In: YES user_id=413 eew yuck. yes use poll at the very least. we should also consider using epoll (linux) and kevent (bsd) in the future as both of those scale better than O(n) unlike select and poll. ---------------------------------------------------------------------- Comment By: Martin v. Löwis (loewis) Date: 2006-06-04 13:14 Message: Logged In: YES user_id=21627 A patch to replace internal_select with poll(2) where available is acceptable. The current version should be conditionally kept. That Windows doesn't have poll(2) is no problem: its select implementation supports arbitrarily large FDs (just like poll). Relaxing the IS_SELECTABLE usage to cases where select is actually going to be called is optional: there shouldn't be too many systems left without select where people still want to open many file descriptors. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2006-05-31 15:27 Message: Logged In: YES user_id=31435 akuchling: No, poll() is not part of the Windows socket API. ---------------------------------------------------------------------- Comment By: Michael Smith (mlrsmith) Date: 2006-05-31 14:39 Message: Logged In: YES user_id=1488997 That, of course, is the problem - select() is available more or less universally, but poll() is not. However, it's not terribly difficult to write a poll() wrapper using select(), though doing so in general would have serious performance issues (the specific uses in socketmodule.c do not hit this problem), and retains the limitations of select. It's not clear that the complexity of doing this is worthwhile compared to just implementing the simple API needed internally to socketmodule.c using both APIs (i.e. just adding a poll() option). Regardless, it'd be nice if at least the basic fix I wrote up went in - it solves the immediate problem, at least. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2006-05-31 11:39 Message: Logged In: YES user_id=11375 I expect such a patch would be acceptable. The largest issue is probably whether poll() is available everywhere, or if we'd be stuck maintaining both select() and poll() based versions of internal_select(). Does Windows support poll() at all? ---------------------------------------------------------------------- Comment By: Michael Smith (mlrsmith) Date: 2006-05-29 07:13 Message: Logged In: YES user_id=1488997 Yes, I had my ulimit set appropriately. There's no problem with _opening_ many sockets (at least, I don't think there is) - the problem is with trying to do something (like call socket.recv()) with them. The code in socketmodule.c is pretty clear - and having upgraded to 2.4.3 with ubuntu dapper, I _am_ running into this. For now, we're having to keep all our production machines on 2.4.2. ---------------------------------------------------------------------- Comment By: Gabriel Wicke (gabrielwicke) Date: 2006-05-29 07:11 Message: Logged In: YES user_id=956757 Please disregard my comments completely- just opening more than 1024 files does not trigger this bug, but doing a socket.send() on the 1025th socket does. ---------------------------------------------------------------------- Comment By: Gabriel Wicke (gabrielwicke) Date: 2006-05-29 07:00 Message: Logged In: YES user_id=956757 Never mind- you already tried ulimit. It still works for me though, 2.4.3 from Ubuntu Dapper. ---------------------------------------------------------------------- Comment By: Gabriel Wicke (gabrielwicke) Date: 2006-05-29 06:57 Message: Logged In: YES user_id=956757 Try "ulimit -n 8096" (only permitted in a root shell) to set your max socket usage to something larger. Opening more than 1024 sockets works for me in python 2.4.3 after changing the limit for the terminal in question and restarting the interpreter. Just ulimit -n will show you your current limit. Ulimits are enforced by the Linux kernel and can likely be configured system-wide in /etc/sysctl.cfg. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1494314&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
