Re: Py2.7/FreeBSD: maximum number of open files
In article mailman.2711.1321299276.27778.python-l...@python.org Christian Heimes li...@cheimes.de wrote: Am 14.11.2011 19:28, schrieb Tobias Oberstein: Thanks! This is probably the most practical option I can go. I've just tested: the backported new IO on Python 2.7 will indeed open 32k files on FreeBSD. It also creates the files much faster. The old, non-monkey-patched version was getting slower and slower as more files were opened/created .. I wonder what's causing the O(n^2) behavior. Is it the old file type or BSD's fopen() fault? It is code in libc. My old stdio (still in use on FreeBSD) was never designed to be used in situations with more than roughly 256 file descriptors -- hence the short in the file number field. (The OS used to be full of other places that kept the maximum allowable file descriptor fairly small, such as the on-stack copies of fd_set objects in select() calls.) You will want to redesign the code that finds or creates a free FILE object, and probably some of the things that work with line-buffered FILEs (specifically calls to _fwalk() when reading from a line-buffered stream). -- In-Real-Life: Chris Torek, Wind River Systems Intel require I note that my opinions are not those of WRS or Intel Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603 email: gmail (figure it out) http://web.torek.net/torek/index.html -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 16:57, schrieb Tobias Oberstein: I am trying to convince Python to open more than 32k files .. this is on FreeBSD. Now I know I have to set appropriate limits .. I did: $ sysctl kern.maxfiles kern.maxfiles: 204800 $ sysctl kern.maxfilesperproc kern.maxfilesperproc: 20 $ sysctl kern.maxvnodes kern.maxvnodes: 20 $ ulimit unlimited Here is what happens with a Python freshly built from sources .. it'll tell me I can open 200k files .. but will bail out at 32k: I'm not familiar with BSD but Linux has similar Kernel options. The kernel options might be *global* flags to set the total upper limit of open file descriptors for the entire system, not for a single process. Also on Linux ulimit doesn't display the fd limit. You have to use ulimit -n. Why do you need more than 32k file descriptors anyway? It's an insanely high amount of FDs. Most programs need less than 100 and the default value of 1024 on my Linux servers is usually high enough. I've never increased the fd limit over 8192 and our biggest installation servers more than 80 TB data in about 20 to 25 million files. Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 17:36, schrieb Tobias Oberstein: This is a dedicated machine doing nothing else .. I'm monitoring global FD usage sysctl kern.openfiles and it's way beyond the configured limit $ ulimit -n 20 Apparently you did everything right here. Well, it was worth the try. ;) It's not for files: This is a network service .. I tested it with up to 50k TCP connections .. however at this point, when the service tries to open a file, it'll bail out. Sockets+Files both contribute to open FDs. I need 50k sockets + 100 files. Thus, this is even more strange: the Python (a Twisted service) will happily accept 50k sockets, but as soon as you do open() a file, it'll bail out. A limit of 32k smells like a overflow in a signed int. Perhaps your system is able and configured to handle more than 32k FDs but you hit an artificial limit because some C code or API has a overflow. This seems to be a known bug in FreeBSD http://lists.freebsd.org/pipermail/freebsd-bugs/2010-July/040689.html Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 18:03, schrieb Tobias Oberstein: This is unbelievable. I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3 ... both on i386 _and_ amd64. Now I'm f***d;( A last chance: is it possible to compile Python for not using libc fopen(), but the Posix open()? Thanks anyway for this hint! A fix would break the ABI compatibility. I guess that you won't see a fix for the issue until FreeBSD 9.0 is released. And no, you can't re-compile Python to use the open() API instead of fopen(). The built-in file type is developed around the file pointer API, not the file descriptor API. Luckily Python 2.7 has a backport of Python 3.x new IO library. The new IO API doesn't use file pointers at all and implements its own layer around file descriptors. Good luck! Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
On Nov 14, 5:03 pm, Tobias Oberstein tobias.oberst...@tavendo.de wrote: I need 50k sockets + 100 files. Thus, this is even more strange: the Python (a Twisted service) will happily accept 50k sockets, but as soon as you do open() a file, it'll bail out. A limit of 32k smells like a overflow in a signed int. Perhaps your system is able and configured to handle more than 32k FDs but you hit an artificial limit because some C code or API has a overflow. This seems to be a known bug in FreeBSDhttp://lists.freebsd.org/pipermail/freebsd-bugs/2010- July/040689.html This is unbelievable. I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3 ... both on i386 _and_ amd64. Now I'm f***d;( A last chance: is it possible to compile Python for not using libc fopen(), but the Posix open()? Thanks anyway for this hint! Have you tried/or is it possible to get your 100 or whatever files first, before your sockets? hth Jon -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 18:46, schrieb Tobias Oberstein: I just confirmed that the bug is even there for FreeBSD 9 RC1 ! This is most unfortunate. Seriously. W00t, that sucks! You could migrate to another BSD (NetBSD) or Linux ... :) I am running out of options, since I am willing to make my stuff Python 3 compatible, but Twisted is not yet there. Using the backported new IO on Python 2.7 will not make open() automatically use the new IO, will it? No, the open() function of Python 2.7 will still use the file class which in return uses fopen(). You could try to monkey patch the built-in open() function. It's mostly API compatible with the current open() function: import io, __builtin__ __builtin__.open = io.open It works as long as no codes checks for isinstance(obj, file). If your app only has to worry about log files, you might want to overwrite the _open() method of logging.FileHandler and its subclasses. Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 19:28, schrieb Tobias Oberstein: Thanks! This is probably the most practical option I can go. I've just tested: the backported new IO on Python 2.7 will indeed open 32k files on FreeBSD. It also creates the files much faster. The old, non-monkey-patched version was getting slower and slower as more files were opened/created .. I wonder what's causing the O(n^2) behavior. Is it the old file type or BSD's fopen() fault? There seem to be slight differences though: Non-monkey patched: I can write to the file a non-Unicode string, even when the file was opened non-Binary. With monkey patch: either open the file Binary-mode, or write Unicode strings .. Python 3.x doesn't collate data and text but distinguishes between bytes (str type in Python 2.x) and unicode. You can't write unicode to a binary file and str to a text file. See it as an opportunity! You are taking the first step towards Python 3 compatibility. :) Christian -- http://mail.python.org/mailman/listinfo/python-list
