socket: too many open file descriptors (Re: Python with many threads)
Hello, Answering my own mail ;) Ott Köstner wrote: Still, something locks up in the system, when I increase the number Python threads with http://grub.org/?q=en/node/204 [...] The frequency of these messages increases with 70 threads. And with 200 threads the system locks up totally -- almost no access from web. [I can still access the system and kill the process :) ] Clearly, there is some limit in the system. It is not memory, cpu or bandwidth. If the number of threads is also not limited, it must be something else. What might it be? Eventually, I found out that the limiting factor here is named, not capable of handling more request and causing Apache to lock up. But now I am trapped here. Is there anybody who can help? In /var/log/messages: named[63198]: socket: too many open file descriptors I have googled this, but no help. Recompiled bind, adding -DFD_SETSIZE=2048 to CFLAGS. No help. sockstat |grep -c named does not big numbers. With 60 Grubng/Pyhton threads this number is around 20, rises over 100 with 100 threads, but this is still much lower than 2048... Any ideas? Regards, O.K. -- Testi oma Interneti kiirust / Test Your Internet speed: http://speedtest.zzz.ee/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: socket: too many open file descriptors (Re: Python with many threads)
On Wednesday 17 December 2008 4:17:46 pm Michel Talon wrote: Ott Köstner wrote: In /var/log/messages: named[63198]: socket: too many open file descriptors See the sysctl variables: kern.maxfiles kern.maxfilesperproc kern.maxfiles: 12328 kern.maxfilesperproc: 11095 Note that Google leads immediately to this: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-07/msg00251.html Yes, actually I saw this. 11095 seemed a big number in this case. Increased to: kern.maxfiles: 25000 kern.maxfilesperproc: 2 ... and the problem remains... :( There are also messages like this: named[63198]: client 127.0.0.1#60448: error sending response: host unreachable Regards, O.K. -- Mõõda oma inteneti kiirust / Test Your Internet speed http://speedtest.zzz.ee/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: socket: too many open file descriptors (Re: Python with many threads)
Ott Köstner wrote: In /var/log/messages: named[63198]: socket: too many open file descriptors See the sysctl variables: kern.maxfiles kern.maxfilesperproc Note that Google leads immediately to this: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-07/msg00251.html -- Michel TALON ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Python with many threads
Michel Talon wrote: Nothing limits the number of concurrent threads. Personnally i have checked i can run Grub Next Generation Python Client with 600 threads without any problem. niobe% uname -a FreeBSD niobe.lpthe.jussieu.fr 7.0-STABLE FreeBSD 7.0-STABLE #0: Tue Jul 22 10:31:01 CEST 2008 niobe% python Python 2.5.2 (r252:60911, Jul 5 2008, 13:44:44) Still, something locks up in the system, when I increase the number Python threads with http://grub.org/?q=en/node/204 With more than 50 threads I start to get random messages like: Exception in thread 40: Traceback (most recent call last): File /usr/local/lib/python2.5/threading.py, line 486, in __bootstrap_inner self.run() The frequency of these messages increases with 70 threads. And with 200 threads the system locks up totally -- almost no access from web. [I can still access the system and kill the process :) ] Clearly, there is some limit in the system. It is not memory, cpu or bandwidth. If the number of threads is also not limited, it must be something else. What might it be? Network traffic is moderate: IP traffic Recall that python threads are just native threads, but these threads are protected from trashing the python state by a single lock, the GIL, which is released when you do IO. This allows to effectively thread IO, but not python computation. The FreeBSD thread library has no particular limitations, you can run hundreds or thousands of threads without much problem, for example under Java. Regards, O.K. [sorry for HTML message in the list] -- Testi oma Interneti kiirust / Test Your Internet speed: http://speedtest.zzz.ee/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Python with many threads
Otto wrote: Exception in thread 30: Traceback (most recent call last): File /usr/local/lib/python2.5/threading.py, line 486, in __bootstrap_in= ner self.run() The number of errors increases rapidly with even bigger number of threads. Is there any way I can increase the maximum number of Python threads? Nothing limits the number of concurrent threads. Personnally i have checked i can run Grub Next Generation Python Client with 600 threads without any problem. niobe% uname -a FreeBSD niobe.lpthe.jussieu.fr 7.0-STABLE FreeBSD 7.0-STABLE #0: Tue Jul 22 10:31:01 CEST 2008 niobe% python Python 2.5.2 (r252:60911, Jul 5 2008, 13:44:44) Recall that python threads are just native threads, but these threads are protected from trashing the python state by a single lock, the GIL, which is released when you do IO. This allows to effectively thread IO, but not python computation. The FreeBSD thread library has no particular limitations, you can run hundreds or thousands of threads without much problem, for example under Java. -- Michel TALON ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Python with many threads
Hello list, Trying to run Grub Next Generation Python Client (http://grub.org/?q=en/node/204) on my FreeDSD 7.1 box with as many threads as possible. Python version is 2.5.2. The question is, what limits the number of concurrent threads I can run? I have free memory and cpu, but starting from around 50 threads, Python starts to give random errors like this: Exception in thread 30: Traceback (most recent call last): File /usr/local/lib/python2.5/threading.py, line 486, in __bootstrap_inner self.run() The number of errors increases rapidly with even bigger number of threads. Is there any way I can increase the maximum number of Python threads? With best regards, O.K. -- Mõõda oma inteneti kiirust / Test Your Internet speed http://speedtest.zzz.ee/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
