Re: [sqlite] Changing file descriptor of database file
The select() limit has nothing to do with sqlite. You already noted it's a limit on sockets -- it's really an OS limit. Do you have any idea what your max is or what you think you need? I see some solutions and upcoming problems (by the way, you forgot to mention what OS you're on). #1 You're apparently handling your socket connections all from the same executing binary. Just give each client their own executable and you won't hit that limit. Two ways to do that are to fork() and also to spawn the executable uniquely for each client (so you don't run a server process -- you let xinetd do it for you so you get one process per socket. #2 Use poll() instead of select(). poll() doesnt have the 1024 limit. Michael D. Black Senior Scientist Northrop Grumman Mission Systems From: sqlite-users-boun...@sqlite.org on behalf of Pavel Ivanov Sent: Fri 5/7/2010 3:43 PM To: General Discussion of SQLite Database Subject: [sqlite] Changing file descriptor of database file Hi, all! I'm writing a server application that uses SQLite database as a storage. In fact for various reasons I use a lot of SQLite database files at once. This application can serve hundreds of clients spread all over the network. Communication with those clients goes over non-blocking sockets and currently select() is used to listen for events in those sockets. select() has limitation that it can be used only with file descriptors less than 1024. On a heavy loaded server my application can easily run out of those 1024 file descriptors which will mean that it will no longer accept connections from new clients. To avoid such situation or at least to make possibility of such event less I'd like to force SQLite to open databases with file descriptors greater than 1024. Is there a way how I can do that without patching SQLite sources and maintaining this patch over SQLite updates? Pavel ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Changing file descriptor of database file
For people wanting to re-do their networking code I would reccommend Asio (or Boost::Asio), see: http://think-async.com/Asio/ Ad. On 8/05/2010 9:14 AM, Roger Binns wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 05/07/2010 01:43 PM, Pavel Ivanov wrote: >> select() has limitation that it can be used >> only with file descriptors less than 1024. > > This is rarely true and I only know of one crappy operating system that had > that limitation 13 years ago. You can simply recompile with > - -DFD_SETSIZE=1 in all files that use fd_set to have a different "limit". > > However select() is a horrendous interface, especially beyond a trivial > number of file descriptors and there are lots of better ones out there with > a variety of attributes and support. I'd recommend using libevent which > abstracts all this stuff away. Heck, it even works on Windows! > >http://en.wikipedia.org/wiki/Libevent > > Roger > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkvknu8ACgkQmOOfHg372QRbTQCeKx8ZA++bGsvNCybLlg3P3uwl > eXkAoML1M+wGVvaoCWZaRT7OnwvpgPdD > =WLrY > -END PGP SIGNATURE- > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Changing file descriptor of database file
On Fri, May 07, 2010 at 05:24:58PM -0400, Mikhail Terekhov wrote: > IMHO it would be better to switch to poll/epoll from select instead of > fighting file descriptor numbers. Better: use libevent. (select()'s limit of 1024 fildes is... odd and derives mainly from the FD*() macros and their semantics. Not all operating systems enforce a limit of 1024 fildes as the obvious thing to do is to treat the fildes sets as bitmasks of nfds bits rounded up to a multiple of NFDBITS. Solaris does enfore a 1024 limit for select(), at run-time.) Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Changing file descriptor of database file
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/07/2010 01:43 PM, Pavel Ivanov wrote: > select() has limitation that it can be used > only with file descriptors less than 1024. This is rarely true and I only know of one crappy operating system that had that limitation 13 years ago. You can simply recompile with - -DFD_SETSIZE=1 in all files that use fd_set to have a different "limit". However select() is a horrendous interface, especially beyond a trivial number of file descriptors and there are lots of better ones out there with a variety of attributes and support. I'd recommend using libevent which abstracts all this stuff away. Heck, it even works on Windows! http://en.wikipedia.org/wiki/Libevent Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvknu8ACgkQmOOfHg372QRbTQCeKx8ZA++bGsvNCybLlg3P3uwl eXkAoML1M+wGVvaoCWZaRT7OnwvpgPdD =WLrY -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Changing file descriptor of database file
> IMHO it would be better to switch to poll/epoll from select instead of > fighting file descriptor numbers. Sure, that's my opinion too. It's just some significant amount of work which looks like less prioritized at this moment than other application improvements. So I'm looking for some solution for the short term. Pavel On Fri, May 7, 2010 at 5:24 PM, Mikhail Terekhov wrote: > On Fri, May 7, 2010 at 4:43 PM, Pavel Ivanov wrote: > >> Hi, all! >> >> I'm writing a server application that uses SQLite database as a >> storage. In fact for various reasons I use a lot of SQLite database >> files at once. This application can serve hundreds of clients spread >> all over the network. Communication with those clients goes over >> non-blocking sockets and currently select() is used to listen for >> events in those sockets. select() has limitation that it can be used >> only with file descriptors less than 1024. On a heavy loaded server my >> application can easily run out of those 1024 file descriptors which >> will mean that it will no longer accept connections from new clients. >> To avoid such situation or at least to make possibility of such event >> less I'd like to force SQLite to open databases with file descriptors >> greater than 1024. Is there a way how I can do that without patching >> SQLite sources and maintaining this patch over SQLite updates? >> >> > IMHO it would be better to switch to poll/epoll from select instead of > fighting file descriptor numbers. > > > Regards, > -- > Mikhail Terekhov > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Changing file descriptor of database file
On Fri, May 7, 2010 at 4:43 PM, Pavel Ivanov wrote: > Hi, all! > > I'm writing a server application that uses SQLite database as a > storage. In fact for various reasons I use a lot of SQLite database > files at once. This application can serve hundreds of clients spread > all over the network. Communication with those clients goes over > non-blocking sockets and currently select() is used to listen for > events in those sockets. select() has limitation that it can be used > only with file descriptors less than 1024. On a heavy loaded server my > application can easily run out of those 1024 file descriptors which > will mean that it will no longer accept connections from new clients. > To avoid such situation or at least to make possibility of such event > less I'd like to force SQLite to open databases with file descriptors > greater than 1024. Is there a way how I can do that without patching > SQLite sources and maintaining this patch over SQLite updates? > > IMHO it would be better to switch to poll/epoll from select instead of fighting file descriptor numbers. Regards, -- Mikhail Terekhov ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Changing file descriptor of database file
Hi, all! I'm writing a server application that uses SQLite database as a storage. In fact for various reasons I use a lot of SQLite database files at once. This application can serve hundreds of clients spread all over the network. Communication with those clients goes over non-blocking sockets and currently select() is used to listen for events in those sockets. select() has limitation that it can be used only with file descriptors less than 1024. On a heavy loaded server my application can easily run out of those 1024 file descriptors which will mean that it will no longer accept connections from new clients. To avoid such situation or at least to make possibility of such event less I'd like to force SQLite to open databases with file descriptors greater than 1024. Is there a way how I can do that without patching SQLite sources and maintaining this patch over SQLite updates? Pavel ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
