Wow, thanks for such a detailed reply. :)
>
>
> In message <[EMAIL PROTECTED]>, you wrote:
>
> >What's the practical number of TCP connections per server?
>
> I've gotten over 8,000 at one time on one FreeBSD box.
Yeah, best case, I've had several thousand myself, but not really doing
anything. :) I'm going to try to come up with a test suite for this sorta
thing, unless someone pipes in that it already exists.
> The biggest memory issue for a box that's handling a lot of TCP con-
> nections at a time is the socket I/O buffer sizes. Unless you take
> steps, programatically, to reduce these, you will get (I think) one
> 4KB buffer for input and another 4KB buffer for output. This is for
> EACH active TCP connection.
I hadn't considered this much buffering, but it makes sense.
>
> This can add up to a substantial amount of memory if you have a lot of
> connections.
>
> The way to solve that is to include calls to setsockopt() in your server
> that will have the effect of reducing the per-connection I/O buffer sizes
> just after you accept() each new connection.
Speaking of accepting... What's the upper limit on listen queues? Something
around 64, correct?
>
> Quite a lot of memory (either virtual or real) will also get sucked up
> *if* you have a separate and independent process handling each separate
> connection. A simple experiment I did awhile back indicated that on
> recent-vintage versions of FreeBSD, the absolute minimum per-process
> overhead was 12KB. That is a *minimum*, e.g. for a process that contains
> essentially no code and no data. But you will probably never see that in
> practice, which is to say your minimum per-process overhead is going to
> be bigger than that.
Yeah, I don't plan on doing things the apache way. :) One process per client
seems silly here, since nearly every client will be getting the exact same
data.
> The _clean_ way of doing it would be to write your multi-user server using
> threads, and to assign one thread to each connection. If you can do that,
> then the logic in the program becomes quite simple. Each thread just sits
> there, blocked on a call to read(), until something comes in, and then it
> just parses the command, does whatever it is supposed to do in response to
> that command, and then goes back to the read() again.
>
> But as I understand it, there is not yet sufficient threads support in the
> FreeBSD kernel to make this work well/properly. (I may perhaps be misinformed
> about that, but that's what I have been told anyway.)
I believe this is how ConferenceRoom works, so it seems ok, but I remember
the comments that FreeBSD was their least preferred platform because of
thread problems.
> The other way is just have your server be a single thread/process, and to
> have it keep one big list of all of the connections (i.e. socket fds) that
> it has open at present. Then it just executes mail loop, over and over
> again. At the top of the main look is one big honkin' call to select()
> in which you find out which of your connections is ready to be read or
> written. Then you go off and read/write those as appropriate, and then
> just come back and do the big select() again. (You could do this using
> calls to poll() instead of calls to select(), and that might be a bit
> more efficient.)
This is how traditional ircd's handle it, and how I was planning to. It
seems the most efficient, especially since 99.999% of the clients will be
receiving the same data, it'll make transmission pretty easy.
> If you want a lot of connections, start by increasing the values of
> "maxusers" and NMBCLUSTERS in your kernel config file. Then build and
> install a new kernel. How much is enough for these parameters? Beats
> me. If you find that you are running out of resources, increase them
> some more and then try again.
I really wish more of those options were dynamically tunable, instead of a
magic formula that maxusers controls. :)
Thanks again,
Kevin
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message