Hi Willy :) On Sat, May 12, 2012 at 10:06 PM, Willy Tarreau <[email protected]> wrote:
> On Sat, May 12, 2012 at 08:43:43PM +0300, Bar Ziony wrote: > > So session rate is the number of requests per second ? Why is it called > > session then if it's really requests? > > You have the two. Initially in haproxy, you had no keepalive, so > 1 req = 1 session. Now you have the numbers in the session column, > and if you pass your mouse over the number, you'll see the requests > too. > Great, just saw that. and that's only for the frontend because only it enables keepalive and then req/sec > session/sec ? > > > And "Sessions" is just plain sessions number, without caring for how much > > of them were happening in 1 sec? > > "sessions cur" is the number of concurrent sessions observed at the moment > it is reported. It is not related to any timing, it's what is observed at > one instant. To keep the analogy with the highway, it's how many lanes are > occupied at the precise instant you're taking the snapshot. > > > How can I know the average response time of my servers? haproxy provides > > that data somewhere? > > Yes you have each response time value in your logs :-) > But can I get the average response time? Also, this is the response time of the backend, or the full response time ? > > > I have a max of 800 requests in the backend queue (none in the servers > > queue since there is no persistence). Is that a lot ? :| > > It depends. If you're doing 800 reqs/s, you know that on average it will > take one second to drain these 800 requests, so it can be a lot. But if > you're facing an exceptional event, maybe you accept to delay requests > by up to 1s instead of seeing your servers die or stop responding. > > > I also see 3,400 sessions in the frontend, and only ~100 in the dynamic > > backend and 15 in the static backend (in the "cur" column). How is that > > possible? So many requests are not valid, or sessions are kept and are > not > > for 1 request only ? :\ I don't understand that.. > > It's because a connection is only forwarded to the backend once the client > has sent a request. And for some clients, sending a request takes some time > (eg: large requests, or simply because of poor network connectivity), so > you're always having more connections on the frontend than on the backend. > Also, haproxy closes the connection to the server as soon as it has the > last byte of the response, but it still forwards those data to the client > (so it acts as a TCP buffer). During this response buffering, the clients > are still connected to the frontend but the backend is already released. > > This behaviour enables some multiplexing of the server connections, because > they never remain idle, even if the clients are slow to read the responses. > > OK, got why there are more frontend sessions than backend sessions. But is it usual to see so much more? > > I'm sorry but I didn't quite get what does Concurrency means. > > It is the number of parallel sessions you have at one instant. When you do > "netstat -an | grep -c ESTAB", you get a number of concurrent connections. > > > Connection/sec * response time ? Why is that = concurrency? > > If you're not used to this, you need to draw it on paper to understand. > > Imagine a road passing on a bridge. Your bridge is designed to support > 100 cars. This is its concurrency limit. The response time is the time > a car spends on the bridge. The cars enter the bridge at a rate of 4 > per second. If the cars last more than 25 seconds on the bridge, you'll > have more than 100 cars on your bridge and it might break. If you make > your cars run faster on the bridge, they will last there less time and > there will be less cars on it. If you are on a holiday season, you'll > get a higher "car rate" at the input of the bridge, and if they don't > run faster, you'll break the limit again. > > Cool, thanks :) > > Here they are: > > net.ipv4.tcp_mem = 24372 32496 48744 > > net.ipv4.tcp_wmem = 4096 16384 1039872 > > net.ipv4.tcp_rmem = 4096 87380 1039872 > > > > Are those valid? > > So you have between 24372*4096 and 48744*4096 = 100..200 MB of RAM > assigned to the TCP stack, which is fine considering your VM size. > Your read and write buffers are correct too (the kernel automatically > adjusts them depending on the available memory). > > However you have to be aware that a socket buffer needs at least 4kB > in each direction (hence why the min limit is 4kB), so 200 MB limits > you to 200/2/4 = 25k sockets, 20k of which can be on the frontend side > and 5k of which may be on the backend side. > Didn't get much of that, I'll read about tcp_[rm]?mem some more :) > > > > You can reduce haproxy's memory usage by reducing buffer sizes this > > > way here : > > > tune.bufsize 8030 > > > tune.maxrewrite 1030 > > > > But would this hurt somehow? > > It would only hurt if your average object size is larger than 7kB. And it > would not hurt that much, it would only eat a bit more CPU because haproxy > would have to perform twice the number of recv/send to forward data between > sockets. If you're using mostly large objects (more than twice the buffer > size), you can also enable "option splice-response" which will permit > forwarding between the two sockets without user-land copy. It becomes > insensible to the buffer size and generally saves some CPU cycles. > How can I know if I mostly use small or large objects? > > > I can increase the RAM if that will solve the > > problem! I just wonder how it is possible that haproxy is using so much > > RAM, when I didn't see so much RAM usage from my old single web server > > (nginx). > > As I said, it all depends on how many connections you were having when > you observed the issue. With the 16kB default buffers, 20k conns * (2 > buffers + ~1kB for internal structs) will take around 650 MB of RAM. > This plus the 200 MB for the kernel buffers almost reaches your VM > memory. I forgot to say that you also have conntrack to account for. > What is unsure is if you really reached those 20k conns. > why 2 buffers? frontend + backend? > > > I don't want to configure stuff for a low-RAM machine if I actually need > > more RAM. We have no problem paying for a bigger VPS (but unfortunately > we > > must stay on this VPS infrastructure). > > OK, then let's tune more finely and increase the RAM size at the same > time : use the 8kB buffers in haproxy as I indicated above, and increase > your RAM to 1.5-2GB to be safe. > I'm just now upgrading my LB's (one at a time, failing them over as I upgrade) to 2GB RAM, but I don't want to make the buffers change before I understand the thing with the object sizes - it's really not dangerous ? It's a regular web app, some file uploads, lots of images downloads (small and big)... I don't know the average size of a "dynamic" response. > > > Only nginx is running on this machine as well to terminate SSL, but it > > seems like haproxy is the one that consumes all memory. Nothing else is > > running on the machines besides syslog for haproxy and the machine > itself, > > regular processes and munin plugins every 5 minutes (which are not > causing > > any RAM issues)... > > OK, so it should be easy to get the correct numbers once for all. > > Regards, > Willy > > Thanks! Bar.
