On Sat, May 12, 2012 at 10:42:24PM +0300, Bar Ziony wrote: > > > How can I know the average response time of my servers? haproxy provides > > > that data somewhere? > > > > Yes you have each response time value in your logs :-) > > But can I get the average response time?
By doing an average of what appears in your logs ! Oh if you want, halog can do it for you per URL or per server (halog -ua / halog -srv). > Also, this is the response time of > the backend, or the full response time ? These are the server response times, as well as the total times so you know what part is server-side computing and what part is data transfer (which might partially include the client's download time). > OK, got why there are more frontend sessions than backend sessions. But is > it usual to see so much more? Sometimes yes. I know a website which has excellent response times (<< 1ms) and very good connectivity between haproxy and the servers. For a long time we believed there was a bug with the reporting of the session counts, until we figured that since the time was 100* larger between the client and haproxy than between haproxy and the server, it was normal to have that large a ratio between the frontend and the backend! But if your server's response time is larger than the client-to-haproxy RTT, or if you have objects much larger than the buffers, the ratio between frontend and backend will be closer to 1. There is no rule, it all depends on your site. What you can say for sure is that the higher the ratio, the more savings you're doing on backend servers since you don't need to add more to stand the same load. > > It would only hurt if your average object size is larger than 7kB. And it > > would not hurt that much, it would only eat a bit more CPU because haproxy > > would have to perform twice the number of recv/send to forward data between > > sockets. If you're using mostly large objects (more than twice the buffer > > size), you can also enable "option splice-response" which will permit > > forwarding between the two sockets without user-land copy. It becomes > > insensible to the buffer size and generally saves some CPU cycles. > > > > How can I know if I mostly use small or large objects? You have this in your logs too, but more generally, I think you're the one who knows your site the best. You certainly have an idea of the object size distribution on your site, and of the dynamic object sizes, otherwise it's quite hard for you to guess how large your infrastructure has to be before being hit by the load. > > > I can increase the RAM if that will solve the > > > problem! I just wonder how it is possible that haproxy is using so much > > > RAM, when I didn't see so much RAM usage from my old single web server > > > (nginx). > > > > As I said, it all depends on how many connections you were having when > > you observed the issue. With the 16kB default buffers, 20k conns * (2 > > buffers + ~1kB for internal structs) will take around 650 MB of RAM. > > This plus the 200 MB for the kernel buffers almost reaches your VM > > memory. I forgot to say that you also have conntrack to account for. > > What is unsure is if you really reached those 20k conns. > > why 2 buffers? frontend + backend? No, just because of system call optimizations. You need at least one recv() to parse a request (or response), and at least another one to detect a close(). So each connection will have at least 2 recv() calls per direction, and since each recv() can fill a full buffer, there is generally nothing to save by enabling splice if you don't have enough data to fill between 1 and 2 buffers. > > > I don't want to configure stuff for a low-RAM machine if I actually need > > > more RAM. We have no problem paying for a bigger VPS (but unfortunately > > we > > > must stay on this VPS infrastructure). > > > > OK, then let's tune more finely and increase the RAM size at the same > > time : use the 8kB buffers in haproxy as I indicated above, and increase > > your RAM to 1.5-2GB to be safe. > > > > I'm just now upgrading my LB's (one at a time, failing them over as I > upgrade) to 2GB RAM, but I don't want to make the buffers change before I > understand the thing with the object sizes - it's really not dangerous ? Really not. For instance, I've just run a benchmark here. On a 2.6 GHz core2duo, with 30kB objects, I'm seeing : - 13600 req/s and 3.42 Gbps with 16kB buffers - 13440 req/s and 3.38 Gbps with 8kB buffers - 15440 req/s and 3.90 Gbps with 8kB buffers + splice on 64kB pipes - 16950 req/s and 4.26 Gbps with 8kB buffers + splice on 128kB pipes So you see, the difference between 16 and 8kB is very minor compared to what you can have with splicing (but splicing can vary a lot depending on network cards and drivers). > It's a regular web app, some file uploads, lots of images downloads (small > and big)... I don't know the average size of a "dynamic" response. Well, you have the network bandwidth of the dynamic servers at least (well I hope you're monitoring at least this). You just divide this bandwidth with these servers request rate and you'll have the average object size for these servers. It's very important for a web site to know its average response size. It directly impacts page load time for users, that's the most important indicator after the number of objects per page. You must absolutely figure out these elements otherwise I can predict that your site will never take off because you'll never be able to optimize it for your users. Regards, Willy
