Re: Configuration API?
> Do you have an example of what purpose it would serve ? I'm asking > because it's not very easy to implement with table-based algorithms, > since the size of the table is determined by the GCD of all active > servers' weights. Thus adding a new server will change the size of > the table. > > It's also a feature I've never seen on other products either, which > makes be doubt about its usefulness. Hey Willy, It's really useful in big organizations, when you have to manage tens and tens of LBs and you want to ease the management. An API available remotely allows you to write your own centralized management tools: - configuration backup - configuration edit and push - collection of statistics etc... cheers
Re: node frequently goes down on another physical machine
Hi Amit, Try a "netstat -in" and see if you have any errors on your interfaces :) might help to figure out if you have a duplex mismatch. cheers On Wed, Dec 29, 2010 at 8:04 AM, Amit Nigam wrote: > Hi Willy, > > Thanks for your support, makes me believe I would solve this riddle. > After updating to 1.4.10, sync-ing TC2 and LB1 times thru NTP, and using > options tcp-smart-connect and tcp-smart-accept, I have seen significant > improvments in server downtimes, retries and redispatches. But still I see > lots of retries though there are only 1 redispatch at TC2. > >>> Now in new stats page I noticed one thing which was not in 1.3.22 is >>> LastChk, but I wonder tc1 is showing L7OK/302 in 324ms _and tc2 is >>> showing >>> L7OK/302 in 104ms _ while currently haproxy is running on LB1 and there >>> are >>> 13 retries at TC2. >> >> The only explanation I can see is a network connection issue. What you >> describe looks like packet loss over the wire. It's possible that one >> of your NICs is dying, or that the network cable or switch port is >> defective. >> >> You should try to perform a file transfer between the machine showing >> issues and another one from the local network to verify this hypothesis. >> If you can't achieve wire speed, it's possible you're having such a >> problem. Then you should first move to another switch port (generally >> easy), then swap the cable with another one (possibly swap the cables >> between your two LBs if they're close) then try another port on the >> machine. > > We are on production, and servers are also in a data center, so wont be > possible to swap cabels. > To ascertain packet loss I carried out ping between LB1 and TC2 and TC1. LB1 > and TC1 avg time was 0.101 ms and LB1 to TC2 avg time was 0.382 ms on 64 > byte packet with 0% loss. >> >> Another possible explanation which becomes quite rare nowadays would >> be that you'd be using a forced 100Mbps full duplex port on your switch >> with a gigabit port on your server, which would negociate half duplex. >> You can check for that with "ethtool eth0" on your LBs and TCs. > > I checked we are using vSwitch for external and internal server > communication 1000mb full among v servers on a physical machine. v-server's > are using 1000mb full v-adapters. > > Following are the current stats: > TC1 Retr: 0, Redis: 0 Status OPEN 5h 52m UP, LastChk L70K/302 in 321 ms, > Server Chk: 4, Dwn 1, Dwntime 4m 17s. > TC2 Retr:1326 , Redis: 1 Status OPEN 4h 1m UP, LastChk L70K/302 in 87 ms, > Server Chk: 90, Dwn 2, Dwntime 26s. > Backend 5d 6m UP > > anyways what does LastChk signify? > > Regards, > Amit > - Original Message - From: "Willy Tarreau" > To: "Amit Nigam" > Cc: "Guillaume Bourque" ; > > Sent: Monday, December 27, 2010 11:25 AM > Subject: Re: node frequently goes down on another physical machine > > >> Hi Amit, >> >> On Fri, Dec 24, 2010 at 12:24:55PM +0530, Amit Nigam wrote: >> (...) >> I see nothing wrong in your configs which could justify your issues. >> >>> Now in new stats page I noticed one thing which was not in 1.3.22 is >>> LastChk, but I wonder tc1 is showing L7OK/302 in 324ms _and tc2 is >>> showing >>> L7OK/302 in 104ms _ while currently haproxy is running on LB1 and there >>> are >>> 13 retries at TC2. >> >> The only explanation I can see is a network connection issue. What you >> describe looks like packet loss over the wire. It's possible that one >> of your NICs is dying, or that the network cable or switch port is >> defective. >> >> You should try to perform a file transfer between the machine showing >> issues and another one from the local network to verify this hypothesis. >> If you can't achieve wire speed, it's possible you're having such a >> problem. Then you should first move to another switch port (generally >> easy), then swap the cable with another one (possibly swap the cables >> between your two LBs if they're close) then try another port on the >> machine. >> >> Another possible explanation which becomes quite rare nowadays would >> be that you'd be using a forced 100Mbps full duplex port on your switch >> with a gigabit port on your server, which would negociate half duplex. >> You can check for that with "ethtool eth0" on your LBs and TCs. >> >>> Also can this issue be due to time differences between cluster nodes? as >>> I >>> have seen there is a time difference of around 2 minutes between physical >>> machine 1 vms and physical machine 2 vms. >> >> While it's a bad thing to have machines running at different times, I >> don't see why it could cause any such issue. >> >> Regards, >> Willy >> > > > >
Re: feature request - slowdeath
Hey, A slowdown would be interesting if you want to avoid any huge traffic to be redirected to quickly to other backends. A nice feature, maybe already implemented, would be the "grace shutdown" of a backend. In case you use sticky session, "grace shutdown" would balance new incoming request to other backends and wait the established sessions to finish to consider the backend completely down. It's longer to shutdown a backend, but it's safer for your buisness ;) cheers On Sun, Dec 19, 2010 at 9:29 PM, David Birdsong wrote: > Hey Willy, > > Haproxy rocks! > > I've been using slowstart lately and was wondering if it would be > possible to add the opposite--something like 'slowdeath'. Some > external event would trigger, perhaps a very specific HTTP code, or a > command coming through the control socket, haproxy would un-weight the > server over a preconfigured interval down to zero. > > I have a script that essentially does this--before I discovered > slowstart it took care to bring the server up slowly. If slowdeath > existed, I could delete more code from this script! > >
Re: Server chk sanity check
On Thu, Dec 2, 2010 at 7:54 AM, Willy Tarreau wrote: > Hi Richard, > > On Tue, Nov 30, 2010 at 01:10:01PM +, Richard Shaw wrote: >> >> Hi, >> >> I've got two identical web servers physically close and on the same subnet, >> one of them shows a significantly higher number of server chk's (5320) than >> the other (0) and is periodically marked, very briefly (1-2s) as down. >> >> name,chk,dwn,dwntime >> w1,0,0,0s >> w2,5320,0,2s >> >> The configs for both are as follows >> >> server w1 neww1:80 weight 10 maxconn 120 cookie w1 check inter 2000 rise 1 >> fall 3 >> server w2 neww2:80 weight 10 maxconn 120 cookie w2 check inter 2000 rise 1 >> fall 3 >> >> and the check i'm performing is option httpchk /check.txt >> >> My understanding is that the 5320 is the number of httpchk's HAProxy has >> recorded as fail, is this correct ? > > Yes, that's correct. The 2s downtime with zero down transitions imply that > the server was detected as down upon start up. > > If both servers are the same, it is very likely that one of them has a faulty > network card or that the switch it's connected to has a problem on its port. Or maybe a wrong VLAN ;) cheers > > Willy > > >
Re: health check on httpd load average
Hi, I'm not (yet) an HAproxy expert, but your scenario is good enough if the application hosted on the backend servers is "sessionless" or not... If you don't need session, then your scenario could work. (be careful that all your backend won't answer a 5xx at the same time ;) ) If you need session then some users will loose information... To avoid loosing session, HAproxy should consider your backend as down for new incoming requests while keep on serving the established sessions: "graceful" mode. that way the load on your backend can decrease slowly and your user won't notice anything. Note: if your application needs session but it's smart enough to share sessions between backend servers, then your safe :) regards On Wed, Nov 17, 2010 at 11:40 PM, Maxime Ducharme wrote: > Hi guys > > I would like to accomplish to following with HAProxy : if a server show > a load average too high, indicate this server as down and remove it from > live traffic. When load avg get to a better state, put server back > online. > > Looking into documentation I think the following could work, I would > like to have your opinion. > > 1. create a PHP script that will check actual server status (we need to > ensure load average is not too high, maybe other checks would be acheive > also). According to the value we consider "bad", change current HTTP > status to something like 5xx. > > 2. Configure HAProxy to check on that page like this : > option httpchk HEAD /check-status.php HTTP/1.0 > > Does that make sense ? Is there other method of checking health of > server according to its current load average ? > > We are running high traffic sites, we are also wondering if this would > be more efficient to use > > http-check disable-on-404 > or > http-check expect > > instead. > > We think of not using an y regex pattern would be better for our > situation. > > Thanks and have a nice day > > -- > Maxime Ducharme > Systems Architect > > > > >
Re: Support for SSL
I wish I could use OpenSource solution. But my company refused so I had to follow their requirements (actually, the requirement was to use this specific software :D) and yes, our oldies do their job on SSL :) (If it works, don't fix it!!!) On Wed, Nov 17, 2010 at 5:05 PM, John Marrett wrote: > Bedis, > > At that kind of connection volume (I assume that your 20k/s includes a > certain quantity of keepalive, but a large volume of new connections as > well) I'm not that surprised that you needed dedicated hardware. That > said, I wouldn't expect the load to necessarily be that bad. I have > little experience at this level of transactional volume. > > If you have doubts about your application you could test using an open > source web server and see what kind of performance you're able to obtain > on the SSL component on the same hardware. > > We handle SSL using an open source web server that acts as a reverse > proxy sending the underlying http requests to haproxy which does the > actual load balancing. Seperating the two in this way might let you > avoid licensing costs, though it's probably unnecessary if the Nortel > box is holding up well for you. > >> Note: the software is not Varnish ;) > > Even if you pay for varnish support I doubt it's per node :) > > -JohnF > >
Re: Support for SSL
Hi John, Without entering too much in details, we have a mutualized reverse proxy cache platform in order to accelerate HTTP content (you can call it CDN ;) ) on which we use an HTTP reverse proxy caches coded by a third party company. The reverse proxy software run over a centos linux and has a cost (licence). The hardware is HP server (2U) with 2 quad cores, 32G of memory and a lot of hard drives. I can't remember the numbers exactly, but when we tested the SSL capacity inside the HTTP accelerator, we decreased a lot the overall performance (maybe because of the code) and to keep the same capacity (HTTP throughput) on our CDN we should have bought more servers and more licences. Without SSL enabled, we tested the HTTP accelerator with live traffic at more than 700MB/s, more than 20K HTTP Req/s and 80% of CPU... With only a small percentage of this traffic encrypted the performance decreased a lot, but I can't remember how much :/ Note that in a normal day, our caches run only at 10% of CPU and 100 Mb... Note that we did not saturated our servers with SSL, I'm just saying that to keep enough free capacity to absord customer's pike traffic, we should have bought more servers and licences and the cost would have been too much. It was cheaper to let our old vpn3050 in the racks doing the job :) Maybe it's related to the code of our supplier ;) Anyway, they were working on improving their SSL capacity by taking advantage of offloading the SSL computation to a daughter card into the chassis and keep on using the CPUs and memory to do HTTP acceleration, URL rewrite, ACLs, etc All the SMART stuff :) Note: the software is not Varnish ;) On Wed, Nov 17, 2010 at 3:46 PM, John Marrett wrote: > Bedis, > >> Cause using the cores to decrypt traffic would reduce drastically >> overall performance. >> Well, this is what we saw on our HTTP cache server (running CentOS) on >> 8 cores hardware: when enabling SSL, the performance were so bad that > >> So we kept our old Nortel vpn 3050 to handle the SSL traffic. > > I'm astonished to hear that you had these kinds of issues on modern > hardware. We stopped using dedicated SSL hardware quite some time ago. > > Of course, everyone's traffic is different. May I ask what volume of > traffic (Connections / second, Megabits) you are dealing with that > saturated an 8 core machine? > >> we should have ordered more chassis and licences to handle the same >> traffic... leading to earn less money :) > > What web/ssl server were you using and what version of CentOS. The use > of the word licenses is interesting :) > > Were you already high in CPU consumption without the SSL traffic on the > machine? > > -JohnF > >
Re: Support for SSL
Hi, note that SSL hardware acceleration card can allow you to offload your encrypted traffic and let the OS and software use 100% of the hardware for smarter stuff :) Cause using the cores to decrypt traffic would reduce drastically overall performance. Well, this is what we saw on our HTTP cache server (running CentOS) on 8 cores hardware: when enabling SSL, the performance were so bad that we should have ordered more chassis and licences to handle the same traffic... leading to earn less money :) So we kept our old Nortel vpn 3050 to handle the SSL traffic. regards On Tue, Nov 16, 2010 at 12:27 PM, Willy Tarreau wrote: > Hello, > > On Sun, Nov 07, 2010 at 04:15:18PM +0100, Sebastien Estienne wrote: >> Hello, >> >> Is there any news about SSL support? > > Yes there are some news, we'll have to work on it at Exceliance. > >> With current server's hardware having 8 cores or more, offering SSL is >> quite cheap. > > Hehe one thing at a time : haproxy right now only uses one core. Let's > first have SSL and only then see how we can make use of more cores. > The really difficult part is to try to use more cores without slowing > down the overall processing due to very common synchronization. This > implies massive changes to ensure that there's almost no shared data > between processes or threads. > >> Moreover with tools like firesheep getting widespread offering SSL to >> our users become an important feature > > Firesheep is doing nothing more than what has been done for decades with > many other tools. The same people who believe their traffic cannot be > sniffed by their coworker because they connect via a switch won't care > about having their SSL session hijacked with an invalid certificate. > >> I know that it's possible to use stunnel, but it would be better to >> have SSL support built in haproxy > > Yes indeed. At least stunnel already lets us assemble the bricks to > build whatever we want, eventhough the configs are sometimes tough ! > > Regards, > Willy > > >
Re: Limiting throughput with a cold cache
Hi, By the way, why using Squid? Have you already tried Varnish? It has a "grace" function which might help you :) During grace period, varnish serves stale (but cacheable) objects while retriving object from backend. Of course, it depends if that solution makes sense with your application. And Varnish would reduce disk IOs compared to squid, but forget ICP, they don't do it. You could then do a real tiered caching: users ==> haproxy ==> edge caches ==> backend caches ==> origin. edge caches can be Varnish and can do URL hashes to get content from backend layer to limit also the MISS rate and the load on origin. cheers
Re: Limiting throughput with a cold cache
Hi Dmitri, First, let me summarize your issue and tell me if I'm wrong. you have a haproxy balancing traffic to squid in reverse-proxy mode using hash URL as metric. The problem you have, is when a cache gets up after a crash, it's in trouble because of getting too much MISS requests. Are your HTTP backend servers slow? Have you tried some "tiered" caching? I mean, having a "backend" squid server the "edge" ones would request before going to the HTTP backend. the advantage: the backend squid will be HOT for all your objects. If a squid goes down, haproxy would balance his traffic to other squids, but the backend one will keep on learning objects. When the edge failed squid comes back, then haproxy will balance traffic to it again, whatever the number of request is. That squid will learn object from the backend one. I would do this configuration using ICP. all your edge squids would ask first your backend one for the object. If the backend has the object, then the edge will learn it, otherwise it has to go to the origin. All the most accessed object would remain in the back squid's memory. If the backend squid does not work, you can configure your edge squid to get content from the origin server directly. That way, you can limit the "extra" load on the edge squid when doing cold start. my 2 cents. if you have a lot of traffic from your edge to your backend, you can loadbalance traffic too :) cheers On Tue, Nov 16, 2010 at 1:43 AM, Dmitri Smirnov wrote: > Willy, > > thank you for taking time to respond. This is always thought provoking. > > On 11/13/2010 12:18 AM, Willy Tarreau wrote: >> >> Why reject instead of redispatching ? Do you fear that non-cached requests >> will have a domino effect on all your squids ? > > Yes, this indeed happens. Also, the objective is not to exceed the number of > connections from squids to backend database. In case of cold cache the > redispatch will cause a cache entry to be brought into the wrong shard which > is unlikely to be reused. Thus this would use up a valuable connection just > to satisfy one request. > > However, even this is an optimistic scenario. These cold cache situations > happen due to external factors like AWS issue, our home grown DNS messed up > (AWS does not provide DNS) and etc which causes not all of the squids to be > reported to proxies and messes up the distribution. This is because haproxy > is restarted after the config file is regenerated. > > I have been thinking about preserving some of the distribution using server > IDs when the set of squids partially changes but that's another story, let's > not digress. > > Thus even with redispatch enabled the other squid is unlikely to have free > connection slots because when it goes cold, most of them do. > > Needless to say, most of the other components in the system are also in > distress in case something happens on a large scale. So I choose the > stability of the system to be the priority even though some of the clients > will be refused service which happens to be the least of the evils. > >> >> I don't see any "http-server-close" there, which makes me think that your >> squids are selected upon the first request of a connection and will still >> have to process all subsequent requests, even if their URL don't hash to >> them. > > Good point. This is not the case, however, forceclose takes care of it and I > can see that most of the time the number of concurrently open connections to > any particular squid changes very quickly in a range of 0-3 even though each > of them handles a chunk requests per second. > > >> Your client timeout at 100ms seems way too low to me. I'm sure you'll >> regularly get truncated responses due to packet losses. In my opinion, >> both your client and server timeouts must cover at least a TCP retransmit >> (3s), so that makes sense to set them to at least 4-5 seconds. Also, do >> not forget that sometimes your cache will have to fetch an object before >> responding, so it could make sense to have a server timeout which is at >> least as big as the squid timeout. > > Agree. Right now server timeout in prod is 5s according to what is > recommended in the docs. In fact, I will probably reverse my timeout changes > to be inline with your recommendations. > > Having slept on the problem I came up with a fairly simple idea which is not > perfect but I think does most of the bang for such a simple change. > > It revolves around of adding a maxconn restriction for every individual > squid on in the backend. > > And the number can be easily calculated and then tuned after a loadtest. > > Let's assume I have 1 haproxy in front of a single squid. > > Furthermore, HIT latency: 5ms, MISS latency 200ms for simplicity. > > Incoming traffic 1,000 rps at peak. > > From squids to backend lets have 50 connections max, i.e. with 250 rps max. > > So through a single connection allowance for hot caches we will be able to > process 200 rps. For cold cache we will do only 5 rps. >
