Secondary load balancing method (fallback)
Is it possible to configure a secondary load balancing method, something to fall back on if the first method isn't met? For example, if I balance on the url_param email: balance url_param email Can it instead balance on another url_param: balance url_param id Or have it balance based on source address? I tried setting the following: balance url_param email balance url_param id But it only balanced on the second one, id. I haven't found anything saying this is possible, but I'd just like to make sure it isn't. Thanks.
Re: haproxy not creating stick-table entries fast enough
On Fri, May 12, 2017 at 10:46 AM, Willy Tarreau <w...@1wt.eu> wrote: > On Fri, May 12, 2017 at 10:20:02AM -0700, redundantl y wrote: > > As I've said before, the issue here is these objects aren't hosted on the > > same server that they're being called from. > > > > "A separately hosted application will generate HTML with several (20-30) > > elements that will be loaded simultaneously by the end user's browser." > > > > So a user might go to www.example.com and that page will load the > objects > > from assets.example.com, which is a wholly separate server. > > OK but *normally* if there's parallelism when downloading objects from > assets.example.com, then there's no dependency between them. > > > > The principle of stickiness is to ensure that subsequent requests will > go > > > to the same server that served the previous ones. The main goal is to > > > ensure that all requests carrying a session cookie will end up on the > > > server which holds this session. > > > > > > Here as Lukas explained, you're simulating a browser sending many > totally > > > independant requests in parallel. There's no reason (nor any way) that > > > any equipment in the chain would guess they are related since they > could > > > arrive in any order, and even end up on multiple nodes. > > > > > > > > Well, all of these requests will have the url_param email=, so the load > > balancer has the ability to know they are related. The issue here, at > > least how it appears to me, is since they come in so fast the stick-table > > entry doesn't get generated quickly enough and the requests get > distributed > > to multiple backend servers and eventually stick to just one. > > It's not fast WRT the stick table but WRT the time to connect to the > server. > As I mentionned, the principle of stickiness is to send subsequent requests > to the same server which *served* the previous ones. So if the first > request > is sent to server 1, the connection fails several times, then it's > redispathed > to server 2 and succeeds, it will be server 2 which will be put into the > table > so that next connections will go there as well. > > In your workload, there isn't even the time to validate the connection to > the > server, and *this* is what causes the problem you're seeing. > > Thank you for explaining what I'm seeing. This makes a lot of sense. > > Since changing to load balancing on the url_param our issue has been > > resolved. > > So indeed you're facing the type of workloads requiring a hash. > > > > Also, most people prefer not to apply stickiness for static objects so > that > > > they can be retrieved in parallel from all static servers instead of > all > > > hammering the same server. It might possibly not be your case based on > your > > > explanation, but this is what people usually do for a better user > > > experience. > > > > > > > > The objects aren't static. When they're loaded the application makes > some > > calls to external services (3rd party application, database server) to > > produce the desired objects and links. > > OK I see. Then better stick to the hash using url_param. You can improve > this by combining it with stick anyway if your url_params are frequently > reused (eg: many requests per client). This will avoid redistributing > innocent connections in the event a server is added or removed due to > the hash being recomputed. That can be especially true of your 3rd party > application sometimes has long response times and the probability of a > server outage between the first and the last request for a client becomes > high. > > Thank you for pointing this out, we hadn't considered this scenario. > > > In conclusion, your expected use case still seem quite obscure to me > :-/ > > > > > > Willy > > > > > > > I agree, our use case is fairly unique. > > It looks so :-) > > Willy > Thanks for taking the time to read and respond. It was very informative and helpful.
Re: haproxy not creating stick-table entries fast enough
On Fri, May 12, 2017 at 12:51 AM, Willy Tarreau <w...@1wt.eu> wrote: > On Tue, May 09, 2017 at 09:43:22PM -0700, redundantl y wrote: > > For example, I have tried with the latest versions of Firefox, Safari, > and > > Chrome. With 30 elements on the page being loaded from the server > they're > > all being loaded within 70ms of each other, the first 5 or so happening > on > > the same millisecond. I'm seeing similar behaviour, being sent to > > alternating backend servers until it "settles" and sticks to just one. > > That's only true after the browser starts to retrieve the main page which > gives it the indication that it needs to request such objects. You *always* > have a first request before all other ones. The browser cannot guess it > will have to retrieve many objects out of nowhere. > > As I've said before, the issue here is these objects aren't hosted on the same server that they're being called from. "A separately hosted application will generate HTML with several (20-30) elements that will be loaded simultaneously by the end user's browser." So a user might go to www.example.com and that page will load the objects from assets.example.com, which is a wholly separate server. > The principle of stickiness is to ensure that subsequent requests will go > to the same server that served the previous ones. The main goal is to > ensure that all requests carrying a session cookie will end up on the > server which holds this session. > > Here as Lukas explained, you're simulating a browser sending many totally > independant requests in parallel. There's no reason (nor any way) that > any equipment in the chain would guess they are related since they could > arrive in any order, and even end up on multiple nodes. > > Well, all of these requests will have the url_param email=, so the load balancer has the ability to know they are related. The issue here, at least how it appears to me, is since they come in so fast the stick-table entry doesn't get generated quickly enough and the requests get distributed to multiple backend servers and eventually stick to just one. > If despite this that's what you need (for a very obscure reason), then > you'd rather use hashing for this. It will ensure that the same > distribution > algorithm is applied to all these requests regardless of their ordering. > But > let me tell you that it still makes me feel like you're trying to address > the wrong problem. > > Since changing to load balancing on the url_param our issue has been resolved. > Also, most people prefer not to apply stickiness for static objects so that > they can be retrieved in parallel from all static servers instead of all > hammering the same server. It might possibly not be your case based on your > explanation, but this is what people usually do for a better user > experience. > > The objects aren't static. When they're loaded the application makes some calls to external services (3rd party application, database server) to produce the desired objects and links. > In conclusion, your expected use case still seem quite obscure to me :-/ > > Willy > I agree, our use case is fairly unique.
Re: haproxy not creating stick-table entries fast enough
On Tue, May 9, 2017 at 2:11 PM, Lukas Tribus <lu...@gmx.net> wrote: > Hello, > > > Am 09.05.2017 um 02:52 schrieb redundantl y: > > The way ab is being executed is inline with our real world use. A > > separately hosted application will generate HTML with several (20-30) > > elements that will be loaded simultaneously by the end user's > > browser. There isn't a delay, the elements aren't loaded sequentially. > > I understand that, but I still doubt that a browser will open 10 > concurrent TCP connections within a millisecond. Also browser ALWAYS > keepalive, which you don't consider here (I don't know if you enabled > keepalive in haproxy though). I strongly suggest you look at the actual > browsers behavior (when keepalive is enabled in haproxy), before you > continue investing your time with a problem that may be hypothetical. > > It does open a lot of concurrent connections, within tens of milliseconds of each other. For example, I have tried with the latest versions of Firefox, Safari, and Chrome. With 30 elements on the page being loaded from the server they're all being loaded within 70ms of each other, the first 5 or so happening on the same millisecond. I'm seeing similar behaviour, being sent to alternating backend servers until it "settles" and sticks to just one. >From what I've read, keep-alive is enabled by default with haproxy 1.5 on CentOS 7. > > > So is this behaviour we're seeing with haproxy expected? There's no > > additional options we can try to make it create and use a stick-table > > entry faster? > > This is not about stick-table being slow. Its about requests being > elaborated concurrently in haproxy, so one cannot depend on the other. > To get what you need, haproxy would have to block/queue incoming > requests (I guess you could "maxconn 1" in your frontend, but don't you > run this in production). > > >From what I can tell this is about the stick-table entry not being created fast enough. I'd love to be wrong about this. > > > In the mean time, we've changed the load balancing method from round > > robin to URI and are seeing the behaviour we desire, it'll just carry > > the risk of not having an even distribution of load to the backend > > servers. > > Concurrent requests from ab combined with concurrent request handling in > haproxy with roundrobin balancing will lead to this behavior. I don't > see how haproxy could behave differently in this configuration. > > > cheers, > lukas > > I misspoke, I didn't change it to URI. Instead of sticking on a url_param I'm now load balancing on the url_param desired. Specifically I have the following in my backend configuration: balance url_param email haproxy is creating a hash based on the value of that parameter and deciding which backend server to send it to, so all connections (whether I use ab or one of three web browsers) are being sent to a single backend server. My concern with this was if the hashing method being used was random enough to create an even split between the backend servers. So far in our testing it appears that it is pretty evenly split.
Re: haproxy not creating stick-table entries fast enough
On Mon, May 8, 2017 at 4:01 PM, Lukas Tribus <lu...@gmx.net> wrote: > Hello, > > > Am 09.05.2017 um 00:38 schrieb redundantl y: > > I am running haproxy 1.5.18-3 on CentOS 7 and need to use the > > stick-table feature to make sure traffic for a specific user persists > > to a given server. > > > > Things work fine when connections come in slowly, however when there's > > numerous simultaneous connections and a stick-table entry doesn't > > exist yet some requests will be sent to both backend servers until > > they eventually stick to just one. > > > > For example, using Apache Bench to test: > > > > ab -c 10 -n 30 'http://example.com/index.php?email=a...@example.com' > > > > I see this in the haproxy log: > > > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50812 > > <http://1.2.3.4:50812> [08/May/2017:14:49:10.934] http_front > > backend/server-1 0/0/0/7/7 200 222 - - 9/9/9/4/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50811 > > <http://1.2.3.4:50811> [08/May/2017:14:49:10.933] http_front > > backend/server-2 0/0/0/8/8 200 222 - - 8/8/8/4/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50816 > > <http://1.2.3.4:50816> [08/May/2017:14:49:10.935] http_front > > backend/server-1 0/0/0/7/7 200 222 - - 7/7/7/1/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50819 > > <http://1.2.3.4:50819> [08/May/2017:14:49:10.935] http_front > > backend/server-2 0/0/1/6/7 200 222 - - 6/6/6/1/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50814 > > <http://1.2.3.4:50814> [08/May/2017:14:49:10.935] http_front > > backend/server-1 0/0/0/7/7 200 222 - - 5/5/5/1/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50810 > > <http://1.2.3.4:50810> [08/May/2017:14:49:10.933] http_front > > backend/server-1 0/0/0/9/9 200 222 - - 4/4/4/0/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50813 > > <http://1.2.3.4:50813> [08/May/2017:14:49:10.934] http_front > > backend/server-2 0/0/0/8/8 200 222 - - 3/3/3/0/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50815 > > <http://1.2.3.4:50815> [08/May/2017:14:49:10.935] http_front > > backend/server-2 0/0/0/7/7 200 222 - - 2/2/2/0/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50817 > > <http://1.2.3.4:50817> [08/May/2017:14:49:10.935] http_front > > backend/server-2 0/0/0/7/8 200 222 - - 1/1/1/0/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50818 > > <http://1.2.3.4:50818> [08/May/2017:14:49:10.935] http_front > > backend/server-1 0/0/1/6/8 200 222 - - 0/0/0/0/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50820 > > <http://1.2.3.4:50820> [08/May/2017:14:49:10.967] http_front > > backend/server-1 0/0/0/5/5 200 222 - - 3/3/2/2/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50821 > > <http://1.2.3.4:50821> [08/May/2017:14:49:10.968] http_front > > backend/server-1 0/0/0/4/4 200 222 - - 2/2/2/2/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50823 > > <http://1.2.3.4:50823> [08/May/2017:14:49:10.972] http_front > > backend/server-1 0/0/1/5/6 200 222 - - 7/7/7/7/0 0/0 "GET > > /index.php?email=a...@example.com <mailto:a...@example.com> HTTP/1.0" > > May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50822 > > <http://1.2.3.4:50822> [08/May/2017
haproxy not creating stick-table entries fast enough
I am running haproxy 1.5.18-3 on CentOS 7 and need to use the stick-table feature to make sure traffic for a specific user persists to a given server. Things work fine when connections come in slowly, however when there's numerous simultaneous connections and a stick-table entry doesn't exist yet some requests will be sent to both backend servers until they eventually stick to just one. For example, using Apache Bench to test: ab -c 10 -n 30 'http://example.com/index.php?email=a...@example.com' I see this in the haproxy log: May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50812 [08/May/2017:14:49:10.934] http_front backend/server-1 0/0/0/7/7 200 222 - - 9/9/9/4/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50811 [08/May/2017:14:49:10.933] http_front backend/server-2 0/0/0/8/8 200 222 - - 8/8/8/4/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50816 [08/May/2017:14:49:10.935] http_front backend/server-1 0/0/0/7/7 200 222 - - 7/7/7/1/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50819 [08/May/2017:14:49:10.935] http_front backend/server-2 0/0/1/6/7 200 222 - - 6/6/6/1/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50814 [08/May/2017:14:49:10.935] http_front backend/server-1 0/0/0/7/7 200 222 - - 5/5/5/1/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50810 [08/May/2017:14:49:10.933] http_front backend/server-1 0/0/0/9/9 200 222 - - 4/4/4/0/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50813 [08/May/2017:14:49:10.934] http_front backend/server-2 0/0/0/8/8 200 222 - - 3/3/3/0/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50815 [08/May/2017:14:49:10.935] http_front backend/server-2 0/0/0/7/7 200 222 - - 2/2/2/0/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50817 [08/May/2017:14:49:10.935] http_front backend/server-2 0/0/0/7/8 200 222 - - 1/1/1/0/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50818 [08/May/2017:14:49:10.935] http_front backend/server-1 0/0/1/6/8 200 222 - - 0/0/0/0/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50820 [08/May/2017:14:49:10.967] http_front backend/server-1 0/0/0/5/5 200 222 - - 3/3/2/2/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50821 [08/May/2017:14:49:10.968] http_front backend/server-1 0/0/0/4/4 200 222 - - 2/2/2/2/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50823 [08/May/2017:14:49:10.972] http_front backend/server-1 0/0/1/5/6 200 222 - - 7/7/7/7/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" May 8 14:49:10 localhost haproxy[4996]: 1.2.3.4:50822 [08/May/2017:14:49:10.972] http_front backend/server-1 0/0/0/8/8 200 222 - - 6/6/6/6/0 0/0 "GET /index.php?email=a...@example.com HTTP/1.0" [...] After this point haproxy correctly sends all traffic to server-1. When the stick-table entry expires the problem occurs again. I have tried persisting off a url parameter and source address, both exhibit the same issue. Is haproxy unable to properly handle numerous simultaneous (concurrent) requests like this? Is there something I can do to get this to work as desired? Thanks.
Re: Using multiple url parameters to create sticky-table entries
That seems to work exactly as needed, thank you! I have so much to learn about haproxy, this helps immensely. I didn't even consider looking into defining custom headers and using them this way. Thanks! On Sat, Mar 25, 2017 at 3:13 AM, Jarno Huuskonen <jarno.huusko...@uef.fi> wrote: > Hi, > > On Fri, Mar 24, redundantl y wrote: > > I take it this isn't possible? > > How about something like this: > frontend xyz > http-request set-header X-STICK %[urlp(email),lower]%[urlp( > newsletter_id),lower] > > backend ... > stick on hdr(X-STICK) table detailspage > > -Jarno > > > Thanks. > > > > On Wed, Mar 22, 2017 at 6:01 PM, redundantl y <jerry...@gmail.com> > wrote: > > > > > I'm very, very new to HAproxy (started trying it out an hour ago) and > am > > > looking at setting up a sticky-table based on multiple url_params. > > > > > > I've got it working using a single parameter, but I'm wondering if I > can > > > set it up to create table entries (keys) based on multiple url_param > > > entries when found. For example using a URL like this: > > > > > > http://example.com/details?email=j...@example.com_id=12345 > > > > > > I'd like it to create a key based on the strings in both the email and > > > newsletter_id parameters. Right now if I use the following it'll use > just > > > email or newsletter_id if the former isn't found: > > > > > > stick on url_param(email) table detailspage > > > stick on url_param(newsletter_id) table detailspage > > > > > > What the sticky-table currently looks like: > > > > > > $ echo "show table detailspage" | socat unix:/var/run/haproxy.sock - > > > # table: detailspage, type: string, size:5242880, used:3 > > > 0x7f90f0cc9a24: key=12345 use=0 exp=596822 server_id=1 > > > 0x7f90f0cc9754: key=j...@example.com use=0 exp=593655 server_id=1 > > > 0x7f90f0cc9964: key=harryca...@example.com use=0 exp=595455 > server_id=2 > > > > > > Thanks. > > > > > -- > Jarno Huuskonen >
Re: Using multiple url parameters to create sticky-table entries
I take it this isn't possible? Thanks. On Wed, Mar 22, 2017 at 6:01 PM, redundantl y <jerry...@gmail.com> wrote: > I'm very, very new to HAproxy (started trying it out an hour ago) and am > looking at setting up a sticky-table based on multiple url_params. > > I've got it working using a single parameter, but I'm wondering if I can > set it up to create table entries (keys) based on multiple url_param > entries when found. For example using a URL like this: > > http://example.com/details?email=j...@example.com_id=12345 > > I'd like it to create a key based on the strings in both the email and > newsletter_id parameters. Right now if I use the following it'll use just > email or newsletter_id if the former isn't found: > > stick on url_param(email) table detailspage > stick on url_param(newsletter_id) table detailspage > > What the sticky-table currently looks like: > > $ echo "show table detailspage" | socat unix:/var/run/haproxy.sock - > # table: detailspage, type: string, size:5242880, used:3 > 0x7f90f0cc9a24: key=12345 use=0 exp=596822 server_id=1 > 0x7f90f0cc9754: key=j...@example.com use=0 exp=593655 server_id=1 > 0x7f90f0cc9964: key=harryca...@example.com use=0 exp=595455 server_id=2 > > Thanks. >
Using multiple url parameters to create sticky-table entries
I'm very, very new to HAproxy (started trying it out an hour ago) and am looking at setting up a sticky-table based on multiple url_params. I've got it working using a single parameter, but I'm wondering if I can set it up to create table entries (keys) based on multiple url_param entries when found. For example using a URL like this: http://example.com/details?email=j...@example.com_id=12345 I'd like it to create a key based on the strings in both the email and newsletter_id parameters. Right now if I use the following it'll use just email or newsletter_id if the former isn't found: stick on url_param(email) table detailspage stick on url_param(newsletter_id) table detailspage What the sticky-table currently looks like: $ echo "show table detailspage" | socat unix:/var/run/haproxy.sock - # table: detailspage, type: string, size:5242880, used:3 0x7f90f0cc9a24: key=12345 use=0 exp=596822 server_id=1 0x7f90f0cc9754: key=j...@example.com use=0 exp=593655 server_id=1 0x7f90f0cc9964: key=harryca...@example.com use=0 exp=595455 server_id=2 Thanks.
