On Tue, Oct 22, 2019, 10:27 PM Morotti, Romain D < romain.d.moro...@jpmorgan.com> wrote:
> Hello, > > > > The use case is to load balance applications in multiple datacenters or > regions. > > The common pattern today to cover multiple locations is to deploy services > in each location separately and independently. > > > > This happens with kubernetes for example, where a cluster is typically > limited to a datacenter. Covering multiple locations is done by having > independent clusters and provisioning the application to each of them. > > e.g. myapp.kube-naest1.example.com + myapp.kube-euwest.example.com. > > > > As a developer, I want to present the application on a single consistent > URL with failover, e.g. myapp.example.com. > > Without getting into too much details, this is done with a layer of load > balancing and this requires careful consideration around DNS, Host header, > TLS SNI, certificates and healthcheck to work end-to-end. > > > > Let's take the two most basic use cases. > > > > # multi clusters use case > > backend foo > > mode http > > option httpchk GET /healthcheck > > http-send-name-header Host > > server myapp.kube-naest1.example.com myapp.kube-naest1.example.com:80 > > server myapp.kube-naest2.example.com myapp.kube-naest2.example.com:80 > > > > # multi regions use case > > backend foo > > mode http > > option httpchk GET /healthcheck > > http-send-name-header Host > > server myapp.kube-naest1.example.com myapp.kube-naest1.example.com:80 > > server myapp.kube-euwest.example.com myapp.kube-euwest.example.com:80 > > > > Each backend expects its own Host header, otherwise kubernetes cannot > route and serve the request. > Can't you just set an Alias on your backend with all the expected domains to serve for that vhost? > In haproxy, this can be made to work using the "http-send-name-header > Host" directive, that overrides the Host header per backend. > > > > This setup fails in practice because of the healthcheck failing. The > healthcheck request sent by haproxy doesn't have a host header. It ignores > the "http-send-name-header" directive and there is no option to set the > healthcheck host per backend. > > (Similar challenge with TLS SNI, that is easily worked around by disabling > TLS checks. There are settings "sni req.hdr(host)" and "check-sni" to > adjust, with little information on all these settings combine). > > > > The proposed patch intended to add one setting per backend to manage the > host header end-to-end consistently and reliably: adjusting healthchecks, > forwarding requests and TLS domain. > > > > If you prefer not to do this, I can think of a less intrusive patch to > configure the healthcheck. > > Since there is already the "http-send-name-header Host" directive, whose > main use case is to adjust the Host header per backend. > > I could patch the healthcheck code to follow that directive when running > https healthchecks, if that’s okay with you. > > > > Thinking of it, it could be considered a bug that the healthcheck doesn't > do that already. If I configure all my servers to get a header, it’s > certainly important and I expect the healthcheck to get it too. > > > > Related questions and issues: > > > https://serverfault.com/questions/876871/configure-haproxy-to-include-host-headers-for-different-backends > > > https://serverfault.com/questions/770737/making-haproxy-pass-a-host-name-in-httpcheck > > > https://serverfault.com/questions/594669/haproxy-health-checking-multiple-servers-with-different-host-names > > > > > > Regards. > > > > > > *From:* Willy Tarreau [mailto:w...@1wt.eu] > *Sent:* 03 October 2019 05:51 > *To:* Morotti, Romain D (CIB Tech, GBR) <romain.d.moro...@jpmorgan.com> > *Cc:* haproxy@formilux.org; Sayar, Guy H (CIB Tech, GBR) < > guy.h.sa...@jpmorgan.com> > *Subject:* Re: [PR/FEATURE] support for virtual hosts / Host header per > server > > > > Hello Romain, > > On Tue, Oct 01, 2019 at 12:08:03PM +0000, Morotti, Romain D wrote: > > What is the status on this? > > Sorry, but it took some time to work on other priorities, and to be > honest, the subject looked scary enough to deserve enough time to > study it. Sadly, if I can say, the subject was pretty descriptive of > what it does, and this is fundamentally wrong. > > So just to summarize for those who haven't had a look at the patch, > what this patch does is to replace the host header in requests sent to > a server with one specified for this server, resulting in each server > within the same farm to run on a different vhost. This goes back to > the errors that plagued many hosting infrastructures in the late 90s > and early 2000s where redirects, hosts in pages etc were wrong because > the application was called with an internal name instead of the correct > one. Moreover this used to prevent servers from being shared between > multiple hosts since the host header was containing rubish. For > reference, Apache merged the ProxyPreserveHost in 2.0.31 in 2002 to put > an end to that madness. But here you're escalating this to a new level > and involving it in load balancing, 15 years after everyone managed to > fix that early mistake and bury it deeply forever. So in short for a > request sent to foo.bar.com, you'll randomly forward it as being for > foo1.bar.com, bar.foo2.com, www.example.org etc... all in the same farm! > It's as if when launching my browser on Amazon I was presented eBay, > Amazon and Aliexpress in a round robin fashion. I'm exagerating but it > really is the same principle. > > It is critically important that Host headers are not tampered with > anymore, especially in a load balanced setup where all nodes must > absolutely receive the exact same requests. Nowadays they're used > way more than in the early 2000, you have them in Set-Cookie, > Access-Control-Allow-Origin header with CORS, Location for redirects > (obviously), logs, statistics. It makes absolutely no sense to deploy > multiple servers in a same farm on different host names for the same > service. And no, migrating from an old Apache 1.3 config relying on > ProxyPass is not a valid excuse in 2019 for reintroducing such > architectural flaws! > > Such mistakes are still occasionally encountered in field, but extremely > rarely. Usually it's after their author, and single defender, quits the > company and their substitute discovers the mess and says "Houston, we > have a problem". But fortunately while I've met it very few times with > reverse proxies over the last decade, I never met it at all within a > load balancing farm in 18 years! > > I'm seeing this as the outcome of someone having shoot himself in the > foot, then aiming at the knee in order not to feel the foot's pain > anymore. But here I don't want to engage every users' knee in front > of this gun. > > For the part concerning the health checks however your point is totally > valid and I absolutely hate the "\r\nHost:foo" hack that we're relying on > from time to time. I'd like to see checks reworked so that we can pass > various headers etc. I'm not fond of having just a vhost directive to be > used by checks because 1) it's confusing and 2) we're just hiding the dust > under the carpet. We do need really improved checks and dealing with Host > only is not sufficient. Some users need to pass cookies, auth headers, > user-agent, or x-forwarded-for headers for example, and such configs are > really really dirty. Usually they switch to tcp-checks but then they cannot > use fancy features like disable-on-404. > > > > The lack of this functionality is a blocker to use HAProxy with kubernetes. > > I'd say that many existing haproxy users in kubernetes would probably > disagree with this bold claim. Saying that you're facing difficulties > and are seeking for configuration help would possibly be more accurate. > > In my opinion at this point it's preferable that you clearly describe > the problem you're facing so that it can be split in 3 parts : > > - architecture issues > - configuration issues > - haproxy limitations > > I really suspect that you're facing a bit of each, and that for example > you're having an issue with your checks being unusable or unreliable on > the *real* application, forcing you to check a dummy one running on the > server itself, which does expect to be called with its own name and that > since haproxy uses the same check requests for all hosts you're facing > difficulties running the dummy checks. But maybe fixing the application > so that it can really be checked would be the best solution, maybe > configuring the dummy check application with a proper host would be the > second best solution, maybe using HTTP/1.0 checks to get rid of Host > (as many people do notably on static servers to end up on the default > vhost) would work fine, or maybe we need to perform minimal changes to > the checks to deal with a specific case. > > Among the things I've been thinking about in the past regarding short > term improvements to the checks that could remain compatible with a > whole rewrite, I'd enumerate : > - a new directive "http-check send-cookie" to bring the ability to > automatically send the cookie header with the server's value when > configured ; some would probably suggest that it should always be > set and does not even require an option maybe. > > - the ability to define a general purpose argument on a server line > ("check-arg") that would be sent in any header after the checks, > and as a side effect could be used to send a Host header. > > For example we could have this : > > backend foo > # send "X-Farm-Name: foo" in all checks > server foo1 1.1.1.1:8000 check-arg foo > server foo2 1.1.1.2:8000 check-arg foo > http-check send-arg X-Farm-Name > > As well as this : > > backend foo > # send "X-Server-Name: foo1/2" in all checks > server foo1 1.1.1.1:8000 check-arg foo1 > server foo2 1.1.1.2:8000 check-arg foo2 > http-check send-arg X-Server-Name > > Or this : > > backend foo > # send "Host: foo1/2.example.internal" in all checks > server foo1 1.1.1.1:8000 check-arg foo1.example.internal > server foo2 1.1.1.2:8000 check-arg foo2.example.internal > http-check send-arg Host > > We could even be more forward-thinking and allow to define and reference > multiple such arguments : > > backend foo > # send "Host: foo1/2.example.internal" in all checks > server foo1 1.1.1.1:8000 check-arg(foo,foo1.example.internal) > server foo2 1.1.1.2:8000 check-arg(foo,foo2.example.internal) > http-check send-arg(1) X-Farm-Name > http-check send-arg(2) Host > > These are just ideas of course and code speaks better. > > If you think that your issue really is only related to health checks > (which would then be quite a relief) and that something like above would > suit your needs, I'm willing to help you get this merged before the code > freeze. I just don't want to spend extra energy for nothing :-/ > > Ideas and suggestions are welcome. > > Thanks, > Willy > > This message is confidential and subject to terms at: > https://www.jpmorgan.com/emaildisclaimer including on confidential, > privileged or legal entity information, viruses and monitoring of > electronic messages. If you are not the intended recipient, please delete > this message and notify the sender immediately. Any unauthorized use is > strictly prohibited. >
