Hi,
this is an addition to the cases I sent previously, I accidentally found out
that haproxy behaves differently when the last header is not a "Connection:"
header.
This is a case for config #2 ("reqidel ^X-Forwarded-For:.*" is set):
case i): stays on the same backend.
nc 127.0.0.1 8085 <<EOF
GET / HTTP/1.1
Host: www.google.de
X-Forwarded-For: 127.0.0.1
Connection: keep-alive
Some-Header: foobar
EOF
Expected behaviour:
Case i) and h) *must* behave the same. Some browsers reorder the headers, so
you might have a "Cookie:" after a "Connection:" header, thus some users will
be loadbalanced to the same backend, some will jump between backends.
A really mean bug...
----- original Nachricht --------
Betreff: Re: Re: balance (hdr) problem (maybe bug?)
Gesendet: Fr, 11. Feb 2011
Von: Craig Craig<[email protected]>
> Hi,
>
> I decided to narrow the bug a bit and deleted all other backends/frontends
> we have; I've defined three servers in the backend which all query
> www.google.de. Thus you do not have to set up your own server if you want to
> test this config, google can take the load. ;)
>
> The problem is reproducable with this config, I used need netcat here.
>
> Running with #1 config ("reqidel ^X-Forwarded-For:.*" in frontend_btg not
> set).
>
> case a) jumps between backends:
> nc 127.0.0.1 8085 <<EOF
> GET / HTTP/1.1
> Host: www.google.de
> Connection: keep-alive
>
> EOF
>
> case b) stays on same backend:
> nc 127.0.0.1 8085 <<EOF
> GET / HTTP/1.1
> Host: www.google.de
> Connection: close
>
> EOF
>
> case c) stays on same backend:
> nc 127.0.0.1 8085 <<EOF
> GET / HTTP/1.1
> Host: www.google.de
> X-Forwarded-For: 127.0.0.1
> Connection: close
>
> EOF
>
> case d) stays on same backend:
> nc 127.0.0.1 8085 <<EOF
> GET / HTTP/1.1
> Host: www.google.de
> X-Forwarded-For: 127.0.0.1
> Connection: keep-alive
>
> EOF
>
> Expected behaviour:
> Case a) should not jump between servers. An empty x-forwarded-for header
> means that always the "same" header (an empty one) should be hashed, you
> should always end up on the same server.
> Case a) and b) should behave the same. What does it matter if the connection
> is set to keep-alive or close, I've set option httpclose anyways.
>
>
> Running with #2 config ("reqidel ^X-Forwarded-For:.*" in frontend_btg is
> set).
>
> case e) jumps between backends:
> nc 127.0.0.1 8085 <<EOF
> GET / HTTP/1.1
> Host: www.google.de
> Connection: keep-alive
>
> EOF
>
> case f) stays on same backend:
> nc 127.0.0.1 8085 <<EOF
> GET / HTTP/1.1
> Host: www.google.de
> Connection: close
>
> EOF
>
> case g) stays on same backend:
> nc 127.0.0.1 8085 <<EOF
> GET / HTTP/1.1
> Host: www.google.de
> X-Forwarded-For: 127.0.0.1
> Connection: close
>
> EOF
>
> case h) jumps between backends:
> nc 127.0.0.1 8085 <<EOF
> GET / HTTP/1.1
> Host: www.google.de
> X-Forwarded-For: 127.0.0.1
> Connection: keep-alive
>
> EOF
>
> Expected behaviour:
> Case e) same expections as with case a) and config-1
> Case h) should really stay on one backend. I haproxy to delete
> X-Forwarded-For on the frontend, add new "X-Forwarded-For: SRC-IP", and
> balance based on that header in the backend.
>
> With this behaviour you will get some problems with http/https and sessions,
> stunnel will add an X-Forwarded-For header which contains the actual IP, but
> the user might have sent a different one (or none) resulting in the client
> to access different backends with http than with https.
>
>
> Best regards,
>
> Craig
>
>
>
> ----- original Nachricht --------
>
> Betreff: Re: balance (hdr) problem (maybe bug?)
> Gesendet: Do, 10. Feb 2011
> Von: Willy Tarreau<[email protected]>
>
> > Hi Craig,
> >
> > On Mon, Feb 07, 2011 at 09:24:24PM +0100, Craig wrote:
> > > Hi,
> > >
> > > >> The X-Forwarded-For header is only added once at the end of all
> > > processing.
> > > >> Otherwise, having it in the defaults section would result in both
> your
> > > >> frontend and your backend adding it.
> > > Then the possibility to add it only to a frontend or a backend in the
> > > defaults section would be nice?
> >
> > It is already the case. The fact is that we're telling haproxy that we
> > want an outgoing request to have the header. If you set the option in
> > the frontend, it will have it. If you set it in the backend, it will
> > have it. If you set it in both, it will only be added once. It's really
> > a flag : when the request passes through a frontend or backend which
> > has the option, then it will have the header appended.
> >
> > > >> So in your case, what happens is that you delete it in the frontend
> > > (using
> > > >> reqidel) then you tag the session for adding a new one after all
> > > processing
> > > >> is done.
> > > >>
> > > >> When at the last point we have to establish a connection to the
> > > server, we
> > > >> check the header and balance based on it. I agree we should always
> > > have it
> > > >> filled with the same value, so there's a bug.
> > > So if I got it right, I cannot balance based on the new header because
> > > it was not added yet. That behaviour comes really unexpected because
> one
> > > usually would believe it was already added in the frontend.
> >
> > It can come unexpected when you reason with header addition, but it's
> sort
> > of an implicit header addition. The opposite would be much more
> unexpected,
> > you'd really not want the header to be added twice because it was enable
> in
> > both sections. It's possible that the doc is not clear enough :
> >
> > "This option may be specified either in the frontend or in the backend.
> If
> > at
> > least one of them uses it, the header will be added. Note that the
> > backend's
> > setting of the header subargument takes precedence over the frontend's
> if
> > both are defined."
> >
> > Maybe we should insist on the fact that it's done only at the end.
> >
> > We could try to add it in the frontend and tag the session to know it was
> > already performed. But this would slightly change the semantics to a new
> > one which might not necessarily be desirable. For instance, it's possible
> > in a backend to delete the header and set the option. That way you know
> > that your servers will receive exactly one occurrence of it. Many people
> > are doing that because their servers are having issues with this header
> > passed as a list. Changing the behaviour would result in the backend's
> > delete rule to suppress the header that was just added, and the new one
> > won't be added anymore since it already was.
> >
> > > >> My guess is that you're running a version prior to 1.4.10 which has
> > the
> > > >> header deletion bug : the header list can become corrupted when
> > exactly
> > > >> two consecutive headers are removed from the request (eg: connection
> > and
> > > >> x-forwarded-for). Then the newly added X-Forwarded-For could not be
> > seen
> > > >> by the code responsible for hashing it.
> > > >>
> > > >> If so, please try to upgrade to the last bug fix (1.4.10) and see if
> > the
> > > >> problem persists.
> > > I am already using 1.4.10 - sorry, it seems I somehow forgot to mention
> > > it! :/
> >
> > OK so I'm interested in any reliabe reproducer for this bug (eg: config
> > and/or
> > request exhibiting the issue). You can send me your config privately if
> you
> > don't want to post it to the list.
> >
> > > That is a good hint, but I also have a frontend for SSL (with stunnel
> > > which adds the X-Forward-For header) that I'd want to use the same
> > > backend. I did not like defining backends twice as it introduces
> > > redundancy and might lead to inconsistency, it is a good workaround
> > > though. Note: my testing and the bug happened with the normal frontend.
> >
> > OK I see. Be aware that this setup is not compatible with keep-alive
> > though,
> > as stunnel will only add the header in the first request. An alternative
> is
> > to apply the patch for the proxy protocol to stunnel and use it with
> either
> > haproxy 1.5-dev, or use the 1.4 backports that were recently posted to
> the
> > list.
> >
> > > Also, I could leave out the reqidel of the header, but then a malicious
> > > party could theoretically choose the server it accesses (by forging
> > > x-forwarded-for) and overload one after another; I prefer to take away
> > > this possibility (yea I am overdoing it, maybe). ;)
> >
> > Targetting a server is a false problem. It can also be done by forcing
> > cookies when persistence is used. And even when persistence cookies are
> > encrypted and not predictible, it's enough to make one valid request and
> > replay very fast with the assigned cookie to always hit the same server
> ;-)
> >
> > Your infrastructure and configuration should ensure that your servers
> don't
> > die from overloading. That will protect you from this specific issue.
> >
> > Regards,
> > Willy
> >
> >
> >
>
> --- original Nachricht Ende ----
>
>
--- original Nachricht Ende ----
