Re: Backend sends 204, haproxy sends 502
On Mit 28.10.2009 17:12, Dirk Taggesell wrote: On Wed, Oct 28, 2009 at 2:19 PM, Jeffrey 'jf' Lim wrote: what version of haproxy is this? Ah sorry. It is 1.3.17 Is it possible to update to 1.3.22 yust to be on the save side ;-) ### http://haproxy.1wt.eu/ Let's put it short : those of you running 1.3.15.2, 1.3.16 or 1.3.17 are doomed ### And one other thing to look at - what is the log line like for this particular request? Oct 28 13:50:57 127.0.0.1 haproxy[3282]: 88.217.248.214:42160[28/Oct/2009:13:50:57.690] cookietracker cookietracker/cookietracker 1/0/0/-1/3 502 204 - - SL-- 2000/0/0/0/0 0/0 "GET /c HTTP/1.1" S : the TCP session was unexpectedly aborted by the server, or the server explicitly refused it. L : the proxy was still transmitting LAST data to the client while the server had already finished. This one is very rare as it can only happen when the client dies while receiving the last packets. Do you have any error messages in the Server logs? followed after some seconds by about several dozen of these lines: Oct 28 13:52:01 127.0.0.1 haproxy[3282]: 10.224.115.160:43562[28/Oct/2009:13:51:11.732] trackertest trackertest/ -1/1/0/-1/5 0 0 - - sL-- 1902/1902/1902/0/0 0/0 "" 10.224.115.160 is the server's ip NATed address (Amazon EC2) s : the server-side timeout expired while waiting for the server to send or receive data. L : the proxy was still transmitting LAST data to the client while the server had already finished. This one is very rare as it can only happen when the client dies while receiving the last packets. As the previous poster have said haproxy wait for content of the request. As in rfc 2616 described: ### 10.2.5 204 No Content The server has fulfilled the request but does not need to return an entity-body, and might want to return updated metainformation. The response MAY include new or updated metainformation in the form of entity-headers, which if present SHOULD be associated with the requested variant. If the client is a user agent, it SHOULD NOT change its document view from that which caused the request to be sent. This response is primarily intended to allow input for actions to take place without causing a change to the user agent's active document view, although any new or updated metainformation SHOULD be applied to the document currently in the user agent's active view. The 204 response MUST NOT include a message-body, and thus is always terminated by the first empty line after the header fields. ### maybe haproxy should take care of this ;-) Aleks
Re: Backend sends 204, haproxy sends 502
BTW: sorry for the somewhat wierd looking thread. I forgot to manually include haproxy@formilux.org as recipient and manually re-sent it afterwards to the list - for the sake of completeness. And I am forced to use this abomination of a mail reader called google mail, because my mail relay at 1und1.de won't deliver mails to the formilux.orgmail server. It complains with: A message that you sent could not be delivered to one or more of its recipients. The following addresses failed: domain name system error: host mail.formilux.org: mail exchanger has no ip addresses Which is not entirely true, but mail.formilux,org is a CNAME instead of an A record and maybe the 1und1 server is complaining about that.
Re: Backend sends 204, haproxy sends 502
On Wed, Oct 28, 2009 at 3:30 PM, Jeffrey 'jf' Lim wrote: > it looks like Karsten's suspicion is correct. Try adding the > 'Content-length: 0' header. haproxy is still expecting more data from the > backend. (It apparently does not know about status 204???). > thanks, Jeffrey and Karsten, I forwarded the suggestion to our developers (I am only the humble infrastructure guy :D ) and we'll see whether it works.
Re: Backend sends 204, haproxy sends 502
On Wed, Oct 28, 2009 at 2:19 PM, Jeffrey 'jf' Lim wrote: what version of haproxy is this? > Ah sorry. It is 1.3.17 > do 200 requests from the same backend passed through haproxy work? > Yes, haproxy generally works when i test it with an ordinary Apache as back-end instead of the custom app. > I can't say that i've looked too closely at the code for this, but, I get > the impression that haproxy generally returns 502 for stuff that it cannot > recognize. > I am afraid it is so. There's some paragraphs in the documentation which suggest that. > And one other thing to look at - what is the log line like for this > particular request? > Oct 28 13:50:57 127.0.0.1 haproxy[3282]: 88.217.248.214:42160[28/Oct/2009:13:50:57.690] cookietracker cookietracker/cookietracker 1/0/0/-1/3 502 204 - - SL-- 2000/0/0/0/0 0/0 "GET /c HTTP/1.1" followed after some seconds by about several dozen of these lines: Oct 28 13:52:01 127.0.0.1 haproxy[3282]: 10.224.115.160:43562[28/Oct/2009:13:51:11.732] trackertest trackertest/ -1/1/0/-1/5 0 0 - - sL-- 1902/1902/1902/0/0 0/0 "" 10.224.115.160 is the server's ip NATed address (Amazon EC2)
Re: Backend sends 204, haproxy sends 502
On Wed, Oct 28, 2009 at 2:05 PM, John Lauro wrote: > You could run mode tcp if you setup haproxy in transparent mode . > The docs say: Note that contrary to a common belief, this option does NOT make HAProxy present the client's IP to the server when establishing the connection. Which makes sense. And it doesn't work.
RE: Backend sends 204, haproxy sends 502
Oops, me bad. that's technically right. I was burned by this terminology too. What is considered transparent mode is actually good if you want to proxy the world instead of your servers, and it can be combined with usesrc. Anyways, what I should of said was you can make Haproxy present the client's IP with "source haproxyinterfaceip usesrc client" Might be good if the transparent mode had a reference to usesrc.. From: Dirk Taggesell [mailto:dirk.tagges...@googlemail.com] Sent: Wednesday, October 28, 2009 9:48 AM To: John Lauro Subject: Re: Backend sends 204, haproxy sends 502 On Wed, Oct 28, 2009 at 2:05 PM, John Lauro wrote: You could run mode tcp if you setup haproxy in transparent mode . The docs say: Note that contrary to a common belief, this option does NOT make HAProxy present the client's IP to the server when establishing the connection. Which makes sense. And it doesn't work. No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.423 / Virus Database: 270.14.29/2455 - Release Date: 10/28/09 09:34:00
Re: Backend sends 204, haproxy sends 502
On Wed, Oct 28, 2009 at 9:54 PM, Dirk Taggesell < dirk.tagges...@googlemail.com> wrote: > On Wed, Oct 28, 2009 at 2:19 PM, Jeffrey 'jf' Lim wrote: > > what version of haproxy is this? >> > > Ah sorry. It is 1.3.17 > > >> do 200 requests from the same backend passed through haproxy work? >> > > Yes, haproxy generally works when i test it with an ordinary Apache as > back-end instead of the custom app. > > >> I can't say that i've looked too closely at the code for this, but, I get >> the impression that haproxy generally returns 502 for stuff that it cannot >> recognize. >> > > I am afraid it is so. There's some paragraphs in the documentation which > suggest that. > > >> And one other thing to look at - what is the log line like for this >> particular request? >> > > Oct 28 13:50:57 127.0.0.1 haproxy[3282]: > 88.217.248.214:42160[28/Oct/2009:13:50:57.690] cookietracker > cookietracker/cookietracker > 1/0/0/-1/3 502 204 - - SL-- 2000/0/0/0/0 0/0 "GET /c HTTP/1.1" > > it looks like Karsten's suspicion is correct. Try adding the 'Content-length: 0' header. haproxy is still expecting more data from the backend. (It apparently does not know about status 204???). And to answer Karsten's question: the content-length header isn't strictly mandated (it's a 'SHOULD'). -jf > followed after some seconds by about several dozen of these lines: > Oct 28 13:52:01 127.0.0.1 haproxy[3282]: > 10.224.115.160:43562[28/Oct/2009:13:51:11.732] trackertest > trackertest/ -1/1/0/-1/5 0 > 0 - - sL-- 1902/1902/1902/0/0 0/0 "" > > 10.224.115.160 is the server's ip NATed address (Amazon EC2) >
Re: Backend sends 204, haproxy sends 502
Hi, most 502 errors in haproxy responses come from "bad" backend responses. Could you try adding a "Content-Length: 0" header to the backend response? I don't know if RFC requires it in a 204 response. btw. the expires date in your setcookie looks a bit strange. 0059, 1959 or 2059? Karsten Am Mittwoch, 28. Oktober 2009 schrieben Sie: > bash-3.2$ curl --verbose "http://cm01.example.com:8000/c"; > * About to connect() to cm01.example.com port 8000 (#0) > * Trying 22.33.44.55... connected > * Connected to cm01.example.com (22.33.44.55) port 8000 (#0) > > > > GET /c HTTP/1.1 > > > User-Agent: curl/7.19.6 (i386-apple-darwin9.8.0) libcurl/7.19.6 > > OpenSSL/0.9.8k zlib/1.2.3 > > > > Host: cm01.example.com:8000 > > > Accept: */* > > < HTTP/1.1 204 No Content > < Date: Wed, 28 Oct 2009 11:56:44 GMT > < Server: Jetty/5.1.11RC0 (Linux/2.6.21.7-2.fc8xen amd64 java/1.6.0_16 > < Expires: Thu, 01 Jan 1970 00:00:00 GMT > < Set-Cookie: pid=08f0b764185;Path=/;Domain=.example.com;Expires=Thu, > 16-Oct-59 11:56:44 GMT > < Connection: close > < > * Closing connection #0 >
Re: Backend sends 204, haproxy sends 502
On Wed, Oct 28, 2009 at 9:02 PM, Dirk Taggesell < dirk.tagges...@googlemail.com> wrote: > Hi all, > > I want to load balance a new server application that generally sends > http code 204 - to save bandwidth and to avoid client-side caching. > In fact it only exchanges cookie data, thus no real content is delivered > anyway. > > When requests are made via haproxy, the backend - as intended - delivers > a code 204 but haproxy instead turns it into a code 502. Unfortunately I > cannot use tcp mode because the server app needs the client's IP > address. Is there something else I can do? > > what version of haproxy is this? do 200 requests from the same backend passed through haproxy work? I can't say that i've looked too closely at the code for this, but, I get the impression that haproxy generally returns 502 for stuff that it cannot recognize. And one other thing to look at - what is the log line like for this particular request? -jf -- In the meantime, here is your PSA: "It's so hard to write a graphics driver that open-sourcing it would not help." -- Andrew Fear, Software Product Manager, NVIDIA Corporation http://kerneltrap.org/node/7228
RE: Backend sends 204, haproxy sends 502
You could run mode tcp if you setup haproxy in transparent mode . From: Dirk Taggesell [mailto:dirk.tagges...@googlemail.com] Sent: Wednesday, October 28, 2009 9:03 AM To: haproxy@formilux.org Subject: Backend sends 204, haproxy sends 502 Hi all, I want to load balance a new server application that generally sends http code 204 - to save bandwidth and to avoid client-side caching. In fact it only exchanges cookie data, thus no real content is delivered anyway. When requests are made via haproxy, the backend - as intended - delivers a code 204 but haproxy instead turns it into a code 502. Unfortunately I cannot use tcp mode because the server app needs the client's IP address. Is there something else I can do? Request directly to the appserver: bash-3.2$ curl --verbose "http://cm01.example.com:8000/c"; * About to connect() to cm01.example.com port 8000 (#0) * Trying 22.33.44.55... connected * Connected to cm01.example.com (22.33.44.55) port 8000 (#0) > > GET /c HTTP/1.1 > > User-Agent: curl/7.19.6 (i386-apple-darwin9.8.0) libcurl/7.19.6 OpenSSL/0.9.8k zlib/1.2.3 > > Host: cm01.example.com:8000 > > Accept: */* > > < HTTP/1.1 204 No Content < Date: Wed, 28 Oct 2009 11:56:44 GMT < Server: Jetty/5.1.11RC0 (Linux/2.6.21.7-2.fc8xen amd64 java/1.6.0_16 < Expires: Thu, 01 Jan 1970 00:00:00 GMT < Set-Cookie: pid=08f0b764185;Path=/;Domain=.example.com;Expires=Thu, 16-Oct-59 11:56:44 GMT < Connection: close < * Closing connection #0 The above is how it is intended to look. And now via haproxy: bash-3.2$ curl --verbose "http://cm01.example.com/c"; * About to connect() to cm01.example.com port 80 (#0) * Trying 22.33.44.55... connected * Connected to cm01.example.com (22.33.44.55) port 80 (#0) > > GET /c HTTP/1.1 > > User-Agent: curl/7.19.6 (i386-apple-darwin9.8.0) libcurl/7.19.6 OpenSSL/0.9.8k zlib/1.2.3 > > Host: cm01.example.com > > Accept: */* > > * HTTP 1.0, assume close after body < HTTP/1.0 502 Bad Gateway < Cache-Control: no-cache < Connection: close < Content-Type: text/html < 502 Bad Gateway The server returned an invalid or incomplete response. * Closing connection #0 No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.423 / Virus Database: 270.14.29/2455 - Release Date: 10/28/09 09:34:00
