RE: Apache translates 500 to 502 from haproxy
Well, all the problems, the original one that we hit a couple of months ago and the current one are related to one thing: Apache expects some request/response to be read by the downstream haproxy ( and its backends) which refuse to do it due to some error condition and instead sends back a error status like 404, 502, 401 abruptly. Haproxy seem to send a correct response back to Apache as we have seen before, it's the apache that misinterprets it. Yeah, I definitely need to reproduce this problem in test and see what could be the real cause. I will keep you posted. Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Tuesday, September 20, 2011 10:31 AM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org; 'Amrit Jassal' Subject: Re: Apache translates 500 to 502 from haproxy Hi Sachin, On Mon, Sep 19, 2011 at 01:47:28PM +0530, Sachin Shetty wrote: Hey Willy, So we are now hit by the side effect of this fix i.e. disabling httpclose. Two problems: 1. Entries in the log are missing, I guess you already warned me about it. Do you think if we disable keep alive in our Apache fronting haproxy, this will problem will go away? Yes it will solve this issue at least. BTW, with what I saw in your trace, I really see no reason why http-server-close would not work, because the server advertises a correct content-length so haproxy should wait for both streams to synchronize. Are you sure you had http-server-close in both the frontend and the backend, and that you didn't have any remains of forceclose nor httpclose ? Just in doubt, if you're willing to make a new test, I'm interested in a new trace :-) 2. Related to one, but an interesting one. - A request comes to haproxy, as configured after waiting in haproxy queue for 10 seconds due to backend free connection unavailable, it sends a 503 back, logged correctly in haproxy and apache - The client retries, I think with Keep Alive over the same connection and it sees a 400 status back. Now this request is no where in haproxy logs so there is no way to see what happened in haproxy and who really dropped the ball. The connection never made it to the backed cherrypy server since it logs each request it receives. When you see the 400, is it the standard haproxy response or is it apache ? If it is haproxy, you should see it in its logs, which doesn't seem to be the case. It is possible that the client (or apache ?) continues to send a bit of the remaining POST data before the request and that this confuses the next hop (apache or haproxy). That's just a guess, of course. Cheers, Willy
RE: Apache translates 500 to 502 from haproxy
Hey Willy, So we are now hit by the side effect of this fix i.e. disabling httpclose. Two problems: 1. Entries in the log are missing, I guess you already warned me about it. Do you think if we disable keep alive in our Apache fronting haproxy, this will problem will go away? 2. Related to one, but an interesting one. - A request comes to haproxy, as configured after waiting in haproxy queue for 10 seconds due to backend free connection unavailable, it sends a 503 back, logged correctly in haproxy and apache - The client retries, I think with Keep Alive over the same connection and it sees a 400 status back. Now this request is no where in haproxy logs so there is no way to see what happened in haproxy and who really dropped the ball. The connection never made it to the backed cherrypy server since it logs each request it receives. Thanks Sachin -Original Message- From: Sachin Shetty [mailto:sshe...@egnyte.com] Sent: Wednesday, June 15, 2011 5:23 PM To: 'Willy Tarreau' Cc: 'Cassidy, Bryan'; 'haproxy@formilux.org' Subject: RE: Apache translates 500 to 502 from haproxy tried with option http-server-close instead of option httpclose - no luck it does not work. The only way I can get it to work is without either. Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Wednesday, June 15, 2011 12:40 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Wed, Jun 15, 2011 at 12:11:58PM +0530, Sachin Shetty wrote: I also think apache is the issue. I think we have a few mischievous modules like mod_rpaf that I need to disable and test. I will keep you posted. For now do you see any severe problem if I disabe httpclose as a workaround? As I said, in theory no, unless you need haproxy to analyse more than one request per connection. But please do make a test with http-server-close. While there is little chance that it helps, it's not completely impossible because it is able to reinforce keep-alive on the client side even if it is disable on the server side. Willy
Re: Apache translates 500 to 502 from haproxy
Hi Sachin, On Mon, Sep 19, 2011 at 01:47:28PM +0530, Sachin Shetty wrote: Hey Willy, So we are now hit by the side effect of this fix i.e. disabling httpclose. Two problems: 1. Entries in the log are missing, I guess you already warned me about it. Do you think if we disable keep alive in our Apache fronting haproxy, this will problem will go away? Yes it will solve this issue at least. BTW, with what I saw in your trace, I really see no reason why http-server-close would not work, because the server advertises a correct content-length so haproxy should wait for both streams to synchronize. Are you sure you had http-server-close in both the frontend and the backend, and that you didn't have any remains of forceclose nor httpclose ? Just in doubt, if you're willing to make a new test, I'm interested in a new trace :-) 2. Related to one, but an interesting one. - A request comes to haproxy, as configured after waiting in haproxy queue for 10 seconds due to backend free connection unavailable, it sends a 503 back, logged correctly in haproxy and apache - The client retries, I think with Keep Alive over the same connection and it sees a 400 status back. Now this request is no where in haproxy logs so there is no way to see what happened in haproxy and who really dropped the ball. The connection never made it to the backed cherrypy server since it logs each request it receives. When you see the 400, is it the standard haproxy response or is it apache ? If it is haproxy, you should see it in its logs, which doesn't seem to be the case. It is possible that the client (or apache ?) continues to send a bit of the remaining POST data before the request and that this confuses the next hop (apache or haproxy). That's just a guess, of course. Cheers, Willy
RE: Apache translates 500 to 502 from haproxy
I also think apache is the issue. I think we have a few mischievous modules like mod_rpaf that I need to disable and test. I will keep you posted. For now do you see any severe problem if I disabe httpclose as a workaround? Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Wednesday, June 15, 2011 11:08 AM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Tue, Jun 14, 2011 at 06:16:23PM +0530, Sachin Shetty wrote: man We already had option httpclose in our config and removing it fixed it. We haven't tweaked any configs for a few months, I dont even know why we had this in the first place :) I read through the documentation, I dont think we need it. Do you have any reservations about taking it out? If you don't have the option, then only the first request and first response of each connection is analysed. So if Apache does keep-alive with the server over haproxy, then haproxy won't parse the second and subsequent requests. If you were already having httpclose, then haproxy did not close by itself, so that means that the server was closing the connection after it had nothing else to send. In other words, we have two servers (haproxy and the server behind it) who agree on acting the same way and the Apache definitely is the issue here. Could you just make a try with option http-server-close then ? I think it won't work due to the server closing, but if it does it would be better. I'll have to think about implementing a drain mode over keep-alive connections for this precise case : if the connection to the server is dead while the connection to the client is active with data still coming, we should silently drain those data. Regards, Willy
Re: Apache translates 500 to 502 from haproxy
On Wed, Jun 15, 2011 at 12:11:58PM +0530, Sachin Shetty wrote: I also think apache is the issue. I think we have a few mischievous modules like mod_rpaf that I need to disable and test. I will keep you posted. For now do you see any severe problem if I disabe httpclose as a workaround? As I said, in theory no, unless you need haproxy to analyse more than one request per connection. But please do make a test with http-server-close. While there is little chance that it helps, it's not completely impossible because it is able to reinforce keep-alive on the client side even if it is disable on the server side. Willy
RE: Apache translates 500 to 502 from haproxy
Does disabling httpclose also mean that haproxy will not even log subsequent requests on the same connection? Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Wednesday, June 15, 2011 12:40 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Wed, Jun 15, 2011 at 12:11:58PM +0530, Sachin Shetty wrote: I also think apache is the issue. I think we have a few mischievous modules like mod_rpaf that I need to disable and test. I will keep you posted. For now do you see any severe problem if I disabe httpclose as a workaround? As I said, in theory no, unless you need haproxy to analyse more than one request per connection. But please do make a test with http-server-close. While there is little chance that it helps, it's not completely impossible because it is able to reinforce keep-alive on the client side even if it is disable on the server side. Willy
Re: Apache translates 500 to 502 from haproxy
On Wed, Jun 15, 2011 at 01:09:13PM +0530, Sachin Shetty wrote: Does disabling httpclose also mean that haproxy will not even log subsequent requests on the same connection? Exactly. But that's only an issue if that ever happens. Willy
RE: Apache translates 500 to 502 from haproxy
tried with option http-server-close instead of option httpclose - no luck it does not work. The only way I can get it to work is without either. Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Wednesday, June 15, 2011 12:40 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Wed, Jun 15, 2011 at 12:11:58PM +0530, Sachin Shetty wrote: I also think apache is the issue. I think we have a few mischievous modules like mod_rpaf that I need to disable and test. I will keep you posted. For now do you see any severe problem if I disabe httpclose as a workaround? As I said, in theory no, unless you need haproxy to analyse more than one request per connection. But please do make a test with http-server-close. While there is little chance that it helps, it's not completely impossible because it is able to reinforce keep-alive on the client side even if it is disable on the server side. Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 11:25:19AM +0530, Sachin Shetty wrote: Hey Willy, We have 1.4.11 - so how do we use the workaround - This is already in prod and I need to work around it before we can get to the root cause. The workaround is part of the code, so you're hitting a different issue I think. Please do not hesitate to send me a network capture of what passes between apache and haproxy. Maybe it's not hard to improve the workaround to cover your case if that's the issue. Regards, Willy
RE: Apache translates 500 to 502 from haproxy
Hey Willy, tcpdump would be fine? Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Tuesday, June 14, 2011 12:14 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Tue, Jun 14, 2011 at 11:25:19AM +0530, Sachin Shetty wrote: Hey Willy, We have 1.4.11 - so how do we use the workaround - This is already in prod and I need to work around it before we can get to the root cause. The workaround is part of the code, so you're hitting a different issue I think. Please do not hesitate to send me a network capture of what passes between apache and haproxy. Maybe it's not hard to improve the workaround to cover your case if that's the issue. Regards, Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 12:22:03PM +0530, Sachin Shetty wrote: Hey Willy, tcpdump would be fine? Yes, for instance : tcpdump -s0 -npi eth0 port 80 -w trace1.cap Please note that -s0 is very important otherwise packets will be truncated. Regards, Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 01:09:12PM +0530, Sachin Shetty wrote: here is the tcp dump containing that precise request: tcpdump -s0 -npi lo port 9910 -w trace1.cap There is nothing abnormal there, the response was complete and not truncated. The connection was eventually closed before all of the request was read, but I see nothing wrong with that and it is quite common with redirects or 401. It's very strange that your apache does return a 502 here, because it received the response long before the connection was closed, so it should already have forwarded it. Also many people are using apache in front of haproxy without such a problem. Are you sure there is not anything in your apache config/modules which buffers the whole response before responding ? It might explain this different behaviour. Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 02:14:15PM +0530, Sachin Shetty wrote: I looked at all the apache modules we have and haven't seen anything glaring there. Attached tcpdump from cherrypy server when Apache is bypassing haproxy and forwarding to cherrypy directly. Obviously cherrypy does it in order, loads the whole file and then sends the response back which works ok for Apache. I understand, but I see nothing in HTTP which makes it mandatory to read all the content once you have responded. If you respond, you don't need those data anymore. The common practice is to read at least some of them in order to avoid the issue with the close causing a reset. Haproxy reads as much as it can until the close. Once the connection is closed, it cannot read anymore. Apache accepts to read a lot more of data,but with a limit too. Willy
RE: Apache translates 500 to 502 from haproxy
Yeah, I understand. So what could we do? I am really stuck with this and not able to figure out any workaround either. -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Tuesday, June 14, 2011 4:17 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Tue, Jun 14, 2011 at 02:14:15PM +0530, Sachin Shetty wrote: I looked at all the apache modules we have and haven't seen anything glaring there. Attached tcpdump from cherrypy server when Apache is bypassing haproxy and forwarding to cherrypy directly. Obviously cherrypy does it in order, loads the whole file and then sends the response back which works ok for Apache. I understand, but I see nothing in HTTP which makes it mandatory to read all the content once you have responded. If you respond, you don't need those data anymore. The common practice is to read at least some of them in order to avoid the issue with the close causing a reset. Haproxy reads as much as it can until the close. Once the connection is closed, it cannot read anymore. Apache accepts to read a lot more of data,but with a limit too. Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 04:27:00PM +0530, Sachin Shetty wrote: Yeah, I understand. So what could we do? I am really stuck with this and not able to figure out any workaround either. I really have no idea, because the server already replies before the data are sent. Maybe you could disable the http-server-close option so that haproxy works in tunnel mode. It will then not analyse any request nor response beyond headers and will let client and server agree on when to close the connection. You may even resort to the plain old option httpclose so that each other are aware that the connection is used only for one request. Willy
RE: Apache translates 500 to 502 from haproxy
man We already had option httpclose in our config and removing it fixed it. We haven't tweaked any configs for a few months, I dont even know why we had this in the first place :) I read through the documentation, I dont think we need it. Do you have any reservations about taking it out? Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Tuesday, June 14, 2011 5:15 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Tue, Jun 14, 2011 at 04:27:00PM +0530, Sachin Shetty wrote: Yeah, I understand. So what could we do? I am really stuck with this and not able to figure out any workaround either. I really have no idea, because the server already replies before the data are sent. Maybe you could disable the http-server-close option so that haproxy works in tunnel mode. It will then not analyse any request nor response beyond headers and will let client and server agree on when to close the connection. You may even resort to the plain old option httpclose so that each other are aware that the connection is used only for one request. Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 06:16:23PM +0530, Sachin Shetty wrote: man We already had option httpclose in our config and removing it fixed it. We haven't tweaked any configs for a few months, I dont even know why we had this in the first place :) I read through the documentation, I dont think we need it. Do you have any reservations about taking it out? If you don't have the option, then only the first request and first response of each connection is analysed. So if Apache does keep-alive with the server over haproxy, then haproxy won't parse the second and subsequent requests. If you were already having httpclose, then haproxy did not close by itself, so that means that the server was closing the connection after it had nothing else to send. In other words, we have two servers (haproxy and the server behind it) who agree on acting the same way and the Apache definitely is the issue here. Could you just make a try with option http-server-close then ? I think it won't work due to the server closing, but if it does it would be better. I'll have to think about implementing a drain mode over keep-alive connections for this precise case : if the connection to the server is dead while the connection to the client is active with data still coming, we should silently drain those data. Regards, Willy
Re: Apache translates 500 to 502 from haproxy
Willy Tarreau w at 1wt.eu writes: On Fri, Jun 10, 2011 at 04:41:08PM +0530, Manoj Kumar wrote: Hi, We are forwarding specific requests from apache to haproxy which interbally forwards it to a pool of cherry py servers. We have seen that certain requests end up in 500 in haproxy and cherry py logs which is ok and understood, but apache instead sends a 502 to the client. Maybe for any reason apache is rejecting the response and aborting the connection, which might explain that message in your logs : [Fri Jun 10 00:46:01 2011] [error] (103)Software caused connection abort: proxy: pass request body failed to 192.168.2.100:9910 http://192.168.2.15:9910/ (192.168.2.15) Willy Hi Willy, I spent some more time looking in to this, notice the error in apache log, it is parsing request body and not response I think this is what is gong on: 1. Apache receives a POST request 2. Forwards to haproxy 3. haproxy forwards to Cherrypy 4. Cherrypy aborts the request due to some internal error, returns 401/500 5. haproxy sends the response back to Apache and terminates the connection 6. Apache however is still expecting somebody will read the posted response and barfs with pass request body failed error Now this is definitely due to haproxy, since if I skip haproxy and make Apache hit cherrypy directly, I see a proper response code from Apache. I think haproxy is terminating the connection prematurely when the backend server returns and error status code Any idea?
Re: Apache translates 500 to 502 from haproxy
Sachin Shetty sshetty@... writes: I see a similar thread here, no solution though https://forums.rightscale.com//showthread.php?t=210
RE: Apache translates 500 to 502 from haproxy
Hello, Check that Cherrypy is serving up valid HTTP. You could also try setting HAProxy to balance in TCP mode instead of HTTP mode, though if this helps it would just be masking any problem that might exist. I once had a backend 500 response translated to 502 by HAProxy balancing in HTTP mode. I wrote 500 in quotes because the backend (Apache improperly configured in my case) served up an HTML document containing the words 500 internal server error, but didn't actually serve up any HTTP headers prior to the document - just the document itself. HAProxy then changed the response to a 502, as it should, because not including headers is obviously invalid HTTP. I was totally stumped until I ran tcpdump and saw what was happening. Your setup is different than mine was, of course. But maybe this will give you a lead... Hope this helps, Bryan -Original Message- From: Sachin Shetty [mailto:sshe...@egnyte.com] Sent: Monday, June 13, 2011 11:12 AM To: haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy Willy Tarreau w at 1wt.eu writes: On Fri, Jun 10, 2011 at 04:41:08PM +0530, Manoj Kumar wrote: Hi, We are forwarding specific requests from apache to haproxy which interbally forwards it to a pool of cherry py servers. We have seen that certain requests end up in 500 in haproxy and cherry py logs which is ok and understood, but apache instead sends a 502 to the client. Maybe for any reason apache is rejecting the response and aborting the connection, which might explain that message in your logs : [Fri Jun 10 00:46:01 2011] [error] (103)Software caused connection abort: proxy: pass request body failed to 192.168.2.100:9910 http://192.168.2.15:9910/ (192.168.2.15) Willy Hi Willy, I spent some more time looking in to this, notice the error in apache log, it is parsing request body and not response I think this is what is gong on: 1. Apache receives a POST request 2. Forwards to haproxy 3. haproxy forwards to Cherrypy 4. Cherrypy aborts the request due to some internal error, returns 401/500 5. haproxy sends the response back to Apache and terminates the connection 6. Apache however is still expecting somebody will read the posted response and barfs with pass request body failed error Now this is definitely due to haproxy, since if I skip haproxy and make Apache hit cherrypy directly, I see a proper response code from Apache. I think haproxy is terminating the connection prematurely when the backend server returns and error status code Any idea?
RE: Apache translates 500 to 502 from haproxy
Hey Bryan, I did check Cherrypy response by directly posting the same request via curl and it looks ok to me. A few interesting things are: 1. haproxy logs the response as 401 correctly - its apache which is calling haproxy marks it 502 2. It's a post request 3. Even via haproxy, it works when posting smaller files, but get the bad proxy error when posting a bigger file like 1.5MB+ file Thanks Sachin -Original Message- From: Cassidy, Bryan [mailto:bcass...@winnipeg.ca] Sent: Tuesday, June 14, 2011 12:19 AM To: Sachin Shetty; haproxy@formilux.org Subject: RE: Apache translates 500 to 502 from haproxy Hello, Check that Cherrypy is serving up valid HTTP. You could also try setting HAProxy to balance in TCP mode instead of HTTP mode, though if this helps it would just be masking any problem that might exist. I once had a backend 500 response translated to 502 by HAProxy balancing in HTTP mode. I wrote 500 in quotes because the backend (Apache improperly configured in my case) served up an HTML document containing the words 500 internal server error, but didn't actually serve up any HTTP headers prior to the document - just the document itself. HAProxy then changed the response to a 502, as it should, because not including headers is obviously invalid HTTP. I was totally stumped until I ran tcpdump and saw what was happening. Your setup is different than mine was, of course. But maybe this will give you a lead... Hope this helps, Bryan -Original Message- From: Sachin Shetty [mailto:sshe...@egnyte.com] Sent: Monday, June 13, 2011 11:12 AM To: haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy Willy Tarreau w at 1wt.eu writes: On Fri, Jun 10, 2011 at 04:41:08PM +0530, Manoj Kumar wrote: Hi, We are forwarding specific requests from apache to haproxy which interbally forwards it to a pool of cherry py servers. We have seen that certain requests end up in 500 in haproxy and cherry py logs which is ok and understood, but apache instead sends a 502 to the client. Maybe for any reason apache is rejecting the response and aborting the connection, which might explain that message in your logs : [Fri Jun 10 00:46:01 2011] [error] (103)Software caused connection abort: proxy: pass request body failed to 192.168.2.100:9910 http://192.168.2.15:9910/ (192.168.2.15) Willy Hi Willy, I spent some more time looking in to this, notice the error in apache log, it is parsing request body and not response I think this is what is gong on: 1. Apache receives a POST request 2. Forwards to haproxy 3. haproxy forwards to Cherrypy 4. Cherrypy aborts the request due to some internal error, returns 401/500 5. haproxy sends the response back to Apache and terminates the connection 6. Apache however is still expecting somebody will read the posted response and barfs with pass request body failed error Now this is definitely due to haproxy, since if I skip haproxy and make Apache hit cherrypy directly, I see a proper response code from Apache. I think haproxy is terminating the connection prematurely when the backend server returns and error status code Any idea?
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 12:28:55AM +0530, Sachin Shetty wrote: Hey Bryan, I did check Cherrypy response by directly posting the same request via curl and it looks ok to me. A few interesting things are: 1. haproxy logs the response as 401 correctly - its apache which is calling haproxy marks it 502 2. It's a post request 3. Even via haproxy, it works when posting smaller files, but get the bad proxy error when posting a bigger file like 1.5MB+ file I don't like this, it smells like the issue we have with Linux dropping all socket contents after a close() if the client sends more data. What version of haproxy are you running ? A workaround for this issue was added into 1.4.9. It made haproxy read ad much as it could from the connection before closing. Ideally, a network trace of what is received and sent by haproxy when the issue happens would be of great help to try to improve the behaviour without turning it into an easy DoS vulnerability. Willy
RE: Apache translates 500 to 502 from haproxy
Hey Willy, We have 1.4.11 - so how do we use the workaround - This is already in prod and I need to work around it before we can get to the root cause. ./haproxy -version HA-Proxy version 1.4.11 2011/02/10 Copyright 2000-2010 Willy Tarreau w...@1wt.eu Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Tuesday, June 14, 2011 11:03 AM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Tue, Jun 14, 2011 at 12:28:55AM +0530, Sachin Shetty wrote: Hey Bryan, I did check Cherrypy response by directly posting the same request via curl and it looks ok to me. A few interesting things are: 1. haproxy logs the response as 401 correctly - its apache which is calling haproxy marks it 502 2. It's a post request 3. Even via haproxy, it works when posting smaller files, but get the bad proxy error when posting a bigger file like 1.5MB+ file I don't like this, it smells like the issue we have with Linux dropping all socket contents after a close() if the client sends more data. What version of haproxy are you running ? A workaround for this issue was added into 1.4.9. It made haproxy read ad much as it could from the connection before closing. Ideally, a network trace of what is received and sent by haproxy when the issue happens would be of great help to try to improve the behaviour without turning it into an easy DoS vulnerability. Willy
Apache translates 500 to 502 from haproxy
Hi, We are forwarding specific requests from apache to haproxy which interbally forwards it to a pool of cherry py servers. We have seen that certain requests end up in 500 in haproxy and cherry py logs which is ok and understood, but apache instead sends a 502 to the client. We see something like this in the apache logs: [Fri Jun 10 00:46:01 2011] [error] (103)Software caused connection abort: proxy: pass request body failed to 192.168.2.100:9910 http://192.168.2.15:9910/ (192.168.2.15) [Fri Jun 10 00:46:01 2011] [error] proxy: pass request body failed to 192.168.2.100:9910 http://192.168.2.15:9910/ (192.168.2.15) from 71.204.146.120 () [Fri Jun 10 00:46:01 2011] [info] Initial (No.1) HTTPS request received for child 71 (server host.domain.com:443 http://traci.egnyte.com:443/ ) Thanks Regards Manoj Kumar | Xoriant Solutions Pvt. Ltd. Winchester, Hiranandani Business Park, Powai, Mumbai 400076, INDIA. Tel: +91 22 30511000 | Voip: + 1 408 834 4495 | http://www.xoriant.com http://www.xoriant.com/
Re: Apache translates 500 to 502 from haproxy
On Fri, Jun 10, 2011 at 04:41:08PM +0530, Manoj Kumar wrote: Hi, We are forwarding specific requests from apache to haproxy which interbally forwards it to a pool of cherry py servers. We have seen that certain requests end up in 500 in haproxy and cherry py logs which is ok and understood, but apache instead sends a 502 to the client. Maybe for any reason apache is rejecting the response and aborting the connection, which might explain that message in your logs : [Fri Jun 10 00:46:01 2011] [error] (103)Software caused connection abort: proxy: pass request body failed to 192.168.2.100:9910 http://192.168.2.15:9910/ (192.168.2.15) Willy
