NOSRV & BADREQ
Hi, we are using HA-Proxy version 1.3.19 2009/07/27
For the second time in the last week we seem to be getting NOSRV in the
haproxy logs for a few minutes without any reason (nothing in any log down
stream).
I noticed one thing during each outage is that before the NOSRV starts
happening this is showing up in the log.
Nov 22 09:20:50 127.0.0.1 haproxy[22459]:
208.54.90.71:47640[22/Nov/2009:09:20:50.637] ads_events ads_events/<
NOSRV> -1/-1/-1/-1/0 400 187 - - PR-- 10/10/0/0/0 0/0 {|} ""
Nov 22 09:20:51 127.0.0.1 haproxy[22459]:
208.54.4.68:23435[22/Nov/2009:09:20:51.707] ads_events
ads_events/ -1/-1/-1/-1/0 400
187 - - PR-- 14/14/0/0/0 0/0 {|} ""
Nov 22 09:20:57 127.0.0.1 haproxy[22459]:
208.54.83.78:28295[22/Nov/2009:09:20:57.506] ads_events
ads_events/ -1/-1/-1/-1/0 400
187 - - PR-- 15/15/0/0/0 0/0 {|} ""
Then after this the NOSRV starts happening for 2-3 minutes and then clears
up.
Any ideas to why this is happening or what I can do to troubleshoot further?
/*
Joe Stein
http://www.medialets.com
*/
URL check and block
Is there a way to block URL terms without degrading performance (so kind of like a WAF type function)? Also, how to block IP? If there is doc a link is kewl too o, np. Thanks!!! Happy Holidays in advance =8^) /* Joe Stein, 973-944-0094 http://www.medialets.com */
haproxy burst
Hi All, anyone have any ideas if this is something in version bug (do not want to upgrade if not) or is something I can control with configuration. So what I see running haproxy is when there are bursts (an additional lets say 1,000 connections) i get NOSRV from all of my backend servers. I set all of the max conn to a VERY low number (as a test) to confirm that the downstream servers were not causing this. They can handle this load without spikes so when the spikes come haproxy goes screwey. I am not opposed to upgrading (currently runngin v 3.19) but need some solid ground that doing so will fix this issue if possible any assistance is welcome. -- /* Joe Stein, 973-944-0094 http://www.medialets.com */
Re: haproxy burst
the problem got serious so I stood up another LB set of instances and installed v1.3.23 all traffic is getting routed over to it. what I used to see in the logs where just NOSRV 503 in the haproxy log once it hit max con. Now that we upgraded (or rather migrated) let me see if the issue continues or another one presents itself. so far so good but no burst yet. Most likely "Everything worked once I upgraded" will be the result (isn't it always), but thanks for the quick response as always. On Wed, Mar 3, 2010 at 2:23 PM, Willy Tarreau wrote: > Hi Joe, > > On Wed, Mar 03, 2010 at 01:30:50PM -0500, Joe Stein wrote: > > Hi All, anyone have any ideas if this is something in version bug (do not > > want to upgrade if not) or is something I can control with configuration. > > > > So what I see running haproxy is when there are bursts (an additional > lets > > say 1,000 connections) i get NOSRV from all of my backend servers. > > > > I set all of the max conn to a VERY low number (as a test) to confirm > that > > the downstream servers were not causing this. They can handle this load > > without spikes so when the spikes come haproxy goes screwey. > > > > I am not opposed to upgrading (currently runngin v 3.19) but need some > solid > > ground that doing so will fix this issue if possible any assistance is > > welcome. > > I suspect that you have a low "timeout queue" (which defaults to "timeout > connect" if unset) which aborts the requests from the queue when the > timeout > is expired. What do your log look like when you get this ? Probably the > flags > start with "sQ" or something like this indicating the request was aborted > in > the queue while waiting for a server. > > Willy > > -- /* Joe Stein, 973-944-0094 http://www.medialets.com */
Re: haproxy burst
how do you change the timeout setting? This is what we have now in default (using same config as before) contimeout 1m clitimeout 240s srvtimeout 240s On Wed, Mar 3, 2010 at 2:40 PM, Willy Tarreau wrote: > On Wed, Mar 03, 2010 at 02:34:46PM -0500, Joe Stein wrote: > > the problem got serious so I stood up another LB set of instances and > > installed v1.3.23 all traffic is getting routed over to it. > > > > what I used to see in the logs where just NOSRV 503 in the haproxy log > once > > it hit max con. > > OK but those are just two words in a line, the most important ones are > around that. > > > Now that we upgraded (or rather migrated) let me see if the > > issue continues or another one presents itself. so far so good but no > burst > > yet. > > > > Most likely "Everything worked once I upgraded" will be the result (isn't > it > > always), but thanks for the quick response as always. > > In my opinion, no such issue was known in 1.3.19, so no fix was applied > either. And since it will be produced by default with a standard config, > I find it very likely that it's just an improper timeout setting. > > It's good that you upgraded from 1.3.19, because it had a bug causing a > risk of segfault if you used the unix socket. > > Regards, > Willy > > -- /* Joe Stein, 973-944-0094 http://www.medialets.com */
Re: haproxy burst
another symptom we saw often was we could not get to the stats page or even ssh into the box when this was happening On Wed, Mar 3, 2010 at 2:48 PM, Joe Stein wrote: > how do you change the timeout setting? > > This is what we have now in default (using same config as before) > > contimeout 1m > clitimeout 240s > srvtimeout 240s > > > > On Wed, Mar 3, 2010 at 2:40 PM, Willy Tarreau wrote: > >> On Wed, Mar 03, 2010 at 02:34:46PM -0500, Joe Stein wrote: >> > the problem got serious so I stood up another LB set of instances and >> > installed v1.3.23 all traffic is getting routed over to it. >> > >> > what I used to see in the logs where just NOSRV 503 in the haproxy log >> once >> > it hit max con. >> >> OK but those are just two words in a line, the most important ones are >> around that. >> >> > Now that we upgraded (or rather migrated) let me see if the >> > issue continues or another one presents itself. so far so good but no >> burst >> > yet. >> > >> > Most likely "Everything worked once I upgraded" will be the result >> (isn't it >> > always), but thanks for the quick response as always. >> >> In my opinion, no such issue was known in 1.3.19, so no fix was applied >> either. And since it will be produced by default with a standard config, >> I find it very likely that it's just an improper timeout setting. >> >> It's good that you upgraded from 1.3.19, because it had a bug causing a >> risk of segfault if you used the unix socket. >> >> Regards, >> Willy >> >> > > > -- > /* > Joe Stein, 973-944-0094 > http://www.medialets.com > */ > -- /* Joe Stein, 973-944-0094 http://www.medialets.com */
502 Gateway
Hi, we have a new server that we are trying to fit behind our existing HAProxy implementation. For some reason it is giving us a 502 error (the server is a tomcat6 server with servlet) and we are trying to hunt down the cause. Any help to what this might be or how we can better interrogate haproxy to why it is giving this header (e.g. some header missing or such). We are running 1.3.23 in production but have the same issue also on the 1.4.8 release too. /* Joe Stein, 973-944-0094 http://www.medialets.com Twitter: @allthingshadoop */
Re: 502 Gateway
Thanks Willy, found the issue with curl -I and it was messed up header like you said. Thanx! /* Joe Stein, 973-944-0094 http://www.medialets.com Twitter: @allthingshadoop */ On Jul 21, 2010, at 3:56 PM, Willy Tarreau wrote: > Hi Joe, > > On Wed, Jul 21, 2010 at 11:38:21AM -0400, Joe Stein wrote: >> Hi, we have a new server that we are trying to fit behind our existing >> HAProxy implementation. For some reason it is giving us a 502 error (the >> server is a tomcat6 server with servlet) and we are trying to hunt down the >> cause. Any help to what this might be or how we can better interrogate >> haproxy to why it is giving this header (e.g. some header missing or such). >> We are running 1.3.23 in production but have the same issue also on the >> 1.4.8 release too. > > If you have enabled the stats socket in the global section, you can > dump the last fault request and response of each backend using "socat" : > >printf "show errors\n" | socat stdio unix-connect:/var/run/haproxy.stat > > You will get the rejected response and the exact location of the first > faulty character. Most likely you're having a forbidden character in a > header name, I've seen that once. > > Regards, > Willy >
convert POST to GET?
Is there anyway for HAPROXY to take a POST and turn it into a GET when sending it to backend server? Thanks, /* Joe Stein, 973-944-0094 http://www.medialets.com Twitter: @allthingshadoop */
HTTPS Backend?
Hey folks, is there a way to create a HTTPS back end without using tunnel? right now we have a lot of acl in our frontend section sending to different backends. we are now going to be sending https traffic through haproxy but I can not get this to work. I tried making the default go to the https backend i setup but that is not working either. any help would be appreiated I would prefer no having to setup stunnel and still having my https done in apache like it is now. thanks in advance /* Joe Stein, 973-944-0094 http://www.medialets.com Twitter: @allthingshadoop */
SPDY support?
Hi, I was wondering if/when SPDY support might be added to HAPROXY? Thanks! /* Joe Stein, 973-944-0094 http://www.medialets.com Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> */
Re: SPDY support?
why never? F5 just announced support for it http://www.slideshare.net/f5dotcom/f5-ado-slide-share I appreciate it is not a standard... yet ... but never is such a strong word and seems shortsighted is there something I am missing why you would say never? On Wed, May 2, 2012 at 6:25 PM, Baptiste wrote: > Hi, > > As far as I know, never. :) > On the other hand, HTTP 2.0 may be integrated asap as soon as proposed by > IETF. > > cheers > > > On Tue, May 1, 2012 at 4:05 AM, Joe Stein wrote: > > Hi, I was wondering if/when SPDY support might be added to HAPROXY? > > > > Thanks! > > > > /* > > Joe Stein, 973-944-0094 > > http://www.medialets.com > > Twitter: @allthingshadoop > > */ > -- /* Joe Stein, 973-944-0094 http://www.medialets.com Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> */
Re: SPDY support?
very much so, thanks Willy On Tue, May 8, 2012 at 2:01 PM, Willy Tarreau wrote: > Hi, > > On Tue, May 08, 2012 at 06:57:04PM +0200, Baptiste wrote: > > "Never" unless SPDY become the new standard for HTTP/2.0, validated by > IETF. > > > > To be honest, I talk from time to time to Willy about SPDY protocol. > > And he does not want to implement a protocol which is not a standard > > within HAProxy. > > He prefers waiting for the standardized HTTP/2.0 and because some > > stuff in SPDY are not > > > > F5 is not the only one, boostedge from Activenetworks, nginx, apache > > (through a module), and others have implemented or are implemting > > SPDY. > > > > But Willy is the best person to answer you, I hope he'll answer you soon > :) > > > > Note that I'm on your side: I'd be keen to have SPDY implemented in > > HAProxy. Unfortunately, it's a long time job and HAProxy is missing > > some major features before implementing SPDY (well that's my point of > > view). > > The point is that SPDY is nice and brings a lot of performance boost, but > at the expense of a much more complex infrastructure and a more fragile > handling of DoS attacks. It's around 100 times easier to DoS a SPDY server > than it is for an HTTP server because you can force the server to parse > and process large requests with very few bytes due to the header > compression. > The header compression also means that double buffering becomes mandatory, > which comes with a cost for intermediaries. > > At the moment, SPDY ensures that HTTP/1.1 can be optimized as much as > possible, but there are inherent issues in HTTP/1.1 that have to be > addressed in HTTP/2.0 (CRLF, long header names, folding, etc...). > > That's why with the guys from Squid, Varnish and Wingate we presented > an concurrent proposal to the IETF one month ago : > > http://tools.ietf.org/html/draft-tarreau-httpbis-network-friendly-00 > > Right now there are 4 drafts for HTTP/2.0 : SPDY, ours (which is really > just a small draft and which we still need to work on), the MS guy's and > hopefully Waka if Roy Fielding finds time to write it and publish it. > > All of these drafts use very different concepts, and with a component > such as haproxy, it can be between 3 and 6 months of work before such > a support is implemented, and maybe more for the most complex ones. > > For this reason, I don't want to implement something which is going to > move soon. It's very likely that most of SPDY will be adopted as HTTP/2, > but better work on HTTP/2 when it takes shape than work on SPDY right > now and throw everything away once it's just finished. > > Hoping this clarifies the situation, > > Willy > > -- /* Joe Stein, 973-944-0094 http://www.medialets.com Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> */
