Re: Haproxy and Ajax / HXR Post

[Willy Tarreau]

On Mon, Oct 10, 2011 at 06:58:21AM +0200, Baptiste wrote:
> 2011/10/9 Andreas Bergman :
> > Hi All,
> > Earlier today we tried a emergency, not pre-tested LB solution for a
> > customer, needless to say this didn't go very well.
> > The LB part worked well, and most of the functions worked well, but among
> > those who didn't work at all were Ajax HXR requests used to upload photos
> > and registering to the site.
> > We did a rollback and the site now works somewhat well.
> > But, we need to figure out why the HXRs and most POST requests didn't work.
> > I've seen in some threads that the HXR requests lacks a length header and
> > therefore Nginx chokes and sends a 405/403.
> > The setup used is Haproxy, Varnish and Nginx, but when we removed varnish it
> > didn't work either, but using only varnish and Nginx works, but Haproxy and
> > Nginx doesn't, that leads to the conclusion that it is something with HXR
> > and Haproxy, probably a config-issue or something related to the HXR header.
> > Any ideas? The config we used was pretty straightforward, we used keep alive
> > thats it. (The LB machine is reinstalled so i haven' got the config anymore)
> > Br
> > Andreas Bergman
> 
> Hi,
> 
> Maybe you enabled "option http-server-close" or "option http-close".
> 
> In order to troubleshoot, we need at least the config and the logs :)
> Can you re-install an HAProxy and run some test again and share with
> us your config and the generated logs?
> 
> Note: you can do that out of production, just make HAProxy listen on
> an "exotic" port and point your own browser on it.

Alternatively, a network capture of some of these XHR requests would help
because we could try to reproduce on development code.

Regards,
Willy

Re: Keep alive with haproxy & stud

[Willy Tarreau]

Hi Erik,

On Sat, Oct 08, 2011 at 06:40:49PM +, Erik Torlen wrote:
> Hi,
> 
> I see different results on the keep alive using http vs https.
> 
> Loadtest against https (through stud) gives me around 69% keep alive effiency 
> (using 3-20 objects per connection in different tests). When testing
> through http directly against haproxy I get 99% keep alive with the same 
> loadtest scripts.
> 
> I have tried changing timeouts and different modes (http-pretend-keepalive 
> etc) but still no improvement.
> 
> Anyone that knows how to improve this and why it's happening?

If you're trying directly then via stud and see different things, then
none of the haproxy options (pretend-keepalive, ...) will have any effect.
It is very possible that timeouts were too low but that would mean you
were using insanely low timeouts (eg: a few ms). It is also possible
that the tool you used for the test can't run as many https concurrent
connections as it runs http connections, and that it closes some of them
by itself. And it is also possible that there are a few issues with stud.
While it performs well, it's still young and it is possible that some
pathological corner cases remain. Haproxy experienced this in its early
age too. You need to enable logging everywhere and get more precise stats
from your load testing tool (eg: all response times, not just an average).

Regards,
Willy

Re: How about server side keep-alive in v1.5?

[Willy Tarreau]

On Sun, Oct 09, 2011 at 02:41:57PM +0200, Baptiste wrote:
> On Sun, Oct 9, 2011 at 4:50 AM, wsq003  wrote:
> > Hi Willy,
> >
> > In the mainpage I saw below: "1.5 will bring keep-alive to the server, but
> > it will probably make sense only with static servers."
> >
> > While in the change-log or source code I did not find this feature
> > (server side keep-alive).
> >
> > Am I missing something, or server side keep-alive still on going?
> >
> > Thanks,
> >
> 
> Sorry for answering on behalf of Willy,
> (Willy, just correct me if I'm wrong).
> 
> HTTP Keepalive has not been developed yet on the server side.
> Actually, this is one of the reason why the 1.5 release of HAProxy is
> still under development.

In fact there was some progress. I have some experimental code which
is able to follow a series of requests over a same connection but which
does not yet know how to change to another server and does not know how
to retry upon server aborts.

> Concerning the delay, only Willy will be able to answer.

That's my biggest concern ! This work has been delayed over one year
now because of other work, I would really like to get it done once
for all.

Regards,
Willy

Re: Quick reqrep question

[Willy Tarreau]

On Tue, Oct 11, 2011 at 10:13:00AM -0500, James Nelson wrote:
> Willy,
> 
> Thanks for your help earlier giving us a quick fix for our TCP problem.
>  This one should be simple:
> 
> I'm looking to create the following transformations with an HTTP request
> (names changed, punctuation correct):
> 
> (change everything)
> from /test1/ to /actual-call/actual.svc?wsdl
> 
> code I tried: reqrep ^([^\]*\/test1/(.*) \1\/actual-call/actual.svc?wsdl\2

Spaces and a comma are missing above, I think you copied it by hand. With
this it will probably work :

  reqrep ^([^\ ]*\ )/test1/(.*) \1\/actual-call/actual.svc?wsdl\2

Regards,
Willy

Re: haproxy, stud and proxy protocol

[Willy Tarreau]

Hi Vincent,

On Fri, Oct 14, 2011 at 08:04:47AM +0200, Vincent Bernat wrote:
> > Still, 40 ms of latency. I have also modified stud to not send the
> > proxy line until there is something else to send (by commenting out
> > "ev_io_start" in "if (OPTIONS.WRITE_PROXY_LINE)" section of
> > handle_connect(). No change. Since the code handling PROXY protocol is
> > so small in stud, I have no other idea of what to do. See here for the
> > relevant code:
> >  https://github.com/bumptech/stud/blob/master/stud.c#L489
> 
> > Any idea where the latency could lie?
> 
> The problem  was stud did not  disable Nagle's algorithm  on the backend
> side. Therefore, nothing to do with HAProxy.

Interesting, thank for the info. This means we all need to re-run tests
then :-) Did you add a setsockopt(TCP_NODELAY) or was there just an option
for that ?

Cheers,
Willy

RE: syslogd dropping requests from haproxy

[Sachin Shetty]

Thanks Willy - I will these and let you know.

-Original Message-
From: Willy Tarreau [mailto:w...@1wt.eu] 
Sent: Saturday, October 15, 2011 11:32 AM
To: Sachin Shetty
Cc: haproxy@formilux.org
Subject: Re: syslogd dropping requests from haproxy

On Sat, Oct 15, 2011 at 01:35:46AM +0530, Sachin Shetty wrote:
> Found some more info, when haproxy configured to send logs to remote host
> instead of local syslogd, it works fine. Definately something to do with
the
> local syslogd under heavy load.

Check that your syslog doesn't log synchronously. For the basic
sysklogd, it means you need to have a "-" in front of your file
names. And even with this, sysklogd's limits are quickly reached
(between 300 and 1000 logs/s depending on the machine).

For high loads, I strongly recommend syslog-ng. It's the only one I
found which managed to log more than 1 logs/s without dropping
any :

 http://www.balabit.com/network-security/syslog-ng

Regards,
Willy

Re: syslogd dropping requests from haproxy

[Willy Tarreau]

On Sat, Oct 15, 2011 at 01:35:46AM +0530, Sachin Shetty wrote:
> Found some more info, when haproxy configured to send logs to remote host
> instead of local syslogd, it works fine. Definately something to do with the
> local syslogd under heavy load.

Check that your syslog doesn't log synchronously. For the basic
sysklogd, it means you need to have a "-" in front of your file
names. And even with this, sysklogd's limits are quickly reached
(between 300 and 1000 logs/s depending on the machine).

For high loads, I strongly recommend syslog-ng. It's the only one I
found which managed to log more than 1 logs/s without dropping
any :

 http://www.balabit.com/network-security/syslog-ng

Regards,
Willy

RE: syslogd dropping requests from haproxy

[Aleksandar Lazic]

Hi,

On 14.10.2011 22:05, Sachin Shetty wrote:


Found some more info, when haproxy configured to send logs to remote
host instead of local syslogd, it works fine. Definately something to
do with the local syslogd under heavy load.


Which syslogd do you use?
Remote:
Local:

Which Distribution do you use?
Which haproxy version do you use?
Please can you show us your haproxy config.

BR
Aleks

RE: syslogd dropping requests from haproxy

[Sachin Shetty]

Found some more info, when haproxy configured to send logs to remote host
instead of local syslogd, it works fine. Definately something to do with the
local syslogd under heavy load.

 

Thanks

Sachin

 

From: Sachin Shetty [mailto:sshe...@egnyte.com] 
Sent: Saturday, October 15, 2011 12:57 AM
To: 'haproxy@formilux.org'
Subject: syslogd dropping requests from haproxy

 

Hi,

 

We have a pretty heavily loaded haproxy server - more than one are running
on a single machine. I am seeing not all requests are being logged to
syslogd - I am sure this is not related to the httpclose parameter since the
same conf file works fine on another machine where the load is pretty low

 

Anybody faced any such problem? any configs in haproxy to write to a
different log file instead of syslogd?

 

Thanks

Sachin

syslogd dropping requests from haproxy

[Sachin Shetty]

Hi,

 

We have a pretty heavily loaded haproxy server - more than one are running
on a single machine. I am seeing not all requests are being logged to
syslogd - I am sure this is not related to the httpclose parameter since the
same conf file works fine on another machine where the load is pretty low

 

Anybody faced any such problem? any configs in haproxy to write to a
different log file instead of syslogd?

 

Thanks

Sachin

Re: Backend server in maintenance mode

[Baptiste]

On Fri, Oct 14, 2011 at 6:32 PM, Mathieu Simon
 wrote:
> Hello,
>
> here my question.
>
> I'm trying to stop gracefully a backend server using HATop.
> The disable command give me a 504 status code for pending request on this
> backend server.
>
> Does it exist a workaround in config file to finish the pending request ?
> (like a force-persist?)
>
> My current cluster is in active/passive mode with one active and the ohter
> in backup
> So no problem for disable the backup with no traffic.
>
> I could easily do a workaround by hot reconfigure haproxy and set the active
> to backup.
> But I prefer to touch only the config file when I need to add a new server
> for example.
>
> I know already the way to do what I want with iptables but I'm looking for
> solution simpler like HATop could do.
>
> I hope my question is clear enough :)
>
> Thanks in advance!
>
> Mathieu Simon.
>
>
>
>

Hi,
You should try to turn the weight of the server to 0.

Cheers

Backend server in maintenance mode

[Mathieu Simon]

Hello,

here my question.

I'm trying to stop gracefully a backend server using HATop.
The disable command give me a 504 status code for pending request on this
backend server.

Does it exist a workaround in config file to finish the pending request ?
(like a force-persist?)

My current cluster is in active/passive mode with one active and the ohter
in backup
So no problem for disable the backup with no traffic.

I could easily do a workaround by hot reconfigure haproxy and set the active
to backup.
But I prefer to touch only the config file when I need to add a new server
for example.

I know already the way to do what I want with iptables but I'm looking for
solution simpler like HATop could do.

I hope my question is clear enough :)

Thanks in advance!

Mathieu Simon.

Re: Haproxy -v 1.4.18 and amazon rds

[Holger Just]

Rhys,

HAProxy resolves IPs of backend servers only once during startup. As new
EC2 instances get an new IP on every startup, HAProxy doesn't find your
new instance. Because of that, it is generally discouraged to use
hostnames in backend sepcifications.

You have basically two ways to solve that:

* You can restart HAProxy to force a re-resolution of DNS names
* You can use elastic IPs for your servers which gives them static IPs.

And if I were you, I'd use elastic IPs as dynamic IPs on servers gives
me bad headaches personally...

--Holger

On 2011-10-14 10:57, Rhys Powell wrote:
> Hello all,
> 
> Have a problem at the moment that I just cant seem to fix. We currently
> run a master RDS instance but between the hours of 8am and 6pm we spawn
> a read only replica up so that there is no affect on writes while some
> of the longer queries are made.
> 
> If we restart haproxy it instantly pick up this slave but it never finds
> it with out a restart. ANyone else had this probelm or know what a fix is?
> 
> config file is below
> 
> TIA
> 
> Rhys
> 
> global
> log 127.0.0.1   local0
> maxconn 4096
> user haproxy
> group haproxy
> daemon
> 
> defaults
> log global
> modetcp
> option  tcplog
> option  dontlognull
> retries 3
> option redispatch
> contimeout  5000
> clitimeout  5
> srvtimeout  5
> 
> listen mysql-pdw
> bind 0.0.0.0:3306 
> option mysql-check user theusername
> hash-type consistent
> balance roundrobin
> server slave2 nameofslave2andid.rds.amazonaws.com:3306
>  weight 100 check inter
> 5s observe layer4
> server slave1 nameofslave1andid.rds.amazonaws.com:3306
>  weight 10 check inter
> 5s observe layer4
> server master nameofmasterandid.rds.amazonaws.com:3306
>  weight 1 check inter
> 5s observe layer4
> 
> 
> 

Haproxy -v 1.4.18 and amazon rds

[Rhys Powell]

Hello all,

Have a problem at the moment that I just cant seem to fix. We currently run
a master RDS instance but between the hours of 8am and 6pm we spawn a read
only replica up so that there is no affect on writes while some of the
longer queries are made.

If we restart haproxy it instantly pick up this slave but it never finds it
with out a restart. ANyone else had this probelm or know what a fix is?

config file is below

TIA

Rhys

global
log 127.0.0.1   local0
maxconn 4096
user haproxy
group haproxy
daemon

defaults
log global
modetcp
option  tcplog
option  dontlognull
retries 3
option redispatch
contimeout  5000
clitimeout  5
srvtimeout  5

listen mysql-pdw
bind 0.0.0.0:3306
option mysql-check user theusername
hash-type consistent
balance roundrobin
server slave2 nameofslave2andid.rds.amazonaws.com:3306 weight 100
check inter 5s observe layer4
server slave1 nameofslave1andid.rds.amazonaws.com:3306 weight 10
check inter 5s observe layer4
server master nameofmasterandid.rds.amazonaws.com:3306 weight 1
check inter 5s observe layer4

无担保个人贷款，6.12%超低利率,最高可贷30万,最快1日即可放款

[摩根王子]

小额贷款、无担保、无抵押贷款
无抵押贷款,6.12%超低利率,最快1日即可放款,专业银行贷款,最高可贷30万
无抵押无担保，门槛低！ 最快1天到账，马上办.

6.12%超低利率,最高可贷30万,最快1日即可放款
6.12%超低利率,最高可贷30万,最快1日即可放款
6.12%超低利率,最高可贷30万,最快1日即可放款
6.12%超低利率,最高可贷30万,最快1日即可放款
6.12%超低利率,最高可贷30万,最快1日即可放款


活动网址： http://www.buyersnet.info/google.asp?q=11
活动网址： http://www.buyersnet.info/google.asp?q=11
活动网址： http://www.buyersnet.info/google.asp?q=11
活动网址： http://www.buyersnet.info/google.asp?q=11
活动网址： http://www.buyersnet.info/google.asp?q=11
活动网址： http://www.buyersnet.info/google.asp?q=11

请记下网址以备用．