Re: HAProxy stops working all of a sudden
Willy, The issue still persists. Not sure what am I missing. -Rahul N. On Friday, August 10, 2012, Rahul Nair wrote: > Willy, > I have upgraded the Linux kernel to and haproxy to 1.4.18 and kernel to 2.6.38-15-server > Will monitor it for few days and will let you know the updates. > -Rahul N. > > On Fri, Aug 10, 2012 at 2:04 AM, Willy Tarreau wrote: >> >> On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote: >> > Willy, >> > >> > >From your description, it could be an issue with some connection >> > tracking somewhere caused by excess of source addr:ports. >> > >> > Ohh ok.. >> > Also I just found that as per the documentation in this link , it says that >> > "it can cause problems when IP connection tracking is enabled on the >> > machine, because a same connection may be seen twice with different states". >> > Does this mean that I need to disable the nf_conntrack module by adding >> > "net.netfilter.nf_conntrack_acct = 0" to "/etc/sysctl.conf" ? >> >> You can't disable nf_conntrack using a sysctl. You need to unload the >> module itself. It's not nf_conntrack_acct but nf_conntrack. >> >> > Bu default this module seems to be enabled. >> > cat /proc/sys/net/netfilter/nf_conntrack_acct >> > 1 >> > >> > Following are the answers to your questions: >> > >> > What's your haproxy version and kernel version ? >> > >> >- HA-Proxy version: 1.4.8 2010/06/16 >> >> Be careful, this is quite outdated ! 2 years of fixes have been merged >> since : >> $ git log --pretty=oneline v1.4.8..|grep -c BUG >> 72 >> >> => Your version has 72 bugs that have already been fixed now. >>I don't remember of any affecting transparent proxying though, but >>when you fix the issue you'd be advised to update it. >> >> >- Kernel Version: 2.6.32-24-server >> >- OS: Ubuntu 10.04 >> >> You should also check that your kernel is up to date, as what you're >> observing might as well simply be a kernel bug. >> >> > Are you sure all your servers route back through your haproxy box ? >> > >> >- Yes the default gateway of all the real servers is HAProxy server. >> >- On real servers I have multiple IPs of two different networks >> > - One which we use for communication between HAproxy server and Real >> > servers. >> > - And One which is used by the real servers to communicate with our >> > internal application servers >> >> OK. >> >> > Did you test only from one source machine or did you have many clients ? >> > >> >- This issue occurs intermittently from one or two different source IPs >> >- At the same time when I check the functionality from another source >> >IP, it works fine. >> >> Fine, then it really makes me think about a conntrack issue. Also, you >> should ensure that your client never directly talks to the server without >> passing via haproxy (which I can imagine you do during your tests when >> observing the issue). It only makes the problem worse with conntrack. >> >> Regards, >> Willy >> > > > > -- > -Rahul N. > IT Department > In2M Technologies Pvt Ltd. (Finicity) > Website: www.finicity.com/india > -- -Rahul N. IT Department In2M Technologies Pvt Ltd. (Finicity) Website: www.finicity.com/india
Re: HAProxy stops working all of a sudden
Willy, I have upgraded the Linux kernel to and haproxy to 1.4.18 and kernel to 2.6.38-15-server Will monitor it for few days and will let you know the updates. -Rahul N. On Fri, Aug 10, 2012 at 2:04 AM, Willy Tarreau wrote: > On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote: > > Willy, > > > > >From your description, it could be an issue with some connection > > tracking somewhere caused by excess of source addr:ports. > > > > Ohh ok.. > > Also I just found that as per the documentation in this link , it says > that > > "it can cause problems when IP connection tracking is enabled on the > > machine, because a same connection may be seen twice with different > states". > > Does this mean that I need to disable the nf_conntrack module by adding > > "net.netfilter.nf_conntrack_acct = 0" to "/etc/sysctl.conf" ? > > You can't disable nf_conntrack using a sysctl. You need to unload the > module itself. It's not nf_conntrack_acct but nf_conntrack. > > > Bu default this module seems to be enabled. > > cat /proc/sys/net/netfilter/nf_conntrack_acct > > 1 > > > > Following are the answers to your questions: > > > > What's your haproxy version and kernel version ? > > > >- HA-Proxy version: 1.4.8 2010/06/16 > > Be careful, this is quite outdated ! 2 years of fixes have been merged > since : > $ git log --pretty=oneline v1.4.8..|grep -c BUG > 72 > > => Your version has 72 bugs that have already been fixed now. >I don't remember of any affecting transparent proxying though, but >when you fix the issue you'd be advised to update it. > > >- Kernel Version: 2.6.32-24-server > >- OS: Ubuntu 10.04 > > You should also check that your kernel is up to date, as what you're > observing might as well simply be a kernel bug. > > > Are you sure all your servers route back through your haproxy box ? > > > >- Yes the default gateway of all the real servers is HAProxy server. > >- On real servers I have multiple IPs of two different networks > > - One which we use for communication between HAproxy server and > Real > > servers. > > - And One which is used by the real servers to communicate with our > > internal application servers > > OK. > > > Did you test only from one source machine or did you have many clients ? > > > >- This issue occurs intermittently from one or two different source > IPs > >- At the same time when I check the functionality from another source > >IP, it works fine. > > Fine, then it really makes me think about a conntrack issue. Also, you > should ensure that your client never directly talks to the server without > passing via haproxy (which I can imagine you do during your tests when > observing the issue). It only makes the problem worse with conntrack. > > Regards, > Willy > > -- -Rahul N. IT Department In2M Technologies Pvt Ltd. (Finicity) Website: www.finicity.com/india
Re: git clone haproxy repo is not working? server returns 404 not found
Hi Han, On Fri, Aug 10, 2012 at 10:51:47AM +0800, Han He wrote: > Hi, > > I have problems to clone the haproxy repo, does anyone have the same problem? > > No response after the following commands: > # git clone http://git.1wt.eu/git/haproxy.git > > I did a packets capture, and found git server return 404 not found for > some requests. This is something I've already encountered when I was using alternate files. I have already noticed this issue with objects which were in a pack and which were still requested as an object of their own by a specific client. The issue disappeared after a git-gc. So I've done it now, please let me know if it's better. Regards, Willy
Re: log format different and CAPTURE_LEN settings
Hi Aleks, On Fri, Aug 10, 2012 at 12:56:18AM +0200, Aleksandar Lazic wrote: > Hi will, > > On 09-08-2012 19:21, Willy Tarreau wrote: > >On Thu, Aug 09, 2012 at 06:57:16PM +0200, Aleksandar Lazic wrote: > > [snip] > > >>after the first minute the log is now as defined ;-) > > > >Thanks guys, patch applied. > > how about the CAPTURE_LEN setting. > > ### > After rebuild I still get the warning that the I can only capture 63 > bytes. > I have change include/common/defaults.h to > > ### > // reserved buffer space for header capture > #ifndef CAPTURE_LEN > #define CAPTURE_LEN 64 > #endif > ### > > and rebuild it. > ### It's unclear to me what is causing you an issue with this one, because it's only used for cookie captures now (you know, the request and response cookie specified in "capture cookie"). Also, you shouldn't set it too high because this memory is allocated for each session. Regards, Willy
Re: log format different and CAPTURE_LEN settings
Hi will, On 09-08-2012 19:21, Willy Tarreau wrote: On Thu, Aug 09, 2012 at 06:57:16PM +0200, Aleksandar Lazic wrote: [snip] after the first minute the log is now as defined ;-) Thanks guys, patch applied. how about the CAPTURE_LEN setting. ### After rebuild I still get the warning that the I can only capture 63 bytes. I have change include/common/defaults.h to ### // reserved buffer space for header capture #ifndef CAPTURE_LEN #define CAPTURE_LEN 64 #endif ### and rebuild it. ### Best regards Aleks
Re: HAProxy stops working all of a sudden
On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote: > Willy, > > >From your description, it could be an issue with some connection > tracking somewhere caused by excess of source addr:ports. > > Ohh ok.. > Also I just found that as per the documentation in this link , it says that > "it can cause problems when IP connection tracking is enabled on the > machine, because a same connection may be seen twice with different states". > Does this mean that I need to disable the nf_conntrack module by adding > "net.netfilter.nf_conntrack_acct = 0" to "/etc/sysctl.conf" ? You can't disable nf_conntrack using a sysctl. You need to unload the module itself. It's not nf_conntrack_acct but nf_conntrack. > Bu default this module seems to be enabled. > cat /proc/sys/net/netfilter/nf_conntrack_acct > 1 > > Following are the answers to your questions: > > What's your haproxy version and kernel version ? > >- HA-Proxy version: 1.4.8 2010/06/16 Be careful, this is quite outdated ! 2 years of fixes have been merged since : $ git log --pretty=oneline v1.4.8..|grep -c BUG 72 => Your version has 72 bugs that have already been fixed now. I don't remember of any affecting transparent proxying though, but when you fix the issue you'd be advised to update it. >- Kernel Version: 2.6.32-24-server >- OS: Ubuntu 10.04 You should also check that your kernel is up to date, as what you're observing might as well simply be a kernel bug. > Are you sure all your servers route back through your haproxy box ? > >- Yes the default gateway of all the real servers is HAProxy server. >- On real servers I have multiple IPs of two different networks > - One which we use for communication between HAproxy server and Real > servers. > - And One which is used by the real servers to communicate with our > internal application servers OK. > Did you test only from one source machine or did you have many clients ? > >- This issue occurs intermittently from one or two different source IPs >- At the same time when I check the functionality from another source >IP, it works fine. Fine, then it really makes me think about a conntrack issue. Also, you should ensure that your client never directly talks to the server without passing via haproxy (which I can imagine you do during your tests when observing the issue). It only makes the problem worse with conntrack. Regards, Willy
Re: HAProxy stops working all of a sudden
Willy, >From your description, it could be an issue with some connection tracking somewhere caused by excess of source addr:ports. Ohh ok.. Also I just found that as per the documentation in this link , it says that "it can cause problems when IP connection tracking is enabled on the machine, because a same connection may be seen twice with different states". Does this mean that I need to disable the nf_conntrack module by adding "net.netfilter.nf_conntrack_acct = 0" to "/etc/sysctl.conf" ? Bu default this module seems to be enabled. cat /proc/sys/net/netfilter/nf_conntrack_acct 1 Following are the answers to your questions: What's your haproxy version and kernel version ? - HA-Proxy version: 1.4.8 2010/06/16 - Kernel Version: 2.6.32-24-server - OS: Ubuntu 10.04 Are you sure all your servers route back through your haproxy box ? - Yes the default gateway of all the real servers is HAProxy server. - On real servers I have multiple IPs of two different networks - One which we use for communication between HAproxy server and Real servers. - And One which is used by the real servers to communicate with our internal application servers Did you test only from one source machine or did you have many clients ? - This issue occurs intermittently from one or two different source IPs - At the same time when I check the functionality from another source IP, it works fine. Thanks Rahul N. On Thu, Aug 9, 2012 at 10:56 PM, Willy Tarreau wrote: > Hello Rahul, > > On Thu, Aug 9, 2012 at 12:13 AM, Rahul Nair > wrote: > > Guys, > > I am in process of implementing HAProxy with TPROXY in our setup for > "mode tcp". > > All of a sudden the website stops working and gives out error in > browser: "Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error." > > When I remove/comment "source 0.0.0.0 usesrc clientip" the website > starts working fine. > > And later on when I again enable "source 0.0.0.0 usesrc clientip" it > starts working fine, It seems that the issue is intermittent. > > Please help me understand what exactly the problem could be. > > Hardware configuration of HAProxy server: > > RAM:256MB > > Processor:Single core > > Thanks, > > Rahul N. > > From your description, it could be an issue with some connection tracking > somewhere caused by excess of source addr:ports. But it could be many > things. > What's your haproxy version and kernel version ? Are you sure all your > servers route back through your haproxy box ? Did you test only from one > source machine or did you have many clients ? > > Willy > > -- -Rahul N. IT Department In2M Technologies Pvt Ltd. (Finicity) Website: www.finicity.com/india
Re: HAProxy stops working all of a sudden
Hello Rahul, On Thu, Aug 9, 2012 at 12:13 AM, Rahul Nair wrote: > Guys, > I am in process of implementing HAProxy with TPROXY in our setup for "mode > tcp". > All of a sudden the website stops working and gives out error in browser: > "Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error." > When I remove/comment "source 0.0.0.0 usesrc clientip" the website starts > working fine. > And later on when I again enable "source 0.0.0.0 usesrc clientip" it starts > working fine, It seems that the issue is intermittent. > Please help me understand what exactly the problem could be. > Hardware configuration of HAProxy server: > RAM:256MB > Processor:Single core > Thanks, > Rahul N. >From your description, it could be an issue with some connection tracking somewhere caused by excess of source addr:ports. But it could be many things. What's your haproxy version and kernel version ? Are you sure all your servers route back through your haproxy box ? Did you test only from one source machine or did you have many clients ? Willy
Re: log format different and CAPTURE_LEN settings
On Thu, Aug 09, 2012 at 06:57:16PM +0200, Aleksandar Lazic wrote: > Hi William, > > On 09-08-2012 16:52, William Lallemand wrote: > >On Thu, Aug 09, 2012 at 03:16:07PM +0200, Aleksandar Lazic wrote: > >>Hi, > >> > > > >Hello, > > > >>[...] > >> > >>As you can see I have not 'option logasap' but get the '+'-sign?! > >> > >>Please can anybody help me to find the error, thanks. > >> > >>Best regards > >>Aleks > >> > > > >It looks like a bug with the option unique-id-format. > > > >Can you try this patch ? > > after the first minute the log is now as defined ;-) Thanks guys, patch applied. Willy
Re: log format different and CAPTURE_LEN settings
Hi William, On 09-08-2012 16:52, William Lallemand wrote: On Thu, Aug 09, 2012 at 03:16:07PM +0200, Aleksandar Lazic wrote: Hi, Hello, [...] As you can see I have not 'option logasap' but get the '+'-sign?! Please can anybody help me to find the error, thanks. Best regards Aleks It looks like a bug with the option unique-id-format. Can you try this patch ? after the first minute the log is now as defined ;-) Thanks. Cheers Aleks
Re: HAProxy stops working all of a sudden
Group, Any clues on this issue..? Thanks Rahul N. On Thursday, August 9, 2012, Rahul Nair wrote: > Hello All, > Please help me on this issue. > Thanks, > Rahul N. > > On Thu, Aug 9, 2012 at 12:13 AM, Rahul Nair wrote: >> >> Guys, >> I am in process of implementing HAProxy with TPROXY in our setup for "mode tcp". >> All of a sudden the website stops working and gives out error in browser: "Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error." >> When I remove/comment "source 0.0.0.0 usesrc clientip" the website starts working fine. >> And later on when I again enable "source 0.0.0.0 usesrc clientip" it starts working fine, It seems that the issue is intermittent. >> Please help me understand what exactly the problem could be. >> Hardware configuration of HAProxy server: >> RAM:256MB >> Processor:Single core >> Thanks, >> Rahul N. >> > > > > -- > -Rahul N. > IT Department > In2M Technologies Pvt Ltd. (Finicity) > Website: www.finicity.com/india > -- Sent from Gmail Mobile
Re: log format different and CAPTURE_LEN settings
On Thu, Aug 09, 2012 at 03:16:07PM +0200, Aleksandar Lazic wrote:
> Hi,
>
Hello,
> [...]
>
> As you can see I have not 'option logasap' but get the '+'-sign?!
>
> Please can anybody help me to find the error, thanks.
>
> Best regards
> Aleks
>
It looks like a bug with the option unique-id-format.
Can you try this patch ?
--
William Lallemand
>From 7d40e9f6d3f8f1c5ce09e264226a1e5e369d70a0 Mon Sep 17 00:00:00 2001
From: William Lallemand
Date: Thu, 9 Aug 2012 16:41:35 +0200
Subject: [PATCH] BUG/MINOR: to_log erased with unique-id-format
curproxy->to_log was reset to LW_INIT when using unique-id-format,
so logs looked like option logasap
---
src/log.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/log.c b/src/log.c
index 2a3cd16..b1f532a 100644
--- a/src/log.c
+++ b/src/log.c
@@ -309,7 +309,7 @@ void parse_logformat_string(char *str, struct proxy *curproxy, struct list *list
struct logformat_node *tmplf, *back;
int options = 0;
- curproxy->to_log = LW_INIT;
+ curproxy->to_log |= LW_INIT;
/* flush the list first. */
list_for_each_entry_safe(tmplf, back, list_format, list) {
--
1.7.9.5
log format different and CAPTURE_LEN settings
Hi,
I wanted to add the uniq-id logging to the http-log format, I just
copied the format
string from src/log.c but I got different log entries. (see below)
I also needed to capture more then 63 bytes so I have build HAProxy
like this
make TARGET=linux26 USE_LINUX_SPLICE=1 USE_STATIC_PCRE=1 # and added
SMALL_OPTS = -DCAPTURE_LEN=256
After rebuild I still get the warning that the I can only capture 63
bytes.
I have change include/common/defaults.h to
###
// reserved buffer space for header capture
#ifndef CAPTURE_LEN
#define CAPTURE_LEN 64
#endif
###
and rebuild it.
default format %Ci:%Cp [%t] %f %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %st %B
%cc %cs %tsc
%ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r
my log-format %Ci:%Cp\ [%t]\ %f\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\ %B\
%cc\ %cs\ %tsc\
%ac/%fc/%bc/%sc/%rc\ %ID\ %sq/%bq\ %hr\ %hs\ %{+Q}r
Output of default with original CAPTURE_LEN
Aug 8 15:54:00 localhost.localdomain haproxy[7295]:
188.165.15.140:49534 [08/Aug/2012:15:53:58.819]
fe_panomax delivery/nginx 0/0/0/1563/1563 200 214 - - 2/2/0/0/0 32
0/0 {Mozilla/4.0 (compatible;)}
"POST REQUEST HTTP/1.1"
Output of my log-format with new CAPTURE_LEN
Aug 8 16:03:55 localhost.localdomain haproxy[9810]:
188.165.15.140:42660 [08/Aug/2012:16:03:54.455]
fe_panomax delivery/- 0/0/0/1344/+1344 200 +128 - - 5/5/1/0/0 143
0/0 "POST REQUEST HTTP/1.1"
now original with new CAPTURE_LEN
Aug 8 16:11:42 localhost.localdomain haproxy[15396]:
188.165.15.140:56583 [08/Aug/2012:16:11:42.199]
fe_panomax delivery/- 0/0/0/716/+716 200 +128 - - 4/4/1/0/0 0/0
"POST REQUEST HTTP/1.1"
now original with original CAPTURE_LEN <= currently running
Aug 8 16:24:52 localhost.localdomain haproxy[22350]:
188.165.15.140:49382 [08/Aug/2012:16:24:50.269]
fe_panomax delivery/- 0/0/0/2690/+2690 200 +128 - - 5/5/1/0/0 0/0
"POST REQUEST HTTP/1.1"
Currently running
haproxy-1.5-dev11# ./haproxy -vv
HA-Proxy version 1.5-dev11 2012/06/04
Copyright 2000-2012 Willy Tarreau
Build options :
TARGET = linux26
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_LINUX_SPLICE=1 USE_STATIC_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents =
200
Encrypted password support via crypt(3): yes
Available polling systems :
sepoll : pref=400, test result OK
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 4 (4 usable), will use sepoll.
my config:
grep -v '^(#|$)' /home/al/download/haproxy/panomax.cfg
global
daemon
pidfile /var/run/haproxy.pid
maxconn 6 # warning: this has to be 3 times the expected
value!
log 127.0.0.1 local0
ulimit-n 120022
defaults
modehttp
balance roundrobin
option dontlognull
option httplog
option http-server-close
option redispatch
option forwardfor
option http-no-delay
option httpchk GET /favicon.ico HTTP/1.0
unique-id-format %{+X}o\ %Ci:%Cp_%Ts_%rt
unique-id-header X-Unique-ID
balance source
retries 1
maxconn 2000
contimeout 5000
clitimeout 5
srvtimeout 5
frontend fe_panomax
bind :80
option forwardfor # add 'X-Forwarded-For: IP'
log global
capture request header User-Agent len 128
capture cookie PHPSESSID len 128
rspdel ^X-Powered-By:.*
acl stat_request url_beg /haproxy_stats
use_backend stats_backend if stat_request
acl fpm_request url_beg /fpm_status
use_backend default if fpm_request
acl host_delivery hdr_beg(host) -i delivery
use_backend delivery if host_delivery
acl host_admin hdr_beg(host) -i admin
use_backend admin if host_admin
acl host_static hdr_beg(host) -i static
use_backend static if host_static
acl host_panodata hdr_beg(host) -i panodata
use_backend panodata if host_panodata
# send everything to next stage
default_backend default
backend default
log 127.0.0.1 local1
option httplog
option httpchk GET /favicon.ico HTTP/1.0
server nginx :81 maxconn 500 check inter 5s fall 3
backend panodata
#127.0.0.3:8002
log global
option httplog
option httpchk GET /crossdomain.xml HTTP/1.0
server nginx :80 check inter 5s fall 3
backend delivery
#127.0.0.3:8002
log global
option httplog
option httpchk GET /crossdomain.xml HTTP/1.0
server nginx :81 check inter 5s fall 3
backend static
#127.0.0.3:8002
log global
option httplog
option httpchk GET /crossdomain.xml HTTP/1.0
server nginx :80 check inter 5s fall 3
backend admin
#127.0.0.3:8002
log global
option httplog
option httpchk GET /img/css-nav.gif HTTP/1.0
server nginx :81 check inter 5s fall 3
backend stats_backend
mode http
balance
timeout connect 4000
timeout server 3
stats uri /haproxy_stats
..
Test, please ignore
This is a test to check if list is back online. Please ignore. Willy
