Re: HAProxy 1.5-dev7 attempting to connect to real server port of twice virtual server port
Hello Willy. On 23 September 2011 09:35, Willy Tarreau w...@1wt.eu wrote: thanks to your diags and detailed reports, I found the bug and I have the fix. Thanks - that's working for us. Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: HAProxy 1.5-dev7 attempting to connect to real server port of twice virtual server port
Hello Willy. On 22 September 2011 22:08, Willy Tarreau w...@1wt.eu wrote: Wow, you've hit an amazing bug. I seem to remember we touched that area reacently when a server port was ignored. I'll have a look at this, because it's very likely that we fixed it wrong. Thanks. I had another look at it today, but didn't make much progress. Attempting to work around the problem, I experimented with not setting the SRV_MAPPORTS flag when there is no port specified in the server directive - simply commenting-out the else clause on lines 3933-3934 of cfgparse.c. I then added a test for ((struct sockaddr_in *)s-req-cons-addr.s.to)-sin_port == 0 in assign_server_address(), and copying the port from s-req-prod-addr.c.to to s-req-cons-addr.s.to Interestingly, this works when HAProxy is run in debug mode, but not as a daemon. In the latter case, it seems that some of the structure elements are not set - in particular s-req-prod-addr.c.to.ss_family and s-req-prod-addr.c.to-sin_port. I'm not sure if that's expected behaviour when the session structure doesn't have SRV_MAPPORTS set, but it was certainly confusing! Thanks for your help. Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
HAProxy 1.5-dev7 attempting to connect to real server port of twice virtual server port
Afternoon all.
We have come across an interesting problem with HAProxy 1.5-dev7: with
the config below, the proxy attempts to connect to a real server port
of twice the virtual server's port.
listen v1
bind 192.168.69.32:80
mode http
balance leastconn
server backup 127.0.0.1:9081 backup
option httpclose
option redispatch
option abortonclose
maxconn 4
log global
option httplog
server r41 192.168.68.41 weight 1 check port 80 inter 2000 rise 2
fall 3 minconn 0 maxconn 0
In this example, the client's requests to 192.168.69.32:80 are
forwarded to 192.168.68.41:160.
A bit of digging in the source has led to the assign_server_address()
function in backend.c, in particular the following section:
if (target_srv(s-target)-state SRV_MAPPORTS) {
int base_port;
if (!(s-be-options PR_O_TRANSP)
!(s-flags SN_FRT_ADDR_SET))
get_frt_addr(s);
/* First, retrieve the port from the incoming
connection */
base_port = get_host_port(s-req-prod-addr.c.to);
/* Second, assign the outgoing connection's port */
base_port += get_host_port(s-req-prod-addr.s.to);
set_host_port(s-req-cons-addr.s.to, base_port);
}
cfg_parse_listen() appears to set the SRV_MAPPORTS flag, as there is
no destination port specified in the server directive. The proxy
therefore uses the port from s-req-prod-addr.c.to plus
s-req-prod-addr.s.to. With the configuration above, both of these
structure elements have the value 80, resulting in a destination port
of 160.
At that point, I hit a dead end in my understanding of the code. It
would appear that either s-req-prod-addr.c.to or
s-req-prod-addr.s.to are holding the wrong port value, or one of
these structures is being used incorrectly in the sum.
Any pointers for further investigation would be appreciated.
Thanks,
Nick.
--
Nick Chalk.
Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/
Re: HAProxy 1.5-dev7 attempting to connect to real server port of twice virtual server port
Hello Brane. On 21 September 2011 21:30, Brane F. Gračnar brane.grac...@najdi.si wrote: Try replacing this line with: server r41 192.168.68.41:80 weight 1 check inter 2000 rise 2 fall 3 minconn 0 maxconn 0 Yes, explicitly specifying the real server port works fine - but that's exercising a different code path. Originally, the virtual service was defined with a range of ports, so the real server had to be specified without a port. I simplified the configuration for the purposes of testing. Thanks, Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: Issues with haproxy 1.5-dev6 2011/04/08 and tproxy Cannot bind to tproxy source address before connect()
On 18 April 2011 14:39, Mark Brooks m...@loadbalancer.org wrote:
We were having problems using transparent proxy and haproxy and
1.5-dev6 2011/04/08, where haproxy would not work with tproxy enabled.
...
We were able to get it working in 1.5-dev6 by reversing the change
made from dev5 to dev6 below is the patch -
--- proto_tcp.c-dev6 2011-04-18 14:05:38.0 +0100
+++ proto_tcp.c 2011-04-18 14:12:31.0 +0100
@@ -150,7 +150,7 @@
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, (char *) one, sizeof(one));
if (foreign_ok) {
- ret = bind(fd, (struct sockaddr *)bind_addr,
get_addr_len(bind_addr));
+ ret = bind(fd, (struct sockaddr *)bind_addr,
sizeof(bind_addr));
if (ret 0)
return 2;
}
It appears that bind_addr.ss_family is not being set in
tcp_bind_socket() before the call to get_addr_len(bind_addr).
The following allows the use of get_addr_len():
--- proto_tcp.c-dev62011-04-18 14:05:38.0 +0100
+++ proto_tcp.c 2011-04-18 15:23:41.0 +0100
@@ -138,12 +138,14 @@
((struct sockaddr_in
*)bind_addr)-sin_addr = ((struct sockaddr_in *)remote)-sin_addr;
if (flags 2)
((struct sockaddr_in
*)bind_addr)-sin_port = ((struct sockaddr_in *)remote)-sin_port;
+ bind_addr.ss_family = AF_INET;
break;
case AF_INET6:
if (flags 1)
((struct sockaddr_in6
*)bind_addr)-sin6_addr = ((struct sockaddr_in6 *)remote)-sin6_addr;
if (flags 2)
((struct sockaddr_in6
*)bind_addr)-sin6_port = ((struct sockaddr_in6 *)remote)-sin6_port;
+ bind_addr.ss_family = AF_INET6;
break;
}
}
Nick.
--
Nick Chalk.
Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/
Re: httpchk - checking request body
Hello Anze. On 13 March 2010 10:28, Anze anzen...@volja.net wrote: Here is my config: ... listen webfarm 123.123.123.123:80 mode http stats enable balance roundrobin stats auth user:pass cookie SERVERID insert indirect option httpclose option forwardfor except 123.123.123.123 # option httpchk HEAD /check.php HTTP/1.0 # option httpchk /check.php # http-check expect string all_is_ok server a0 10.0.0.1:8080 cookie a0 check server a1 10.0.0.2:8080 cookie a1 check - I have played with the 3 uncommented lines in listen section - these are the results: ... With 1.4.1 WITH patch applied: - all 3 lines disabled: system works (serves content, no problem, but no check) - only 1. line enabled: ERROR, haproxy tells all hosts are down (when they are not) - only 2. line enabled: ERROR, haproxy tells all hosts are down (when they are not) - 2. and 3. line or 1. and 3. line enabled: ERROR as above. On the ToDo list is better logging of what httpchk is doing - along with working out all the options to the original patch. As a quick test, try adding the following options in place of the commented-out lines above: option httpchk GET /check.php HTTP/1.0 http-check expect rstring all_is_ok The 'string' test should work, but it's had less testing than 'rstring'. I've not tried the HEAD method; I can't see it working with 'expect'. Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: httpchk - checking request body
Hello Anze. On 13 March 2010 20:08, Anze anzen...@volja.net wrote: Interesting. Have you run haproxy -d from the command line to check the debugging output? option log-health-checks is also useful. I was running it from the command line (too) because logging was not working (rsyslogd's UDP support was disabled). That was the output I have sent in the first mails. Ah, missed that email. It's a half-close. HAProxy only sends one request, and then waits for the response, so we might as well close the transmit channel. Are you sure nothing is being sent anymore through this socket? Never? Not in my packet traces. HAProxy sends one request, receives the response from the real server, real server closes its channel, HAProxy replies with RST. Future tests bind a new socket. When tcpdump'ing the problem system that led me to work on this, I found that HAProxy was not closing the transmit channel properly. The real servers were sending FIN to close their side, and HAProxy was responding with RST. So I put in the half-close to make sure the connection was cleanly shut down. I think this is not the proper way to deal with this, but as I said, I am no expert in sockets programming. Neither am I - my background is in System/Network Admin. I'm also just learning the HAProxy code, so there may be a better way to ensure that the check connection is closed properly. I would say that haproxy still tries to sent RST but can't - which leads to an error somewhere. The RST should not be sent - that's a response to an already-closed socket. However, HAProxy is not signalling that the connection should be closed by sending a FIN. It's interesting that this has caused problems in your system. As well as running HAProxy in debug mode, would you be able to do a tcpdump on the system running haproxy? If you could do that both with and without the half-close, that would be very useful. Hmmm, I am a bit out of depths here. I would love to help out, but please send me more specific instructions on what you need. What arguments should I use with tcpdump so it helps you? Also, if you don't mind, I'd like to send it privately to you and Willy (so I don't need to clean up too much private / security sensitive info from it). I have no problem with you sending it to me directly. My usual tcpdump invocation is: tcpdump -i any -s 0 -w output_file Substitute an appropriate filename for output_file. You won't want to leave this running for too long, as it listens on all interfaces and captures the whole of every packet. I'd suggest starting the tcpdump, then start HAProxy, wait for a few health checks to be sent, then stop the tcpdump with Ctrl-C. If required, I can send you a version of checks.c with debugging code in it - that will dump a lot of data on the checks if you run it in debug. Sure, bring it on. ;) I'll send that directly to you, rather than the list... Those two will not check the content - I'll need to trawl through the config code to check whether it defaults to checking the HTTP return code in these cases. I have tested GET with the fixed version (no half-closed socket) and it works. Interesting. I'll test that locally. Thanks for your help debugging this. Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: Truncated health check response from real servers
Hello Willy. On 8 March 2010 21:25, Willy Tarreau w...@1wt.eu wrote: On Mon, Mar 08, 2010 at 04:32:54PM +, Nick Chalk wrote: Attached is a patch against v1.4.1. It contains the updated ECV patch, and the hacks to work around check responses that span multiple packets. At first glance, it seems fine. I'll have to find my notes about the original patch to see if anything I spot back then is still present, but in my opinion it looks OK. Thanks. I don't remember if there were any doc in the original patch either, I'll have to dig. No, I don't think we have any documentation. I'll try to find some time to add that to the patch. Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: Truncated health check response from real servers
Hello Willy.
On 3 March 2010 20:31, Willy Tarreau w...@1wt.eu wrote:
OK that's perfect then. If you don't manage to sort out your issue
with small packets, do not hesitate to post your work in progress
to the list, it often helps a lot to work iteratively.
The small-packet problem turned out to be a testing fault - the apache
servers I'm using for testing were not sending the whole page when I
had set a small MTU.
Attached is a patch against v1.4.1. It contains the updated ECV patch,
and the hacks to work around check responses that span multiple
packets.
It seems to be working fine in my tests, and handles check responses
that are bigger than the buffer.
I'd be grateful for anyone's comments and suggestions.
Nick.
--
Nick Chalk.
Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/
diff -ur haproxy-1.4.1/include/types/proxy.h haproxy-1.4.1-ecv/include/types/proxy.h
--- haproxy-1.4.1/include/types/proxy.h 2010-03-04 22:39:19.0 +
+++ haproxy-1.4.1-ecv/include/types/proxy.h 2010-03-08 10:38:22.0 +
@@ -137,6 +137,8 @@
#define PR_O2_MYSQL_CHK 0x0002 /* use MYSQL check for server health */
#define PR_O2_USE_PXHDR 0x0004 /* use Proxy-Connection for proxy requests */
#define PR_O2_CHK_SNDST 0x0008 /* send the state of each server along with HTTP health checks */
+#define PR_O2_EXPECT 0x0010 /* http-check expect sth */
+#define PR_O2_NOEXPECT 0x0020 /* http-check expect ! sth */
/* end of proxy-options2 */
/* bits for sticking rules */
@@ -274,6 +276,9 @@
int grace;/* grace time after stop request */
char *check_req; /* HTTP or SSL request to use for PR_O_HTTP_CHK|PR_O_SSL3_CHK */
int check_len;/* Length of the HTTP or SSL3 request */
+ char *expect_str; /* http-check expected content */
+ regex_t *expect_regex; /* http-check expected content */
+ char *expect_type; /* type of http-check, such as status, string */
struct chunk errmsg[HTTP_ERR_SIZE]; /* default or customized error messages for known errors */
int uuid;/* universally unique proxy ID, used for SNMP */
unsigned int backlog; /* force the frontend's listen backlog */
diff -ur haproxy-1.4.1/include/types/server.h haproxy-1.4.1-ecv/include/types/server.h
--- haproxy-1.4.1/include/types/server.h 2010-03-04 22:39:19.0 +
+++ haproxy-1.4.1-ecv/include/types/server.h 2010-03-08 10:38:22.0 +
@@ -147,6 +147,9 @@
struct freq_ctr sess_per_sec; /* sessions per second on this server */
int puid;/* proxy-unique server ID, used for SNMP */
+ char *check_data; /* storage of partial check results */
+ int check_data_len; /* length of partial check results stored in check_data */
+
struct {
const char *file; /* file where the section appears */
int line; /* line where the section appears */
diff -ur haproxy-1.4.1/src/cfgparse.c haproxy-1.4.1-ecv/src/cfgparse.c
--- haproxy-1.4.1/src/cfgparse.c 2010-03-04 22:39:19.0 +
+++ haproxy-1.4.1-ecv/src/cfgparse.c 2010-03-08 10:38:22.0 +
@@ -2874,8 +2874,65 @@
/* enable emission of the apparent state of a server in HTTP checks */
curproxy-options2 |= PR_O2_CHK_SNDST;
}
+ else if (strcmp(args[1], expect) == 0) {
+ if (strcmp(args[2], status) == 0 || strcmp(args[2], string) == 0) {
+curproxy-options2 |= PR_O2_EXPECT;
+if (*(args[3]) == 0) {
+ Alert(parsing [%s:%d] : '%s %s %s' expects regex as an argument.\n,
+ file, linenum, args[0], args[1], args[2]);
+ return -1;
+}
+curproxy-expect_type = strdup(args[2]);
+curproxy-expect_str = strdup(args[3]);
+ }
+else if (strcmp(args[2], rstatus) == 0 || strcmp(args[2], rstring) == 0) {
+curproxy-options2 |= PR_O2_EXPECT;
+if (*(args[3]) == 0) {
+Alert(parsing [%s:%d] : '%s %s %s' expects regex as an argument.\n,
+file, linenum, args[0], args[1], args[2]);
+return -1;
+}
+curproxy-expect_regex = calloc(1, sizeof(regex_t));
+if (regcomp(curproxy-expect_regex, args[3], REG_EXTENDED) != 0) {
+Alert(parsing [%s:%d] : bad regular expression '%s'.\n, file, linenum, args[0]);
+return -1;
+}
+curproxy-expect_type = strdup(args[2]);
+}
+else if (strcmp(args[2], !) == 0 ) {
+curproxy-options2 |= PR_O2_NOEXPECT;
+if (strcmp(args[3], status) == 0 || strcmp(args[3], string) == 0) {
+ if (*(args[4]) == 0) {
+ Alert(parsing [%s:%d] : '%s %s %s %s' expects regex as an argument
Re: Truncated health check response from real servers
Hello Willy. On 2 March 2010 21:45, Willy Tarreau w...@1wt.eu wrote: If your quick ack already works for one single check, then simply allocate a buffer for each server in cfgparse.c, and have the checks functions use that server-specific buffer instead of trash. Thanks for the pointer. I've added buffer and length variables to struct server, and allocated space with the rest of the server initialisation. Looks to be stable so far, although there's still a problem with very small response packets - something I'll work on tomorrow. If you come up with something working well, I'm willing to merge it into 1.4, so please do not hesitate to submit your work in progress. You'll also get more testers. When I have something working and presentable, I'll submit the patch. Thanks, Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: Truncated health check response from real servers
Hello Willy. On 25 February 2010 20:25, Willy Tarreau w...@1wt.eu wrote: On Mon, Feb 15, 2010 at 10:05:57AM +, Nick Chalk wrote: On 13 February 2010 10:40, Willy Tarreau w...@1wt.eu wrote: Indeed, with MSG_PEEK we have no way to tell the connection was closed. For the time being, I've hacked together a patch to get our customer up and running. I've allocated a new static character buffer, to store the intermediate results from the real server. I'm relying on recv() returning a length of 0 to indicate the server has closed the connection - not sure if that's a reliable method, but it seems to be repeatable. It will not always work with MSG_PEEK. Sorry, I should have made it clear that I'd removed the MSG_PEEK flag. The other temporary solution will be to allocate a receive buffer to every server to store health check responses. Hmm, I hadn't considered that multiple checks could be running in parallel. I'll have a look at whether that can break my quick hack. Thanks, Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: Truncated health check response from real servers
Hello Willy, Krzysztof. On 13 February 2010 10:40, Willy Tarreau w...@1wt.eu wrote: On Fri, Feb 12, 2010 at 05:47:41PM +0100, Krzysztof Olędzki wrote: There are several issues with the fix: - we need to check if connection is not closed, as it is pointless to use MSG_PEEK and restarting such check if there is no more data we are able to read Indeed, with MSG_PEEK we have no way to tell the connection was closed. For the time being, I've hacked together a patch to get our customer up and running. I've allocated a new static character buffer, to store the intermediate results from the real server. I'm relying on recv() returning a length of 0 to indicate the server has closed the connection - not sure if that's a reliable method, but it seems to be repeatable. - some servers return empty description so increasing minimum response length prevents haproxy from accepting such checks. Of course, if you are not using such server, it should be safe to do it in your locally patched version, but we mustn't do it on a public version. In fact, we should re-parse the response each time we call recv(). As long as we don't find a complete response, we can wait. This still implies a non-trivial change to current code. I decided not to run the content check for every received packet, as I couldn't see an easy way to deal with the case where the string to match is split between two packets. Nick. -- Nick Chalk. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
