Re: Transparent Proxy
On Fri, Sep 23, 2011 at 11:53 PM, Jason J. W. Williams jasonjwwilli...@gmail.com wrote: Hello, My understanding has been that HAProxy can be set up in conjunction with TPROXY support in the Linux kernel so that the backend servers see the original client's source IP address on incoming packets? So is the option transparent (http://code.google.com/p/haproxy-docs/wiki/transparent) not related to that type of transparent proxying or am I mistaken and there's no way to make HAProxy preserve the original client IP on the way to the backend servers? Thank you in advance. -J Hi, You have to patch your kernel with TProxy and then to use the source keyword: http://code.google.com/p/haproxy-docs/wiki/source Note that the default gateway of your servers must be the HAProxy box in that kind of architecture. cheers
Re: Transparent Proxy
Jason, No that option is not relevant for TPROXY (client source IP transparency) Its an old blog but take a look at: http://blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy/ Ignore the kernel re-compile stuff, as its all pretty standard in modern kernels. But it should show you how to construct the haproxy.cfg file. On 23 September 2011 22:53, Jason J. W. Williams jasonjwwilli...@gmail.com wrote: Hello, My understanding has been that HAProxy can be set up in conjunction with TPROXY support in the Linux kernel so that the backend servers see the original client's source IP address on incoming packets? So is the option transparent (http://code.google.com/p/haproxy-docs/wiki/transparent) not related to that type of transparent proxying or am I mistaken and there's no way to make HAProxy preserve the original client IP on the way to the backend servers? Thank you in advance. -J -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Feature idea (maybe)
Hi, I have an idea for a possibly useful feature. How about some possibility to keep a certain IP in a sandbox, where we can filter all it's traffic trough a virus/intrusion detection filter, then allow it to be trusted for X hours/days and pass it out to the general loadbalancing queue? I recently had an attack where it would have been great if I could filter the first 100 connections from that IP trouugh a squid/clamd/etc supported backend, it would have caught the attack (namely: upload of malware). What do you think. Levente
Re: Feature idea (maybe)
I forgot to mention: It would be useful to have the possibility to filter only incoming or outgoing traffic or both in this fashion. Hope this has some merit... Levente 2011.09.24. 9:12 keltezéssel, Levente Peres írta: Hi, I have an idea for a possibly useful feature. How about some possibility to keep a certain IP in a sandbox, where we can filter all it's traffic trough a virus/intrusion detection filter, then allow it to be trusted for X hours/days and pass it out to the general loadbalancing queue? I recently had an attack where it would have been great if I could filter the first 100 connections from that IP trouugh a squid/clamd/etc supported backend, it would have caught the attack (namely: upload of malware). What do you think. Levente --- avast! Antivirus: Inbound message clean. Virus Database (VPS): 110923-1, 2011.09.23 Tested on: 2011.09.24. 9:14:55 avast! - copyright (c) 1988-2011 AVAST Software. http://www.avast.com
Re: Transparent Proxy
Thank you. I've been reading that, but wanted to confirm. -J Sent via iPhone Is your email Premiere? On Sep 24, 2011, at 0:57, Malcolm Turnbull malc...@loadbalancer.org wrote: Jason, No that option is not relevant for TPROXY (client source IP transparency) Its an old blog but take a look at: http://blog.loadbalancer.org/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy/ Ignore the kernel re-compile stuff, as its all pretty standard in modern kernels. But it should show you how to construct the haproxy.cfg file. On 23 September 2011 22:53, Jason J. W. Williams jasonjwwilli...@gmail.com wrote: Hello, My understanding has been that HAProxy can be set up in conjunction with TPROXY support in the Linux kernel so that the backend servers see the original client's source IP address on incoming packets? So is the option transparent (http://code.google.com/p/haproxy-docs/wiki/transparent) not related to that type of transparent proxying or am I mistaken and there's no way to make HAProxy preserve the original client IP on the way to the backend servers? Thank you in advance. -J -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: Haproxy notifications
Hi Carlo, I tried to install it but i got that error when run it. -# service haproxyctl show health /etc/init.d/haproxyctl:17:in `require': no such file to load -- haproxyctl (LoadError) from /etc/init.d/haproxyctl:17 On 22 September 2011 21:42, carlo flores ca...@petalphile.com wrote: Here's an HAProxy ctl/init script that includes Nagios and Cloudkick checks (to the UNIX socket and lsof listeners) to go with the regular start/stop/init stuff: https://github.com/flores/haproxyctl The advantage of using a wrapper around the UNIX socket is you can also add checks for available backends, hosts down, and so forth. This should be in addition to whatever TCP/HTTP checks that come default in any monitoring solution. (The script above is easy to edit to add such things.) On Thu, Sep 22, 2011 at 10:02 AM, Ben Timby bti...@gmail.com wrote: On Thu, Sep 22, 2011 at 11:30 AM, Guillaume Bourque guillaume.bour...@gmail.com wrote: option log-health-checks :-) I took notification to mean something other than logging. -- *İbrahim Ercan * *Gsm: 0 505 608 24 37*
Re: Need help about reqrep (urgent)
Hi, On Fri, Sep 23, 2011 at 11:20:52AM +, DerekLiao wrote: Hey there, I have trouble with direct ??reqrep?? what I want is : My main web site is : caipiao.aaa.com My second web site is: www.bbb.com My goal is redirect URL from caipiao.aaa.com/ kaijiang -www.bbb.com/result_aaa , but the URL in the browse still need to display caipiao.aaa.com/kaijiang what I did is: Frontend: acl acl_dom_caipiao.aaa.com hdr_dom(host) caipiao.aaa.com acl acl_url_kaijiang path_beg -i /kaijiang use_backend cs_aaa_caipiaos_1 if acl_dom_caipiao.aaa.com acl_url_kaijiang backend: backend cs_aaa_caipiaos_1 reqrep ^([^\ ]*)\ /kaijiang(/.*) \1\ /result_aaa\2 option httpchk GET / HTTP/1.1\r\nHost:\ www.bbb.com server srv_aaa_caipiaos_1_01 www.bbb.com:80 check Are you sure that your server does not also consider the Host header and refuses to serve /result_aaa from caipiao.aaa.com ? when I access caipiao.aaa.com/kaijiang, I get a 404 error, looks like reqrep did not working~ but if make a little change as below: acl acl_url_result path_beg -i/result_aaa use_backend cs_aaa_caipiaos_1 if acl_dom_caipiao.aaa.com acl_url_result then I can access caipiao.aaa.com/result_aaa directly, So there should be a problem on reqrep Strange then. Could you help to analyze the root cause of this ? What does your server say in its logs ? You'll have the useful information there since you'll know how the request will have been rewritten. Regards, Willy
RE: Need help about reqrep (urgent)
Is it set correctly I only see 503/404 error on haproxy log, I already set debug level ! version 1.4.14 I really need your help My Skype account : derek.liao66 Date: Sat, 24 Sep 2011 12:43:35 +0200 From: w...@1wt.eu To: y...@hotmail.com CC: haproxy@formilux.org Subject: Re: Need help about reqrep (urgent) Hi, On Fri, Sep 23, 2011 at 11:20:52AM +, DerekLiao wrote: Hey there, I have trouble with direct ??reqrep?? what I want is : My main web site is : caipiao.aaa.com My second web site is: www.bbb.com My goal is redirect URL from caipiao.aaa.com/ kaijiang -www.bbb.com/result_aaa , but the URL in the browse still need to display caipiao.aaa.com/kaijiang what I did is: Frontend: acl acl_dom_caipiao.aaa.com hdr_dom(host) caipiao.aaa.com acl acl_url_kaijiang path_beg -i /kaijiang use_backend cs_aaa_caipiaos_1 if acl_dom_caipiao.aaa.com acl_url_kaijiang backend: backend cs_aaa_caipiaos_1 reqrep ^([^\ ]*)\ /kaijiang(/.*) \1\ /result_aaa\2 option httpchk GET / HTTP/1.1\r\nHost:\ www.bbb.com server srv_aaa_caipiaos_1_01 www.bbb.com:80 check Are you sure that your server does not also consider the Host header and refuses to serve /result_aaa from caipiao.aaa.com ? when I access caipiao.aaa.com/kaijiang, I get a 404 error, looks like reqrep did not working~ but if make a little change as below: acl acl_url_result path_beg -i/result_aaa use_backend cs_aaa_caipiaos_1 if acl_dom_caipiao.aaa.com acl_url_result then I can access caipiao.aaa.com/result_aaa directly, So there should be a problem on reqrep Strange then. Could you help to analyze the root cause of this ? What does your server say in its logs ? You'll have the useful information there since you'll know how the request will have been rewritten. Regards, Willy
Re: Need help about reqrep (urgent)
On Sat, Sep 24, 2011 at 10:46:23AM +, DerekLiao wrote: Is it set correctly It does not indicate what's in it. What do you mean by correctly ? I only see 503/404 error on haproxy log, I already set debug level ! version 1.4.14 503s indicate that sometimes there is no server to connect to, possibly because some health checks sometimes fail. Regards, Willy
RE: Need help about reqrep (urgent)
health check is normal, I checked from haproxy monitor screen , How to check reqrep is working fine or not ? I double the URI has not been replaced Date: Sat, 24 Sep 2011 12:52:01 +0200 From: w...@1wt.eu To: y...@hotmail.com CC: haproxy@formilux.org Subject: Re: Need help about reqrep (urgent) On Sat, Sep 24, 2011 at 10:46:23AM +, DerekLiao wrote: Is it set correctly It does not indicate what's in it. What do you mean by correctly ? I only see 503/404 error on haproxy log, I already set debug level ! version 1.4.14 503s indicate that sometimes there is no server to connect to, possibly because some health checks sometimes fail. Regards, Willy
Re: Need help about reqrep (urgent)
On Sat, Sep 24, 2011 at 11:06:07AM +, DerekLiao wrote: health check is normal, I checked from haproxy monitor screen , OK, maybe it's your server which is returning 503 then. How to check reqrep is working fine or not ? That's what I explained, in apache's logs. I double the URI has not been replaced How did you check that then ? If you look at your request line in haproxy's logs and compare it to the line in apache's logs, it should become obvious if the replacement was partially, totally applied or not at all. Willy
RE: Need help about reqrep (urgent)
is it possible to get detail rewrite infomation like apache error log does? tcpdump, little bit diffcult ,too much taffice need to analyse From: luky...@hotmail.com To: y...@hotmail.com; w...@1wt.eu CC: haproxy@formilux.org Subject: RE: Need help about reqrep (urgent) Date: Sat, 24 Sep 2011 13:20:49 +0200 tcpdump it on the haproxy box then. Somehow you need to understand what happens between haproxy and the backend. Lukas From: y...@hotmail.com To: w...@1wt.eu CC: haproxy@formilux.org Subject: RE: Need help about reqrep (urgent) Date: Sat, 24 Sep 2011 11:15:09 + from haproxy, I can see the URI is /kaijiang which was not changed I can't see the backend server's log, because it was out of my control, actually that is thirdparty company's web site Date: Sat, 24 Sep 2011 13:11:05 +0200 From: w...@1wt.eu To: y...@hotmail.com CC: haproxy@formilux.org Subject: Re: Need help about reqrep (urgent) On Sat, Sep 24, 2011 at 11:06:07AM +, DerekLiao wrote: health check is normal, I checked from haproxy monitor screen , OK, maybe it's your server which is returning 503 then. How to check reqrep is working fine or not ? That's what I explained, in apache's logs. I double the URI has not been replaced How did you check that then ? If you look at your request line in haproxy's logs and compare it to the line in apache's logs, it should become obvious if the replacement was partially, totally applied or not at all. Willy
Re: Need help about reqrep (urgent)
On Sat, Sep 24, 2011 at 11:23:07AM +, DerekLiao wrote: is it possible to get detail rewrite infomation like apache error log does? No, what haproxy logs is what it *receives*. That's why we're asking for what is seen on the other side. tcpdump, little bit diffcult ,too much taffice need to analyse You don't need to let it run for minutes, just a few seconds during a click. Alternatively you can try to add a second rule after it to block the rewritten request. If you see that the request is blocked, then you can conclude it was properly rewritten : reqrep ^([^\ ]*)\ /kaijiang(/.*) \1\ /result_aaa\2 reqdeny ^([^\ ]*)\ /result_aaa BTW, we're all assuming that you have appropriately set option httpclose or option http-server-close so that all requests are rewritten, not only the first one. You might want to double-check. Regards, Willy
RE: Need help about reqrep (urgent)
All, I tried what you suggested, I think it s also a problem on rewrite URI: configuration files: reqrep ^([^\ ]*)\ /kaijiang(/.*) \1\ /result_bbb\2 reqirep ^Host: Host:\ www.bbb.com #reqdeny ^([^\ ]*)\ /kaijiang reqdeny ^([^\ ]*)\ /result_bbb option httpchk GET / HTTP/1.1\r\nHost:\ www.bbb.com server srv_aaa_caipiaos_1_01 www.bbb.com:80 check The problem show up like : srv_aaa_caipiaos_1_01 20993/0/0/0/20993 404 484 - - 4/4/0/1/0 0/0 GET /kaijiang HTTP/1.1 and reqdeny is not working , after i changed to #reqdeny ^([^\ ]*)\ /kaijiang, reqdeny works as expected From: cyril.bo...@free.fr To: y...@hotmail.com CC: haproxy@formilux.org; w...@1wt.eu Subject: Re: Need help about reqrep (urgent) Date: Sat, 24 Sep 2011 14:35:25 +0200 Hi all, Le Samedi 24 Septembre 2011 12:43:35 Willy Tarreau a écrit : Hi, On Fri, Sep 23, 2011 at 11:20:52AM +, DerekLiao wrote: backend cs_aaa_caipiaos_1 reqrep ^([^\ ]*)\ /kaijiang(/.*) \1\ /result_aaa\2 option httpchk GET / HTTP/1.1\r\nHost:\ www.bbb.com server srv_aaa_caipiaos_1_01 www.bbb.com:80 check Are you sure that your server does not also consider the Host header and refuses to serve /result_aaa from caipiao.aaa.com ? I take the thread at the top because for me Willy is right from the beginning. Your issue is probably not in your reqrep line but because you don't modify the Host header before sending the request to the remote server (as you do in your httpchk test). You should try to add a second reqrep (or reqirep) to change the host : reqirep ^Host: Host:\ www.bbb.com when I access caipiao.aaa.com/kaijiang, I get a 404 error, looks like reqrep did not working~ And this is probably due to the fact that you try to contact a virtualhost caipiao.aaa.com on the remote server, which doesn't exist. -- Cyril Bonté
RE: Need help about reqrep (urgent)
configuration files: reqrep ^([^\ ]*)\ /kaijiang([/\?\ ].*)\1\ /result_aaa\2 reqirep ^Host: Host:\ www.bbb.com Logs: srv_aaa_caipiaos_01 0/0/0/2/2 404 735 - - 0/0/0/0/0 0/0 GET /result_aaa/ HTTP/1.1 but i can access www.bbb.com/result_aaa/ From: y...@hotmail.com To: cyril.bo...@free.fr CC: haproxy@formilux.org; w...@1wt.eu Subject: RE: Need help about reqrep (urgent) Date: Sat, 24 Sep 2011 15:01:33 + configuration files: reqrep ^([^\ ]*)\ /kaijiang([/\?\ ].*)\1\ /result_aaa\ reqirep ^Host: Host:\ www.bbb.com Logs: srv_aaa_caipiaos_01 0/0/0/2/2 404 735 - - 0/0/0/0/0 0/0 GET /result_aaa/ HTTP/1.1 but i can access www.bbb.com/result_aaa/ -- From: cyril.bo...@free.fr To: y...@hotmail.com CC: haproxy@formilux.org; w...@1wt.eu Subject: Re: Need help about reqrep (urgent) Date: Sat, 24 Sep 2011 16:32:05 +0200 Le Samedi 24 Septembre 2011 14:10:19 DerekLiao a écrit : All, I tried what you suggested, I think it s also a problem on rewrite URI: configuration files: reqrep ^([^\ ]*)\ /kaijiang(/.*) \1\ /result_bbb\2 reqirep ^Host: Host:\ www.bbb.com #reqdeny ^([^\ ]*)\ /kaijiang reqdeny ^([^\ ]*)\ /result_bbb option httpchk GET / HTTP/1.1\r\nHost:\ www.bbb.com server srv_aaa_caipiaos_1_01 www.bbb.com:80 check The problem show up like : srv_aaa_caipiaos_1_01 20993/0/0/0/20993 404 484 - - 4/4/0/1/0 0/0 GET /kaijiang HTTP/1.1 Ok, now your log shows that you don't have a / at the end of your request but your regexp requires one. If you want it to work, you need to adapt the regexp depending on your needs. For example : reqrep ^([^\ ]*)\ /kaijiang([/\?\ ].*) \1\ /result_bbb\2 This will allow requests like : /kaijiang /kaijiang/... /kaijiang?param=value... but won't match things like : /kaijiangfoo /kaijiangbar /kaijiang;param=value -- Cyril Bonté
Re: Need help about reqrep (urgent)
Le Samedi 24 Septembre 2011 15:02:59 DerekLiao a écrit : configuration files: reqrep ^([^\ ]*)\ /kaijiang([/\?\ ].*)\1\ /result_aaa\2 reqirep ^Host: Host:\ www.bbb.com Logs: srv_aaa_caipiaos_01 0/0/0/2/2 404 735 - - 0/0/0/0/0 0/0 GET /result_aaa/ HTTP/1.1 I'm lost, I don't understand what you are testing. Why is your log showing /result_aaa/ if you want to test /kaijiang ? Also, you don't provide enough information at the same time: Why your server is not the same as the one you provided at the beginning : srv_aaa_caipiaos_1_01 ? I think you forgot to give other pieces of configuration. It looks like you are testing another backend configuration. Confirmed by your frontend acls if they didn't change : acl acl_dom_caipiao.aaa.com hdr_dom(host) caipiao.aaa.com acl acl_url_kaijiang path_beg -i /kaijiang use_backend cs_aaa_caipiaos_1 if acl_dom_caipiao.aaa.com acl_url_kaijiang If you ask for /result_aaa/ the condition on the last line won't match. Can you provide everything (hide the IP/host names/passwords for security reasons) ? This will save time to everybody ;-) but i can access www.bbb.com/result_aaa/ I suppose that you missed something about the Host headers. OK, you can access to www.bbb.com/result_aaa/ But it looks like your hostname in front of haproxy is caipiao.aaa.com (or aaa.com) Are you sure the server www.bbb.com accepts your virtualhost ? You can verify it : curl -i -H Host: caipiao.aaa.com www.bbb.com/result_aaa/ Isn't it a 404 ? -- Cyril Bonté
RE: Need help about reqrep (urgent)
caipiao.aaa.com/kaijiang(my website) - www.bbb.com/result_aaa(third party website) when i enter caipiao.aaa.com/kaijiang, it should show me the content of www.bbb.com/result_aaa From: cyril.bo...@free.fr To: y...@hotmail.com CC: haproxy@formilux.org; w...@1wt.eu Subject: Re: Need help about reqrep (urgent) Date: Sat, 24 Sep 2011 17:27:37 +0200 Le Samedi 24 Septembre 2011 15:02:59 DerekLiao a écrit : configuration files: reqrep ^([^\ ]*)\ /kaijiang([/\?\ ].*)\1\ /result_aaa\2 reqirep ^Host: Host:\ www.bbb.com Logs: srv_aaa_caipiaos_01 0/0/0/2/2 404 735 - - 0/0/0/0/0 0/0 GET /result_aaa/ HTTP/1.1 I'm lost, I don't understand what you are testing. Why is your log showing /result_aaa/ if you want to test /kaijiang ? Also, you don't provide enough information at the same time: Why your server is not the same as the one you provided at the beginning : srv_aaa_caipiaos_1_01 ? I think you forgot to give other pieces of configuration. It looks like you are testing another backend configuration. Confirmed by your frontend acls if they didn't change : acl acl_dom_caipiao.aaa.com hdr_dom(host) caipiao.aaa.com acl acl_url_kaijiang path_beg -i /kaijiang use_backend cs_aaa_caipiaos_1 if acl_dom_caipiao.aaa.com acl_url_kaijiang If you ask for /result_aaa/ the condition on the last line won't match. Can you provide everything (hide the IP/host names/passwords for security reasons) ? This will save time to everybody ;-) but i can access www.bbb.com/result_aaa/ I suppose that you missed something about the Host headers. OK, you can access to www.bbb.com/result_aaa/ But it looks like your hostname in front of haproxy is caipiao.aaa.com (or aaa.com) Are you sure the server www.bbb.com accepts your virtualhost ? You can verify it : curl -i -H Host: caipiao.aaa.com www.bbb.com/result_aaa/ Isn't it a 404 ? -- Cyril Bonté
Re: Need help about reqrep (urgent)
On Sat, Sep 24, 2011 at 03:02:59PM +, DerekLiao wrote: configuration files: reqrep ^([^\ ]*)\ /kaijiang([/\?\ ].*)\1\ /result_aaa\2 reqirep ^Host: Host:\ www.bbb.com Logs: srv_aaa_caipiaos_01 0/0/0/2/2 404 735 - - 0/0/0/0/0 0/0 GET /result_aaa/ HTTP/1.1 but i can access www.bbb.com/result_aaa/ You still did not comment on the point I made below : BTW, we're all assuming that you have appropriately set option httpclose or option http-server-close so that all requests are rewritten, not only the first one. You might want to double-check. We still have no trace of the request sent to the server either, which will help find where your error is. It's particularly difficult to help you because the discussion seems to be one-way, and we're playing riddles with your configuration. With all the requested info, maybe your site would already be running now. If you want people to help you, please try to be a bit more open about your config, and to respond to their questions. Till now your requests look more like complaints than requests for help. Willy
Re: Haproxy notifications
Hey Ibrahim. Continued on your GitHub issue! On Sat, Sep 24, 2011 at 1:25 AM, İbrahim Ercan ibrahim.er...@engineer.comwrote: Hi Carlo, I tried to install it but i got that error when run it. -# service haproxyctl show health /etc/init.d/haproxyctl:17:in `require': no such file to load -- haproxyctl (LoadError) from /etc/init.d/haproxyctl:17 On 22 September 2011 21:42, carlo flores ca...@petalphile.com wrote: Here's an HAProxy ctl/init script that includes Nagios and Cloudkick checks (to the UNIX socket and lsof listeners) to go with the regular start/stop/init stuff: https://github.com/flores/haproxyctl The advantage of using a wrapper around the UNIX socket is you can also add checks for available backends, hosts down, and so forth. This should be in addition to whatever TCP/HTTP checks that come default in any monitoring solution. (The script above is easy to edit to add such things.) On Thu, Sep 22, 2011 at 10:02 AM, Ben Timby bti...@gmail.com wrote: On Thu, Sep 22, 2011 at 11:30 AM, Guillaume Bourque guillaume.bour...@gmail.com wrote: option log-health-checks :-) I took notification to mean something other than logging. -- *İbrahim Ercan * *Gsm: 0 505 608 24 37*