Re: SMTP loadbalancer haproxy and logs
Look at the mailinglist fom yesterday ;) I was asking the same question ;) Michiel Israel Garcia wrote: My haproxy configuration running prefectly. A simple LB with two smtp servers. But in both smtp servers only report smtp connections from haproxy server and I want to log en every smtp server the original ip from the sending source server. Is it possible? global maxconn 500 log 127.0.0.1 local0 user haproxy group haproxy # debug defaults log global mode http stats enable stats auth hadmin:XX balance roundrobin maxconn 500 option httplog option abortonclose retries 3 option redispatch clitimeout 15000 contimeout 3 srvtimeout 3 listen stats :80 mode http stats enable stats auth hadmin:X stats refresh20s listen smtp :25 mode tcp maxconn 500 option tcplog balance roundrobin server smtp1 xx.xx.xx.xx:25 check inter 1 server smtp2 xx.xx.xx.xx:25 check inter 1
Re: question about transparancy
Original Message Subject: Re: question about transparancy From: Angelo Höngens To: haproxy@formilux.org Date: 04/23/2010 11:31 AM On 23-4-2010 11:19, Michiel van Es wrote: Angelo Höngens wrote: On 22-4-2010 20:28, Michiel van Es wrote: Yes That is the default smtp failover setup but I want to balance the load via a load balancer setup Mx records can not balance load If you have 2 mx records with the same priority, your load should be balanced.. Or you could have a single mx record pointing to a hostname which has 2 A records.. DNS round robin will take care of the balancing. That is why there are almost no smtp balancers, because it is not needed. In the 1980's they already designed smtp for balancing and failover. For other protocols this was not so easy, that's why people wrote http balancers :) Yes I understand, but what about settings features as weight or doe advanced load balancing? You can't do advanced balancing, true.. If you *must* have weight, you could go for the host records approach. Make 1 MX record pointing to mx-in.example.com, and create three host records: mx-in -> x.x.x.1, mx-in -> x.x.x.1, mx-in -> x.x.x.2. This way, server 1 gets around 66% of the sessions, and server 2 gets around 33% of the sessions. What is one of the mailservers are broken and you want to take it offline. With a normal TTL in dns it can take 1 or 2 days before other mailservers know it should not send a mail to that server and use the other. I like load balancers because they can let you decide how traffic must flow. No problem if you use the MX way, just take the server offline, no need to change dns.. Remote mail servers will just try one mail server, and if it's down, they will use the other, failover is built into the way smtp and dns work together. I'm not saying you should not do what you are doing. If you really want to use your own balancer, and you feel better doing that, then by all means please do. What's I'm saying is that people have been balancing smtp servers for 30 years using the ways they though of in the 80's, and since that works for most organisations, it might work for you. KISS. Don't look blindly at the tools you're using, but choose the tools you need based on the goal you're trying to reach. Ah, who am I kidding, I'm just an IT-nerd wanting to play with cool balancers as well.. That is the whole idea ;) I know DNS load balancing works (I have it up & running right now) but I want to do load balancing with a load balancer..I want to have a setup which is flexible and a blueprint for let say: a 500+ mailserver setup with very heavy load (think Google or Hotmail). Michiel
Re: question about transparancy
Angelo Höngens wrote: On 22-4-2010 20:28, Michiel van Es wrote: Yes That is the default smtp failover setup but I want to balance the load via a load balancer setup Mx records can not balance load If you have 2 mx records with the same priority, your load should be balanced.. Or you could have a single mx record pointing to a hostname which has 2 A records.. DNS round robin will take care of the balancing. That is why there are almost no smtp balancers, because it is not needed. In the 1980's they already designed smtp for balancing and failover. For other protocols this was not so easy, that's why people wrote http balancers :) Yes I understand, but what about settings features as weight or doe advanced load balancing? What is one of the mailservers are broken and you want to take it offline. With a normal TTL in dns it can take 1 or 2 days before other mailservers know it should not send a mail to that server and use the other. I like load balancers because they can let you decide how traffic must flow. I am trying now LVS on my HAproxy load balancer but can not get it working with just 1 public ip-adres (it is a vps). I am buying a 2nd ip to use as vip for LVS..perhaps that works. Kind Regards, Michiel
Re: question about transparancy
Hmm, I found out that it is not supported for mode tcp: Starting HAproxy: [WARNING] 111/213701 (5089) : config : 'option forwardfor' ignored for proxy 'load_balanced_smtp' as it requires HTTP mode. [ OK ] Is this possible for the mode tcp and smtp and Haproxy? Michiel On 4/22/10 9:34 PM, Michiel van Es wrote: I am running ASSP on the mailservers, since 1.6.2 ASSP understands headres like x-forward-for : - Text to Identify Originating IP Header* (OriginatingIP) If ASSP runs behind another server(s), no IP/HELO based filter will work. If a special header with the originating IP is inserted from the frontend serber ASSP will use the originating IP to perform IP and HELO checks. For example: X-Forwarded-For|X-Originating-IP Is it possible to let HAproxy push such an header for smtp without recompiling the kernel on my load balancer with Tproxy support? I hope it works as it works for http out of the box :) Kind regards, Michiel On 4/22/10 8:28 PM, Michiel van Es wrote: Yes That is the default smtp failover setup but I want to balance the load via a load balancer setup Mx records can not balance load Regards Michiel van Es Op 22 apr 2010 om 19:06 heeft Guillaume Bourque het volgende geschreven:\ Hi, Why dont you use 2 MX record with the same weight All email will come in to youre 2 servers. Just my 2¢ Bye Michiel van Es a écrit : Hi, I am running 2 mailservers with ASSP as a spam proxy. I use haproxy to load balance between the 2 mailservers. The problem is that the spam filter is seeing every connection coming from the load balancer and not the originating ip. Do you know how I make the connections from my HAproxy load balancer to my 2 mailservers transparent? Kind regards, Michiel -- Guillaume Bourque, B.Sc., consultant, infrastructures technologiques libres ! 514 576-7638
Re: question about transparancy
I am running ASSP on the mailservers, since 1.6.2 ASSP understands headres like x-forward-for : - Text to Identify Originating IP Header* (OriginatingIP) If ASSP runs behind another server(s), no IP/HELO based filter will work. If a special header with the originating IP is inserted from the frontend serber ASSP will use the originating IP to perform IP and HELO checks. For example: X-Forwarded-For|X-Originating-IP Is it possible to let HAproxy push such an header for smtp without recompiling the kernel on my load balancer with Tproxy support? I hope it works as it works for http out of the box :) Kind regards, Michiel On 4/22/10 8:28 PM, Michiel van Es wrote: Yes That is the default smtp failover setup but I want to balance the load via a load balancer setup Mx records can not balance load Regards Michiel van Es Op 22 apr 2010 om 19:06 heeft Guillaume Bourque het volgende geschreven:\ Hi, Why dont you use 2 MX record with the same weight All email will come in to youre 2 servers. Just my 2¢ Bye Michiel van Es a écrit : Hi, I am running 2 mailservers with ASSP as a spam proxy. I use haproxy to load balance between the 2 mailservers. The problem is that the spam filter is seeing every connection coming from the load balancer and not the originating ip. Do you know how I make the connections from my HAproxy load balancer to my 2 mailservers transparent? Kind regards, Michiel -- Guillaume Bourque, B.Sc., consultant, infrastructures technologiques libres ! 514 576-7638
Re: question about transparancy
Yes That is the default smtp failover setup but I want to balance the load via a load balancer setup Mx records can not balance load Regards Michiel van Es Op 22 apr 2010 om 19:06 heeft Guillaume Bourque > het volgende geschreven:\ Hi, Why dont you use 2 MX record with the same weight All email will come in to youre 2 servers. Just my 2¢ Bye Michiel van Es a écrit : Hi, I am running 2 mailservers with ASSP as a spam proxy. I use haproxy to load balance between the 2 mailservers. The problem is that the spam filter is seeing every connection coming from the load balancer and not the originating ip. Do you know how I make the connections from my HAproxy load balancer to my 2 mailservers transparent? Kind regards, Michiel -- Guillaume Bourque, B.Sc., consultant, infrastructures technologiques libres ! 514 576-7638
Re: question about transparancy
On 4/22/10 6:03 PM, Chris Sarginson wrote: Michiel van Es wrote: I read this archive from 2008: http://www.formilux.org/archives/haproxy/0801/0507.html But the problem is that I can not recompile the kernel :( Isn't there any other way like the x-forward-for header to make this work? As this runs on layer 4 headers can not be inserted. If you wish to be able to see the originating IP addresses you will probably want to use LVS loadbalancing. Hmmm..that is too bad..then I have to work out a new LVS setup after I finally got HAproxy up & running :( Why isn't it possible with smtp but it is working with http? Chris Michiel
Re: question about transparancy
I read this archive from 2008: http://www.formilux.org/archives/haproxy/0801/0507.html But the problem is that I can not recompile the kernel :( Isn't there any other way like the x-forward-for header to make this work? Kind regards, Michiel On 4/22/10 5:55 PM, Michiel van Es wrote: Hi, I am running 2 mailservers with ASSP as a spam proxy. I use haproxy to load balance between the 2 mailservers. The problem is that the spam filter is seeing every connection coming from the load balancer and not the originating ip. Do you know how I make the connections from my HAproxy load balancer to my 2 mailservers transparent? Kind regards, Michiel
question about transparancy
Hi, I am running 2 mailservers with ASSP as a spam proxy. I use haproxy to load balance between the 2 mailservers. The problem is that the spam filter is seeing every connection coming from the load balancer and not the originating ip. Do you know how I make the connections from my HAproxy load balancer to my 2 mailservers transparent? Kind regards, Michiel
Re: newbie question
On 4/20/10 11:45 PM, Willy Tarreau wrote: On Tue, Apr 20, 2010 at 11:31:02PM +0200, Michiel van Es wrote: listen load_balanced :80 mode http option httpclose balance roundrobin# Load Balancing algorithm option forwardfor # This sets X-Forwarded-For server server1 194.145.200.17 server server2 194.145.200.171 none of your servers are checked (neither here nor in any other section). You should add "check" on each "server" line for that. Shall I enable check on every server entry for every 'service' entry? If you think that any of your service may individually fail, yes you should. For instance, the SMTP service might die without the HTTP service being affected. However, if some services are provided by a same daemon which cannot "half-fail" (eg: pop/imap), then you can enable server tracking. The principle is that only one of the service does the checks, and the other one says "track xxx/yyy" instead of "check", where "xxx" is the other instance name (the "listen" section in your case) and "yyy" the server name. That way, the tracking server will always be in the same state as the tracked server, and both will be up or down simultaneously. I like the check method, so you think my posted 2nd config looks better? Michiel Willy
Re: newbie question
On 4/20/10 11:19 PM, Willy Tarreau wrote:
Hi Michiel,
Hi Willy
On Tue, Apr 20, 2010 at 10:07:10PM +0200, Michiel van Es wrote:
Hello,
I am trying to set up HAproxy for 2 servers servicing:
smtp, imaps, http, https and xmpp over ssl
I am getting problems with squirrelmail over https, as I login I get the
message:
Your session has expired, but will be resumed after logging in again.
Go to the login page
I think it because the HTTPS requests are balanced roundrobin so server1
sets the cookie and server2 does not know about the session.
What I want is that a users is sticky connected over https (perhaps all
my serviced protocols?) and if the server is down , the user is services
at the other server.
First for that you have to define how you identify a "user". In HTTP mode,
we generally use cookie insertion so that the user presents himself the
indication of the server it was on. For other protocols however, we don't
have this information. So most likely in your case, you'll want to use the
source IP address as a poor identifier of the "user". It will generally
work OK for most of your users but some of them will experience issues
when they browse from multiple source addresses (rare for end users, but
fairly common from enterprises). What is generally suggested for that is
to stick on the cookie in HTTP mode and source IP for other protocols,
hoping that the user will not switch to another IP when switching from
HTTP to another protocol.
I have changed my config somewhat and it seems to work much better for
failover and stickyness:
global
maxconn 4096 # Total Max Connections.
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
daemon
user haproxy
grouphaproxy
chroot /var/chroot/haproxy
defaults
log global
modehttp
#option httplog
option dontlognull
retries 2
option redispatch
maxconn 2000
contimeout 5000
clitimeout 5
srvtimeout 5
listen load_balanced_https :443
mode tcp
balance source
option ssl-hello-chk
server webserver1 194.145.200.17 weight 1 maxconn 2000 check
server webserver2 194.145.200.171 weight 1 maxconn 2000 check
listen load_balanced_http :80
mode http
stats enable
stats auth admin:vegansxe
balance roundrobin
cookie JSESSIONID prefix
option httplog
option httpclose
option forwardfor
option httpchk HEAD /src/login.php HTTP/1.0
server webA 194.145.200.17:80 cookie A check
server webB 194.145.200.171:80 cookie B check
listen load_balanced_xmpp :5222
mode tcp
balance source
server webserver1 194.145.200.17 weight 1 maxconn 2000 check
server webserver2 194.145.200.171 weight 1 maxconn 2000 check
listen load_balanced_smtp :25
mode tcp
balance source
server webserver1 194.145.200.17 weight 1 maxconn 2000 check
server webserver2 194.145.200.171 weight 1 maxconn 2000 check
listen load_balanced_smtps :27
mode tcp
balance source
server webserver1 194.145.200.17 weight 1 maxconn 2000 check
server webserver2 194.145.200.171 weight 1 maxconn 2000 check
listen load_balanced_imaps :993
mode tcp
balance source
server webserver1 194.145.200.17 weight 1 maxconn 2000 check
server webserver2 194.145.200.171 weight 1 maxconn 2000 check
listen admin_stats :8080
mode http
stats uri /my_stats
stats realm Global\ statistics
stats auth adminlogin:adminpassword
In version 1.4, you can simply declare a "stick-table" with a set of
parameters (number of entries, expiration, etc...) and indicate in all
of your "listen" sections that you want to use it to stick on the source
IP address ("stick on src table XXX").
In version 1.3 there is no "stick table" so you'll have to fallback to
source IP hashing, which is not as smooth and can cause major redispatch
issues when a server dies or comes back. Still this is a very commonly
accepted tradeoff.
I have one comment about your config below :
My haproxy.conf looks like:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096 # Total Max Connections. This is dependent on
ulimit
daemon
nbproc 2 # Number of processing cores. Dual Dual-core Opteron is
4 cores for example.
Never use that "nbproc" parameter unless you have a really valid
reason to do so (and if so you'll know why you want to use it).
Running on more than one process will prevent any form of stickiness
from working because incoming connections are randomly balanced on
any process with its own stick table. Also, it causes the checks
to be sent twice to the servers, and prevents you from getting
correct stats because any of the two processes can process the
stats requests. You should then simply remove this line.
Removed it!
user haproxy
grouphaproxy
chroot /var/chroot/hapr
newbie question
Hello, I am trying to set up HAproxy for 2 servers servicing: smtp, imaps, http, https and xmpp over ssl I am getting problems with squirrelmail over https, as I login I get the message: Your session has expired, but will be resumed after logging in again. Go to the login page I think it because the HTTPS requests are balanced roundrobin so server1 sets the cookie and server2 does not know about the session. What I want is that a users is sticky connected over https (perhaps all my serviced protocols?) and if the server is down , the user is services at the other server. My haproxy.conf looks like: global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 4096 # Total Max Connections. This is dependent on ulimit daemon nbproc 2 # Number of processing cores. Dual Dual-core Opteron is 4 cores for example. user haproxy grouphaproxy chroot /var/chroot/haproxy defaults log global modehttp retries 3 clitimeout 6 srvtimeout 3 contimeout 4000 # option httpclose # Disable Keepalive listen load_balanced :80 mode http option httpclose balance roundrobin# Load Balancing algorithm option forwardfor # This sets X-Forwarded-For server server1 194.145.200.17 server server2 194.145.200.171 listen http_proxy :443 mode tcp option ssl-hello-chk balance roundrobin server server1 194.145.200.17 server server2 194.145.200.171 listen smtp_proxy 0.0.0.0:25 mode tcp balance roundrobin server server1 194.145.200.17:25 weight 1 maxconn 512 server server2 194.145.200.171:25 weight 1 maxconn 512 listen ftp_proxy :20,:21 #,:6-6 mode tcp balance roundrobin server server1 194.145.200.17 weight 1 maxconn 512 server server2 194.145.200.171 weight 1 maxconn 512 listen smtps_proxy 0.0.0.0:27 mode tcp option ssl-hello-chk balance roundrobin server server1 194.145.200.17:27 weight 1 maxconn 512 server server2 194.145.200.171:27 weight 1 maxconn 512 listen xmpps_proxy :5222 mode tcp option ssl-hello-chk balance roundrobin server server1 194.145.200.17 weight 1 maxconn 512 server server2 194.145.200.171 weight 1 maxconn 512 listen imaps_proxy 0.0.0.0:993 mode tcp option ssl-hello-chk balance roundrobin server server1 194.145.200.17:993 weight 1 maxconn 512 server server2 194.145.200.171:993 weight 1 maxconn 512 I tried to read the configuration.txt manual time after time but I can nog et it right :( Can anyone help me? :) Kind regards, Michiel
