Re: configuration problem with "backup" backends
It looks like answer to question #1 is the use of a stick-table stick-table type ip size 1 nopurge stick on dst On 09/12/16 11:13, Michele Mazzucco wrote: Hello, I have two HAProxy servers configured with a floating IP, providing a hot standby solution. The listen configuration looks like: listen my_backend bind option mysql-check ... server mysql1 source server mysql2 source backup ... I would like to know if the following are possible: 1 - Due to the "backup" keyword, mysql2 is employed *only* when mysql1 is unavailable. Whenever mysql1 becomes available, however, it will be used, leading to possible consistency problems. Is it possible to keep mysql2 as primary once mysq1 comes back online? 2 - By using the "source" keywork in the server lines the floating IP is employed when communicating with the MySQL servers, including periodic checks. Is is possible to use the static HAProxy IP for checks and the floating IP for SQL transactions? The rationale behind that is that checks are performed by both HAProxy servers while SQL transactions are carried out only by one. Thank you, Michele
configuration problem with "backup" backends
Hello, I have two HAProxy servers configured with a floating IP, providing a hot standby solution. The listen configuration looks like: listen my_backend bind option mysql-check ... server mysql1 source server mysql2 source backup ... I would like to know if the following are possible: 1 - Due to the "backup" keyword, mysql2 is employed *only* when mysql1 is unavailable. Whenever mysql1 becomes available, however, it will be used, leading to possible consistency problems. Is it possible to keep mysql2 as primary once mysq1 comes back online? 2 - By using the "source" keywork in the server lines the floating IP is employed when communicating with the MySQL servers, including periodic checks. Is is possible to use the static HAProxy IP for checks and the floating IP for SQL transactions? The rationale behind that is that checks are performed by both HAProxy servers while SQL transactions are carried out only by one. Thank you, Michele
Questions regarding HA setup for MySQL
Hello, I am considering using HAProxy to build an HA system where HAProxy instances are used to forward SQL queries to a MySQL cluster with a master-master configuration. I went through an old email of Willy, see http://www.formilux.org/archives/haproxy/1003/3259.html, but still could not clarify some doubts I have. 1 - Regarding HAProxy availability, I have successfully tried keepalived, however it does not prevent split-brain scenarios, e.g., all HAPRoxy instances may decide to take the floating IP address they share. Would you recommend using pacemaker instead?, I would be willing to sacrifice availability in order to ensure 100% consistency. 2 - Regarding the MySQL farm, I would like writes to be sent to only one server due to the use of auto-increment within MySQL transactions. I am aware that the "backup" keyword in the haproxy configuration can be used to achieve that, however I would like to have "stickyness", i.e., once the failover has occurred, I do not want the failed node to be promoted as "master" whenever it becomes available, as long as the "backup" is up and running. 3 - Is it possible to perform custom checks/operations? e.g., in order to prevent accidental modifications I would like all MySQL nodes apart from the "active" one to be in read only mode, see http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_super_read_only, and change that *only* when the backup is promoted. Thanks, Michele
Re: problems with enabling/disabling servers via socket
Hello Willy, I have just upgraded HAProxy from 1.4.18 to 1.4.20 and it looks like the problem occurs when enabling servers only. 2012-03-15 08:13:55,191 Disabling reserves 2012-03-15 08:13:55,191 disable server www/i-95f89df1 ; disable server www/i-8bf89def ; disable server www/i-97f89df3 2012-03-15 08:14:18,112 Enabling reserves 2012-03-15 08:14:18,113 enable server www/i-95f89df1 ; enable server www/i-8bf89def ; enable server www/i-97f89df3 2012-03-15 08:14:18,113 Expected 5 active serves, have 2, calling change_state() again 2012-03-15 08:14:18,113 enable server www/i-95f89df1 ; enable server www/i-8bf89def ; enable server www/i-97f89df3 ... 012-03-15 08:14:53,963 Enabling reserves 2012-03-15 08:14:53,963 enable server www/i-95f89df1 ; enable server www/i-8bf89def ; enable server www/i-97f89df3 2012-03-15 08:14:53,964 Expected 5 active serves, have 2, calling change_state() again 2012-03-15 08:14:53,964 enable server www/i-95f89df1 ; enable server www/i-8bf89def ; enable server www/i-97f89df3 . 2012-03-15 08:14:54,971 Disabling reserves 2012-03-15 08:14:54,972 disable server www/i-95f89df1 ; disable server www/i-8bf89def ; disable server www/i-97f89df3 2012-03-15 08:15:45,289 Enabling reserves 2012-03-15 08:15:45,290 enable server www/i-95f89df1 ; enable server www/i-8bf89def ; enable server www/i-97f89df3 enabling works fine ... 2012-03-15 08:15:46,297 Disabling reserves 2012-03-15 08:15:46,297 disable server www/i-95f89df1 ; disable server www/i-8bf89def ; disable server www/i-97f89df3 Michele On Mar 15, 2012, at 8:46 AM, Willy Tarreau wrote: Hi Michele, On Wed, Mar 14, 2012 at 01:07:51PM +0200, Michele Mazzucco wrote: Hello Willy, I have tried to follow your advice, but it didn't solve the matter -- it looks like the problem is not the web interface. After executing the code above, I use a different socket to retrieve the stats and check that the number of active servers corresponds to the expected one. This is the log 2012-03-14 10:50:56,568 Enabling reserves 2012-03-14 10:50:56,568 enable server www/i-932dbef7 -- command sent over the socket 2012-03-14 10:50:56,569 Expected 5 active serves, have 4, calling change_state() again 2012-03-14 10:50:56,569 enable server www/i-932dbef7 --- command sent over the socket ... server enabled Just to be sure, is it the disabling of servers which causes trouble or enabling them back ? I'm asking because we (very) recently fixed an issues related to servers leaving maintenance mode which was introduced by fixing another issue with server tracking, but the regression did not affect any 1.4 releases. However, since both fixes have been backported into 1.4.20, it is possible there was another corner case we did not identify and which is solved now. Just in case, would you please check if 1.4.20 still behaves the same ? Maybe we're trying to troubleshoot an already fixed issue. Regards, Willy
Re: problems with enabling/disabling servers via socket
Hi Carlo, On Mar 15, 2012, at 9:04 AM, Carlo Flores wrote: Hey Michele. I'm confused as to how show stat is correct... According to the statistics I have retrieved by means of show stat after enabling disabling servers, it looks like the socket is working as expected. But your script does not correspond to this even though it uses show stat... I have changed the script. If you wish to see it, I can send it to you. After executing the code above, I use a different socket to retrieve the stats and check that the number of active servers corresponds to the expected one. You're sure it's not an issue with this script and the autoheal you have going on to see the number of expected servers? No, I have tried with hatop as well, and the same problem occurs. If you stop that script and simply disable a server via socat, then refresh the interface as well as get a new show stat, then enable a server via socat, then refresh the interface as well as get a new show stat, are your results as expected? I don't think it's a problem related to the web interface -- socat behaves like my script, i.e., it fails to enable servers from time to time. Cheers, Michele On Wed, Mar 14, 2012 at 11:46 PM, Willy Tarreau w...@1wt.eu wrote: Hi Michele, On Wed, Mar 14, 2012 at 01:07:51PM +0200, Michele Mazzucco wrote: Hello Willy, I have tried to follow your advice, but it didn't solve the matter -- it looks like the problem is not the web interface. After executing the code above, I use a different socket to retrieve the stats and check that the number of active servers corresponds to the expected one. This is the log 2012-03-14 10:50:56,568 Enabling reserves 2012-03-14 10:50:56,568 enable server www/i-932dbef7 -- command sent over the socket 2012-03-14 10:50:56,569 Expected 5 active serves, have 4, calling change_state() again 2012-03-14 10:50:56,569 enable server www/i-932dbef7 --- command sent over the socket ... server enabled Just to be sure, is it the disabling of servers which causes trouble or enabling them back ? I'm asking because we (very) recently fixed an issues related to servers leaving maintenance mode which was introduced by fixing another issue with server tracking, but the regression did not affect any 1.4 releases. However, since both fixes have been backported into 1.4.20, it is possible there was another corner case we did not identify and which is solved now. Just in case, would you please check if 1.4.20 still behaves the same ? Maybe we're trying to troubleshoot an already fixed issue. Regards, Willy
Re: problems with enabling/disabling servers via socket
Hello Willy, when the 'enable' command fails, it ignores ALL the servers in the list, e.g., see line 2 and line 4 of the logs below. The command is exactly the same, however it fails at the first attempt. For testing purposes, I have now added a small sleep (40 ms) between the enable/disable command and the check. It looks like things are working as expected now. Please note, however, that this is not due to the fact that I am using 2 separate sockets -- the spurious behavior occurs even if a single socket (open in interactive mode) is employed. Thanks, Michele On Mar 15, 2012, at 11:49 AM, Willy Tarreau wrote: Hi again, On Thu, Mar 15, 2012 at 11:42:24AM +0200, Michele Mazzucco wrote: Hi Willy, I apologize if I wasn't clear. The comment in my previous email means that the servers were enabled, as expected. In other words, disabling servers seems to work fine ALL the times, while enabling servers works from time to time only. In the scenario described by those logs (see my previous email), 2 servers are always enabled, while 3 are enabled/disabled at runtime. The following log 2012-03-15 08:14:18,112 Enabling reserves 2012-03-15 08:14:18,113 enable server www/i-95f89df1 ; enable server www/i-8bf89def ; enable server www/i-97f89df3 --- line 2 2012-03-15 08:14:18,113 Expected 5 active serves, have 2, calling change_state() again --- line 3 2012-03-15 08:14:18,113 enable server www/i-95f89df1 ; enable server www/i-8bf89def ; enable server www/i-97f89df3 --- line 4 means that 3 servers were supposed to be enabled after line 2 (e.g., 5 servers in total), but were not (only the 2 'always on' servers are enabled). Hence, the command was issued again in line 4. This time the command succeeded. Here, instead: 2012-03-15 08:15:45,289 Enabling reserves 2012-03-15 08:15:45,290 enable server www/i-95f89df1 ; enable server www/i-8bf89def ; enable server www/i-97f89df3 enabling works fine the command succeeded at the first attempt. I should mention that line 3 above uses a (different) socket which is open in interactive mode. There, the updated state is retrieved by means of 'show stat'. Does it make any sense now? Yes it's clearer that way. Still, if you have a means to report which servers fail to be enabled, it would help. For instance we might discover it's always the beginning of the list or the end of the list which is ignored. Regards, Willy
Re: problems with enabling/disabling servers via socket
Hello, the attached Python script can be used to reproduce the problem. No external library is necessary. Cheers, Michele ''' Created on Mar 15, 2012 @author: Michele Mazzucco michelemazzu...@gmail.com ''' import sys, time, subprocess, string # backend backend = 'www' # list of servers servers_list = ['i-932dbef7', 'i-c368faa7', 'i-c968faad', 'i-cf68faab', 'i-c168faa5'] def test(sock='tmp/haproxy'): ''' * param sock: path to the socket used by HAProxy ''' enable = 'disable' while True: l = [] for i in servers_list: tmp = '%s server %s/%s' % (enable, backend, i) l.append(tmp) command = ';'.join(l) #socat = 'echo %s | socat stdio %s' % (command, sock) p1 = subprocess.Popen(['echo', command], stdout=subprocess.PIPE, shell=False) p2 = subprocess.Popen(['socat', 'stdio', sock], stdin=p1.stdout, shell=False, stderr=subprocess.STDOUT, stdout=subprocess.PIPE) output = p2.communicate() if p2.returncode != 0: raise RuntimeError('Unable to issue command') #socat = 'echo show stat | socat stdio %s' % sock # check p1 = subprocess.Popen(['echo', 'show stat'], stdout=subprocess.PIPE, shell=False) p2 = subprocess.Popen(['socat', 'stdio', sock], stdin=p1.stdout, shell=False, stderr=subprocess.STDOUT, stdout=subprocess.PIPE) output = p2.communicate() if p2.returncode != 0: raise RuntimeError('Unable to retrieve stats') res = output[0].split('\n') for line in res: if line.startswith(backend): tokens = string.split(line, ',') if tokens[1] in servers_list: # ignore BACKEND if enable == 'disable' and tokens[17] != 'MAINT': # see sec. 9.1 # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt print 'Expected MAINT, got %s' % tokens[17] print line raise RuntimeError() if enable == 'enable' and tokens[17] == 'MAINT': print 'Expected not MAINT!' print line raise RuntimeError() if enable == 'enable': enable = 'disable' else: enable = 'enable' time.sleep(0.1) # the longer sleep is, the less likely the problem is likely to occur if __name__ == '__main__': sock = '/tmp/haproxy' if len(sys.argv) 1: sock = sys.argv[1] test(sock) On Mar 15, 2012, at 4:59 PM, Jonathan Matthews wrote: On 15 March 2012 08:21, Michele Mazzucco michelemazzu...@gmail.com wrote: I don't think it's a problem related to the web interface -- socat behaves like my script, i.e., it fails to enable servers from time to time. Michele - I think the most useful thing you can do is to provide a socat invocation that causes the problem, and let the list know how frequently it's caused by that *exact* invocation. Or, to put it another way, I'm not entirely sure that we're not simply remote-debugging your script, without actually having a copy of it! However, it's not really in-scope for anyone on list to fix your script, hence providing a simple command using very simple primitives (bash; socat; etc) that we can repeat until failure occurs will really help to clarify the failure you're observing. FWIW, I suggest you /don't/ provide the script :-) All the best, Jonathan -- Jonathan Matthews London, Oxford, UK http://www.jpluscplusm.com/contact.html
Re: problems with enabling/disabling servers via socket
Hello Jonathan, I have already sent Willy the strace logs. Cheers, Michele On Mar 15, 2012, at 4:59 PM, Jonathan Matthews wrote: On 15 March 2012 08:21, Michele Mazzucco michelemazzu...@gmail.com wrote: I don't think it's a problem related to the web interface -- socat behaves like my script, i.e., it fails to enable servers from time to time. Michele - I think the most useful thing you can do is to provide a socat invocation that causes the problem, and let the list know how frequently it's caused by that *exact* invocation. Or, to put it another way, I'm not entirely sure that we're not simply remote-debugging your script, without actually having a copy of it! However, it's not really in-scope for anyone on list to fix your script, hence providing a simple command using very simple primitives (bash; socat; etc) that we can repeat until failure occurs will really help to clarify the failure you're observing. FWIW, I suggest you /don't/ provide the script :-) All the best, Jonathan -- Jonathan Matthews London, Oxford, UK http://www.jpluscplusm.com/contact.html
Re: problems with enabling/disabling servers via socket
Hello Willy, I have tried to follow your advice, but it didn't solve the matter -- it looks like the problem is not the web interface. After executing the code above, I use a different socket to retrieve the stats and check that the number of active servers corresponds to the expected one. This is the log 2012-03-14 10:50:56,568 Enabling reserves 2012-03-14 10:50:56,568 enable server www/i-932dbef7 -- command sent over the socket 2012-03-14 10:50:56,569 Expected 5 active serves, have 4, calling change_state() again 2012-03-14 10:50:56,569 enable server www/i-932dbef7 --- command sent over the socket ... server enabled Cheers, Michele On Mar 10, 2012, at 10:16 AM, Willy Tarreau wrote: Hello Michele, On Mon, Mar 05, 2012 at 03:41:20PM +0200, Michele Mazzucco wrote: Hello everyone, I think I might have tracked down the problem. According to the statistics I have retrieved by means of show stat after enabling disabling servers, it looks like the socket is working as expected. The might be in the web interface. In my test, involving HAProxy 1.4.18 and 2 Apache servers being enabled/disabled every 5 seconds, show stat was issued after every state change (using a different socket) and reported the expected number of active servers (i.e., either 0 or 2). On the other hand in a number of times the web interface marked only 1 of the two serves in brown (MAINT). Since your script sends the commands one at a time, it is fairly possible that the stats page refreshes between the two commands and only shows one server disabled. I suggest that if you absolutely need to have them shut at the same time, then you disable them on the same command line which will be executed at once : disable server b1/s1 ; disable server b1/s2 enable server b1/s1 ; enable server b1/s2 Regards, Willy
Re: problems with enabling/disabling servers via socket
Hello everyone, I think I might have tracked down the problem. According to the statistics I have retrieved by means of show stat after enabling disabling servers, it looks like the socket is working as expected. The might be in the web interface. In my test, involving HAProxy 1.4.18 and 2 Apache servers being enabled/disabled every 5 seconds, show stat was issued after every state change (using a different socket) and reported the expected number of active servers (i.e., either 0 or 2). On the other hand in a number of times the web interface marked only 1 of the two serves in brown (MAINT). Cheers, Michele It looks like servers are enabled/disabled On Mar 2, 2012, at 7:25 PM, Cyril Bonté wrote: Hi Michele, Le 02/03/2012 17:26, Michele Mazzucco a écrit : Hello, I am trying to enable/disable servers at runtime by means of the UNIX socket created by HAProxy. In order to do that, I am using the Python script below. I'm testing the code with 2 servers, and I have noticed that quite often only one of the 2 servers is enabled/disable (sometimes the first, others the second). However sometimes none of them changes state, of both of them do. I have tried to open the socket in non-interactive mode as well (i.e., re-open the socket at every command), but I didn't notice any difference. Am I doing anything wrong?, servers' names are something like i-a3c832c7. As a side not, I should add that replacing the send() on the socket with socat Maybe you didn't set your socket in admin level ? global stats socket ... level admin echo disable server www/i-9bb943ff | socat stdio unix-connect:/tmp/haproxy does not change the behavior of the system. Thanks, Michele def change_state(self, servers_list, enable=True): try: s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) s.connect(self.socket_path) # interactive mode (see documentation, sec. 9.2) s.send('prompt\n') self.wait(s) # waits for the prompt for i in servers_list: command = None if enable == True: command = 'enable server www/%s' % i else: command = 'disable server www/%s' % i log.info(command) s.send('%s\n' % command) self.wait(s) s.shutdown(socket.SHUT_RDWR) s.close() except SocketError, e: log.error('socket error: %s' % e) self.__go = False -- Cyril Bonté
problems with enabling/disabling servers via socket
Hello, I am trying to enable/disable servers at runtime by means of the UNIX socket created by HAProxy. In order to do that, I am using the Python script below. I'm testing the code with 2 servers, and I have noticed that quite often only one of the 2 servers is enabled/disable (sometimes the first, others the second). However sometimes none of them changes state, of both of them do. I have tried to open the socket in non-interactive mode as well (i.e., re-open the socket at every command), but I didn't notice any difference. Am I doing anything wrong?, servers' names are something like i-a3c832c7. As a side not, I should add that replacing the send() on the socket with socat echo disable server www/i-9bb943ff | socat stdio unix-connect:/tmp/haproxy does not change the behavior of the system. Thanks, Michele def change_state(self, servers_list, enable=True): try: s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) s.connect(self.socket_path) # interactive mode (see documentation, sec. 9.2) s.send('prompt\n') self.wait(s) # waits for the prompt for i in servers_list: command = None if enable == True: command = 'enable server www/%s' % i else: command = 'disable server www/%s' % i log.info(command) s.send('%s\n' % command) self.wait(s) s.shutdown(socket.SHUT_RDWR) s.close() except SocketError, e: log.error('socket error: %s' % e) self.__go = False
Re: problems with enabling/disabling servers via socket
Hello Cyril, no, those settings should be fine. Here is my configuration file. global stats socket/tmp/haproxy mode 666 level admin stats timeout 1h pidfile /var/run/haproxy.pid daemon spread-checks 5 defaults log 127.0.0.1 local2 info modehttp log global option tcp-smart-accept option tcp-smart-connect retries 3 maxconn 1 contimeout 4s clitimeout 10s srvtimeout 10s timeout server 10s frontend http-in bind *:80 default_backend www backend www option httplog stats enable mode http grace 5000 stats uri /admin?stats stats refresh 5s stats auth admin:admin option forceclose balance leastconn option httpchk # sends an http request to monitor the servers option redispatch . servers here Michele On Mar 2, 2012, at 7:25 PM, Cyril Bonté wrote: Hi Michele, Le 02/03/2012 17:26, Michele Mazzucco a écrit : Hello, I am trying to enable/disable servers at runtime by means of the UNIX socket created by HAProxy. In order to do that, I am using the Python script below. I'm testing the code with 2 servers, and I have noticed that quite often only one of the 2 servers is enabled/disable (sometimes the first, others the second). However sometimes none of them changes state, of both of them do. I have tried to open the socket in non-interactive mode as well (i.e., re-open the socket at every command), but I didn't notice any difference. Am I doing anything wrong?, servers' names are something like i-a3c832c7. As a side not, I should add that replacing the send() on the socket with socat Maybe you didn't set your socket in admin level ? global stats socket ... level admin echo disable server www/i-9bb943ff | socat stdio unix-connect:/tmp/haproxy does not change the behavior of the system. Thanks, Michele def change_state(self, servers_list, enable=True): try: s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) s.connect(self.socket_path) # interactive mode (see documentation, sec. 9.2) s.send('prompt\n') self.wait(s) # waits for the prompt for i in servers_list: command = None if enable == True: command = 'enable server www/%s' % i else: command = 'disable server www/%s' % i log.info(command) s.send('%s\n' % command) self.wait(s) s.shutdown(socket.SHUT_RDWR) s.close() except SocketError, e: log.error('socket error: %s' % e) self.__go = False -- Cyril Bonté
question about reloading haproxy
Hello, I am trying to control a haproxy instance running as a daemon via python: my script monitors the status of haproxy (e.g., number of queued requests), adds/remove some apache servers and updates the configuration of haproxy. Currently my script runs as a root. Would it be possible to restart haproxy (e.g. by issuing the command /etc/init.d/haproxy restart) with lower privileges? The problem seems to be that I cannot re-bind port 80, unless the script runs with root privileges. Thanks, Michele
Re: question about reloading haproxy
Hello Brane, yes, but that is equivalent to running the script as root... the effective user ID is 0. Cheers, Michele On Feb 10, 2012, at 11:52 AM, Brane F. Gračnar wrote: On 02/10/2012 10:35 AM, Michele Mazzucco wrote: Currently my script runs as a root. Would it be possible to restart haproxy (e.g. by issuing the command /etc/init.d/haproxy restart) with lower privileges? The problem seems to be that I cannot re-bind port 80, unless the script runs with root privileges. sudo(8) is your friend. Brane
Re: HAProxy in Amazon EC2 in peer mode
You can also write a script that gather instances' data (using the EC2 APIs) in order to get the IPs. That way you can update the HAProxy configuration files when a change in your setup occurs. Cheers, Michele On Feb 8, 2012, at 5:33 PM, Mir Islam wrote: Right, the internal IP changes since it is on DHCP. But I was thinking of assigning static IP (EIP in Amazon term) to the HAProxies. And then refer to each other via those EIP. Here is a quick and dirty diagram of howI am thinking of implementing. Ofcourse the main cost is the static IP addresses that I have to get from Amazon.
problem with logging on Mac OS X
Hello everybody, I am trying to configure haproxy to run on Mac OS X, but I am unable to write logs using syslog. I have modified the syslog configuration according to http://superuser.com/questions/131578/how-do-you-enable-syslogd-to-accept-incoming-connections-on-snow-leopard-from-re and http://hints.macworld.com/article.php?story=20110724103552640 in order to accept incoming network connections, and I have restarted the syslog daemon, howver I have only succeeded in printing lines such as Dec 14 16:47:36 Albook sudo[6245]: michele : TTY=ttys001 ; PWD=/Users/michele/devel/haproxy-1.4.18 ; USER=root ; COMMAND=./haproxy -f config_dec_14.cfg -d when haproxy starts. My configuration, after stripping many options (stats, timeouts, etc.), is the following: global log 172.17.36.220 local2 defaults modehttp log global option httplog listen http-in option httplog option forceclose bind *:80 server server1 127.0.0.1:8002 maxconn 32 My /etc/syslog.conf contains the following line local2.*/var/log/haproxy.log Is my configuration file wrong or the problem is elsewhere? Thanks, Michele