Re: regular disconnection every 2-3 mins on haproxy cloudstack mgmt server LB implementation
You're welkome! Great to hear it now works :-) Sent from my iPhone > On 30 Apr 2016, at 23:13, Indra Pramana wrote: > > Hi Remi, > > Many thanks for your tips! You are right indeed, by default haproxy has > timeout set to 50 seconds (or 50,000ms). I changed it to 120 seconds (2 > minutes) and have monitored the agent connection and it's no longer > flapping. Monitoring now and will let you know if there are any further > issues. > > On /etc/haproxy/haproxy.cfg, modify this under defaults section: > >timeout client 12 >timeout server 12 >#timeout client 5 >#timeout server 5 > > and restart the haproxy service for the changes to take effect. > > Again, many thanks Remi, greatly appreciated. :) > > -ip- > > > On Sun, May 1, 2016 at 4:30 AM, Remi Bergsma > wrote: > >> Hi, >> >> You may want to check if HAproxy keeps the connection open long enough. >> The system vms send traffic every minute or so and you want haproxy to keep >> the connection open longer than that. If haproxy is set to close idle >> connections after say 30sec then this will result in flapping connections. >> >> Regards, Remi >> >> Sent from my iPhone >> >>> On 30 Apr 2016, at 22:00, Indra Pramana wrote: >>> >>> Dear all, >>> >>> We are running CloudStack 4.2.0 with KVM hypervisor and Ceph RBD storage. >>> We implemented secondary CloudStack management server and haproxy load >>> balancer, and tonight we changed our configuration so that the CloudStack >>> agents will be connecting to the LB IP rather than the CS mgmt server >>> directly. >>> >>> However, we noted that the agent will be regularly disconnected every 2-3 >>> minutes. Here are the excerpts on the agent.log: >>> >>> >>> 2016-05-01 01:30:10,982 DEBUG [utils.nio.NioConnection] >>> (Agent-Selector:null) Location 1: Socket >>> Socket[addr=/X.X.X.8,port=8250,localport=50613] closed on read. Pro >>> bably -1 returned: Connection closed with -1 on reading size. >>> 2016-05-01 01:30:10,983 DEBUG [utils.nio.NioConnection] >>> (Agent-Selector:null) Closing socket >>> Socket[addr=/X.X.X.8,port=8250,localport=50613] >>> 2016-05-01 01:30:10,983 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) >>> Clearing watch list: 2 >>> 2016-05-01 01:30:15,984 INFO [cloud.agent.Agent] (Agent-Handler-3:null) >>> Lost connection to the server. Dealing with the remaining commands... >>> 2016-05-01 01:30:20,985 INFO [cloud.agent.Agent] (Agent-Handler-3:null) >>> Reconnecting... >>> 2016-05-01 01:30:20,986 INFO [utils.nio.NioClient] (Agent-Selector:null) >>> Connecting to X.X.X.8:8250 >>> 2016-05-01 01:30:21,101 INFO [utils.nio.NioClient] (Agent-Selector:null) >>> SSL: Handshake done >>> 2016-05-01 01:30:21,101 INFO [utils.nio.NioClient] (Agent-Selector:null) >>> Connected to X.X.X.8:8250 >>> 2016-05-01 01:30:21,133 DEBUG [kvm.resource.LibvirtCapXMLParser] >>> (Agent-Handler-1:null) Found /usr/bin/kvm as a suiteable emulator >>> 2016-05-01 01:30:21,134 DEBUG [kvm.resource.LibvirtComputingResource] >>> (Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep >> convert >>> 2016-05-01 01:30:21,152 DEBUG [kvm.resource.LibvirtComputingResource] >>> (Agent-Handler-1:null) Execution is successful. >>> 2016-05-01 01:30:21,152 DEBUG [kvm.resource.LibvirtComputingResource] >>> (Agent-Handler-1:null) convert [-c] [-p] [-q] [-n] [-f fmt] [-t cache] >>> [-T src_cache] [-O output >>> _fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S >>> sparse_size] filename [filename2 [...]] output_filename >>> options are: 'none', 'writeback' (default, except for convert), >>> 'writethrough', >>> 'directsync' and 'unsafe' (default for convert) >>> >>> >>> and then it will reconnected again, and then got disconnected again, etc. >>> It will continue on that loop. >>> >>> haproxy.cfg configuration for the NIC facing the hypervisors: >>> >>> >>> listen cloudstack_systemvm_8250 >>> bind X.X.X.8:8250 >>> mode tcp >>> option tcplog >>> balance source >>> server management-server-01.xxx.com X.X.X.3:8250 maxconn 32 check >>> server management-server-02.xxx.com X.X.X.6:8250 maxconn 32 check >>> >>> >>> Note that .3 and .6 are the first and second CloudStack management >> servers >>> respectively, while .8 is the IP of the load balancer. We are using one >> LB >>> at the moment. >>> >>> Nothing much is found on the haproxy.log, see below. >>> >>> >>> May 1 01:14:41 cs-haproxy-02 haproxy[923]: X.X.X.28:50401 >>> [01/May/2016:01:12:50.803] cloudstack_systemvm_8250 >>> cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/110340 8584 cD >>> 0/0/0/0/0 0/0 >>> May 1 01:15:46 cs-haproxy-02 haproxy[923]: X.X.X.28:50402 >>> [01/May/2016:01:14:51.150] cloudstack_systemvm_8250 >>> cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/54920 8234 cD >>> 0/0/0/0/0 0/0 >>> May 1 01:16:47 cs-haproxy-02 haproxy[923]: X.X.X.28:50403 >>> [01/May/2016:01:15:56.075] cloudstack_systemvm_8250 >>> cloudstack_systemv
Re: regular disconnection every 2-3 mins on haproxy cloudstack mgmt server LB implementation
Hi Remi, Many thanks for your tips! You are right indeed, by default haproxy has timeout set to 50 seconds (or 50,000ms). I changed it to 120 seconds (2 minutes) and have monitored the agent connection and it's no longer flapping. Monitoring now and will let you know if there are any further issues. On /etc/haproxy/haproxy.cfg, modify this under defaults section: timeout client 12 timeout server 12 #timeout client 5 #timeout server 5 and restart the haproxy service for the changes to take effect. Again, many thanks Remi, greatly appreciated. :) -ip- On Sun, May 1, 2016 at 4:30 AM, Remi Bergsma wrote: > Hi, > > You may want to check if HAproxy keeps the connection open long enough. > The system vms send traffic every minute or so and you want haproxy to keep > the connection open longer than that. If haproxy is set to close idle > connections after say 30sec then this will result in flapping connections. > > Regards, Remi > > Sent from my iPhone > > > On 30 Apr 2016, at 22:00, Indra Pramana wrote: > > > > Dear all, > > > > We are running CloudStack 4.2.0 with KVM hypervisor and Ceph RBD storage. > > We implemented secondary CloudStack management server and haproxy load > > balancer, and tonight we changed our configuration so that the CloudStack > > agents will be connecting to the LB IP rather than the CS mgmt server > > directly. > > > > However, we noted that the agent will be regularly disconnected every 2-3 > > minutes. Here are the excerpts on the agent.log: > > > > > > 2016-05-01 01:30:10,982 DEBUG [utils.nio.NioConnection] > > (Agent-Selector:null) Location 1: Socket > > Socket[addr=/X.X.X.8,port=8250,localport=50613] closed on read. Pro > > bably -1 returned: Connection closed with -1 on reading size. > > 2016-05-01 01:30:10,983 DEBUG [utils.nio.NioConnection] > > (Agent-Selector:null) Closing socket > > Socket[addr=/X.X.X.8,port=8250,localport=50613] > > 2016-05-01 01:30:10,983 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) > > Clearing watch list: 2 > > 2016-05-01 01:30:15,984 INFO [cloud.agent.Agent] (Agent-Handler-3:null) > > Lost connection to the server. Dealing with the remaining commands... > > 2016-05-01 01:30:20,985 INFO [cloud.agent.Agent] (Agent-Handler-3:null) > > Reconnecting... > > 2016-05-01 01:30:20,986 INFO [utils.nio.NioClient] (Agent-Selector:null) > > Connecting to X.X.X.8:8250 > > 2016-05-01 01:30:21,101 INFO [utils.nio.NioClient] (Agent-Selector:null) > > SSL: Handshake done > > 2016-05-01 01:30:21,101 INFO [utils.nio.NioClient] (Agent-Selector:null) > > Connected to X.X.X.8:8250 > > 2016-05-01 01:30:21,133 DEBUG [kvm.resource.LibvirtCapXMLParser] > > (Agent-Handler-1:null) Found /usr/bin/kvm as a suiteable emulator > > 2016-05-01 01:30:21,134 DEBUG [kvm.resource.LibvirtComputingResource] > > (Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep > convert > > 2016-05-01 01:30:21,152 DEBUG [kvm.resource.LibvirtComputingResource] > > (Agent-Handler-1:null) Execution is successful. > > 2016-05-01 01:30:21,152 DEBUG [kvm.resource.LibvirtComputingResource] > > (Agent-Handler-1:null) convert [-c] [-p] [-q] [-n] [-f fmt] [-t cache] > > [-T src_cache] [-O output > > _fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S > > sparse_size] filename [filename2 [...]] output_filename > >options are: 'none', 'writeback' (default, except for convert), > > 'writethrough', > >'directsync' and 'unsafe' (default for convert) > > > > > > and then it will reconnected again, and then got disconnected again, etc. > > It will continue on that loop. > > > > haproxy.cfg configuration for the NIC facing the hypervisors: > > > > > > listen cloudstack_systemvm_8250 > > bind X.X.X.8:8250 > > mode tcp > > option tcplog > > balance source > > server management-server-01.xxx.com X.X.X.3:8250 maxconn 32 check > > server management-server-02.xxx.com X.X.X.6:8250 maxconn 32 check > > > > > > Note that .3 and .6 are the first and second CloudStack management > servers > > respectively, while .8 is the IP of the load balancer. We are using one > LB > > at the moment. > > > > Nothing much is found on the haproxy.log, see below. > > > > > > May 1 01:14:41 cs-haproxy-02 haproxy[923]: X.X.X.28:50401 > > [01/May/2016:01:12:50.803] cloudstack_systemvm_8250 > > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/110340 8584 cD > > 0/0/0/0/0 0/0 > > May 1 01:15:46 cs-haproxy-02 haproxy[923]: X.X.X.28:50402 > > [01/May/2016:01:14:51.150] cloudstack_systemvm_8250 > > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/54920 8234 cD > > 0/0/0/0/0 0/0 > > May 1 01:16:47 cs-haproxy-02 haproxy[923]: X.X.X.28:50403 > > [01/May/2016:01:15:56.075] cloudstack_systemvm_8250 > > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/51344 7868 cD > > 0/0/0/0/0 0/0 > > May 1 01:17:48 cs-haproxy-02 haproxy[923]: X.X.X.28:50404 > > [01/May/2016:01:16:57.426] cloudstack_systemvm_8250 > >
Re: regular disconnection every 2-3 mins on haproxy cloudstack mgmt server LB implementation
Hi, You may want to check if HAproxy keeps the connection open long enough. The system vms send traffic every minute or so and you want haproxy to keep the connection open longer than that. If haproxy is set to close idle connections after say 30sec then this will result in flapping connections. Regards, Remi Sent from my iPhone > On 30 Apr 2016, at 22:00, Indra Pramana wrote: > > Dear all, > > We are running CloudStack 4.2.0 with KVM hypervisor and Ceph RBD storage. > We implemented secondary CloudStack management server and haproxy load > balancer, and tonight we changed our configuration so that the CloudStack > agents will be connecting to the LB IP rather than the CS mgmt server > directly. > > However, we noted that the agent will be regularly disconnected every 2-3 > minutes. Here are the excerpts on the agent.log: > > > 2016-05-01 01:30:10,982 DEBUG [utils.nio.NioConnection] > (Agent-Selector:null) Location 1: Socket > Socket[addr=/X.X.X.8,port=8250,localport=50613] closed on read. Pro > bably -1 returned: Connection closed with -1 on reading size. > 2016-05-01 01:30:10,983 DEBUG [utils.nio.NioConnection] > (Agent-Selector:null) Closing socket > Socket[addr=/X.X.X.8,port=8250,localport=50613] > 2016-05-01 01:30:10,983 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) > Clearing watch list: 2 > 2016-05-01 01:30:15,984 INFO [cloud.agent.Agent] (Agent-Handler-3:null) > Lost connection to the server. Dealing with the remaining commands... > 2016-05-01 01:30:20,985 INFO [cloud.agent.Agent] (Agent-Handler-3:null) > Reconnecting... > 2016-05-01 01:30:20,986 INFO [utils.nio.NioClient] (Agent-Selector:null) > Connecting to X.X.X.8:8250 > 2016-05-01 01:30:21,101 INFO [utils.nio.NioClient] (Agent-Selector:null) > SSL: Handshake done > 2016-05-01 01:30:21,101 INFO [utils.nio.NioClient] (Agent-Selector:null) > Connected to X.X.X.8:8250 > 2016-05-01 01:30:21,133 DEBUG [kvm.resource.LibvirtCapXMLParser] > (Agent-Handler-1:null) Found /usr/bin/kvm as a suiteable emulator > 2016-05-01 01:30:21,134 DEBUG [kvm.resource.LibvirtComputingResource] > (Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep convert > 2016-05-01 01:30:21,152 DEBUG [kvm.resource.LibvirtComputingResource] > (Agent-Handler-1:null) Execution is successful. > 2016-05-01 01:30:21,152 DEBUG [kvm.resource.LibvirtComputingResource] > (Agent-Handler-1:null) convert [-c] [-p] [-q] [-n] [-f fmt] [-t cache] > [-T src_cache] [-O output > _fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S > sparse_size] filename [filename2 [...]] output_filename >options are: 'none', 'writeback' (default, except for convert), > 'writethrough', >'directsync' and 'unsafe' (default for convert) > > > and then it will reconnected again, and then got disconnected again, etc. > It will continue on that loop. > > haproxy.cfg configuration for the NIC facing the hypervisors: > > > listen cloudstack_systemvm_8250 > bind X.X.X.8:8250 > mode tcp > option tcplog > balance source > server management-server-01.xxx.com X.X.X.3:8250 maxconn 32 check > server management-server-02.xxx.com X.X.X.6:8250 maxconn 32 check > > > Note that .3 and .6 are the first and second CloudStack management servers > respectively, while .8 is the IP of the load balancer. We are using one LB > at the moment. > > Nothing much is found on the haproxy.log, see below. > > > May 1 01:14:41 cs-haproxy-02 haproxy[923]: X.X.X.28:50401 > [01/May/2016:01:12:50.803] cloudstack_systemvm_8250 > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/110340 8584 cD > 0/0/0/0/0 0/0 > May 1 01:15:46 cs-haproxy-02 haproxy[923]: X.X.X.28:50402 > [01/May/2016:01:14:51.150] cloudstack_systemvm_8250 > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/54920 8234 cD > 0/0/0/0/0 0/0 > May 1 01:16:47 cs-haproxy-02 haproxy[923]: X.X.X.28:50403 > [01/May/2016:01:15:56.075] cloudstack_systemvm_8250 > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/51344 7868 cD > 0/0/0/0/0 0/0 > May 1 01:17:48 cs-haproxy-02 haproxy[923]: X.X.X.28:50404 > [01/May/2016:01:16:57.426] cloudstack_systemvm_8250 > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/50854 7630 cD > 0/0/0/0/0 0/0 > May 1 01:18:49 cs-haproxy-02 haproxy[923]: X.X.X.28:50405 > [01/May/2016:01:17:58.285] cloudstack_systemvm_8250 > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/50955 7630 cD > 0/0/0/0/0 0/0 > May 1 01:24:49 cs-haproxy-02 haproxy[923]: X.X.X.28:50406 > [01/May/2016:01:18:59.245] cloudstack_systemvm_8250 > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/350361 14638 cD > 0/0/0/0/0 0/0 > May 1 01:28:00 cs-haproxy-02 haproxy[923]: X.X.X.28:50571 > [01/May/2016:01:27:09.638] cloudstack_systemvm_8250 > cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/50602 2852 cD > 0/0/0/0/0 0/0 > May 1 01:30:11 cs-haproxy-02 haproxy[923]: X.X.X.28:50613 > [01/May/2016:01:29:20.260] cloudstack_systemvm_8250 > cloudstack_systemvm_8250/man
regular disconnection every 2-3 mins on haproxy cloudstack mgmt server LB implementation
Dear all, We are running CloudStack 4.2.0 with KVM hypervisor and Ceph RBD storage. We implemented secondary CloudStack management server and haproxy load balancer, and tonight we changed our configuration so that the CloudStack agents will be connecting to the LB IP rather than the CS mgmt server directly. However, we noted that the agent will be regularly disconnected every 2-3 minutes. Here are the excerpts on the agent.log: 2016-05-01 01:30:10,982 DEBUG [utils.nio.NioConnection] (Agent-Selector:null) Location 1: Socket Socket[addr=/X.X.X.8,port=8250,localport=50613] closed on read. Pro bably -1 returned: Connection closed with -1 on reading size. 2016-05-01 01:30:10,983 DEBUG [utils.nio.NioConnection] (Agent-Selector:null) Closing socket Socket[addr=/X.X.X.8,port=8250,localport=50613] 2016-05-01 01:30:10,983 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) Clearing watch list: 2 2016-05-01 01:30:15,984 INFO [cloud.agent.Agent] (Agent-Handler-3:null) Lost connection to the server. Dealing with the remaining commands... 2016-05-01 01:30:20,985 INFO [cloud.agent.Agent] (Agent-Handler-3:null) Reconnecting... 2016-05-01 01:30:20,986 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to X.X.X.8:8250 2016-05-01 01:30:21,101 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done 2016-05-01 01:30:21,101 INFO [utils.nio.NioClient] (Agent-Selector:null) Connected to X.X.X.8:8250 2016-05-01 01:30:21,133 DEBUG [kvm.resource.LibvirtCapXMLParser] (Agent-Handler-1:null) Found /usr/bin/kvm as a suiteable emulator 2016-05-01 01:30:21,134 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Executing: /bin/bash -c qemu-img --help|grep convert 2016-05-01 01:30:21,152 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Execution is successful. 2016-05-01 01:30:21,152 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) convert [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output _fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] filename [filename2 [...]] output_filename options are: 'none', 'writeback' (default, except for convert), 'writethrough', 'directsync' and 'unsafe' (default for convert) and then it will reconnected again, and then got disconnected again, etc. It will continue on that loop. haproxy.cfg configuration for the NIC facing the hypervisors: listen cloudstack_systemvm_8250 bind X.X.X.8:8250 mode tcp option tcplog balance source server management-server-01.xxx.com X.X.X.3:8250 maxconn 32 check server management-server-02.xxx.com X.X.X.6:8250 maxconn 32 check Note that .3 and .6 are the first and second CloudStack management servers respectively, while .8 is the IP of the load balancer. We are using one LB at the moment. Nothing much is found on the haproxy.log, see below. May 1 01:14:41 cs-haproxy-02 haproxy[923]: X.X.X.28:50401 [01/May/2016:01:12:50.803] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/110340 8584 cD 0/0/0/0/0 0/0 May 1 01:15:46 cs-haproxy-02 haproxy[923]: X.X.X.28:50402 [01/May/2016:01:14:51.150] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/54920 8234 cD 0/0/0/0/0 0/0 May 1 01:16:47 cs-haproxy-02 haproxy[923]: X.X.X.28:50403 [01/May/2016:01:15:56.075] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/51344 7868 cD 0/0/0/0/0 0/0 May 1 01:17:48 cs-haproxy-02 haproxy[923]: X.X.X.28:50404 [01/May/2016:01:16:57.426] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/50854 7630 cD 0/0/0/0/0 0/0 May 1 01:18:49 cs-haproxy-02 haproxy[923]: X.X.X.28:50405 [01/May/2016:01:17:58.285] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/50955 7630 cD 0/0/0/0/0 0/0 May 1 01:24:49 cs-haproxy-02 haproxy[923]: X.X.X.28:50406 [01/May/2016:01:18:59.245] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/350361 14638 cD 0/0/0/0/0 0/0 May 1 01:28:00 cs-haproxy-02 haproxy[923]: X.X.X.28:50571 [01/May/2016:01:27:09.638] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/50602 2852 cD 0/0/0/0/0 0/0 May 1 01:30:11 cs-haproxy-02 haproxy[923]: X.X.X.28:50613 [01/May/2016:01:29:20.260] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/50876 7630 cD 0/0/0/0/0 0/0 May 1 01:32:11 cs-haproxy-02 haproxy[923]: X.X.X.28:50614 [01/May/2016:01:30:21.142] cloudstack_systemvm_8250 cloudstack_systemvm_8250/management-server-01.xxx.com 1/0/110308 8870 cD 0/0/0/0/0 0/0 Note that .28 is the IP address of the hypervisor I tested to connect to the LB IP. Did I missed out anything on the haproxy configuration? Any advice is greatly appreciated. Looking forward to your reply, thank you. Cheers.
