Re: Database High Availability
Sure, thanks a lot!
On Tue, May 3, 2022 at 12:53 PM Ivan Kudryavtsev wrote:
> Well,
>
> You would better consult with real-life mysql experts, as for me, I
> referred to great severalnines.com articles like:
>
> https://severalnines.com/resources/database-management-tutorials/galera-cluster-mysql-tutorial
>
> https://severalnines.com/database-blog/avoiding-deadlocks-galera-setting-haproxy-single-node-writes-and-multi-node-reads
>
> Just take a look for best practices there.
>
>
>
> On Tue, May 3, 2022 at 10:18 AM Jayanth Reddy
> wrote:
>
> > Hi,
> >
> > Thanks again for the tips! Below is the current configuration, please
> > suggest changes if any.
> >
> > HAProxy
> >
> > frontend galera-fe
> > mode tcp
> > bind 10.231.4.112:3306
> > use_backend galera-be
> >
> > backend galera-be
> > balance source
> > mode tcp
> > option tcpka
> > option mysql-check user haproxy
> > server galera-0 10.231.4.36:3306 check
> > server galera-1 10.231.4.37:3306 check
> > server galera-2 10.231.4.38:3306 check
> >
> > Keepalived
> >
> > vrrp_script check_backend {
> > script "killall -0 haproxy"
> > weight -20
> > interval 2
> > rise 2
> > fall 2
> > }
> >
> > vrrp_instance DB_0 {
> > state MASTER # BACKUP on others
> > priority 100
> > interface enp1s0
> > virtual_router_id 50
> > advert_int 1
> > unicast_peer {
> > 10.231.4.87 # Relevant on others
> > 10.231.4.88 # Relevant on others
> > }
> > virtual_ipaddress {
> > 10.231.4.112/24
> > }
> > track_script {
> > check_backend
> > }
> > }
> >
> > Best Regards,
> > Jayanth
> >
> > On Tue, May 3, 2022 at 12:33 PM Ivan Kudryavtsev wrote:
> >
> > > Sounds cool,
> > >
> > > Just ensure that in any failure case (db, haproxy, OS or hardware
> crash)
> > > all the Management servers are switched to the same Galera instance,
> > > otherwise, this could lead to operational problems.
> > > Also, backups are still mandatory, recommend doing them from one of
> > > Galera's hot-swap nodes, not from the main operational node.
> > >
> > > Best wishes, Ivan
> > >
> > > On Tue, May 3, 2022 at 9:53 AM Jayanth Reddy <
> jayanthreddy5...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Thank you. Have set up MariaDB Galera Cluster with the required
> > > HAProxy
> > > > configuration with MYSQL health checks. Everything is working fine.
> > > >
> > > > On Mon, May 2, 2022 at 10:48 AM Ivan Kudryavtsev
> > wrote:
> > > >
> > > > > Hi, I use MariaDB Galera cluster.
> > > > >
> > > > > But you have to pin all the CS management to the same galera node
> to
> > > make
> > > > > cloudstack transactioned operations work correctly. HAproxy or
> shared
> > > > > common ip solve that.
> > > > >
> > > > > пн, 2 мая 2022 г., 7:34 AM Jayanth Reddy <
> jayanthreddy5...@gmail.com
> > >:
> > > > >
> > > > > > Hello guys,
> > > > > >
> > > > > > How are you doing database High Availability? Any inputs on
> DB
> > > > > > Clustering and CloudStack configuration would really help me.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Database High Availability
Well,
You would better consult with real-life mysql experts, as for me, I
referred to great severalnines.com articles like:
https://severalnines.com/resources/database-management-tutorials/galera-cluster-mysql-tutorial
https://severalnines.com/database-blog/avoiding-deadlocks-galera-setting-haproxy-single-node-writes-and-multi-node-reads
Just take a look for best practices there.
On Tue, May 3, 2022 at 10:18 AM Jayanth Reddy
wrote:
> Hi,
>
> Thanks again for the tips! Below is the current configuration, please
> suggest changes if any.
>
> HAProxy
>
> frontend galera-fe
> mode tcp
> bind 10.231.4.112:3306
> use_backend galera-be
>
> backend galera-be
> balance source
> mode tcp
> option tcpka
> option mysql-check user haproxy
> server galera-0 10.231.4.36:3306 check
> server galera-1 10.231.4.37:3306 check
> server galera-2 10.231.4.38:3306 check
>
> Keepalived
>
> vrrp_script check_backend {
> script "killall -0 haproxy"
> weight -20
> interval 2
> rise 2
> fall 2
> }
>
> vrrp_instance DB_0 {
> state MASTER # BACKUP on others
> priority 100
> interface enp1s0
> virtual_router_id 50
> advert_int 1
> unicast_peer {
> 10.231.4.87 # Relevant on others
> 10.231.4.88 # Relevant on others
> }
> virtual_ipaddress {
> 10.231.4.112/24
> }
> track_script {
> check_backend
> }
> }
>
> Best Regards,
> Jayanth
>
> On Tue, May 3, 2022 at 12:33 PM Ivan Kudryavtsev wrote:
>
> > Sounds cool,
> >
> > Just ensure that in any failure case (db, haproxy, OS or hardware crash)
> > all the Management servers are switched to the same Galera instance,
> > otherwise, this could lead to operational problems.
> > Also, backups are still mandatory, recommend doing them from one of
> > Galera's hot-swap nodes, not from the main operational node.
> >
> > Best wishes, Ivan
> >
> > On Tue, May 3, 2022 at 9:53 AM Jayanth Reddy >
> > wrote:
> >
> > > Hi,
> > >
> > > Thank you. Have set up MariaDB Galera Cluster with the required
> > HAProxy
> > > configuration with MYSQL health checks. Everything is working fine.
> > >
> > > On Mon, May 2, 2022 at 10:48 AM Ivan Kudryavtsev
> wrote:
> > >
> > > > Hi, I use MariaDB Galera cluster.
> > > >
> > > > But you have to pin all the CS management to the same galera node to
> > make
> > > > cloudstack transactioned operations work correctly. HAproxy or shared
> > > > common ip solve that.
> > > >
> > > > пн, 2 мая 2022 г., 7:34 AM Jayanth Reddy >:
> > > >
> > > > > Hello guys,
> > > > >
> > > > > How are you doing database High Availability? Any inputs on DB
> > > > > Clustering and CloudStack configuration would really help me.
> > > > >
> > > >
> > >
> >
>
Re: Database High Availability
Hi,
Thanks again for the tips! Below is the current configuration, please
suggest changes if any.
HAProxy
frontend galera-fe
mode tcp
bind 10.231.4.112:3306
use_backend galera-be
backend galera-be
balance source
mode tcp
option tcpka
option mysql-check user haproxy
server galera-0 10.231.4.36:3306 check
server galera-1 10.231.4.37:3306 check
server galera-2 10.231.4.38:3306 check
Keepalived
vrrp_script check_backend {
script "killall -0 haproxy"
weight -20
interval 2
rise 2
fall 2
}
vrrp_instance DB_0 {
state MASTER # BACKUP on others
priority 100
interface enp1s0
virtual_router_id 50
advert_int 1
unicast_peer {
10.231.4.87 # Relevant on others
10.231.4.88 # Relevant on others
}
virtual_ipaddress {
10.231.4.112/24
}
track_script {
check_backend
}
}
Best Regards,
Jayanth
On Tue, May 3, 2022 at 12:33 PM Ivan Kudryavtsev wrote:
> Sounds cool,
>
> Just ensure that in any failure case (db, haproxy, OS or hardware crash)
> all the Management servers are switched to the same Galera instance,
> otherwise, this could lead to operational problems.
> Also, backups are still mandatory, recommend doing them from one of
> Galera's hot-swap nodes, not from the main operational node.
>
> Best wishes, Ivan
>
> On Tue, May 3, 2022 at 9:53 AM Jayanth Reddy
> wrote:
>
> > Hi,
> >
> > Thank you. Have set up MariaDB Galera Cluster with the required
> HAProxy
> > configuration with MYSQL health checks. Everything is working fine.
> >
> > On Mon, May 2, 2022 at 10:48 AM Ivan Kudryavtsev wrote:
> >
> > > Hi, I use MariaDB Galera cluster.
> > >
> > > But you have to pin all the CS management to the same galera node to
> make
> > > cloudstack transactioned operations work correctly. HAproxy or shared
> > > common ip solve that.
> > >
> > > пн, 2 мая 2022 г., 7:34 AM Jayanth Reddy :
> > >
> > > > Hello guys,
> > > >
> > > > How are you doing database High Availability? Any inputs on DB
> > > > Clustering and CloudStack configuration would really help me.
> > > >
> > >
> >
>
Re: Database High Availability
Sounds cool, Just ensure that in any failure case (db, haproxy, OS or hardware crash) all the Management servers are switched to the same Galera instance, otherwise, this could lead to operational problems. Also, backups are still mandatory, recommend doing them from one of Galera's hot-swap nodes, not from the main operational node. Best wishes, Ivan On Tue, May 3, 2022 at 9:53 AM Jayanth Reddy wrote: > Hi, > > Thank you. Have set up MariaDB Galera Cluster with the required HAProxy > configuration with MYSQL health checks. Everything is working fine. > > On Mon, May 2, 2022 at 10:48 AM Ivan Kudryavtsev wrote: > > > Hi, I use MariaDB Galera cluster. > > > > But you have to pin all the CS management to the same galera node to make > > cloudstack transactioned operations work correctly. HAproxy or shared > > common ip solve that. > > > > пн, 2 мая 2022 г., 7:34 AM Jayanth Reddy : > > > > > Hello guys, > > > > > > How are you doing database High Availability? Any inputs on DB > > > Clustering and CloudStack configuration would really help me. > > > > > >
Re: Database High Availability
Hi, Thank you. Have set up MariaDB Galera Cluster with the required HAProxy configuration with MYSQL health checks. Everything is working fine. On Mon, May 2, 2022 at 10:48 AM Ivan Kudryavtsev wrote: > Hi, I use MariaDB Galera cluster. > > But you have to pin all the CS management to the same galera node to make > cloudstack transactioned operations work correctly. HAproxy or shared > common ip solve that. > > пн, 2 мая 2022 г., 7:34 AM Jayanth Reddy : > > > Hello guys, > > > > How are you doing database High Availability? Any inputs on DB > > Clustering and CloudStack configuration would really help me. > > >
Re: Database High Availability
Hi, I use MariaDB Galera cluster. But you have to pin all the CS management to the same galera node to make cloudstack transactioned operations work correctly. HAproxy or shared common ip solve that. пн, 2 мая 2022 г., 7:34 AM Jayanth Reddy : > Hello guys, > > How are you doing database High Availability? Any inputs on DB > Clustering and CloudStack configuration would really help me. >
Database High Availability
Hello guys, How are you doing database High Availability? Any inputs on DB Clustering and CloudStack configuration would really help me.
Re: database high availability question vs haproxy
Hi Simon, thanks for the link - actually I have already read this - but Im still seaking for some answeres :) : - real world experience with DB HA in general - is i better to use haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to reference 2 nodes (1 as master, another as slave) with native ACS DB HA - silly question but anyway... - my understanding - ACS just pings and connects to master or slave (all replication etc, is done from my side, not from ACS) ? Thanks again and any info is greatly appreciated. Andrija On 4 June 2015 at 16:23, Simon Weller swel...@ena.com wrote: Andrija, Here is the original design document, and it should give you a better idea of what is implemented today: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207 We have plans to test this in our lab soon, but just haven't got around to it yet. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Thursday, June 4, 2015 9:08 AM To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly, it is up to the admin to provide appropriate mysql HA (active-active, galera, etc) and ACS management server will JUST try to connect to slaves if the master is down ? We are running Galera, with haproxy/keepalived, and by using stoping haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and failover IP to another host. During these 6 seconds, ACS managemnt server goes dead, because of this DB unavailability. So my wondering, is better to use ACS db HA feature, instead of loadbalancer for this specific purpose ? (we are also using haproxy/keepalived for management server loadbalancing - 2 servers in backend...) Any experience shared is really appreciated ! -- Andrija Panić -- Andrija Panić -- Andrija Panić
Re: database high availability question vs haproxy
Thanks Simon - that is what I thought... So my question would be then, haproxy vs native ACS/mysql connector going to galera1/galera2/etc...will figure out, for now we use haproxy for mysql/galera loadbalancing... THanks a lot Simon, Andrija On 5 June 2015 at 15:18, Simon Weller swel...@ena.com wrote: Personally, I think that Gallera is always going to be a safer option, as it handles conflict resolution natively. Having said that, it appears care has been taken in designing the ACS MGMT DB integration so that the chance of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot of hardware unless you've got plans to use it elsewhere in your organisation. The downside to Galera, is that it's synchronous replication, so it needs very low latency between nodes. That doesn't make it a good candidate for geographic separation between DB nodes for a DR scenario. You're understanding of the replication structure, as based on the design document is correct. MySQL (or Galera) handles all the replication. ACS just handles which node it's writing and reading from. In a 2 node native MySQL cluster, it's expected that you are setup for cross master-master replication. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 2:41 AM To: d...@cloudstack.apache.org Cc: users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Hi Simon, thanks for the link - actually I have already read this - but Im still seaking for some answeres :) : - real world experience with DB HA in general - is i better to use haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to reference 2 nodes (1 as master, another as slave) with native ACS DB HA - silly question but anyway... - my understanding - ACS just pings and connects to master or slave (all replication etc, is done from my side, not from ACS) ? Thanks again and any info is greatly appreciated. Andrija On 4 June 2015 at 16:23, Simon Weller swel...@ena.com wrote: Andrija, Here is the original design document, and it should give you a better idea of what is implemented today: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207 We have plans to test this in our lab soon, but just haven't got around to it yet. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Thursday, June 4, 2015 9:08 AM To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly, it is up to the admin to provide appropriate mysql HA (active-active, galera, etc) and ACS management server will JUST try to connect to slaves if the master is down ? We are running Galera, with haproxy/keepalived, and by using stoping haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and failover IP to another host. During these 6 seconds, ACS managemnt server goes dead, because of this DB unavailability. So my wondering, is better to use ACS db HA feature, instead of loadbalancer for this specific purpose ? (we are also using haproxy/keepalived for management server loadbalancing - 2 servers in backend...) Any experience shared is really appreciated ! -- Andrija Panić -- Andrija Panić -- Andrija Panić -- Andrija Panić
Re: database high availability question vs haproxy
Personally, I think that Gallera is always going to be a safer option, as it handles conflict resolution natively. Having said that, it appears care has been taken in designing the ACS MGMT DB integration so that the chance of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot of hardware unless you've got plans to use it elsewhere in your organisation. The downside to Galera, is that it's synchronous replication, so it needs very low latency between nodes. That doesn't make it a good candidate for geographic separation between DB nodes for a DR scenario. You're understanding of the replication structure, as based on the design document is correct. MySQL (or Galera) handles all the replication. ACS just handles which node it's writing and reading from. In a 2 node native MySQL cluster, it's expected that you are setup for cross master-master replication. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 2:41 AM To: d...@cloudstack.apache.org Cc: users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Hi Simon, thanks for the link - actually I have already read this - but Im still seaking for some answeres :) : - real world experience with DB HA in general - is i better to use haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to reference 2 nodes (1 as master, another as slave) with native ACS DB HA - silly question but anyway... - my understanding - ACS just pings and connects to master or slave (all replication etc, is done from my side, not from ACS) ? Thanks again and any info is greatly appreciated. Andrija On 4 June 2015 at 16:23, Simon Weller swel...@ena.com wrote: Andrija, Here is the original design document, and it should give you a better idea of what is implemented today: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207 We have plans to test this in our lab soon, but just haven't got around to it yet. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Thursday, June 4, 2015 9:08 AM To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly, it is up to the admin to provide appropriate mysql HA (active-active, galera, etc) and ACS management server will JUST try to connect to slaves if the master is down ? We are running Galera, with haproxy/keepalived, and by using stoping haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and failover IP to another host. During these 6 seconds, ACS managemnt server goes dead, because of this DB unavailability. So my wondering, is better to use ACS db HA feature, instead of loadbalancer for this specific purpose ? (we are also using haproxy/keepalived for management server loadbalancing - 2 servers in backend...) Any experience shared is really appreciated ! -- Andrija Panić -- Andrija Panić -- Andrija Panić
Re: database high availability question vs haproxy
Exactly...thanks Simon for the time and help :) Is there any mysql timeout/retry parameter that is set in db.properties (like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to detect haproxy is down (while testing I shutdown haproxu on active node), and then keepalvied moves IP to another node - meaning I have 5-6sec of no connection between ACS mgmt servers and database/haproxy. Is this timeout configurabile on ACS/mysql connector side ? Thanks again, Andrija On 5 June 2015 at 15:54, Simon Weller swel...@ena.com wrote: I think haproxy is a better design in my opinion. You're going to have to use haproxy to balance the host agents to multiple management servers anyway, so you'll already be using it. haproxy can then manage the health checks to Galera rather than over complicating it with the CS Management configuration. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 8:37 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Thanks Simon - that is what I thought... So my question would be then, haproxy vs native ACS/mysql connector going to galera1/galera2/etc...will figure out, for now we use haproxy for mysql/galera loadbalancing... THanks a lot Simon, Andrija On 5 June 2015 at 15:18, Simon Weller swel...@ena.com wrote: Personally, I think that Gallera is always going to be a safer option, as it handles conflict resolution natively. Having said that, it appears care has been taken in designing the ACS MGMT DB integration so that the chance of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot of hardware unless you've got plans to use it elsewhere in your organisation. The downside to Galera, is that it's synchronous replication, so it needs very low latency between nodes. That doesn't make it a good candidate for geographic separation between DB nodes for a DR scenario. You're understanding of the replication structure, as based on the design document is correct. MySQL (or Galera) handles all the replication. ACS just handles which node it's writing and reading from. In a 2 node native MySQL cluster, it's expected that you are setup for cross master-master replication. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 2:41 AM To: d...@cloudstack.apache.org Cc: users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Hi Simon, thanks for the link - actually I have already read this - but Im still seaking for some answeres :) : - real world experience with DB HA in general - is i better to use haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to reference 2 nodes (1 as master, another as slave) with native ACS DB HA - silly question but anyway... - my understanding - ACS just pings and connects to master or slave (all replication etc, is done from my side, not from ACS) ? Thanks again and any info is greatly appreciated. Andrija On 4 June 2015 at 16:23, Simon Weller swel...@ena.com wrote: Andrija, Here is the original design document, and it should give you a better idea of what is implemented today: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207 We have plans to test this in our lab soon, but just haven't got around to it yet. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Thursday, June 4, 2015 9:08 AM To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly, it is up to the admin to provide appropriate mysql HA (active-active, galera, etc) and ACS management server will JUST try to connect to slaves if the master is down ? We are running Galera, with haproxy/keepalived, and by using stoping haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and failover IP to another host. During these 6 seconds, ACS managemnt server goes dead, because of this DB unavailability. So my wondering, is better to use ACS db HA feature, instead of loadbalancer for this specific purpose ? (we are also using haproxy/keepalived for management server loadbalancing - 2 servers in backend...) Any experience shared
Re: database high availability question vs haproxy
I'm not sure whether that's configurable and I haven't seen any examples out there that point to it being configurable. Having said that, there are people a lot more familiar with that than I am on the list. One option you do have for this particular failure situation, is to use monit to monitor the pid of the CS management service and if it's not running, force it to start up again. Losing a load balancer completely is very rare, so it's not likely to happen often at all. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 9:07 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Exactly...thanks Simon for the time and help :) Is there any mysql timeout/retry parameter that is set in db.properties (like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to detect haproxy is down (while testing I shutdown haproxu on active node), and then keepalvied moves IP to another node - meaning I have 5-6sec of no connection between ACS mgmt servers and database/haproxy. Is this timeout configurabile on ACS/mysql connector side ? Thanks again, Andrija On 5 June 2015 at 15:54, Simon Weller swel...@ena.com wrote: I think haproxy is a better design in my opinion. You're going to have to use haproxy to balance the host agents to multiple management servers anyway, so you'll already be using it. haproxy can then manage the health checks to Galera rather than over complicating it with the CS Management configuration. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 8:37 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Thanks Simon - that is what I thought... So my question would be then, haproxy vs native ACS/mysql connector going to galera1/galera2/etc...will figure out, for now we use haproxy for mysql/galera loadbalancing... THanks a lot Simon, Andrija On 5 June 2015 at 15:18, Simon Weller swel...@ena.com wrote: Personally, I think that Gallera is always going to be a safer option, as it handles conflict resolution natively. Having said that, it appears care has been taken in designing the ACS MGMT DB integration so that the chance of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot of hardware unless you've got plans to use it elsewhere in your organisation. The downside to Galera, is that it's synchronous replication, so it needs very low latency between nodes. That doesn't make it a good candidate for geographic separation between DB nodes for a DR scenario. You're understanding of the replication structure, as based on the design document is correct. MySQL (or Galera) handles all the replication. ACS just handles which node it's writing and reading from. In a 2 node native MySQL cluster, it's expected that you are setup for cross master-master replication. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 2:41 AM To: d...@cloudstack.apache.org Cc: users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Hi Simon, thanks for the link - actually I have already read this - but Im still seaking for some answeres :) : - real world experience with DB HA in general - is i better to use haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to reference 2 nodes (1 as master, another as slave) with native ACS DB HA - silly question but anyway... - my understanding - ACS just pings and connects to master or slave (all replication etc, is done from my side, not from ACS) ? Thanks again and any info is greatly appreciated. Andrija On 4 June 2015 at 16:23, Simon Weller swel...@ena.com wrote: Andrija, Here is the original design document, and it should give you a better idea of what is implemented today: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207 We have plans to test this in our lab soon, but just haven't got around to it yet. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Thursday, June 4, 2015 9:08 AM To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly
Re: database high availability question vs haproxy
I think haproxy is a better design in my opinion. You're going to have to use haproxy to balance the host agents to multiple management servers anyway, so you'll already be using it. haproxy can then manage the health checks to Galera rather than over complicating it with the CS Management configuration. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 8:37 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Thanks Simon - that is what I thought... So my question would be then, haproxy vs native ACS/mysql connector going to galera1/galera2/etc...will figure out, for now we use haproxy for mysql/galera loadbalancing... THanks a lot Simon, Andrija On 5 June 2015 at 15:18, Simon Weller swel...@ena.com wrote: Personally, I think that Gallera is always going to be a safer option, as it handles conflict resolution natively. Having said that, it appears care has been taken in designing the ACS MGMT DB integration so that the chance of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot of hardware unless you've got plans to use it elsewhere in your organisation. The downside to Galera, is that it's synchronous replication, so it needs very low latency between nodes. That doesn't make it a good candidate for geographic separation between DB nodes for a DR scenario. You're understanding of the replication structure, as based on the design document is correct. MySQL (or Galera) handles all the replication. ACS just handles which node it's writing and reading from. In a 2 node native MySQL cluster, it's expected that you are setup for cross master-master replication. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 2:41 AM To: d...@cloudstack.apache.org Cc: users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Hi Simon, thanks for the link - actually I have already read this - but Im still seaking for some answeres :) : - real world experience with DB HA in general - is i better to use haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to reference 2 nodes (1 as master, another as slave) with native ACS DB HA - silly question but anyway... - my understanding - ACS just pings and connects to master or slave (all replication etc, is done from my side, not from ACS) ? Thanks again and any info is greatly appreciated. Andrija On 4 June 2015 at 16:23, Simon Weller swel...@ena.com wrote: Andrija, Here is the original design document, and it should give you a better idea of what is implemented today: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207 We have plans to test this in our lab soon, but just haven't got around to it yet. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Thursday, June 4, 2015 9:08 AM To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly, it is up to the admin to provide appropriate mysql HA (active-active, galera, etc) and ACS management server will JUST try to connect to slaves if the master is down ? We are running Galera, with haproxy/keepalived, and by using stoping haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and failover IP to another host. During these 6 seconds, ACS managemnt server goes dead, because of this DB unavailability. So my wondering, is better to use ACS db HA feature, instead of loadbalancer for this specific purpose ? (we are also using haproxy/keepalived for management server loadbalancing - 2 servers in backend...) Any experience shared is really appreciated ! -- Andrija Panić -- Andrija Panić -- Andrija Panić -- Andrija Panić
Re: database high availability question vs haproxy
True..Thanks Simon... On 5 June 2015 at 16:28, Simon Weller swel...@ena.com wrote: I'm not sure whether that's configurable and I haven't seen any examples out there that point to it being configurable. Having said that, there are people a lot more familiar with that than I am on the list. One option you do have for this particular failure situation, is to use monit to monitor the pid of the CS management service and if it's not running, force it to start up again. Losing a load balancer completely is very rare, so it's not likely to happen often at all. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 9:07 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Exactly...thanks Simon for the time and help :) Is there any mysql timeout/retry parameter that is set in db.properties (like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to detect haproxy is down (while testing I shutdown haproxu on active node), and then keepalvied moves IP to another node - meaning I have 5-6sec of no connection between ACS mgmt servers and database/haproxy. Is this timeout configurabile on ACS/mysql connector side ? Thanks again, Andrija On 5 June 2015 at 15:54, Simon Weller swel...@ena.com wrote: I think haproxy is a better design in my opinion. You're going to have to use haproxy to balance the host agents to multiple management servers anyway, so you'll already be using it. haproxy can then manage the health checks to Galera rather than over complicating it with the CS Management configuration. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 8:37 AM To: users@cloudstack.apache.org Cc: d...@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Thanks Simon - that is what I thought... So my question would be then, haproxy vs native ACS/mysql connector going to galera1/galera2/etc...will figure out, for now we use haproxy for mysql/galera loadbalancing... THanks a lot Simon, Andrija On 5 June 2015 at 15:18, Simon Weller swel...@ena.com wrote: Personally, I think that Gallera is always going to be a safer option, as it handles conflict resolution natively. Having said that, it appears care has been taken in designing the ACS MGMT DB integration so that the chance of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot of hardware unless you've got plans to use it elsewhere in your organisation. The downside to Galera, is that it's synchronous replication, so it needs very low latency between nodes. That doesn't make it a good candidate for geographic separation between DB nodes for a DR scenario. You're understanding of the replication structure, as based on the design document is correct. MySQL (or Galera) handles all the replication. ACS just handles which node it's writing and reading from. In a 2 node native MySQL cluster, it's expected that you are setup for cross master-master replication. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Friday, June 5, 2015 2:41 AM To: d...@cloudstack.apache.org Cc: users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Hi Simon, thanks for the link - actually I have already read this - but Im still seaking for some answeres :) : - real world experience with DB HA in general - is i better to use haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to reference 2 nodes (1 as master, another as slave) with native ACS DB HA - silly question but anyway... - my understanding - ACS just pings and connects to master or slave (all replication etc, is done from my side, not from ACS) ? Thanks again and any info is greatly appreciated. Andrija On 4 June 2015 at 16:23, Simon Weller swel...@ena.com wrote: Andrija, Here is the original design document, and it should give you a better idea of what is implemented today: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207 We have plans to test this in our lab soon, but just haven't got around to it yet. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Thursday, June 4, 2015 9:08 AM To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would
Re: database high availability question vs haproxy
Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly, it is up to the admin to provide appropriate mysql HA (active-active, galera, etc) and ACS management server will JUST try to connect to slaves if the master is down ? We are running Galera, with haproxy/keepalived, and by using stoping haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and failover IP to another host. During these 6 seconds, ACS managemnt server goes dead, because of this DB unavailability. So my wondering, is better to use ACS db HA feature, instead of loadbalancer for this specific purpose ? (we are also using haproxy/keepalived for management server loadbalancing - 2 servers in backend...) Any experience shared is really appreciated ! -- Andrija Panić -- Andrija Panić
Re: database high availability question vs haproxy
Andrija, Here is the original design document, and it should give you a better idea of what is implemented today: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207 We have plans to test this in our lab soon, but just haven't got around to it yet. - Si From: Andrija Panic andrija.pa...@gmail.com Sent: Thursday, June 4, 2015 9:08 AM To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Re: database high availability question vs haproxy Anyone :) ? On 31 May 2015 at 00:26, Andrija Panic andrija.pa...@gmail.com wrote: Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly, it is up to the admin to provide appropriate mysql HA (active-active, galera, etc) and ACS management server will JUST try to connect to slaves if the master is down ? We are running Galera, with haproxy/keepalived, and by using stoping haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and failover IP to another host. During these 6 seconds, ACS managemnt server goes dead, because of this DB unavailability. So my wondering, is better to use ACS db HA feature, instead of loadbalancer for this specific purpose ? (we are also using haproxy/keepalived for management server loadbalancing - 2 servers in backend...) Any experience shared is really appreciated ! -- Andrija Panić -- Andrija Panić
database high availability question vs haproxy
Hi, I would have a question on database HA feature in db.properties ( http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability ) If I understand correctly, it is up to the admin to provide appropriate mysql HA (active-active, galera, etc) and ACS management server will JUST try to connect to slaves if the master is down ? We are running Galera, with haproxy/keepalived, and by using stoping haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and failover IP to another host. During these 6 seconds, ACS managemnt server goes dead, because of this DB unavailability. So my wondering, is better to use ACS db HA feature, instead of loadbalancer for this specific purpose ? (we are also using haproxy/keepalived for management server loadbalancing - 2 servers in backend...) Any experience shared is really appreciated ! -- Andrija Panić
