Re: [openstack-dev] [Fuel][MySQL][DLM][Oslo][DB][Trove][Galera][operators] Multi-master writes look OK, OCF RA and more things
Hi Roman. That's interesting, although’s hard to believe (there is no slave lag in galera multi master). I can only suggest us to create another jepsen test to verify exactly scenario you describe. As well as other OpenStack specific patterns. Regards, Bogdan. Od: Roman Podoliaka Wysłano: piątek, 29 kwietnia 2016 21:04 Do: OpenStack Development Mailing List (not for usage questions) DW: openstack-operat...@lists.openstack.org Hi Bogdan, Thank you for sharing this! I'll need to familiarize myself with this Jepsen thing, but overall it looks interesting. As it turns out, we already run Galera in multi-writer mode in Fuel unintentionally in the case, when the active MySQL node goes down, HAProxy starts opening connections to a backup, then the active goes up again, HAProxy starts opening connections to the original MySQL node, but OpenStack services may still have connections opened to the backup in their connection pools - so now you may have connections to multiple MySQL nodes at the same time, exactly what you wanted to avoid by using active/backup in the HAProxy configuration. ^ this actually leads to an interesting issue [1], when the DB state committed on one node is not immediately available on another one. Replication lag can be controlled via session variables [2], but that does not always help: e.g. in [1] Nova first goes to Neutron to create a new floating IP, gets 201 (and Neutron actually *commits* the DB transaction) and then makes another REST API request to get a list of floating IPs by address - the latter can be served by another neutron-server, connected to another Galera node, which does not have the latest state applied yet due to 'slave lag' - it can happen that the list will be empty. Unfortunately, 'wsrep_sync_wait' can't help here, as it's two different REST API requests, potentially served by two different neutron-server instances. Basically, you'd need to *always* wait for the latest state to be applied before executing any queries, which Galera is trying to avoid for performance reasons. Thanks, Roman [1] https://bugs.launchpad.net/fuel/+bug/1529937 [2] http://galeracluster.com/2015/06/achieving-read-after-write-semantics-with-galera/ On Fri, Apr 22, 2016 at 10:42 AM, Bogdan Dobrelya wrote: > [crossposting to openstack-operat...@lists.openstack.org] > > Hello. > I wrote this paper [0] to demonstrate an approach how we can leverage a > Jepsen framework for QA/CI/CD pipeline for OpenStack projects like Oslo > (DB) or Trove, Tooz DLM and perhaps for any integration projects which > rely on distributed systems. Although all tests are yet to be finished, > results are quite visible, so I better off share early for a review, > discussion and comments. > > I have similar tests done for the RabbitMQ OCF RA clusterers as well, > although have yet wrote a report. > > PS. I'm sorry for so many tags I placed in the topic header, should I've > used just "all" :) ? Have a nice weekends and take care! > > [0] https://goo.gl/VHyIIE > > -- > Best regards, > Bogdan Dobrelya, > Irc #bogdando > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [HA][RabbitMQ][messaging][Pacemaker][operators] Improved OCF resource agent for dynamic active-active mirrored clustering
Thank you Andrew. Answers below. >>> Sounds interesting, can you give any comment about how it differs to the other[i] upstream agent? Am I right that this one is effectively A/P and wont function without some kind of shared storage? Any particular reason you went down this path instead of full A/A? [i] https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/rabbitmq-cluster <<< It is based on multistate clone notifications. It requries nothing shared but Corosync info base CIB where all Pacemaker resources stored anyway. And it is fully A/A. All running rabbit nodes may process AMQP connections. Master state is only for a cluster initial point at wich other slaves may join to it. Note, here you can find events flow charts as well [0] [0] https://www.rabbitmq.com/pacemaker.html Regards, Bogdan__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel][puppet] CI gate for regressions detection in deployment data
> Why do you use [puppet] tag? > Is there anything related to Puppet OpenStack modules we should take > care of? There are several deployment automation tools leveraging the puppet-openstack modules. Those are under the Big Tent and each has its own composition layer and deployment patterns with specific data flows. That is why I opened this discussion public and put the puppet tag as well. I believe the idea of automated checks of the data layer applies to them as well and I’d like to have opinions on that from a broad audience. What is the correct tag for deploymnent automation with a Puppet? -- Best regards, Bogdan Dobrelya, Irc #bogdando__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone][Fernet] HA SQL backend for Fernet keys
I suggest to use pacemaker multistate clone resource to rotate and rsync fernet tokens from local directories across cluster nodes. The resource prototype is described here https://etherpad.openstack.org/p/fernet_tokens_pacemaker Pros: Pacemaker will care about CAP/split-brain stuff for us, we just design rotate and rsync logic. Also no shared FS/DB involved but only Corosync CIB - to store few internal resource state related params, not tokens. Cons: Keystone nodes hosting fernet tokens directories must be members of pacemaker cluster. Also custom OCF script should be created to implement this. __ Regards, Bogdan Dobrelya. IRC: bogdando Matt Fischer also discusses key rotation here: http://www.mattfischer.com/blog/?p=648 And here: http://www.mattfischer.com/blog/?p=665 On Mon, Jul 27, 2015 at 2:30 PM, Dolph Mathews wrote: …__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] 5.1 branch created
Please also note, that all open patches created *before* the stable/5.1 branch should now be reconsidered as targeted to 6.0 release instead of 5.1 as they initially were upon the submission. Therefore, all backporting stuff needed should be not forgotten and related bugs should be updated as appropriate. E.g. bug https://bugs.launchpad.net/fuel/+bug/1366712 initially was nominated for 5.1 milestone only, but after the stable/5.1 branch creation, it was updated in order to reflect all its new backporting status as well. Also note, if there were no more patches but one 5.1 targeted (i.e. master branch, before stable/5.1 creation), related bugs should be updated as 6.0 in progress and 5.1 as not started(confirmed) or triaged - in order to not forgot to backport it later once merged in master branch. Please let’s not forget an important backports. Best regards, Bogdan Dobrelya. From: Aleksandra Fedorova Sent: Wednesday, September 10, 2014 2:45 PM To: openstack-dev@lists.openstack.org, fuel-core-t...@mirantis.com Hi, everyone, as a follow-up for discussion [1] we created stable/5.1 branch for all Fuel projects. Since now on all patches targeted for upcoming 5.1 release should go to master branch first and then should be cherry-picked to stable/5.1. CI Infrastructure branching is in progress. With any branching-related issues please contact devops team at #fuel-devops IRC channel. [1] https://www.mail-archive.com/openstack-dev%40lists.openstack.org/msg34607.html -- Aleksandra Fedorova bookwar___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
