Re: [openstack-dev] [Openstack-operators] [Fuel][Oslo][RabbitMQ][Shovel] Deprecate mirrored queues from HA AMQP cluster scenario
On 8 June 2015 at 15:10:15, Davanum Srinivas (dava...@gmail.com) wrote: > I'd like to bring out a poll about deprecating the RabbitMQ mirrored > queues for HA layout and replacing the AMQP clustering by shovel > [0], > [1]. I guess the federation would not be a good option, but let's > consider it as well. RabbitMQ team member here. Neither Shovel nor Federation will replace mirroring. Shovel moves messages from a queue to an exchange (within a single node or between remote nodes and/or clusters). It doesn't replicate anything. Federation has two parts to it: * Queue federation: no replicate, distributes messages from a single logical queue between N nodes or clusters, when there are no local consumers to consume them. * Exchange federation replicates a stream of messages going through an exchange. As messages are consumed upstream, downstream has no way of knowing about it. > Why this must be done? The answer is that the rabbit cluster cannot > detect and survive "micro outages" well and just ending up with > some > queues stuck and as a result, the rabbitmqctl control plane hanged > completely unresponsive (until the rabbit node erased and recovered > its > cluster membership). These outages could be caused either by > the network > *or* by CPU load spikes. For example, like this bug in Fuel project > [2] > and this mail thread [3]. The right thing to do here is introduce timeouts to rabbitmqctl, which was 99% finished in the past but some RabbitMQ team members felt it should produce more detailed error messages, which extended the scope of the change significantly. > This seems rather the Erlang's > Mnesia generic clustering issue, than something what could be just fixed > in RabbitMQ, unless the mnesia based clustering would be dropped > completely ;) While Mnesia indeed needs to be replaced to introduce AP (as in CAP) style mirroring, the issue you're bringing up here has nothing to do with Mnesia. Mnesia is not used by rabbitmqctl, and it is not used to store messages. It's a rabbitmqctl issue, and potentially a hint that you may want to reduce net_ticktime value (say, to 5-10 seconds) to make queue master unavailability detected faster. 1. http://www.rabbitmq.com/nettick.html -- MK Staff Software Engineer, Pivotal/RabbitMQ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [Fuel][Oslo][RabbitMQ][Shovel] Deprecate mirrored queues from HA AMQP cluster scenario
Bogdan, I'd also like to ask: c) Does anyone have any experience with shovel in a "realistic" openstack environment? (or even a devstack one) Thanks, dims On Mon, Jun 8, 2015 at 6:24 AM, Bogdan Dobrelya wrote: > Hello, stackers. > > I'd like to bring out a poll about deprecating the RabbitMQ mirrored > queues for HA layout and replacing the AMQP clustering by shovel [0], > [1]. I guess the federation would not be a good option, but let's > consider it as well. > > Why this must be done? The answer is that the rabbit cluster cannot > detect and survive "micro outages" well and just ending up with some > queues stuck and as a result, the rabbitmqctl control plane hanged > completely unresponsive (until the rabbit node erased and recovered its > cluster membership). These outages could be caused either by the network > *or* by CPU load spikes. For example, like this bug in Fuel project [2] > and this mail thread [3]. > > So, let's please vote and discuss. > > But the questions also are: > a) Would be there changes in Oslo.messaging required as well in order to > support the underlying AMQP layer architecture changes? > b) Are there any volunteers for this research to be done for the > Oslo.messaging AMQP rabbit driver? > > PS. Note, I'm not bringing RabbitMQ versions here as the issue seems > unresolved for any of existing ones. This seems rather the Erlang's > Mnesia generic clustering issue, than something what could be just fixed > in RabbitMQ, unless the mnesia based clustering would be dropped > completely ;) > > [0] https://www.rabbitmq.com/shovel-dynamic.html > [1] https://www.rabbitmq.com/shovel.html > [2] https://bugs.launchpad.net/fuel/+bug/1460762 > [3] https://groups.google.com/forum/#!topic/rabbitmq-users/iZWokxvhlaU > > -- > Best regards, > Bogdan Dobrelya, > Skype #bogdando_at_yahoo.com > Irc #bogdando > > ___ > OpenStack-operators mailing list > openstack-operat...@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
