Re: [openstack-dev] [Fuel] Rabbitmq 3.4.0 upgrade for the 6.1 release, is it worth it?

2015-04-29 Thread Jay Pipes 

Agree with Vova and Tomasz. It's too late and risky for 6.1, in my opinion.

Best,
-jay

On 04/29/2015 07:55 AM, Vladimir Kuklin wrote:

I am 100% against the upgrade, folks. We need to ensure that user can
use different network for GRE segmentation for high-load cases and
mention this in Installation and Operations Guides - there is no time
for it.

On Wed, Apr 29, 2015 at 2:33 PM, Tomasz Napierala
mailto:tnapier...@mirantis.com>> wrote:

I’m -1 for it.

Considering how much time we needed to troubleshoot the problems
already, I don’t think we have time to properly test the upgrade.


 > On 29 Apr 2015, at 12:37, Sergii Golovatiuk
mailto:sgolovat...@mirantis.com>> wrote:
 >
 > -1 for upgrading it in 6.1. Known devil is better than unknown
angel :)
 >
 > In 7.0 we can try 3.5.0 with updated Erlang.
 >
 > ~thanks
 >
 >
 > --
 > Best regards,
 > Sergii Golovatiuk,
 > Skype #golserge
 > IRC #holser
 >
 > On Wed, Apr 29, 2015 at 12:20 PM, Davanum Srinivas
mailto:dsrini...@mirantis.com>> wrote:
 > Bogdan,
 >
 > Pacemaker, corosync etc, we picked vivid packages right? So don't
we test what's in vivid for this too? Apparently it's 3.4.3-2 per [0].
 >
 > I agree, we should not do this in 6.1, However, we should start
testing this ASAP.
 >
 > Another data point, Alexander Nevenchannyy pointed out to me that
3.5.0 came with an updated Erlang that has the following fix:
 > OTP-11497  To prevent a race condition if there is a short
communication
 >
 > problem when node-down and node-up events are received. They
 >
 > are now stored and later checked if the node came up just
 >
 > before mnesia flagged the node as down. (Thanks to Jonas Falkevik )
 >
 > which seems interesting as well.
 >
 > thanks,
 >
 > dims
 >
 > [0] https://launchpad.net/ubuntu/vivid/+package/rabbitmq-server
 > [1] http://www.erlang.org/download/otp_src_17.0.readme
 >
 > On Wed, Apr 29, 2015 at 4:04 AM, Bogdan Dobrelya
mailto:bdobre...@mirantis.com>> wrote:
 > Hello.
 >
 > There are several concerns why we have to upgrade RabbitMQ to
3.4.0 [0]:
 > 1) At least two bugfixes related to the current high-load issue
with MQ [1]:
 > - 26404 prevent queue synchronisation from hanging if there is a very
 > short partition just as it starts (since 3.1.0)
 > - 26368 prevent autoheal from hanging when loser shuts down
before the
 > winner  learns it is the winner (since 3.1.0)
 > 2) We should as well check how the new 'pause-if-all-down' option
works
 > for split brain recovery.
 > 3) We should address the 'force_boot' recommendations from this mail
 > thread [2] to speed up the MQ cluster assemble time.
 >
 > The question is - is it worth it to do this in the 6.1 release scope?
 > I vote to postpone this for the 7.0 dev cycle as the impact of such
 > changes might be unpredictable.
 >
 > [0] https://www.rabbitmq.com/release-notes/README-3.4.0.txt
 > [1] https://bugs.launchpad.net/fuel/+bug/1447619
 > [2]
 >
http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg51625.html
 >
 > --
 > Best regards,
 > Bogdan Dobrelya,
 > Skype #bogdando_at_yahoo.com 
 > Irc #bogdando
 >
 >

--
Tomasz 'Zen' Napierala
Product Engineering - Poland









--
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
35bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com 
www.mirantis.ru 
vkuk...@mirantis.com 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Rabbitmq 3.4.0 upgrade for the 6.1 release, is it worth it?

2015-04-29 Thread Vladimir Kuklin 
I am 100% against the upgrade, folks. We need to ensure that user can use
different network for GRE segmentation for high-load cases and mention this
in Installation and Operations Guides - there is no time for it.

On Wed, Apr 29, 2015 at 2:33 PM, Tomasz Napierala 
wrote:

> I’m -1 for it.
>
> Considering how much time we needed to troubleshoot the problems already,
> I don’t think we have time to properly test the upgrade.
>
>
> > On 29 Apr 2015, at 12:37, Sergii Golovatiuk 
> wrote:
> >
> > -1 for upgrading it in 6.1. Known devil is better than unknown angel :)
> >
> > In 7.0 we can try 3.5.0 with updated Erlang.
> >
> > ~thanks
> >
> >
> > --
> > Best regards,
> > Sergii Golovatiuk,
> > Skype #golserge
> > IRC #holser
> >
> > On Wed, Apr 29, 2015 at 12:20 PM, Davanum Srinivas <
> dsrini...@mirantis.com> wrote:
> > Bogdan,
> >
> > Pacemaker, corosync etc, we picked vivid packages right? So don't we
> test what's in vivid for this too? Apparently it's 3.4.3-2 per [0].
> >
> > I agree, we should not do this in 6.1, However, we should start testing
> this ASAP.
> >
> > Another data point, Alexander Nevenchannyy pointed out to me that 3.5.0
> came with an updated Erlang that has the following fix:
> > OTP-11497  To prevent a race condition if there is a short communication
> >
> > problem when node-down and node-up events are received. They
> >
> > are now stored and later checked if the node came up just
> >
> > before mnesia flagged the node as down. (Thanks to Jonas Falkevik )
> >
> > which seems interesting as well.
> >
> > thanks,
> >
> > dims
> >
> > [0] https://launchpad.net/ubuntu/vivid/+package/rabbitmq-server
> > [1] http://www.erlang.org/download/otp_src_17.0.readme
> >
> > On Wed, Apr 29, 2015 at 4:04 AM, Bogdan Dobrelya 
> wrote:
> > Hello.
> >
> > There are several concerns why we have to upgrade RabbitMQ to 3.4.0 [0]:
> > 1) At least two bugfixes related to the current high-load issue with MQ
> [1]:
> > - 26404 prevent queue synchronisation from hanging if there is a very
> > short partition just as it starts (since 3.1.0)
> > - 26368 prevent autoheal from hanging when loser shuts down before the
> > winner  learns it is the winner (since 3.1.0)
> > 2) We should as well check how the new 'pause-if-all-down' option works
> > for split brain recovery.
> > 3) We should address the 'force_boot' recommendations from this mail
> > thread [2] to speed up the MQ cluster assemble time.
> >
> > The question is - is it worth it to do this in the 6.1 release scope?
> > I vote to postpone this for the 7.0 dev cycle as the impact of such
> > changes might be unpredictable.
> >
> > [0] https://www.rabbitmq.com/release-notes/README-3.4.0.txt
> > [1] https://bugs.launchpad.net/fuel/+bug/1447619
> > [2]
> >
> http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg51625.html
> >
> > --
> > Best regards,
> > Bogdan Dobrelya,
> > Skype #bogdando_at_yahoo.com
> > Irc #bogdando
> >
> >
>
> --
> Tomasz 'Zen' Napierala
> Product Engineering - Poland
>
>
>
>
>
>
>


-- 
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
35bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com 
www.mirantis.ru
vkuk...@mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Rabbitmq 3.4.0 upgrade for the 6.1 release, is it worth it?

2015-04-29 Thread Tomasz Napierala 
I’m -1 for it.

Considering how much time we needed to troubleshoot the problems already, I 
don’t think we have time to properly test the upgrade.


> On 29 Apr 2015, at 12:37, Sergii Golovatiuk  wrote:
> 
> -1 for upgrading it in 6.1. Known devil is better than unknown angel :)
> 
> In 7.0 we can try 3.5.0 with updated Erlang.
> 
> ~thanks
> 
> 
> --
> Best regards,
> Sergii Golovatiuk,
> Skype #golserge
> IRC #holser
> 
> On Wed, Apr 29, 2015 at 12:20 PM, Davanum Srinivas  
> wrote:
> Bogdan,
> 
> Pacemaker, corosync etc, we picked vivid packages right? So don't we test 
> what's in vivid for this too? Apparently it's 3.4.3-2 per [0].
> 
> I agree, we should not do this in 6.1, However, we should start testing this 
> ASAP.
> 
> Another data point, Alexander Nevenchannyy pointed out to me that 3.5.0 came 
> with an updated Erlang that has the following fix:
> OTP-11497  To prevent a race condition if there is a short communication
> 
> problem when node-down and node-up events are received. They
> 
> are now stored and later checked if the node came up just
> 
> before mnesia flagged the node as down. (Thanks to Jonas Falkevik )
> 
> which seems interesting as well.
> 
> thanks,
> 
> dims
> 
> [0] https://launchpad.net/ubuntu/vivid/+package/rabbitmq-server
> [1] http://www.erlang.org/download/otp_src_17.0.readme
> 
> On Wed, Apr 29, 2015 at 4:04 AM, Bogdan Dobrelya  
> wrote:
> Hello.
> 
> There are several concerns why we have to upgrade RabbitMQ to 3.4.0 [0]:
> 1) At least two bugfixes related to the current high-load issue with MQ [1]:
> - 26404 prevent queue synchronisation from hanging if there is a very
> short partition just as it starts (since 3.1.0)
> - 26368 prevent autoheal from hanging when loser shuts down before the
> winner  learns it is the winner (since 3.1.0)
> 2) We should as well check how the new 'pause-if-all-down' option works
> for split brain recovery.
> 3) We should address the 'force_boot' recommendations from this mail
> thread [2] to speed up the MQ cluster assemble time.
> 
> The question is - is it worth it to do this in the 6.1 release scope?
> I vote to postpone this for the 7.0 dev cycle as the impact of such
> changes might be unpredictable.
> 
> [0] https://www.rabbitmq.com/release-notes/README-3.4.0.txt
> [1] https://bugs.launchpad.net/fuel/+bug/1447619
> [2]
> http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg51625.html
> 
> --
> Best regards,
> Bogdan Dobrelya,
> Skype #bogdando_at_yahoo.com
> Irc #bogdando
> 
> 

-- 
Tomasz 'Zen' Napierala
Product Engineering - Poland







__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Rabbitmq 3.4.0 upgrade for the 6.1 release, is it worth it?

2015-04-29 Thread Sergii Golovatiuk 
-1 for upgrading it in 6.1. Known devil is better than unknown angel :)

In 7.0 we can try 3.5.0 with updated Erlang.

~thanks


--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Wed, Apr 29, 2015 at 12:20 PM, Davanum Srinivas 
wrote:

> Bogdan,
>
> Pacemaker, corosync etc, we picked vivid packages right? So don't we test
> what's in vivid for this too? Apparently it's 3.4.3-2 per [0].
>
> I agree, we should not do this in 6.1, However, we should start testing
> this ASAP.
>
> Another data point, Alexander Nevenchannyy pointed out to me that 3.5.0
> came with an updated Erlang that has the following fix:
>
> OTP-11497  To prevent a race condition if there is a short communication
>
> problem when node-down and node-up events are received. They
>
> are now stored and later checked if the node came up just
>
> before mnesia flagged the node as down. (Thanks to Jonas Falkevik )
>
> which seems interesting as well.
>
> thanks,
>
> dims
> [0] https://launchpad.net/ubuntu/vivid/+package/rabbitmq-server
> [1] http://www.erlang.org/download/otp_src_17.0.readme
>
> On Wed, Apr 29, 2015 at 4:04 AM, Bogdan Dobrelya 
> wrote:
>
>> Hello.
>>
>> There are several concerns why we have to upgrade RabbitMQ to 3.4.0 [0]:
>> 1) At least two bugfixes related to the current high-load issue with MQ
>> [1]:
>> - 26404 prevent queue synchronisation from hanging if there is a very
>> short partition just as it starts (since 3.1.0)
>> - 26368 prevent autoheal from hanging when loser shuts down before the
>> winner  learns it is the winner (since 3.1.0)
>> 2) We should as well check how the new 'pause-if-all-down' option works
>> for split brain recovery.
>> 3) We should address the 'force_boot' recommendations from this mail
>> thread [2] to speed up the MQ cluster assemble time.
>>
>> The question is - is it worth it to do this in the 6.1 release scope?
>> I vote to postpone this for the 7.0 dev cycle as the impact of such
>> changes might be unpredictable.
>>
>> [0] https://www.rabbitmq.com/release-notes/README-3.4.0.txt
>> [1] https://bugs.launchpad.net/fuel/+bug/1447619
>> [2]
>>
>> http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg51625.html
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Skype #bogdando_at_yahoo.com
>> Irc #bogdando
>>
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Rabbitmq 3.4.0 upgrade for the 6.1 release, is it worth it?

2015-04-29 Thread Dina Belova 
Alexei,

actually we do not insist this should b done for the MOS 6.1. That was the
question to the audience if someone is having other idea. All these
discussions have roots in the bug
https://bugs.launchpad.net/fuel/+bug/1447619 - we have found the issue with
RabbitMQ cluster behaviour under the networking load and Bogdan, Alexei
Khivin and Alex Nevenchannyy found that it possibly might be fixed
upgrading the RabbitMQ (trying it now). And actually this RabbitMQ release
contains lots of crucial bug fixes even without mentioned by Bogdan.

For 6.1 the found workaround might be used, section in the docs written,
etc. The question to the company is if that will be enough and what are we
going to do with it in future.

Cheers,
Dina

On Wed, Apr 29, 2015 at 11:24 AM, Alexei Sheplyakov <
asheplya...@mirantis.com> wrote:

> Hi,
>
> Given that
>
> - MOS 6.1 should be released in a few weeks
> - rabbitmq is kind of a heart of OpenStack
>
> upgrading rabbitmq in MOS 6.1 seems to be an extremely bad idea.
>
> There will be always some bugs (both known and unknown), but we can't keep
> updating various
> components forever and should stop at some moment (which is presumably
> called `soft code freeze').
>
> Best regards,
>   Alexei
>
>
> On Wed, Apr 29, 2015 at 11:04 AM, Bogdan Dobrelya 
> wrote:
>
>> Hello.
>>
>> There are several concerns why we have to upgrade RabbitMQ to 3.4.0 [0]:
>> 1) At least two bugfixes related to the current high-load issue with MQ
>> [1]:
>> - 26404 prevent queue synchronisation from hanging if there is a very
>> short partition just as it starts (since 3.1.0)
>> - 26368 prevent autoheal from hanging when loser shuts down before the
>> winner  learns it is the winner (since 3.1.0)
>> 2) We should as well check how the new 'pause-if-all-down' option works
>> for split brain recovery.
>> 3) We should address the 'force_boot' recommendations from this mail
>> thread [2] to speed up the MQ cluster assemble time.
>>
>> The question is - is it worth it to do this in the 6.1 release scope?
>> I vote to postpone this for the 7.0 dev cycle as the impact of such
>> changes might be unpredictable.
>>
>> [0] https://www.rabbitmq.com/release-notes/README-3.4.0.txt
>> [1] https://bugs.launchpad.net/fuel/+bug/1447619
>> [2]
>>
>> http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg51625.html
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Skype #bogdando_at_yahoo.com
>> Irc #bogdando
>>
>
>


-- 

Best regards,

Dina Belova

Software Engineer

Mirantis Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Rabbitmq 3.4.0 upgrade for the 6.1 release, is it worth it?

2015-04-29 Thread Bogdan Dobrelya 
Hello.

There are several concerns why we have to upgrade RabbitMQ to 3.4.0 [0]:
1) At least two bugfixes related to the current high-load issue with MQ [1]:
- 26404 prevent queue synchronisation from hanging if there is a very
short partition just as it starts (since 3.1.0)
- 26368 prevent autoheal from hanging when loser shuts down before the
winner  learns it is the winner (since 3.1.0)
2) We should as well check how the new 'pause-if-all-down' option works
for split brain recovery.
3) We should address the 'force_boot' recommendations from this mail
thread [2] to speed up the MQ cluster assemble time.

The question is - is it worth it to do this in the 6.1 release scope?
I vote to postpone this for the 7.0 dev cycle as the impact of such
changes might be unpredictable.

[0] https://www.rabbitmq.com/release-notes/README-3.4.0.txt
[1] https://bugs.launchpad.net/fuel/+bug/1447619
[2]
http://www.mail-archive.com/openstack-dev@lists.openstack.org/msg51625.html

-- 
Best regards,
Bogdan Dobrelya,
Skype #bogdando_at_yahoo.com
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev