Re: [openstack-dev] Splitting notifications from rpc (and questions + work around this)

[Andy Smith]

On Fri, 2016-11-04 at 10:45 -0400, Ken Giusti wrote:
> On Wed, Nov 2, 2016 at 8:11 PM, Joshua Harlow 
> wrote:
> > 
> > Hi folks,
> > 
> > There was a bunch of chatter at the summit about how there are
> > really two
> > different types of (oslo) messaging usage that exist in openstack
> > and how
> > they need not be backed by the same solution type (rabbitmq, qpid,
> > kafka...).
> > 
> > For those that were not at the oslo sessions:
> > 
> > https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
> > 
> > The general gist was though that we need to make sure people really
> > do know
> > that there are two very different types of messaging usage in
> > openstack and
> > to ensure that operators (and developers) are picking the right
> > backing
> > technology for each type.
> > 
> > So some questions naturally arise out of this.
> > 
> > * Where are the best practices with regard to selection of the best
> > backend
> > type for rpc (and one for notifications); is this something
> > oslo.messaging
> > should work through (or can the docs team and operator group also
> > help in
> > making this)?
> > 
> > * What are the tradeoffs in using the same (or different)
> > technology for rpc
> > and notifications?
> > 
> I think the olso.messaging team should take the lead here and educate
> users as to what the options are, and  how the two supported
> messaging
> services (RPC and Notifications) differ with respect to backend
> requirements.   These topics really should be part of the
> oslo.messaging 'Theory of Operations' documentation that was
> discussed
> during the Arch WG summit meeting.
> 
> Currently the biggest functional difference between the backends is
> the support of store-and-forward (e.g. queueing) verses point-to-
> point
> message transfer.  Oslo could at least explain the pros and cons of
> each approach with respect to the RPC and Notification services so
> that folks understand what the tradeoffs and advantages are in the
> first place.
> 
> Furthermore the team should also document the functional differences
> between the various choices of backends.  For instance it would be
> useful to understand how the two supported point-to-point backends
> (zmq and dispatch router) differ in both behavior and recommended
> deployment.
> 
> 
> > 
> > * Is it even possible for all oslo.messaging consuming projects to
> > be able
> > to choose 2 different backends, are consuming projects consuming
> > the library
> > correctly so that they can use 2 different backends?
> > 
> > * Is devstack able to run with say kafka for notifications and
> > rabbitmq for
> > rpc (if not, is there any work item the oslo group can help with to
> > make
> > this possible) so that we can ensure and test that all projects can
> > work
> > correctly with appropriate (and possibly different) backends?
> > 

If the backend is considered a black box, the goal should be that from
the application's perspective, there is no functional difference
between the use of one backend over another. Rather, the differences
will be non-functional in nature (e.g. quality of service, scalability,
resilience, maintainability, ec.). This is what oslo.messaging will
need to help explain to users so that they will understand the
operational differences between deploying one backend versus another
(e.g. will they be managing a cluster of brokers or a mesh of routers).

The near term challenge will be to make it easy to configure the use of
alternative backends. The introduction of the transport_url
configuration option should help but most of the existing configuration
management frameworks continue to use the legacy (soon to be
deprecated) rpc_backend, host, port and auth configuration options. For
the most part, the configurations have "hard coded" the use of rabbit.

As an example of what will be required, an effort to update the
openstack puppet modules to introduce the transport_url configuration
options took place during Newton. This uses the the puppet-oslo
module that encapsulates the oslo.messaging configuration.

https://review.openstack.org/#/c/321818/

With this change across the core services, a packstack scenario was
prototyped that setup AMQP 1.0 backend for RPC and rabbit for
notifications. The resulting deployment was able to successfully pass
the  tempest smoke tests, etc.

A key enabler for the use of dual backends in practice will be to work
across the spectrum of installation/configuration management frameworks
to adopt the transport_url(s) options. This is a fair bit of scope but
once done, the oslo.messaging backend plug-in model can be fully
leveraged.

> > * Any other messaging, arch-wg work that we (oslo or others) can
> > help out
> > with to make sure that projects (and operators) are using the right
> > technology for the right use (and not just defaulting to RPC over
> > rabbitmq
> > because it exists, when in reality something else might be a better
> > choice)?
> > 
> 

Re: [openstack-dev] Splitting notifications from rpc (and questions + work around this)

[Ken Giusti]

On Wed, Nov 2, 2016 at 8:11 PM, Joshua Harlow  wrote:
> Hi folks,
>
> There was a bunch of chatter at the summit about how there are really two
> different types of (oslo) messaging usage that exist in openstack and how
> they need not be backed by the same solution type (rabbitmq, qpid,
> kafka...).
>
> For those that were not at the oslo sessions:
>
> https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
>
> The general gist was though that we need to make sure people really do know
> that there are two very different types of messaging usage in openstack and
> to ensure that operators (and developers) are picking the right backing
> technology for each type.
>
> So some questions naturally arise out of this.
>
> * Where are the best practices with regard to selection of the best backend
> type for rpc (and one for notifications); is this something oslo.messaging
> should work through (or can the docs team and operator group also help in
> making this)?
>
> * What are the tradeoffs in using the same (or different) technology for rpc
> and notifications?
>

I think the olso.messaging team should take the lead here and educate
users as to what the options are, and  how the two supported messaging
services (RPC and Notifications) differ with respect to backend
requirements.   These topics really should be part of the
oslo.messaging 'Theory of Operations' documentation that was discussed
during the Arch WG summit meeting.

Currently the biggest functional difference between the backends is
the support of store-and-forward (e.g. queueing) verses point-to-point
message transfer.  Oslo could at least explain the pros and cons of
each approach with respect to the RPC and Notification services so
that folks understand what the tradeoffs and advantages are in the
first place.

Furthermore the team should also document the functional differences
between the various choices of backends.  For instance it would be
useful to understand how the two supported point-to-point backends
(zmq and dispatch router) differ in both behavior and recommended
deployment.


> * Is it even possible for all oslo.messaging consuming projects to be able
> to choose 2 different backends, are consuming projects consuming the library
> correctly so that they can use 2 different backends?
>
> * Is devstack able to run with say kafka for notifications and rabbitmq for
> rpc (if not, is there any work item the oslo group can help with to make
> this possible) so that we can ensure and test that all projects can work
> correctly with appropriate (and possibly different) backends?
>
> * Any other messaging, arch-wg work that we (oslo or others) can help out
> with to make sure that projects (and operators) are using the right
> technology for the right use (and not just defaulting to RPC over rabbitmq
> because it exists, when in reality something else might be a better choice)?
>

Ultimately there should be recommendations for which backends are
optimal for a range of different deployment scenarios, but at this
point we really don't have enough data and experience with these
backends to create such recommendations.

> * More(?)
>
> Just wanted to get this conversation started, because afaik it's one that
> has not been widely circulated (and operators have been setting up rabbitmq
> in various HA and clustered and ... modes, when in reality thinking through
> what and how it is used may be more appropriate); this also applies to
> developers since some technical solutions in openstack seem to be created
> due to (in-part) rabbitmq shortcomings (cells v1 afaik was *in* part created
> due to scaling issues).
>
> -Josh
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Ken Giusti  (kgiu...@gmail.com)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Splitting notifications from rpc (and questions + work around this)

[Davanum Srinivas]

Cheran,

Nova and Neutron already supported this split when we started this
exercise. So yes, they are already shipped :)

https://review.openstack.org/#/c/266960/
https://review.openstack.org/#/c/268335/

-- Dims

On Thu, Nov 3, 2016 at 1:43 PM, Elancheran Subramanian
 wrote:
> Hi Dims,
> Thanks for sharing…
>
> Just wanted to check whether there is any development for Nova and Neutron
> going on, which we can leverage?
>
> Thanks,
> Cheran
>
>
>
>
> On 11/3/16, 12:51 AM, "Davanum Srinivas"  wrote:
>
>>Josh,
>>
>>Kirill Bespalov put together this doc of which components will work
>>with separate rpc and notification configurations:
>>https://docs.google.com/document/d/1CU0KjL9iV8vut76hg9cFuWQGSJawuNq_cK7vRF
>>_KyAA/edit?usp=sharing
>>
> >From my team, Oleksii Zamiatin is trying to scale up ZMQ beyond 200+
>>nodes for RPC.
>>
>>Ilya Tyaptin's review is stuck because Monasca folks have trouble
>>using the newer python-kafka version:
>>https://review.openstack.org/#/c/332105/
>>https://review.openstack.org/#/c/316259/
>>
>>As you can tell, we are trying to offer RabbitMQ or ZMQ for RPC and
>>RabbitMQ or Kafka for Notifications.
>>
>>Hope this helps.
>>
>>Thanks,
>>Dims
>>
>>On Wed, Nov 2, 2016 at 8:11 PM, Joshua Harlow 
>>wrote:
>>> Hi folks,
>>>
>>> There was a bunch of chatter at the summit about how there are really
>>>two
>>> different types of (oslo) messaging usage that exist in openstack and
>>>how
>>> they need not be backed by the same solution type (rabbitmq, qpid,
>>> kafka...).
>>>
>>> For those that were not at the oslo sessions:
>>>
>>> https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
>>>
>>> The general gist was though that we need to make sure people really do
>>>know
>>> that there are two very different types of messaging usage in openstack
>>>and
>>> to ensure that operators (and developers) are picking the right backing
>>> technology for each type.
>>>
>>> So some questions naturally arise out of this.
>>>
>>> * Where are the best practices with regard to selection of the best
>>>backend
>>> type for rpc (and one for notifications); is this something
>>>oslo.messaging
>>> should work through (or can the docs team and operator group also help
>>>in
>>> making this)?
>>>
>>> * What are the tradeoffs in using the same (or different) technology
>>>for rpc
>>> and notifications?
>>>
>>> * Is it even possible for all oslo.messaging consuming projects to be
>>>able
>>> to choose 2 different backends, are consuming projects consuming the
>>>library
>>> correctly so that they can use 2 different backends?
>>>
>>> * Is devstack able to run with say kafka for notifications and rabbitmq
>>>for
>>> rpc (if not, is there any work item the oslo group can help with to make
>>> this possible) so that we can ensure and test that all projects can work
>>> correctly with appropriate (and possibly different) backends?
>>>
>>> * Any other messaging, arch-wg work that we (oslo or others) can help
>>>out
>>> with to make sure that projects (and operators) are using the right
>>> technology for the right use (and not just defaulting to RPC over
>>>rabbitmq
>>> because it exists, when in reality something else might be a better
>>>choice)?
>>>
>>> * More(?)
>>>
>>> Just wanted to get this conversation started, because afaik it's one
>>>that
>>> has not been widely circulated (and operators have been setting up
>>>rabbitmq
>>> in various HA and clustered and ... modes, when in reality thinking
>>>through
>>> what and how it is used may be more appropriate); this also applies to
>>> developers since some technical solutions in openstack seem to be
>>>created
>>> due to (in-part) rabbitmq shortcomings (cells v1 afaik was *in* part
>>>created
>>> due to scaling issues).
>>>
>>> -Josh
>>>
>>>
>>>_
>>>_
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>--
>>Davanum Srinivas :: https://twitter.com/dims
>>
>>__
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] Splitting notifications from rpc (and questions + work around this)

[Elancheran Subramanian]

Hi Dims,
Thanks for sharing… 

Just wanted to check whether there is any development for Nova and Neutron 
going on, which we can leverage?  

Thanks,
Cheran




On 11/3/16, 12:51 AM, "Davanum Srinivas"  wrote:

>Josh,
>
>Kirill Bespalov put together this doc of which components will work
>with separate rpc and notification configurations:
>https://docs.google.com/document/d/1CU0KjL9iV8vut76hg9cFuWQGSJawuNq_cK7vRF
>_KyAA/edit?usp=sharing
>
>From my team, Oleksii Zamiatin is trying to scale up ZMQ beyond 200+
>nodes for RPC.
>
>Ilya Tyaptin's review is stuck because Monasca folks have trouble
>using the newer python-kafka version:
>https://review.openstack.org/#/c/332105/
>https://review.openstack.org/#/c/316259/
>
>As you can tell, we are trying to offer RabbitMQ or ZMQ for RPC and
>RabbitMQ or Kafka for Notifications.
>
>Hope this helps.
>
>Thanks,
>Dims
>
>On Wed, Nov 2, 2016 at 8:11 PM, Joshua Harlow  
>wrote:
>> Hi folks,
>>
>> There was a bunch of chatter at the summit about how there are really 
>>two
>> different types of (oslo) messaging usage that exist in openstack and 
>>how
>> they need not be backed by the same solution type (rabbitmq, qpid,
>> kafka...).
>>
>> For those that were not at the oslo sessions:
>>
>> https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
>>
>> The general gist was though that we need to make sure people really do 
>>know
>> that there are two very different types of messaging usage in openstack 
>>and
>> to ensure that operators (and developers) are picking the right backing
>> technology for each type.
>>
>> So some questions naturally arise out of this.
>>
>> * Where are the best practices with regard to selection of the best 
>>backend
>> type for rpc (and one for notifications); is this something 
>>oslo.messaging
>> should work through (or can the docs team and operator group also help 
>>in
>> making this)?
>>
>> * What are the tradeoffs in using the same (or different) technology 
>>for rpc
>> and notifications?
>>
>> * Is it even possible for all oslo.messaging consuming projects to be 
>>able
>> to choose 2 different backends, are consuming projects consuming the 
>>library
>> correctly so that they can use 2 different backends?
>>
>> * Is devstack able to run with say kafka for notifications and rabbitmq 
>>for
>> rpc (if not, is there any work item the oslo group can help with to make
>> this possible) so that we can ensure and test that all projects can work
>> correctly with appropriate (and possibly different) backends?
>>
>> * Any other messaging, arch-wg work that we (oslo or others) can help 
>>out
>> with to make sure that projects (and operators) are using the right
>> technology for the right use (and not just defaulting to RPC over 
>>rabbitmq
>> because it exists, when in reality something else might be a better 
>>choice)?
>>
>> * More(?)
>>
>> Just wanted to get this conversation started, because afaik it's one 
>>that
>> has not been widely circulated (and operators have been setting up 
>>rabbitmq
>> in various HA and clustered and ... modes, when in reality thinking 
>>through
>> what and how it is used may be more appropriate); this also applies to
>> developers since some technical solutions in openstack seem to be 
>>created
>> due to (in-part) rabbitmq shortcomings (cells v1 afaik was *in* part 
>>created
>> due to scaling issues).
>>
>> -Josh
>>
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>-- 
>Davanum Srinivas :: https://twitter.com/dims
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Splitting notifications from rpc (and questions + work around this)

[Joshua Harlow]

Davanum Srinivas wrote:

Josh,

Kirill Bespalov put together this doc of which components will work
with separate rpc and notification configurations:
https://docs.google.com/document/d/1CU0KjL9iV8vut76hg9cFuWQGSJawuNq_cK7vRF_KyAA/edit?usp=sharing

 From my team, Oleksii Zamiatin is trying to scale up ZMQ beyond 200+
nodes for RPC.

Ilya Tyaptin's review is stuck because Monasca folks have trouble
using the newer python-kafka version:
https://review.openstack.org/#/c/332105/
https://review.openstack.org/#/c/316259/

As you can tell, we are trying to offer RabbitMQ or ZMQ for RPC and
RabbitMQ or Kafka for Notifications.

Hope this helps.

Thanks,
Dims



Thanks much, good things to know (and share) :)

-Josh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Splitting notifications from rpc (and questions + work around this)

[Davanum Srinivas]

Josh,

Kirill Bespalov put together this doc of which components will work
with separate rpc and notification configurations:
https://docs.google.com/document/d/1CU0KjL9iV8vut76hg9cFuWQGSJawuNq_cK7vRF_KyAA/edit?usp=sharing

>From my team, Oleksii Zamiatin is trying to scale up ZMQ beyond 200+
nodes for RPC.

Ilya Tyaptin's review is stuck because Monasca folks have trouble
using the newer python-kafka version:
https://review.openstack.org/#/c/332105/
https://review.openstack.org/#/c/316259/

As you can tell, we are trying to offer RabbitMQ or ZMQ for RPC and
RabbitMQ or Kafka for Notifications.

Hope this helps.

Thanks,
Dims

On Wed, Nov 2, 2016 at 8:11 PM, Joshua Harlow  wrote:
> Hi folks,
>
> There was a bunch of chatter at the summit about how there are really two
> different types of (oslo) messaging usage that exist in openstack and how
> they need not be backed by the same solution type (rabbitmq, qpid,
> kafka...).
>
> For those that were not at the oslo sessions:
>
> https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
>
> The general gist was though that we need to make sure people really do know
> that there are two very different types of messaging usage in openstack and
> to ensure that operators (and developers) are picking the right backing
> technology for each type.
>
> So some questions naturally arise out of this.
>
> * Where are the best practices with regard to selection of the best backend
> type for rpc (and one for notifications); is this something oslo.messaging
> should work through (or can the docs team and operator group also help in
> making this)?
>
> * What are the tradeoffs in using the same (or different) technology for rpc
> and notifications?
>
> * Is it even possible for all oslo.messaging consuming projects to be able
> to choose 2 different backends, are consuming projects consuming the library
> correctly so that they can use 2 different backends?
>
> * Is devstack able to run with say kafka for notifications and rabbitmq for
> rpc (if not, is there any work item the oslo group can help with to make
> this possible) so that we can ensure and test that all projects can work
> correctly with appropriate (and possibly different) backends?
>
> * Any other messaging, arch-wg work that we (oslo or others) can help out
> with to make sure that projects (and operators) are using the right
> technology for the right use (and not just defaulting to RPC over rabbitmq
> because it exists, when in reality something else might be a better choice)?
>
> * More(?)
>
> Just wanted to get this conversation started, because afaik it's one that
> has not been widely circulated (and operators have been setting up rabbitmq
> in various HA and clustered and ... modes, when in reality thinking through
> what and how it is used may be more appropriate); this also applies to
> developers since some technical solutions in openstack seem to be created
> due to (in-part) rabbitmq shortcomings (cells v1 afaik was *in* part created
> due to scaling issues).
>
> -Josh
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev