Re: [openstack-dev] [neutron][oslo.messaging][femdc]Topic names for every resource type RPC endpoint

2017-09-25 Thread Miguel Angel Ajo Pelayo 
Yeah, you could may be segment messages by zones/cells etc.

Or group resources by buckets, for example taking the last 2-3 bytes of
each object identifier.

But you may need to do the math on how that's going to work, the more
objects an agent needs to process, the more likely that it will receive
unnecessary objects in that case.

The ideal, anyway is single object subscription (as long as the client and
rabbit would handle such scenario well)


On Sun, Sep 24, 2017 at 11:45 PM, Matthieu Simonin <
matthieu.simo...@inria.fr> wrote:

> Thanks Miguel for your feedback.
>
> I'll definetely dig more into this.
> Having a lot of messages broadcasted to all the neutron agents is not
> something you want especially in the context of femdc[1].
>
> Best,
>
> Matt
>
> [1]: https://wiki.openstack.org/wiki/Fog_Edge_Massively_Distributed_Clouds
>
> - Mail original -
> > De: "Miguel Angel Ajo Pelayo" <majop...@redhat.com>
> > À: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> > Envoyé: Mercredi 20 Septembre 2017 11:15:12
> > Objet: Re: [openstack-dev] [neutron][oslo.messaging][femdc]Topic names
> for every resource type RPC endpoint
> >
> > I wrote those lines.
> >
> > At that time, I tried a couple a publisher and a receiver at that scale.
> It
> > was the receiver side what crashed trying to subscribe, the sender was
> > completely fine.
> >
> > Sadly I don't keep the test examples, I should have stored them in github
> > or something. It shouldn't be hard to replicate though if you follow the
> > oslo_messaging docs.
> >
> >
> >
> > On Wed, Sep 20, 2017 at 9:58 AM, Matthieu Simonin <
> matthieu.simo...@inria.fr
> > > wrote:
> >
> > > Hello,
> > >
> > > In the Neutron docs about RPCs and Callbacks system, it is said[1] :
> > >
> > > "With the underlying oslo_messaging support for dynamic topics on the
> > > receiver
> > > we cannot implement a per “resource type + resource id” topic, rabbitmq
> > > seems
> > > to handle 1’s of topics without suffering, but creating 100’s of
> > > oslo_messaging receivers on different topics seems to crash."
> > >
> > > I wonder if this statements still holds for the new transports
> supported in
> > > oslo.messaging (e.g Kafka, AMQP1.0) or if it's more a design
> limitation.
> > > I'm interested in any relevant docs/links/reviews on the "topic" :).
> > >
> > > Moreover, I'm curious to get an idea on how many different resources a
> > > Neutron
> > > Agent would have to manage and thus how many oslo_messaging receivers
> > > would be
> > > required (e.g how many security groups a neutron agent has to manage
> ?) -
> > > at
> > > least the order of magnitude.
> > >
> > > Best,
> > >
> > > Matt
> > >
> > >
> > >
> > > [1]: https://docs.openstack.org/neutron/latest/contributor/
> > > internals/rpc_callbacks.html#topic-names-for-every-
> > > resource-type-rpc-endpoint
> > >
> > > 
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][oslo.messaging][femdc]Topic names for every resource type RPC endpoint

2017-09-24 Thread Matthieu Simonin 
Thanks Miguel for your feedback.

I'll definetely dig more into this.
Having a lot of messages broadcasted to all the neutron agents is not 
something you want especially in the context of femdc[1]. 

Best,

Matt

[1]: https://wiki.openstack.org/wiki/Fog_Edge_Massively_Distributed_Clouds

- Mail original -
> De: "Miguel Angel Ajo Pelayo" <majop...@redhat.com>
> À: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Envoyé: Mercredi 20 Septembre 2017 11:15:12
> Objet: Re: [openstack-dev] [neutron][oslo.messaging][femdc]Topic names for 
> every resource type RPC endpoint
> 
> I wrote those lines.
> 
> At that time, I tried a couple a publisher and a receiver at that scale. It
> was the receiver side what crashed trying to subscribe, the sender was
> completely fine.
> 
> Sadly I don't keep the test examples, I should have stored them in github
> or something. It shouldn't be hard to replicate though if you follow the
> oslo_messaging docs.
> 
> 
> 
> On Wed, Sep 20, 2017 at 9:58 AM, Matthieu Simonin <matthieu.simo...@inria.fr
> > wrote:
> 
> > Hello,
> >
> > In the Neutron docs about RPCs and Callbacks system, it is said[1] :
> >
> > "With the underlying oslo_messaging support for dynamic topics on the
> > receiver
> > we cannot implement a per “resource type + resource id” topic, rabbitmq
> > seems
> > to handle 1’s of topics without suffering, but creating 100’s of
> > oslo_messaging receivers on different topics seems to crash."
> >
> > I wonder if this statements still holds for the new transports supported in
> > oslo.messaging (e.g Kafka, AMQP1.0) or if it's more a design limitation.
> > I'm interested in any relevant docs/links/reviews on the "topic" :).
> >
> > Moreover, I'm curious to get an idea on how many different resources a
> > Neutron
> > Agent would have to manage and thus how many oslo_messaging receivers
> > would be
> > required (e.g how many security groups a neutron agent has to manage ?) -
> > at
> > least the order of magnitude.
> >
> > Best,
> >
> > Matt
> >
> >
> >
> > [1]: https://docs.openstack.org/neutron/latest/contributor/
> > internals/rpc_callbacks.html#topic-names-for-every-
> > resource-type-rpc-endpoint
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][oslo.messaging][femdc]Topic names for every resource type RPC endpoint

2017-09-20 Thread Miguel Angel Ajo Pelayo 
I wrote those lines.

At that time, I tried a couple a publisher and a receiver at that scale. It
was the receiver side what crashed trying to subscribe, the sender was
completely fine.

Sadly I don't keep the test examples, I should have stored them in github
or something. It shouldn't be hard to replicate though if you follow the
oslo_messaging docs.



On Wed, Sep 20, 2017 at 9:58 AM, Matthieu Simonin  wrote:

> Hello,
>
> In the Neutron docs about RPCs and Callbacks system, it is said[1] :
>
> "With the underlying oslo_messaging support for dynamic topics on the
> receiver
> we cannot implement a per “resource type + resource id” topic, rabbitmq
> seems
> to handle 1’s of topics without suffering, but creating 100’s of
> oslo_messaging receivers on different topics seems to crash."
>
> I wonder if this statements still holds for the new transports supported in
> oslo.messaging (e.g Kafka, AMQP1.0) or if it's more a design limitation.
> I'm interested in any relevant docs/links/reviews on the "topic" :).
>
> Moreover, I'm curious to get an idea on how many different resources a
> Neutron
> Agent would have to manage and thus how many oslo_messaging receivers
> would be
> required (e.g how many security groups a neutron agent has to manage ?) -
> at
> least the order of magnitude.
>
> Best,
>
> Matt
>
>
>
> [1]: https://docs.openstack.org/neutron/latest/contributor/
> internals/rpc_callbacks.html#topic-names-for-every-
> resource-type-rpc-endpoint
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][oslo.messaging][femdc]Topic names for every resource type RPC endpoint

2017-09-20 Thread Matthieu Simonin 
Hello,

In the Neutron docs about RPCs and Callbacks system, it is said[1] :

"With the underlying oslo_messaging support for dynamic topics on the receiver
we cannot implement a per “resource type + resource id” topic, rabbitmq seems
to handle 1’s of topics without suffering, but creating 100’s of
oslo_messaging receivers on different topics seems to crash."

I wonder if this statements still holds for the new transports supported in
oslo.messaging (e.g Kafka, AMQP1.0) or if it's more a design limitation.
I'm interested in any relevant docs/links/reviews on the "topic" :).

Moreover, I'm curious to get an idea on how many different resources a Neutron
Agent would have to manage and thus how many oslo_messaging receivers would be
required (e.g how many security groups a neutron agent has to manage ?) - at
least the order of magnitude.

Best,

Matt



[1]: 
https://docs.openstack.org/neutron/latest/contributor/internals/rpc_callbacks.html#topic-names-for-every-resource-type-rpc-endpoint

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev