Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
Hi Doug, I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn't block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. Is that correct - and is there any worked example of doing this ? I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the compute exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I'm finding it hard to see how to change this model to have multiple notify.info topic queues into the same exchange ? Cheers, Phil From: openstack-bounces+philip.day=hp@lists.launchpad.net [mailto:openstack-bounces+philip.day=hp@lists.launchpad.net] On Behalf Of Doug Hellmann Sent: 08 May 2012 23:34 To: Russell Bryant Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.commailto:rbry...@redhat.com wrote: On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq. In nova, for a given topic such as 'scheduler', all of the consumers are binding to the same queue on the topic exchange, resulting in round-robin delivery to each of the consumers. If instead you make a new queue, you can get your own copy of each message. There is an additional benefit of using a topic exchange here. The topic used for notifications is 'notifications.priority'. That means that when you create your queue, you can set it up to receive all notifications, or only notifications of a certain priority. Topic exchanges make a lot of sense for messages that should only be consumed once, such as tasks. Notifications are different. Lots of different clients might want to know that some event happened in the system. The way things are in Nova today, they can't. The first client who consumes a notification message will prevent all of the other clients from seeing that message at all. I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. Yes, that wasn't obvious from any of the kombu documentation I've seen so far. I'll keep looking. Thanks, Doug I can change Nova's notification system to use a fanout exchange (in impl_kombu.py changing the exchange type used by NotifyPublisher), but before I submit a patch I want to make sure the current implementation using a topic exchange wasn't selected deliberately for some reason. I think using a fanout exchange would be a downgrade. As I mentioned before, a topic exchange allows you to create a queue to get all notifications or only notifications of a specific priority. If the exchange type is changed to fanout, it's everybody gets everything, and that's it. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
Your own queue listener should attempt to declare the exchange, using the same settings as Nova does. If the exchange exists, its a noop. Otherwise it's created for you. After that, if you start up Nova, it will do the same and reuse your exchange. Obviously this works both ways, and either nova or your code can declare the exchange. AMQP is designed to be a configuration-less protocol, where resources are configured by the first consumer attempting to use them. Thanks, Kiall Sent from my phone. On May 9, 2012 9:52 a.m., Day, Phil philip@hp.com wrote: Hi Doug, ** ** I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. ** ** So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn’t block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. ** ** Is that correct – and is there any worked example of doing this ? ** ** I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the “compute” exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I’m finding it hard to see how to change this model to have multiple “notify.info” topic queues into the same exchange ? ** ** Cheers, Phil ** ** ** ** ** ** ** ** *From:* openstack-bounces+philip.day=hp@lists.launchpad.net [mailto: openstack-bounces+philip.day=hp@lists.launchpad.net] *On Behalf Of *Doug Hellmann *Sent:* 08 May 2012 23:34 *To:* Russell Bryant *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? ** ** ** ** On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.com wrote: On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq. In nova, for a given topic such as 'scheduler', all of the consumers are binding to the same queue on the topic exchange, resulting in round-robin delivery to each of the consumers. If instead you make a new queue, you can get your own copy of each message. There is an additional benefit of using a topic exchange here. The topic used for notifications is 'notifications.priority'. That means that when you create your queue, you can set it up to receive all notifications, or only notifications of a certain priority. Topic exchanges make a lot of sense for messages that should only be consumed once, such as tasks. Notifications are different. Lots of different clients might want to know that some event happened in the system. The way things are in Nova today, they can't. The first client who consumes a notification message will prevent all of the other clients from seeing that message at all. I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. ** ** Yes, that wasn't obvious from any of the kombu documentation I've seen so far. I'll keep looking. ** ** Thanks, Doug I can change Nova's notification system to use a fanout exchange (in impl_kombu.py changing the exchange type used by NotifyPublisher), but before I submit a patch I want to make sure the current implementation using a topic exchange wasn't selected deliberately for some reason. I think using a fanout exchange would
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
OK, get that so far - so both consumers need to declare and use the same exchange. But If I understand the next step right, to get multiple consumers of info notification messages they would all need to create separate notifications.info queues into that exchange.And isn't that exactly what Nova currently does to create a shared queue ? Phil From: Kiall Mac Innes [mailto:ki...@managedit.ie] Sent: 09 May 2012 10:51 To: Day, Phil Cc: openstack@lists.launchpad.net; Russell Bryant; Doug Hellmann Subject: Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? Your own queue listener should attempt to declare the exchange, using the same settings as Nova does. If the exchange exists, its a noop. Otherwise it's created for you. After that, if you start up Nova, it will do the same and reuse your exchange. Obviously this works both ways, and either nova or your code can declare the exchange. AMQP is designed to be a configuration-less protocol, where resources are configured by the first consumer attempting to use them. Thanks, Kiall Sent from my phone. On May 9, 2012 9:52 a.m., Day, Phil philip@hp.commailto:philip@hp.com wrote: Hi Doug, I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn't block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. Is that correct - and is there any worked example of doing this ? I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the compute exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I'm finding it hard to see how to change this model to have multiple notify.infohttp://notify.info topic queues into the same exchange ? Cheers, Phil From: openstack-bounces+philip.day=hp@lists.launchpad.netmailto:hp@lists.launchpad.net [mailto:openstack-bounces+philip.daymailto:openstack-bounces%2Bphilip.day=hp@lists.launchpad.netmailto:hp@lists.launchpad.net] On Behalf Of Doug Hellmann Sent: 08 May 2012 23:34 To: Russell Bryant Cc: openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Subject: Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.commailto:rbry...@redhat.com wrote: On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq. In nova, for a given topic such as 'scheduler', all of the consumers are binding to the same queue on the topic exchange, resulting in round-robin delivery to each of the consumers. If instead you make a new queue, you can get your own copy of each message. There is an additional benefit of using a topic exchange here. The topic used for notifications is 'notifications.priority'. That means that when you create your queue, you can set it up to receive all notifications, or only notifications of a certain priority. Topic exchanges make a lot of sense for messages that should only be consumed once, such as tasks. Notifications are different. Lots of different clients might want to know that some event happened in the system. The way things are in Nova today, they can't. The first client who consumes a notification message will prevent all of the other clients from seeing that message at all. I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
Kinda! The queue has a name, but that name has no bearing on the set of messages received. If you create a queue called MyCustomNotificationQueue, you can bind that to the notifications exchange using the notifications.info routing key. (I'm guessing some of the names here.. I know AMQP, and not the specific naming nova uses!) Nova just happens to use the same queue name and routing key. I believe this is causing the confusion. Does this make sense? Anyway - The RabbitMQ docs probably explain it better than I.. http://www.rabbitmq.com/tutorials/tutorial-five-python.html Thanks, Kiall On Wed, May 9, 2012 at 11:30 AM, Day, Phil philip@hp.com wrote: OK, get that so far – so both consumers need to declare and use the same exchange. ** ** But If I understand the next step right, to get multiple consumers of info notification messages they would all need to create separate “ notifications.info” queues into that exchange.And isn’t that exactly what Nova currently does to create a shared queue ? ** ** Phil ** ** *From:* Kiall Mac Innes [mailto:ki...@managedit.ie] *Sent:* 09 May 2012 10:51 *To:* Day, Phil *Cc:* openstack@lists.launchpad.net; Russell Bryant; Doug Hellmann *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? ** ** Your own queue listener should attempt to declare the exchange, using the same settings as Nova does. If the exchange exists, its a noop. Otherwise it's created for you. After that, if you start up Nova, it will do the same and reuse your exchange. Obviously this works both ways, and either nova or your code can declare the exchange. AMQP is designed to be a configuration-less protocol, where resources are configured by the first consumer attempting to use them. Thanks, Kiall Sent from my phone. On May 9, 2012 9:52 a.m., Day, Phil philip@hp.com wrote: Hi Doug, I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn’t block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. Is that correct – and is there any worked example of doing this ? I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the “compute” exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I’m finding it hard to see how to change this model to have multiple “notify.info” topic queues into the same exchange ? Cheers, Phil *From:* openstack-bounces+philip.day=hp@lists.launchpad.net [mailto: openstack-bounces+philip.day=hp@lists.launchpad.net] *On Behalf Of *Doug Hellmann *Sent:* 08 May 2012 23:34 *To:* Russell Bryant *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.com wrote: On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq. In nova, for a given topic such as 'scheduler', all of the consumers are binding to the same queue on the topic exchange, resulting in round-robin delivery to each of the consumers. If instead you make a new queue, you can get your own copy of each message. There is an additional benefit of using
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On Wed, May 9, 2012 at 6:42 AM, Kiall Mac Innes ki...@managedit.ie wrote: Kinda! The queue has a name, but that name has no bearing on the set of messages received. If you create a queue called MyCustomNotificationQueue, you can bind that to the notifications exchange using the notifications.info routing key. (I'm guessing some of the names here.. I know AMQP, and not the specific naming nova uses!) notifications.info is right. Nova just happens to use the same queue name and routing key. I believe this is causing the confusion. Exactly. I ended up creating a separate queue for each client that I have and setting them to auto-delete when the client disconnects. That way I can have as many clients connecting and listening as I want. The code is in https://github.com/dhellmann/metering-prototype if you want to take a look. Does this make sense? Anyway - The RabbitMQ docs probably explain it better than I.. http://www.rabbitmq.com/tutorials/tutorial-five-python.html Thanks, Kiall On Wed, May 9, 2012 at 11:30 AM, Day, Phil philip@hp.com wrote: OK, get that so far – so both consumers need to declare and use the same exchange. ** ** But If I understand the next step right, to get multiple consumers of info notification messages they would all need to create separate “ notifications.info” queues into that exchange.And isn’t that exactly what Nova currently does to create a shared queue ? ** ** Phil ** ** *From:* Kiall Mac Innes [mailto:ki...@managedit.ie] *Sent:* 09 May 2012 10:51 *To:* Day, Phil *Cc:* openstack@lists.launchpad.net; Russell Bryant; Doug Hellmann *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? ** ** Your own queue listener should attempt to declare the exchange, using the same settings as Nova does. If the exchange exists, its a noop. Otherwise it's created for you. After that, if you start up Nova, it will do the same and reuse your exchange. Obviously this works both ways, and either nova or your code can declare the exchange. AMQP is designed to be a configuration-less protocol, where resources are configured by the first consumer attempting to use them. Thanks, Kiall Sent from my phone. On May 9, 2012 9:52 a.m., Day, Phil philip@hp.com wrote: Hi Doug, I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn’t block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. Is that correct – and is there any worked example of doing this ? I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the “compute” exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I’m finding it hard to see how to change this model to have multiple “ notify.info” topic queues into the same exchange ? Cheers, Phil *From:* openstack-bounces+philip.day=hp@lists.launchpad.net [mailto: openstack-bounces+philip.day=hp@lists.launchpad.net] *On Behalf Of *Doug Hellmann *Sent:* 08 May 2012 23:34 *To:* Russell Bryant *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.com wrote: On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
You can allow nova to send notifications to multiple topics by setting this option in your nova.conf. ## (ListOpt) AMQP topic used for Nova notifications notification_topics=notifications,metering,monitoring This will allow you to consume all the messages from a different service. On Wed, May 9, 2012 at 10:34 AM, Doug Hellmann doug.hellm...@dreamhost.comwrote: On Wed, May 9, 2012 at 6:42 AM, Kiall Mac Innes ki...@managedit.iewrote: Kinda! The queue has a name, but that name has no bearing on the set of messages received. If you create a queue called MyCustomNotificationQueue, you can bind that to the notifications exchange using the notifications.info routing key. (I'm guessing some of the names here.. I know AMQP, and not the specific naming nova uses!) notifications.info is right. Nova just happens to use the same queue name and routing key. I believe this is causing the confusion. Exactly. I ended up creating a separate queue for each client that I have and setting them to auto-delete when the client disconnects. That way I can have as many clients connecting and listening as I want. The code is in https://github.com/dhellmann/metering-prototype if you want to take a look. Does this make sense? Anyway - The RabbitMQ docs probably explain it better than I.. http://www.rabbitmq.com/tutorials/tutorial-five-python.html Thanks, Kiall On Wed, May 9, 2012 at 11:30 AM, Day, Phil philip@hp.com wrote: OK, get that so far – so both consumers need to declare and use the same exchange. ** ** But If I understand the next step right, to get multiple consumers of info notification messages they would all need to create separate “ notifications.info” queues into that exchange.And isn’t that exactly what Nova currently does to create a shared queue ? ** ** Phil ** ** *From:* Kiall Mac Innes [mailto:ki...@managedit.ie] *Sent:* 09 May 2012 10:51 *To:* Day, Phil *Cc:* openstack@lists.launchpad.net; Russell Bryant; Doug Hellmann *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? ** ** Your own queue listener should attempt to declare the exchange, using the same settings as Nova does. If the exchange exists, its a noop. Otherwise it's created for you. After that, if you start up Nova, it will do the same and reuse your exchange. Obviously this works both ways, and either nova or your code can declare the exchange. AMQP is designed to be a configuration-less protocol, where resources are configured by the first consumer attempting to use them. Thanks, Kiall Sent from my phone. On May 9, 2012 9:52 a.m., Day, Phil philip@hp.com wrote: Hi Doug, I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn’t block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. Is that correct – and is there any worked example of doing this ? I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the “compute” exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I’m finding it hard to see how to change this model to have multiple “ notify.info” topic queues into the same exchange ? Cheers, Phil *From:* openstack-bounces+philip.day=hp@lists.launchpad.net [mailto: openstack-bounces+philip.day=hp@lists.launchpad.net] *On Behalf Of *Doug Hellmann *Sent:* 08 May 2012 23:34 *To:* Russell Bryant *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.com wrote: On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
Is that the preferred way to do it, rather than attaching another queue to the existing exchange with the same routing key? On Wed, May 9, 2012 at 1:46 PM, Craig Vyvial cp16...@gmail.com wrote: You can allow nova to send notifications to multiple topics by setting this option in your nova.conf. ## (ListOpt) AMQP topic used for Nova notifications notification_topics=notifications,metering,monitoring This will allow you to consume all the messages from a different service. On Wed, May 9, 2012 at 10:34 AM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Wed, May 9, 2012 at 6:42 AM, Kiall Mac Innes ki...@managedit.iewrote: Kinda! The queue has a name, but that name has no bearing on the set of messages received. If you create a queue called MyCustomNotificationQueue, you can bind that to the notifications exchange using the notifications.info routing key. (I'm guessing some of the names here.. I know AMQP, and not the specific naming nova uses!) notifications.info is right. Nova just happens to use the same queue name and routing key. I believe this is causing the confusion. Exactly. I ended up creating a separate queue for each client that I have and setting them to auto-delete when the client disconnects. That way I can have as many clients connecting and listening as I want. The code is in https://github.com/dhellmann/metering-prototype if you want to take a look. Does this make sense? Anyway - The RabbitMQ docs probably explain it better than I.. http://www.rabbitmq.com/tutorials/tutorial-five-python.html Thanks, Kiall On Wed, May 9, 2012 at 11:30 AM, Day, Phil philip@hp.com wrote: OK, get that so far – so both consumers need to declare and use the same exchange. ** ** But If I understand the next step right, to get multiple consumers of info notification messages they would all need to create separate “ notifications.info” queues into that exchange.And isn’t that exactly what Nova currently does to create a shared queue ? ** ** Phil ** ** *From:* Kiall Mac Innes [mailto:ki...@managedit.ie] *Sent:* 09 May 2012 10:51 *To:* Day, Phil *Cc:* openstack@lists.launchpad.net; Russell Bryant; Doug Hellmann *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? ** ** Your own queue listener should attempt to declare the exchange, using the same settings as Nova does. If the exchange exists, its a noop. Otherwise it's created for you. After that, if you start up Nova, it will do the same and reuse your exchange. Obviously this works both ways, and either nova or your code can declare the exchange. AMQP is designed to be a configuration-less protocol, where resources are configured by the first consumer attempting to use them. Thanks, Kiall Sent from my phone. On May 9, 2012 9:52 a.m., Day, Phil philip@hp.com wrote: Hi Doug, I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn’t block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. Is that correct – and is there any worked example of doing this ? I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the “compute” exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I’m finding it hard to see how to change this model to have multiple “ notify.info” topic queues into the same exchange ? Cheers, Phil *From:* openstack-bounces+philip.day=hp@lists.launchpad.net[mailto: openstack-bounces+philip.day=hp@lists.launchpad.net] *On Behalf Of *Doug Hellmann *Sent:* 08 May 2012 23:34 *To:* Russell Bryant *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.com wrote: On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
Hi Doug, not sure if you've hit the same problem, but I've spent some time on that when I started using RabbitMQ. As I see from the example, you've provided: queue = Queue(name='notifications.info', exchange=Exchange(name='nova'... So you set explicitly a name for the queue. If you have two queues declared with the same name, when they are bound to an Exchange, actually each message is received only by the one queue at a time. Just leave the name field empty(it will be auto-generated), and each queue will receive its copy of the message. So the logic is that the first queue with that name acknowledges the message, and the other one receives nothing. Besides that, the topics are more powerful and handy to use than fanout. On Wed, May 9, 2012 at 6:34 PM, Doug Hellmann doug.hellm...@dreamhost.comwrote: On Wed, May 9, 2012 at 6:42 AM, Kiall Mac Innes ki...@managedit.iewrote: Kinda! The queue has a name, but that name has no bearing on the set of messages received. If you create a queue called MyCustomNotificationQueue, you can bind that to the notifications exchange using the notifications.info routing key. (I'm guessing some of the names here.. I know AMQP, and not the specific naming nova uses!) notifications.info is right. Nova just happens to use the same queue name and routing key. I believe this is causing the confusion. Exactly. I ended up creating a separate queue for each client that I have and setting them to auto-delete when the client disconnects. That way I can have as many clients connecting and listening as I want. The code is in https://github.com/dhellmann/metering-prototype if you want to take a look. Does this make sense? Anyway - The RabbitMQ docs probably explain it better than I.. http://www.rabbitmq.com/tutorials/tutorial-five-python.html Thanks, Kiall On Wed, May 9, 2012 at 11:30 AM, Day, Phil philip@hp.com wrote: OK, get that so far – so both consumers need to declare and use the same exchange. ** ** But If I understand the next step right, to get multiple consumers of info notification messages they would all need to create separate “ notifications.info” queues into that exchange.And isn’t that exactly what Nova currently does to create a shared queue ? ** ** Phil ** ** *From:* Kiall Mac Innes [mailto:ki...@managedit.ie] *Sent:* 09 May 2012 10:51 *To:* Day, Phil *Cc:* openstack@lists.launchpad.net; Russell Bryant; Doug Hellmann *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? ** ** Your own queue listener should attempt to declare the exchange, using the same settings as Nova does. If the exchange exists, its a noop. Otherwise it's created for you. After that, if you start up Nova, it will do the same and reuse your exchange. Obviously this works both ways, and either nova or your code can declare the exchange. AMQP is designed to be a configuration-less protocol, where resources are configured by the first consumer attempting to use them. Thanks, Kiall Sent from my phone. On May 9, 2012 9:52 a.m., Day, Phil philip@hp.com wrote: Hi Doug, I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn’t block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. Is that correct – and is there any worked example of doing this ? I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the “compute” exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I’m finding it hard to see how to change this model to have multiple “ notify.info” topic queues into the same exchange ? Cheers, Phil *From:* openstack-bounces+philip.day=hp@lists.launchpad.net [mailto: openstack-bounces+philip.day=hp@lists.launchpad.net] *On Behalf Of *Doug Hellmann *Sent:* 08 May 2012 23:34 *To:* Russell Bryant *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.com wrote: On 05/08/2012 05:59
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On Wed, May 9, 2012 at 3:17 PM, Tihomir Trifonov t.trifo...@gmail.comwrote: Hi Doug, not sure if you've hit the same problem, but I've spent some time on that when I started using RabbitMQ. As I see from the example, you've provided: queue = Queue(name='notifications.info', exchange=Exchange(name='nova'... So you set explicitly a name for the queue. If you have two queues declared with the same name, when they are bound to an Exchange, actually each message is received only by the one queue at a time. Just leave the name field empty(it will be auto-generated), and each queue will receive its copy of the message. So the logic is that the first queue with that name acknowledges the message, and the other one receives nothing. Besides that, the topics are more powerful and handy to use than fanout. Now that I see how it works, I agree. :-) On Wed, May 9, 2012 at 6:34 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Wed, May 9, 2012 at 6:42 AM, Kiall Mac Innes ki...@managedit.iewrote: Kinda! The queue has a name, but that name has no bearing on the set of messages received. If you create a queue called MyCustomNotificationQueue, you can bind that to the notifications exchange using the notifications.info routing key. (I'm guessing some of the names here.. I know AMQP, and not the specific naming nova uses!) notifications.info is right. Nova just happens to use the same queue name and routing key. I believe this is causing the confusion. Exactly. I ended up creating a separate queue for each client that I have and setting them to auto-delete when the client disconnects. That way I can have as many clients connecting and listening as I want. The code is in https://github.com/dhellmann/metering-prototype if you want to take a look. Does this make sense? Anyway - The RabbitMQ docs probably explain it better than I.. http://www.rabbitmq.com/tutorials/tutorial-five-python.html Thanks, Kiall On Wed, May 9, 2012 at 11:30 AM, Day, Phil philip@hp.com wrote: OK, get that so far – so both consumers need to declare and use the same exchange. ** ** But If I understand the next step right, to get multiple consumers of info notification messages they would all need to create separate “ notifications.info” queues into that exchange.And isn’t that exactly what Nova currently does to create a shared queue ? ** ** Phil ** ** *From:* Kiall Mac Innes [mailto:ki...@managedit.ie] *Sent:* 09 May 2012 10:51 *To:* Day, Phil *Cc:* openstack@lists.launchpad.net; Russell Bryant; Doug Hellmann *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout? ** ** Your own queue listener should attempt to declare the exchange, using the same settings as Nova does. If the exchange exists, its a noop. Otherwise it's created for you. After that, if you start up Nova, it will do the same and reuse your exchange. Obviously this works both ways, and either nova or your code can declare the exchange. AMQP is designed to be a configuration-less protocol, where resources are configured by the first consumer attempting to use them. Thanks, Kiall Sent from my phone. On May 9, 2012 9:52 a.m., Day, Phil philip@hp.com wrote: Hi Doug, I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. So just to be clear, if I understand you correctly within the nova service/rpc abstraction layers the code is set up so that all services do bind to the same queue, and hence we get the round-robin delivery. But, if someone wanted to write a separate notification consumer so that they didn’t block anyone else from seeing the same messages then they (the consumer) should create a new queue on the existing topic exchange. Is that correct – and is there any worked example of doing this ? I thought within the nova code both the exchange and topic queues were set up by the consumer (so for example all compute_managers try to create the “compute” exchange and topic queue, but its only created by the first one and the others connect to the same queue). In that context I’m finding it hard to see how to change this model to have multiple “ notify.info” topic queues into the same exchange ? Cheers, Phil *From:* openstack-bounces+philip.day=hp@lists.launchpad.net[mailto: openstack-bounces+philip.day=hp@lists.launchpad.net] *On Behalf Of *Doug Hellmann *Sent:* 08 May 2012 23:34 *To:* Russell Bryant *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] [nova] why does notification use a topic exchange
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On 05/09/2012 01:57 PM, Doug Hellmann wrote: Is that the preferred way to do it, rather than attaching another queue to the existing exchange with the same routing key? I would just do what you're doing now, which is to use the existing exchange. That lets the message broker do the work of duplicating the message as opposed to making nova send the message multiple times. Perhaps using another exchange would make sense if it made ACLs easier for ensuring that an application can only get notifications, and nothing else on the nova exchange. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On 05/09/2012 03:17 PM, Tihomir Trifonov wrote: Hi Doug, not sure if you've hit the same problem, but I've spent some time on that when I started using RabbitMQ. As I see from the example, you've provided: queue = Queue(name='notifications.info', exchange=Exchange(name='nova'... So you set explicitly a name for the queue. If you have two queues declared with the same name, when they are bound to an Exchange, actually each message is received only by the one queue at a time. To be a bit pedantic, the result is that there is only one queue and two consumers from that queue. Only one consumer gets each message. When there is more than one consumer from the same queue, messages will be distributed in a round-robin fashion. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [nova] why does notification use a topic exchange instead of fanout?
I'm working on some code to monitor notification messages coming from nova by listening directly on the AMQP queue. Unfortunately, nova is using a topic exchange so it seems I can't safely consume the messages or they might not go to some other client that is expecting to see them (especially some other part of nova that is expecting to receive the notification). It seems like notifications should be available to more than one subscriber at a time, but I thought I would ask for background before I started working on a patch to change it. Is there a particular reason a topic exchange was used for notifications? Thanks, Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On 05/08/2012 04:19 PM, Doug Hellmann wrote: I'm working on some code to monitor notification messages coming from nova by listening directly on the AMQP queue. Unfortunately, nova is using a topic exchange so it seems I can't safely consume the messages or they might not go to some other client that is expecting to see them (especially some other part of nova that is expecting to receive the notification). It seems like notifications should be available to more than one subscriber at a time, but I thought I would ask for background before I started working on a patch to change it. Is there a particular reason a topic exchange was used for notifications? I believe you can still get what you want (but not by using nova.rpc as your client). Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq. In nova, for a given topic such as 'scheduler', all of the consumers are binding to the same queue on the topic exchange, resulting in round-robin delivery to each of the consumers. If instead you make a new queue, you can get your own copy of each message. There is an additional benefit of using a topic exchange here. The topic used for notifications is 'notifications.priority'. That means that when you create your queue, you can set it up to receive all notifications, or only notifications of a certain priority. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On Tue, May 8, 2012 at 5:45 PM, Russell Bryant rbry...@redhat.com wrote: On 05/08/2012 04:19 PM, Doug Hellmann wrote: I'm working on some code to monitor notification messages coming from nova by listening directly on the AMQP queue. Unfortunately, nova is using a topic exchange so it seems I can't safely consume the messages or they might not go to some other client that is expecting to see them (especially some other part of nova that is expecting to receive the notification). It seems like notifications should be available to more than one subscriber at a time, but I thought I would ask for background before I started working on a patch to change it. Is there a particular reason a topic exchange was used for notifications? I believe you can still get what you want (but not by using nova.rpc as your client). I'm not using nova.rpc in my code. I am using kombu to connect to the rabbit queue directly. Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq. In nova, for a given topic such as 'scheduler', all of the consumers are binding to the same queue on the topic exchange, resulting in round-robin delivery to each of the consumers. If instead you make a new queue, you can get your own copy of each message. There is an additional benefit of using a topic exchange here. The topic used for notifications is 'notifications.priority'. That means that when you create your queue, you can set it up to receive all notifications, or only notifications of a certain priority. Topic exchanges make a lot of sense for messages that should only be consumed once, such as tasks. Notifications are different. Lots of different clients might want to know that some event happened in the system. The way things are in Nova today, they can't. The first client who consumes a notification message will prevent all of the other clients from seeing that message at all. I can change Nova's notification system to use a fanout exchange (in impl_kombu.py changing the exchange type used by NotifyPublisher), but before I submit a patch I want to make sure the current implementation using a topic exchange wasn't selected deliberately for some reason. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq. In nova, for a given topic such as 'scheduler', all of the consumers are binding to the same queue on the topic exchange, resulting in round-robin delivery to each of the consumers. If instead you make a new queue, you can get your own copy of each message. There is an additional benefit of using a topic exchange here. The topic used for notifications is 'notifications.priority'. That means that when you create your queue, you can set it up to receive all notifications, or only notifications of a certain priority. Topic exchanges make a lot of sense for messages that should only be consumed once, such as tasks. Notifications are different. Lots of different clients might want to know that some event happened in the system. The way things are in Nova today, they can't. The first client who consumes a notification message will prevent all of the other clients from seeing that message at all. I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. I can change Nova's notification system to use a fanout exchange (in impl_kombu.py changing the exchange type used by NotifyPublisher), but before I submit a patch I want to make sure the current implementation using a topic exchange wasn't selected deliberately for some reason. I think using a fanout exchange would be a downgrade. As I mentioned before, a topic exchange allows you to create a queue to get all notifications or only notifications of a specific priority. If the exchange type is changed to fanout, it's everybody gets everything, and that's it. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On Tue, May 8, 2012 at 6:04 PM, Russell Bryant rbry...@redhat.com wrote: On 05/08/2012 05:59 PM, Doug Hellmann wrote: Here is a relevant section pulled out of the amqp 0-9-1 spec: 3.1.3.3 The Topic Exchange Type The topic exchange type works as follows: 1. A message queue binds to the exchange using a routing pattern, P. 2. A publisher sends the exchange a message with the routing key R. 3. The message is passed to the message queue if R matches P. The routing key used for a topic exchange MUST consist of zero or more words delimited by dots. Each word may contain the letters A-Z and a-z and digits 0-9. The routing pattern follows the same rules as the routing key with the addition that * matches a single word, and # matches zero or more words. Thus the routing pattern *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq. In nova, for a given topic such as 'scheduler', all of the consumers are binding to the same queue on the topic exchange, resulting in round-robin delivery to each of the consumers. If instead you make a new queue, you can get your own copy of each message. There is an additional benefit of using a topic exchange here. The topic used for notifications is 'notifications.priority'. That means that when you create your queue, you can set it up to receive all notifications, or only notifications of a certain priority. Topic exchanges make a lot of sense for messages that should only be consumed once, such as tasks. Notifications are different. Lots of different clients might want to know that some event happened in the system. The way things are in Nova today, they can't. The first client who consumes a notification message will prevent all of the other clients from seeing that message at all. I think you missed my main point, which was that a topic exchange does not impose a limitation that only one client can consume a given notification. That's only true if each client is consuming from the same queue bound to the exchange. Yes, that wasn't obvious from any of the kombu documentation I've seen so far. I'll keep looking. Thanks, Doug I can change Nova's notification system to use a fanout exchange (in impl_kombu.py changing the exchange type used by NotifyPublisher), but before I submit a patch I want to make sure the current implementation using a topic exchange wasn't selected deliberately for some reason. I think using a fanout exchange would be a downgrade. As I mentioned before, a topic exchange allows you to create a queue to get all notifications or only notifications of a specific priority. If the exchange type is changed to fanout, it's everybody gets everything, and that's it. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
The reason nova uses a topic exchange and queues like it does is so the processing of notifications can be round-robined to multiple workers. You should be able to open a new queue on the same exchange and receive copies of each message. On May 8, 2012, at 3:19 PM, Doug Hellmann wrote: I'm working on some code to monitor notification messages coming from nova by listening directly on the AMQP queue. Unfortunately, nova is using a topic exchange so it seems I can't safely consume the messages or they might not go to some other client that is expecting to see them (especially some other part of nova that is expecting to receive the notification). It seems like notifications should be available to more than one subscriber at a time, but I thought I would ask for background before I started working on a patch to change it. Is there a particular reason a topic exchange was used for notifications? Thanks, Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Monsyne M. Dragon OpenStack/Nova cell 210-441-0965 work x 5014190 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] why does notification use a topic exchange instead of fanout?
On Tue, May 8, 2012 at 7:42 PM, Monsyne Dragon mdra...@rackspace.comwrote: The reason nova uses a topic exchange and queues like it does is so the processing of notifications can be round-robined to multiple workers. You should be able to open a new queue on the same exchange and receive copies of each message. Yeah, this was a case of not reading enough (or the right) documentation. The kombu docs didn't have an example of that, and I was having some trouble mapping the Rabbit docs onto the kombu classes. I've got it working the way I need now, though. Thanks for your help, Doug On May 8, 2012, at 3:19 PM, Doug Hellmann wrote: I'm working on some code to monitor notification messages coming from nova by listening directly on the AMQP queue. Unfortunately, nova is using a topic exchange so it seems I can't safely consume the messages or they might not go to some other client that is expecting to see them (especially some other part of nova that is expecting to receive the notification). It seems like notifications should be available to more than one subscriber at a time, but I thought I would ask for background before I started working on a patch to change it. Is there a particular reason a topic exchange was used for notifications? Thanks, Doug ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Monsyne M. Dragon OpenStack/Nova cell 210-441-0965 work x 5014190 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
