Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
On 22/03/16 17:20 -0400, Adam Young wrote: On 03/22/2016 09:15 AM, Flavio Percoco wrote: On 21/03/16 21:43 -0400, Adam Young wrote: I had a good discussion with the Nova folks in IRC today. My goal was to understand what could talk to what, and the short according to dansmith " any node in nova land has to be able to talk to the queue for any other one for the most part: compute->compute, compute->conductor, conductor->compute, api->everything. There might be a few exceptions, but not worth it, IMHO, in the current architecture." Longer conversation is here: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/ %23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 Right now, the message queue is a nightmare. All sorts of sensitive information flows over the message queue: Tokens (including admin) are the most obvious. Every piece of audit data. All notifications and all control messages. Before we continue down the path of "anything can talk to anything" can we please map out what needs to talk to what, and why? Many of the use cases seem to be based on something that should be kicked off by the conductor, such as "migrate, resize, live-migrate" and it sounds like there are plans to make that happen. So, let's assume we can get to the point where, if node 1 needs to talk to node 2, it will do so only via the conductor. With that in place, we can put an access control rule in place: I don't think this is going to scale well. Eventually, this will require evolving the conductor to some sort of message scheduler, which is pretty much what the message bus is supposed to do. I'll limit this to what happens with Rabbit and QPID (AMQP1.0) and leave 0 our of it for now. I'll use rabbit as shorthand for both these, but the rules are the same for qpid. Sorry for the pedantic nitpick but, it's not Qpid. I'm afraid calling it Qpid will just confuse people on what we're really talking about here. The amqp1 driver is based on the AMQP 1.0 protocol, which is brokerless. The library used in oslo.messaging is qpid-proton (A.K.A Proton). Qpid is just the name of the Apache Foundation family that these projects belong to (including Qpidd the old broker which we don't support anymore in oslo.messaging). For, say, a migrate operation, the call goes to API, controller, and eventually down to one of the compute nodes. Source? Target? I don't know the code well enough to say, but let's say it is the source. It sends an RPC message to the target node. The message goes to the central broker right now, and then back down to the targen node. Meanwhile, the source node has set up a reply queue and that queue name has gone into the message. The target machine responds by getting a reference to the response queue and sends a message. This message goes up to the broker, and then down to the the source node. A man in the middle could sit there and also read off the queue. It could modify a message, with its own response queue, and happily tranfer things back and forth. So, we have the HMAC proposal, which then puts crypto and key distribution all over the place. Yes, it would guard against a MITM attack, but the cost in complexity and processor time it high. Rabbit does not have a very flexible ACL scheme, bascially, a RegEx per Rabbit user. However, we could easily spin up a new queue for direct node to node communication that did meet an ACL regex. For example, if we said that the regex was that the node could only read/write queues that have its name in them, to make a request and response queue between node-1 and node-2 we could create a queues node-1-node-2 node-1-node-2--reply So, instead of a single queue request, there are two. And conductor could tell the target node: start listening on this queue. Or, we could pass the message through the conductor. The request message goes from node-1 to conductor, where conductor validates the businees logic of the message, then puts it into the message queue for node-2. Responses can then go directly back from node-2 to node-1 the way they do now. OR...we could set up a direct socket between the two nodes, with the socket set up info going over the broker. OR we could use a web server, OR send it over SNMP. Or SMTP, OR TFTP. There are many ways to get the messages from node to node. If we are going to use the message broker to do this, we should at least make it possible to secure it, even if it is not the default approach. It might be possible to use a broker specific technology to optimize this, but I am not a Rabbit expert. Maybe there is some way of filtering messages? 1. Compute nodes can only read from the queue compute. -novacompute-.localdomain 2. Compute nodes can only write to response queues in the RPC vhost 3. Com
Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
On 03/22/2016 05:42 PM, Dan Smith wrote: Shouldn't we be trying to remove central bottlenecks by decentralizing communications where we can? I think that's a good goal to continue having. Some deployers have setup firewalls between compute nodes, or between compute nodes and the database, so we use the conductor to facilitate communications between those nodes. But in general we don't want to send all communications through the conductor. Yep, I think we generally look forward to having all the resize and migrate communication coordinated through conductor, but not really for security reasons specifically. However, I don't think that pumping everything through conductor for, say, api->compute communication is something we should do. So, Api to compute is probably fine as is. I assume that most of that goes in the same queue as the conductor uses. This assumes that we equally trust conductor and the API server, but I think if either is compromised, all bets are off anyway. As several of us said in IRC yesterday, I'd really like nodes to be able to authenticate the sender of a message and not do things based on who sent it and whether that makes sense or not. I read that as "we want to do HMAC outside of the Queue" and,as I said before, we tried that. No one picked it up, Key distribution is a nightmare, and unless you do asymmetric cryptography, you need to have a separate shared secret for each reader and writer: there is no pub-sub with symmetric crypto. And we should not be rolling our own security. Adding a bunch of broker-specific configuration requirements to achieve a security goal (and thus assuming the queue is never compromised) is not really where I want to see us go. Nothing here is broker specific. The rules are the same for Rabbit, QPID and 0MQ. Message Brokers are a key piece of technology in a lot of enterprise software. It is possible to secure them. Denying the operators the ability to secure them because we don't trust the brokers is not fair to the operators. --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
>> Shouldn't we be trying to remove central bottlenecks by >> decentralizing communications where we can? > > I think that's a good goal to continue having. Some deployers have > setup firewalls between compute nodes, or between compute nodes and > the database, so we use the conductor to facilitate communications > between those nodes. But in general we don't want to send all > communications through the conductor. Yep, I think we generally look forward to having all the resize and migrate communication coordinated through conductor, but not really for security reasons specifically. However, I don't think that pumping everything through conductor for, say, api->compute communication is something we should do. As several of us said in IRC yesterday, I'd really like nodes to be able to authenticate the sender of a message and not do things based on who sent it and whether that makes sense or not. Adding a bunch of broker-specific configuration requirements to achieve a security goal (and thus assuming the queue is never compromised) is not really where I want to see us go. --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
On 03/22/2016 09:15 AM, Flavio Percoco wrote: On 21/03/16 21:43 -0400, Adam Young wrote: I had a good discussion with the Nova folks in IRC today. My goal was to understand what could talk to what, and the short according to dansmith " any node in nova land has to be able to talk to the queue for any other one for the most part: compute->compute, compute->conductor, conductor->compute, api->everything. There might be a few exceptions, but not worth it, IMHO, in the current architecture." Longer conversation is here: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 Right now, the message queue is a nightmare. All sorts of sensitive information flows over the message queue: Tokens (including admin) are the most obvious. Every piece of audit data. All notifications and all control messages. Before we continue down the path of "anything can talk to anything" can we please map out what needs to talk to what, and why? Many of the use cases seem to be based on something that should be kicked off by the conductor, such as "migrate, resize, live-migrate" and it sounds like there are plans to make that happen. So, let's assume we can get to the point where, if node 1 needs to talk to node 2, it will do so only via the conductor. With that in place, we can put an access control rule in place: I don't think this is going to scale well. Eventually, this will require evolving the conductor to some sort of message scheduler, which is pretty much what the message bus is supposed to do. I'll limit this to what happens with Rabbit and QPID (AMQP1.0) and leave 0 our of it for now. I'll use rabbit as shorthand for both these, but the rules are the same for qpid. For, say, a migrate operation, the call goes to API, controller, and eventually down to one of the compute nodes. Source? Target? I don't know the code well enough to say, but let's say it is the source. It sends an RPC message to the target node. The message goes to the central broker right now, and then back down to the targen node. Meanwhile, the source node has set up a reply queue and that queue name has gone into the message. The target machine responds by getting a reference to the response queue and sends a message. This message goes up to the broker, and then down to the the source node. A man in the middle could sit there and also read off the queue. It could modify a message, with its own response queue, and happily tranfer things back and forth. So, we have the HMAC proposal, which then puts crypto and key distribution all over the place. Yes, it would guard against a MITM attack, but the cost in complexity and processor time it high. Rabbit does not have a very flexible ACL scheme, bascially, a RegEx per Rabbit user. However, we could easily spin up a new queue for direct node to node communication that did meet an ACL regex. For example, if we said that the regex was that the node could only read/write queues that have its name in them, to make a request and response queue between node-1 and node-2 we could create a queues node-1-node-2 node-1-node-2--reply So, instead of a single queue request, there are two. And conductor could tell the target node: start listening on this queue. Or, we could pass the message through the conductor. The request message goes from node-1 to conductor, where conductor validates the businees logic of the message, then puts it into the message queue for node-2. Responses can then go directly back from node-2 to node-1 the way they do now. OR...we could set up a direct socket between the two nodes, with the socket set up info going over the broker. OR we could use a web server, OR send it over SNMP. Or SMTP, OR TFTP. There are many ways to get the messages from node to node. If we are going to use the message broker to do this, we should at least make it possible to secure it, even if it is not the default approach. It might be possible to use a broker specific technology to optimize this, but I am not a Rabbit expert. Maybe there is some way of filtering messages? 1. Compute nodes can only read from the queue compute.-novacompute-.localdomain 2. Compute nodes can only write to response queues in the RPC vhost 3. Compute nodes can only write to notification queus in the notification host. I know that with AMQP, we should be able to identify the writer of a message. This means that each compute node should have its own user. I have identified how to do that for Rabbit and QPid. I assume for 0mq is would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd rather the 0mq maintainers chime in here. NOTE: Gentle reminder that qpidd has been removed from oslo.messaging. Yes, but QPID is proton is AMQP1.0 and I did a proof of concept with it last summer. It supports encryption and authentication over GSSAPI and i
Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
On 03/21/2016 09:43 PM, Adam Young wrote: I had a good discussion with the Nova folks in IRC today. My goal was to understand what could talk to what, and the short according to dansmith " any node in nova land has to be able to talk to the queue for any other one for the most part: compute->compute, compute->conductor, conductor->compute, api->everything. There might be a few exceptions, but not worth it, IMHO, in the current architecture." Longer conversation is here: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 Right now, the message queue is a nightmare. All sorts of sensitive information flows over the message queue: Tokens (including admin) are the most obvious. Every piece of audit data. All notifications and all control messages. It is indeed a nightmare, but not because of any security issues. Solve security issues by isolating the management plane over which the messages are sent. Do this using standard industry practice of firewall rules. Do NOT do this by adding ever more complexity to the setup and configuration of the message queue itself. -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
On 21/03/16 21:43 -0400, Adam Young wrote: I had a good discussion with the Nova folks in IRC today. My goal was to understand what could talk to what, and the short according to dansmith " any node in nova land has to be able to talk to the queue for any other one for the most part: compute->compute, compute->conductor, conductor->compute, api->everything. There might be a few exceptions, but not worth it, IMHO, in the current architecture." Longer conversation is here: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 Right now, the message queue is a nightmare. All sorts of sensitive information flows over the message queue: Tokens (including admin) are the most obvious. Every piece of audit data. All notifications and all control messages. Before we continue down the path of "anything can talk to anything" can we please map out what needs to talk to what, and why? Many of the use cases seem to be based on something that should be kicked off by the conductor, such as "migrate, resize, live-migrate" and it sounds like there are plans to make that happen. So, let's assume we can get to the point where, if node 1 needs to talk to node 2, it will do so only via the conductor. With that in place, we can put an access control rule in place: I don't think this is going to scale well. Eventually, this will require evolving the conductor to some sort of message scheduler, which is pretty much what the message bus is supposed to do. 1. Compute nodes can only read from the queue compute.-novacompute-.localdomain 2. Compute nodes can only write to response queues in the RPC vhost 3. Compute nodes can only write to notification queus in the notification host. I know that with AMQP, we should be able to identify the writer of a message. This means that each compute node should have its own user. I have identified how to do that for Rabbit and QPid. I assume for 0mq is would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd rather the 0mq maintainers chime in here. NOTE: Gentle reminder that qpidd has been removed from oslo.messaging. I think you can configure rabbit, amqp1 and other technologies to do what you're suggesting here without much trouble. TBH, I'm not sure how many chances would be required in Nova (or even oslo.messaging) but I'd dare to say non are required. I think it is safe (and sane) to have the same use on the compute node communicate with Neutron, Nova, and Ceilometer. This will avoid a false sense of security: if one is compromised, they are all going to be compromised. Plan accordingly. Beyond that, we should have message broker users for each of the components that is a client of the broker. Applications that run on top of the cloud, and that do not get presence on the compute nodes, should have their own VHost. I see Sahara on my Tripleo deploy, but I assume there are others. Either they completely get their own vhost, or the apps should share one separate from the RPC/Notification vhosts we currently have. Even Heat might fall into this category. Note that those application users can be allowed to read from the notification queues if necessary. They just should not be using the same vhost for their own traffic. Please tell me if/where I am blindingly wrong in my analysis. I guess my question is: Have you identified things that need to be changed in any of the projects for this to be possible? Or is it a pure deployment recommendation/decision? I'd argue that any change (assuming changes are required) are likely to happen in specific projects (Nova, Neutron, etc) and that once this scenario is supported, it'll remain a deployment choice to follow it or not. If I want my undercloud services to use a single vhost and a single user, I must be able to do that. The proposal in this email complicates deployments significantly, despite it making sense from a security stand point. One more thing. Depending on the messaging technology, having different virtual hosts may have an impact on the performance when running under huge loads given the fact that the data will be partitioned differently and, therefore, written/read differently. I don't have good data at hand about this, sorry. Flavio -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
On Tue, Mar 22, 2016, at 07:00 AM, Doug Hellmann wrote: > > > > On Mar 21, 2016, at 9:43 PM, Adam Young wrote: > > > > I had a good discussion with the Nova folks in IRC today. > > > > My goal was to understand what could talk to what, and the short according > > to dansmith > > > > " any node in nova land has to be able to talk to the queue for any other > > one for the most part: compute->compute, compute->conductor, > > conductor->compute, api->everything. There might be a few exceptions, but > > not worth it, IMHO, in the current architecture." > > > > Longer conversation is here: > > http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 > > > > Right now, the message queue is a nightmare. All sorts of sensitive > > information flows over the message queue: Tokens (including admin) are the > > most obvious. Every piece of audit data. All notifications and all control > > messages. > > > > Before we continue down the path of "anything can talk to anything" can we > > please map out what needs to talk to what, and why? Many of the use cases > > seem to be based on something that should be kicked off by the conductor, > > such as "migrate, resize, live-migrate" and it sounds like there are plans > > to make that happen. > > > > So, let's assume we can get to the point where, if node 1 needs to talk to > > node 2, it will do so only via the conductor. With that in place, we can > > put an access control rule in place: If you specifically mean compute nodes by "node 1" and "node 2" that is something that's being worked towards. If you mean "node" more generally that's not something that is planned. > > Shouldn't we be trying to remove central bottlenecks by decentralizing > communications where we can? I think that's a good goal to continue having. Some deployers have setup firewalls between compute nodes, or between compute nodes and the database, so we use the conductor to facilitate communications between those nodes. But in general we don't want to send all communications through the conductor. > > Doug > > > > > > 1. Compute nodes can only read from the queue > > compute.-novacompute-.localdomain > > 2. Compute nodes can only write to response queues in the RPC vhost > > 3. Compute nodes can only write to notification queus in the notification > > host. > > > > I know that with AMQP, we should be able to identify the writer of a > > message. This means that each compute node should have its own user. I > > have identified how to do that for Rabbit and QPid. I assume for 0mq is > > would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd rather > > the 0mq maintainers chime in here. > > > > I think it is safe (and sane) to have the same use on the compute node > > communicate with Neutron, Nova, and Ceilometer. This will avoid a false > > sense of security: if one is compromised, they are all going to be > > compromised. Plan accordingly. > > > > Beyond that, we should have message broker users for each of the components > > that is a client of the broker. > > > > Applications that run on top of the cloud, and that do not get presence on > > the compute nodes, should have their own VHost. I see Sahara on my Tripleo > > deploy, but I assume there are others. Either they completely get their > > own vhost, or the apps should share one separate from the RPC/Notification > > vhosts we currently have. Even Heat might fall into this category. > > > > Note that those application users can be allowed to read from the > > notification queues if necessary. They just should not be using the same > > vhost for their own traffic. > > > > Please tell me if/where I am blindingly wrong in my analysis. > > > > > > > > > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
On 21/03/2016 9:43 PM, Adam Young wrote: > I had a good discussion with the Nova folks in IRC today. > > My goal was to understand what could talk to what, and the short > according to dansmith > > " any node in nova land has to be able to talk to the queue for any > other one for the most part: compute->compute, compute->conductor, > conductor->compute, api->everything. There might be a few exceptions, > but not worth it, IMHO, in the current architecture." > > Longer conversation is here: > > http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 > > Right now, the message queue is a nightmare. All sorts of sensitive > information flows over the message queue: Tokens (including admin) are > the most obvious. Every piece of audit data. All notifications and all > control messages. > > Before we continue down the path of "anything can talk to anything" can > we please map out what needs to talk to what, and why? Many of the use > cases seem to be based on something that should be kicked off by the > conductor, such as "migrate, resize, live-migrate" and it sounds like > there are plans to make that happen. > i think the community is split on "anything can talk to anything". this was a side topic[1] on another thread a few months ago regarding versioned notifications(FUN!). i agree with you that we should map what talks with what and why. i personally think it would reduce load and security risk on queue since the messages and the content would be tailored to consumer rather than the current grab bag dump. the counter argument was that it's a lot less flexible and contrary to the purpose of pub/sub. [1] http://lists.openstack.org/pipermail/openstack-dev/2015-November/080215.html cheers, -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
> On Mar 21, 2016, at 9:43 PM, Adam Young wrote: > > I had a good discussion with the Nova folks in IRC today. > > My goal was to understand what could talk to what, and the short according to > dansmith > > " any node in nova land has to be able to talk to the queue for any other one > for the most part: compute->compute, compute->conductor, conductor->compute, > api->everything. There might be a few exceptions, but not worth it, IMHO, in > the current architecture." > > Longer conversation is here: > http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 > > Right now, the message queue is a nightmare. All sorts of sensitive > information flows over the message queue: Tokens (including admin) are the > most obvious. Every piece of audit data. All notifications and all control > messages. > > Before we continue down the path of "anything can talk to anything" can we > please map out what needs to talk to what, and why? Many of the use cases > seem to be based on something that should be kicked off by the conductor, > such as "migrate, resize, live-migrate" and it sounds like there are plans to > make that happen. > > So, let's assume we can get to the point where, if node 1 needs to talk to > node 2, it will do so only via the conductor. With that in place, we can put > an access control rule in place: Shouldn't we be trying to remove central bottlenecks by decentralizing communications where we can? Doug > > 1. Compute nodes can only read from the queue > compute.-novacompute-.localdomain > 2. Compute nodes can only write to response queues in the RPC vhost > 3. Compute nodes can only write to notification queus in the notification > host. > > I know that with AMQP, we should be able to identify the writer of a message. > This means that each compute node should have its own user. I have > identified how to do that for Rabbit and QPid. I assume for 0mq is would > make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd rather the 0mq > maintainers chime in here. > > I think it is safe (and sane) to have the same use on the compute node > communicate with Neutron, Nova, and Ceilometer. This will avoid a false > sense of security: if one is compromised, they are all going to be > compromised. Plan accordingly. > > Beyond that, we should have message broker users for each of the components > that is a client of the broker. > > Applications that run on top of the cloud, and that do not get presence on > the compute nodes, should have their own VHost. I see Sahara on my Tripleo > deploy, but I assume there are others. Either they completely get their own > vhost, or the apps should share one separate from the RPC/Notification vhosts > we currently have. Even Heat might fall into this category. > > Note that those application users can be allowed to read from the > notification queues if necessary. They just should not be using the same > vhost for their own traffic. > > Please tell me if/where I am blindingly wrong in my analysis. > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing
I had a good discussion with the Nova folks in IRC today. My goal was to understand what could talk to what, and the short according to dansmith " any node in nova land has to be able to talk to the queue for any other one for the most part: compute->compute, compute->conductor, conductor->compute, api->everything. There might be a few exceptions, but not worth it, IMHO, in the current architecture." Longer conversation is here: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 Right now, the message queue is a nightmare. All sorts of sensitive information flows over the message queue: Tokens (including admin) are the most obvious. Every piece of audit data. All notifications and all control messages. Before we continue down the path of "anything can talk to anything" can we please map out what needs to talk to what, and why? Many of the use cases seem to be based on something that should be kicked off by the conductor, such as "migrate, resize, live-migrate" and it sounds like there are plans to make that happen. So, let's assume we can get to the point where, if node 1 needs to talk to node 2, it will do so only via the conductor. With that in place, we can put an access control rule in place: 1. Compute nodes can only read from the queue compute.-novacompute-.localdomain 2. Compute nodes can only write to response queues in the RPC vhost 3. Compute nodes can only write to notification queus in the notification host. I know that with AMQP, we should be able to identify the writer of a message. This means that each compute node should have its own user. I have identified how to do that for Rabbit and QPid. I assume for 0mq is would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd rather the 0mq maintainers chime in here. I think it is safe (and sane) to have the same use on the compute node communicate with Neutron, Nova, and Ceilometer. This will avoid a false sense of security: if one is compromised, they are all going to be compromised. Plan accordingly. Beyond that, we should have message broker users for each of the components that is a client of the broker. Applications that run on top of the cloud, and that do not get presence on the compute nodes, should have their own VHost. I see Sahara on my Tripleo deploy, but I assume there are others. Either they completely get their own vhost, or the apps should share one separate from the RPC/Notification vhosts we currently have. Even Heat might fall into this category. Note that those application users can be allowed to read from the notification queues if necessary. They just should not be using the same vhost for their own traffic. Please tell me if/where I am blindingly wrong in my analysis. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev