Re: [Openstack] nova cells minimal setup
On May 29, 2013, at 11:39 PM, Chris Behrens cbehr...@codestud.com wrote: On May 28, 2013, at 8:38 PM, Markus Barth launchpad@boecht.de wrote: Hello everyone, Nova-cells filter has been merged now, so I'd like to make use of them and set up an installation for a proof of concept. My goal is to create new filters and then spawn instances in a demo. Awesome. I'd be happy to assist you here with your current (and future) questions. Yay! Setup: - Global: Horizon, Keystone, Glance - API cell: nova-api, nova-conductor, nova-cells, nova-network (flat), mysql, rabbitmq - child cells: nova-cells, nova-conductor, nova-cert, nova-scheduler, nova-compute-qemu; cinder-*, mysql, rabbitmq There's a number of limitations with cells right now, and the above won't *quite* work… but close. The only issue that I see above is with cinder and nova-network. Further info below. 1. Is the above all services I need to spawn instances from Horizon without errors? Generally, yes.. those are the correct services in the correct cells… *except* nova-network *might* work, but it would have to be in each child cell and you'd need different IP blocks configured in each nova-network in order to work. They can all be the same layer 2 network, however. The limitation is because compute needs to talk to nova-network via RPC in order to assign IP addresses… and RPC communication between nova-compute and other services is intra-cell only. And if you tried to configure the same /24 in 2 different child cells, since they are separate DBs, you'll get the same IPs handed out in multiple child cells. More below under #3. I did think of that, but since Horizon needs to get a tenant's networks from a single quantum-server, I thought I'd make Quantum a global installation. I know instances would not get IP adresses, but I don't actually need them to. I'm currently planning that the global services and each cell are in separate subnets. I guessed if there was Quantum in each cell it would be impossible for Horizon to talk to all of them and then join the configured networks in all cells into one. Or would it be enough to have a global quantum database with Quantum in each cell and Horizon pointing to a random (or an additional global) installation to have Quantum working? Then nova-compute can actually use the local Quantum. 2. Do I need cinder-* (or nova-volume)? Is it really optional now? No, you do not need them if you use local storage for your VMs. In fact, cinder does not currently work with cells. I have a patch for this that I need to clean up and submit. 3. Do I need nova-network (or quantum-*)? I know quantum-* does not have to run when spawning instances, but I guess I need it to create a network for a tenant in the first place!? Quantum is a better choice vs nova-network with or without cells...as nova-network will be deprecated some day (AFAIK). If you choose Quantum, you can set this up as a global service to use with cells as long as all of your child cells are on the same layer 2 network. IP address assignments would be managed globally, etc. If you choose quantum and set it up globally, the API (nova-api) extensions for networking should work. In the future, we'll need to find a good way to allow Quantum to work with a config such that different cells can be on different layer 2 networks. Then Quantum it is. I just thought I'd keep it simple and go with nova-network and probably avoid some problems. Hope that helps so far, It really does, thanks! Markus ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] nova cells minimal setup
Hello everyone, Nova-cells filter has been merged now, so I'd like to make use of them and set up an installation for a proof of concept. My goal is to create new filters and then spawn instances in a demo. Setup: - Global: Horizon, Keystone, Glance - API cell: nova-api, nova-conductor, nova-cells, nova-network (flat), mysql, rabbitmq - child cells: nova-cells, nova-conductor, nova-cert, nova-scheduler, nova-compute-qemu; cinder-*, mysql, rabbitmq 1. Is the above all services I need to spawn instances from Horizon without errors? 2. Do I need cinder-* (or nova-volume)? Is it really optional now? 3. Do I need nova-network (or quantum-*)? I know quantum-* does not have to run when spawning instances, but I guess I need it to create a network for a tenant in the first place!? Thanks for your help Markus ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Cells
Hi Chris, What happens to notifications to other compute servers that are generated as side effect of VM creation a result of using the IPtables firewall driver ? Are they somehow propagated to other Cells, or is there something that keeps all VMs in a particular security group within a Cell ? I looked in the doc and code, but I couldn't see anything that seems to indicate either way. Thanks Phil From: openstack-bounces+philip.day=hp@lists.launchpad.net [mailto:openstack-bounces+philip.day=hp@lists.launchpad.net] On Behalf Of Chris Behrens Sent: 12 July 2012 20:13 To: Michael J Fork Cc: openstack@lists.launchpad.net; Chris Behrens Subject: Re: [Openstack] Nova Cells Partially developed. This probably isn't much use, but I'll throw it out there: http://comstud.com/cells.pdf ATM the messy code speaks for itself here: https://github.com/comstud/nova/tree/cells_service The basic architecture is: Top level cell with API service has DB, rabbit, and the nova-cells service. API's compute_api_class is overridden to use a new class that shoves every action on an instance into the nova-cells service, telling it which cell to route the request to based on instance['cell_name']. The nova-cells service routes the request to correct cell as requested... 1 hop at a time to the nova-cells service in each child. (Each child runs this new nova-cells service also) If nova-cells service gets a message destined for itself, it'll call the appropriate compute_api call in the child. DB updates are hooked in the child and pushed up to parent cells. New instance creation is slightly different. API will create the DB entry up front... and pass the uuid and all of the same data to the nova-cells service, which will pick a cell for the instance. When it is decided to use the 'current cell' in some child, it will create the DB entry there as well... push a notification upward... and cast the message over to the host scheduler (current scheduler). And the build continues as normal from there (host is picked, and message is casted to the host, etc). There's some code to sync instances in case of lost DB updates.. but there's improvements to make yet.. Sorry... that's very quick. I'm going to be AFK for a couple days.. - Chris On Jul 12, 2012, at 10:39 AM, Michael J Fork wrote: Outside of the Etherpad (http://etherpad.openstack.org/FolsomComputeCells) and presentation referenced there (http://comstud.com/FolsomCells.pdf), are there additional details available on the architecture / implementation of Cells? Thanks. Michael - Michael Fork Cloud Architect, Emerging Solutions IBM Systems Technology Group ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Nova Cells
Outside of the Etherpad (http://etherpad.openstack.org/FolsomComputeCells) and presentation referenced there (http://comstud.com/FolsomCells.pdf), are there additional details available on the architecture / implementation of Cells? Thanks. Michael - Michael Fork Cloud Architect, Emerging Solutions IBM Systems Technology Group___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Cells
Partially developed. This probably isn't much use, but I'll throw it out there: http://comstud.com/cells.pdf ATM the messy code speaks for itself here: https://github.com/comstud/nova/tree/cells_service The basic architecture is: Top level cell with API service has DB, rabbit, and the nova-cells service. API's compute_api_class is overridden to use a new class that shoves every action on an instance into the nova-cells service, telling it which cell to route the request to based on instance['cell_name']. The nova-cells service routes the request to correct cell as requested… 1 hop at a time to the nova-cells service in each child. (Each child runs this new nova-cells service also) If nova-cells service gets a message destined for itself, it'll call the appropriate compute_api call in the child. DB updates are hooked in the child and pushed up to parent cells. New instance creation is slightly different. API will create the DB entry up front… and pass the uuid and all of the same data to the nova-cells service, which will pick a cell for the instance. When it is decided to use the 'current cell' in some child, it will create the DB entry there as well… push a notification upward… and cast the message over to the host scheduler (current scheduler). And the build continues as normal from there (host is picked, and message is casted to the host, etc). There's some code to sync instances in case of lost DB updates.. but there's improvements to make yet.. Sorry… that's very quick. I'm going to be AFK for a couple days.. - Chris On Jul 12, 2012, at 10:39 AM, Michael J Fork wrote: Outside of the Etherpad (http://etherpad.openstack.org/FolsomComputeCells) and presentation referenced there (http://comstud.com/FolsomCells.pdf), are there additional details available on the architecture / implementation of Cells? Thanks. Michael - Michael Fork Cloud Architect, Emerging Solutions IBM Systems Technology Group ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Cells
That's a good question. I'm also interested in an update on cells. How is progress on cells going? Is there a blueprint for it? Is it targeted to a folsom milestone? Thanks, Nate On Thu, Jul 12, 2012 at 1:39 PM, Michael J Fork mjf...@us.ibm.com wrote: Outside of the Etherpad (http://etherpad.openstack.org/FolsomComputeCells) and presentation referenced there (http://comstud.com/FolsomCells.pdf), are there additional details available on the architecture / implementation of Cells? Thanks. Michael - Michael Fork Cloud Architect, Emerging Solutions IBM Systems Technology Group ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Cells
On Thursday, July 12, 2012 at 15:13 PM, Chris Behrens wrote: Partially developed. This probably isn't much use, but I'll throw it out there: http://comstud.com/cells.pdf We're going to have to sync once more on removing _to_server calls from the RPC layer. With the matchmaker upstream now, we should be able to use this to provide N-broker support to the AMQP drivers, although we'd need to work this into ampq.py. Also, since the design summit, I should note the code has moved in the direction of providing a Bindings/Exchanges metaphor, which I hope should be easier to digest from the perspective of the queue-server buffs. Let me know when you're ready to have a chat about it, it might do better to do this on the phone or IRC than by email. -- Eric Windisch ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Cells
Sorry about this. I've had other priorities at Rackspace lately, but I have a functioning implementation that I can hope to start to merge ASAP. I'm on vacation for a couple days, so I can provide a better update on Monday. On Jul 12, 2012, at 4:29 PM, Jaesuk Ahn bluejay@gmail.com wrote: +1 here. I am also highly interested in an update on cells implementation since the last design summit. I have not seen any blueprints nor implementation progress update from the community. We have been reviewing all the cell related docs and info. and have been trying to make potential reference use cases. It will be super helpful if someone directs me to any kinds of update regarding cell. Thanks in advance. -- Jaesuk Ahn, Ph.D. Team Leader | Cloud OS Dev. Team Cloud Infrastructure Department KT (Korea Telecom) T. +82-10-9888-0328 | F. +82-303-0993-5340 Active member on OpenStack Korea Community On Fri, Jul 13, 2012 at 4:47 AM, Nathanael Burton nathanael.i.bur...@gmail.com wrote: That's a good question. I'm also interested in an update on cells. How is progress on cells going? Is there a blueprint for it? Is it targeted to a folsom milestone? Thanks, Nate On Thu, Jul 12, 2012 at 1:39 PM, Michael J Fork mjf...@us.ibm.com wrote: Outside of the Etherpad (http://etherpad.openstack.org/FolsomComputeCells) and presentation referenced there (http://comstud.com/FolsomCells.pdf), are there additional details available on the architecture / implementation of Cells? Thanks. Michael - Michael Fork Cloud Architect, Emerging Solutions IBM Systems Technology Group ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
