[Openstack] TC candidacy
Hi all, I'd like to announce my candidacy for a seat on the OpenStack Technical Committee. - General background - I have over 15 years of experience designing and building distributed systems. I am currently a Senior Software Developer at Rackspace, where I have been for a 2 and a half years now. Most of my time at Rackspace has been spent working on OpenStack as both a developer and a technical leader. My first week at Rackspace was spent at the very first OpenStack Design Summit in Austin where the project was announced. Prior to working at Rackspace, I held various roles over 14 years at Concentric Network Corporation/XO Communications including Senior Software Architect and eventually Director of Engineering. My main focus there was on an award winning web/email hosting platform which we'd built to be extremely scalable and fault tolerant. While my name is not on this patent, I was heavily involved with the development and design that led to US6611861. - Why am I interested? - I have strong feelings for OpenStack and I want to help take it to the next level. I have a lot of technical knowledge and experience building scalable distributed systems. During most of my past experience, I haven't had the luxury of having access to a lot extremely fast hardware, so it's been important to make software as performant as possible. I've also had to put lots of effort into having 0 downtime, meaning code can be updated seamlessly without dropping clients. I've also been one to lead host and software security efforts so I have a lot of strong feelings in this area. I am extremely interested in using this experience to make OpenStack perform well, be secure, be more easily pluggable, and easy to use! - OpenStack contributions - As I mentioned above, I was at the very first design summit, so I've been involved with the project from the beginning. I started the initial work for nova-scheduler shortly after the project was opened. I also implemented the RPC support for kombu, making sure to properly support reconnecting and so forth which didn't work quite so well with the carrot code. I've contributed a number of improvements designed to make nova-api more performant. I've worked on the filter scheduler as well as designing and implementing the first version of the Zones replacement that we named 'Cells'. I'm currently looking forward to restructuring our use of DB API to better support upgrades w/ schema changes as well as committing an alternative DB backend implementation for mysql that significantly reduces how long we block on DB API calls compared to sqlalchemy. - Summary - I feel my years of experience contributing to and leading large scale technical projects along with my knowledge of the OpenStack projects will provide a good foundation for technical leadership. Thanks, - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] AggregateInstanceExtraSpecs very slow?
I am not understanding why there are secondary joins defined in the models. I suspect this might break other things, but maybe you can test that this at least makes the scheduling faster: http://paste.openstack.org/show/32534/ That seems to generate a much more acceptable query. - Chris On Feb 25, 2013, at 9:40 PM, Sam Morrison sorri...@gmail.com wrote: On 26/02/2013, at 4:31 PM, Chris Behrens cbehr...@codestud.com wrote: After thinking more, it does seem like we're doing something wrong if the query itself is returning 300k rows. :) I can take a better look at it in front of the computer later if no one beats me to it. Yeah I think it's more than a missing index :-) The query does 2 INNER JOINS on aggregate_hosts then 2 INNER JOINS on aggregate_metadata then does a further 2 LEFT OUTER JOINS on aggregate_hosts. Thanks for the help, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] AggregateInstanceExtraSpecs very slow?
On Feb 25, 2013, at 6:39 PM, Joe Gordon j...@cloudscaling.com wrote: It looks like the scheduler issues are related to the rabbitmq issues. host 'qh2-rcc77' ... is disabled or has not been heard from in a while What does 'nova host-list' say? the clocks must all be synced up? Good things to check. It feels like something is spinning way too much within this filter, though. This can also cause the above message. The scheduler pulls all of the records before it starts filtering… and if there's a huge delay somewhere, it can start seeing a bunch of hosts as disabled. The filter doesn't look like a problem.. unless there's a large amount of aggregate metadata… and/or a large amount of key/values for the instance_type's extra specs. There *is* a DB call in the filter. If that's blocking for an extended period of time, the whole process is blocked… But I suspect by the '100% cpu' comment, that this is not the case… So the only thing I can think of is that it returns a tremendous amount of metadata. Adding some extra logging in the filter could be useful. - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] AggregateInstanceExtraSpecs very slow?
After thinking more, it does seem like we're doing something wrong if the query itself is returning 300k rows. :) I can take a better look at it in front of the computer later if no one beats me to it. On Feb 25, 2013, at 9:28 PM, Chris Behrens cbehr...@codestud.com wrote: Replying from my phone, so I can't look, but I wonder if we have an index missing. On Feb 25, 2013, at 8:54 PM, Sam Morrison sorri...@gmail.com wrote: On Tue, Feb 26, 2013 at 3:15 PM, Sam Morrison sorri...@gmail.com wrote: On 26/02/2013, at 2:15 PM, Chris Behrens cbehr...@codestud.com wrote: On Feb 25, 2013, at 6:39 PM, Joe Gordon j...@cloudscaling.com wrote: It looks like the scheduler issues are related to the rabbitmq issues. host 'qh2-rcc77' ... is disabled or has not been heard from in a while What does 'nova host-list' say? the clocks must all be synced up? Good things to check. It feels like something is spinning way too much within this filter, though. This can also cause the above message. The scheduler pulls all of the records before it starts filtering… and if there's a huge delay somewhere, it can start seeing a bunch of hosts as disabled. The filter doesn't look like a problem.. unless there's a large amount of aggregate metadata… and/or a large amount of key/values for the instance_type's extra specs. There *is* a DB call in the filter. If that's blocking for an extended period of time, the whole process is blocked… But I suspect by the '100% cpu' comment, that this is not the case… So the only thing I can think of is that it returns a tremendous amount of metadata. Adding some extra logging in the filter could be useful. - Chris Thanks Chris, I have 2 aggregates and 2 keys defined and each of the 80 hosts has either one or the other. At the moment every flavour has either one or the other too so I don't think it's too much data. I've tracked it down to this call: metadata = db.aggregate_metadata_get_by_host(context, host_state.host) More debugging has got it down to a query In db.api.aggregate_metadata_get_by_host: query = model_query(context, models.Aggregate).join( _hosts).filter(models.AggregateHost.host == host).join( _metadata) .. rows = query.all() With query debug on this resolves to: SELECT aggregates.created_at AS aggregates_created_at, aggregates.updated_at AS aggregates_updated_at, aggregates.deleted_at AS aggregates_deleted_at, aggregates.deleted AS aggregates_deleted, aggregates.id AS aggregates_id, aggregates.name AS aggregates_name, aggregates.availability_zone AS aggregates_availability_zone, aggregate_hosts_1.created_at AS aggregate_hosts_1_created_at, aggregate_hosts_1.updated_at AS aggregate_hosts_1_updated_at, aggregate_hosts_1.deleted_at AS aggregate_hosts_1_deleted_at, aggregate_hosts_1.deleted AS aggregate_hosts_1_deleted, aggregate_hosts_1.id AS aggregate_hosts_1_id, aggregate_hosts_1.host AS aggregate_hosts_1_host, aggregate_hosts_1.aggregate_id AS aggregate_hosts_1_aggregate_id FROM aggregates INNER JOIN aggregate_hosts AS aggregate_hosts_2 ON aggregates.id = aggregate_hosts_2.aggregate_id AND aggregate_hosts_2.deleted = 0 AND aggregates.deleted = 0 INNER JOIN aggregate_hosts ON aggregate_hosts.aggregate_id = aggregates.id AND aggregate_hosts.deleted = 0 AND aggregates.deleted = 0 INNER JOIN aggregate_metadata AS aggregate_metadata_1 ON aggregates.id = aggregate_metadata_1.aggregate_id AND aggregate_metadata_1.deleted = 0 AND aggregates.deleted = 0 INNER JOIN aggregate_metadata ON aggregate_metadata.aggregate_id = aggregates.id AND aggregate_metadata.deleted = 0 AND aggregates.deleted = 0 LEFT OUTER JOIN aggregate_hosts AS aggregate_hosts_3 ON aggregates.id = aggregate_hosts_3.aggregate_id AND aggregate_hosts_3.deleted = 0 AND aggregates.deleted = 0 LEFT OUTER JOIN aggregate_hosts AS aggregate_hosts_1 ON aggregate_hosts_1.aggregate_id = aggregates.id AND aggregate_hosts_1.deleted = 0 AND aggregates.deleted = 0 WHERE aggregates.deleted = 0 AND aggregate_hosts.host = 'qh2-rcc34'; Which in our case returns 328509 rows in set (25.97 sec) Seems a bit off considering there are 80 rows in aggregate_hosts, 2 rows in aggregates and 2 rows in aggregate_metadata In the code rows is only equal to 1 so it seems to be doing something inside to code to do this? Don't know too much how sqlalchemy works. Seems like a bug to me? or maybe our database has something wrong in it? Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Restart devstack errors
[Removed the dev list -- no need to cross-post.]
It looks like you have broken permissions on
'/usr/local/lib/python2.7/dist-packages/httplib2-0.7.7-py2.7.egg' and/or
subdirectories. Make sure everything is world readable.
- Chris
On Feb 20, 2013, at 7:48 PM, harryxiyou harryxi...@gmail.com wrote:
On Wed, Feb 20, 2013 at 3:14 PM, Hirendra Rathor
hirendra.rat...@gmail.com wrote:
Hi Hirendra Rathor,
I was getting same error when I picked up devstack for the first time few
days ago. I could have tried troubleshooting it but I wasn't particularly
happy with the fact that I had to launch stack.sh every time on reboot. The
script updates the source code and then compiles it which means that the
product behavior could be different after running the script. This is not
necessarily a problem but I felt less in control.
I, therefore, modified my local copy of stack.sh to _not_ do operations that
should be required for the first run. Examples of such operations include
creating user accounts, configuration files, mysql database etc. The result
was a script that just launches the openstack processes. Running this lean
script resulted in successful launch of keystone process, so I didn't have
to troubleshoot the original problem!
I am not clear about how you can solve 'keystone did not start'.
Cloud you please attach your modified stack.sh, which solved this problem.
Now, i can get some logs for my 'keystone did not start' like following.
$ screen -S stack -p key -X stuff
$ screen -x stack
jiawei@jiawei:~/workshop1/devstack$ cd /opt/stack/keystone
/opt/stack/keystone/bin/keystone-all --config-file
/etc/keystone/keystone.conf --log-config /etc/keystone/logging.conf -d
--debug || touch /opt/stack/status/stack/key.failure
Traceback (most recent call last):
File /opt/stack/keystone/bin/keystone-all, line 22, in module
from paste import deploy
File
/usr/local/lib/python2.7/dist-packages/Paste-1.7.5.1-py2.7.egg/paste/__init__.py,
line 4, in module
import pkg_resources
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 2727,
in module
add_activation_listener(lambda dist: dist.activate())
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 700,
in subscribe
callback(dist)
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 2727,
in lambda
add_activation_listener(lambda dist: dist.activate())
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 2227,
in activate
self.insert_on(path)
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 2334,
in insert_on
self.check_version_conflict()
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 2373,
in check_version_conflict
for modname in self._get_metadata('top_level.txt'):
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 2221,
in _get_metadata
for line in self.get_metadata_lines(name):
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 1209,
in get_metadata_lines
return yield_lines(self.get_metadata(name))
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 1201,
in get_metadata
return self._get(self._fn(self.egg_info,name))
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 1316, in _get
stream = open(path, 'rb')
IOError: [Errno 13] Permission denied:
'/usr/local/lib/python2.7/dist-packages/httplib2-0.7.7-py2.7.egg/EGG-INFO/top_level.txt'
Could anyone give me some suggestions to this problem? Thanks
in advance.
--
Thanks
Harry Wei
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Restart devstack errors
Well, you probably don't want world writeable, but :) 755 on dirs and 644 on files is probably more appropriate! But at least you know the issue. - Chris On Feb 20, 2013, at 9:21 PM, harryxiyou harryxi...@gmail.com wrote: On Thu, Feb 21, 2013 at 12:14 PM, Chris Behrens cbehr...@codestud.com wrote: [Removed the dev list -- no need to cross-post.] It looks like you have broken permissions on '/usr/local/lib/python2.7/dist-packages/httplib2-0.7.7-py2.7.egg' nd/or subdirectories. Make sure everything is world readable. You are right. After sudo chmod -R 777 dist-packages/ I could solve this problem successfully. Thanks. -- Thanks Harry Wei ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Reinstating Trey Morris for Nova Core
+1 On Jan 22, 2013, at 5:38 PM, Matt Dietz matt.di...@rackspace.com wrote: All, I think Trey Morris has been doing really well on reviews again, so I'd like to propose him to be reinstated for Nova core. Thoughts? -Dietz ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
The simplest explanation is that Cells gives you a way to tie multiple nova deployments together under a single API. Each cell has its own DB and Rabbit, etc. This was my preso from the last summit: http://comstud.com/FolsomCells.pdf Start of diagrams: http://comstud.com/cells.pdf - Chris On Oct 2, 2012, at 11:09 PM, Endre Karlson endre.karl...@gmail.com wrote: What is a cell contra Aggregate or Zone? Den 3. okt. 2012 07:46 skrev Chris Behrens cbehr...@codestud.com følgende: Yup… looking forward to seeing those contributions! I'm working on getting the branch updated right now. - Chris On Oct 2, 2012, at 10:13 PM, Sam Morrison sorri...@gmail.com wrote: OK great, will be good to get this into master. I have some stuff relating to key pairs, security groups that I'd like to contribute. Also we are looking at the ability for you to specify the cell when booting an instance. Cheers, Sam On 02/10/2012, at 1:06 PM, Chris Behrens cbehr...@codestud.com wrote: Yup, it's done. I just have to deal with some conflicts with our internal branch and my public one.. On Oct 1, 2012, at 7:47 PM, Sam Morrison sorri...@gmail.com wrote: On 02/10/2012, at 12:33 PM, Chris Behrens cbehr...@codestud.com wrote: Thanks, Tom! I have changes to push up that add rpc versioning, etc. Maybe I can get those up tomorrow. Great! I was going to start looking into it but will hold off if you've already done it. Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
First implementation requires global Quantum if you use Quantum… and global Glance. Volumes (Cinder) would also need to be global, but I don't have code appropriate for trunk to support it yet… - Chris On Oct 3, 2012, at 4:14 AM, Endre Karlson endre.karl...@gmail.com wrote: Does this tie into Quantum / Cinder as well or? I guess this will be affected by the same thing? Endre. 2012/10/3 Chris Behrens cbehr...@codestud.com The simplest explanation is that Cells gives you a way to tie multiple nova deployments together under a single API. Each cell has its own DB and Rabbit, etc. This was my preso from the last summit: http://comstud.com/FolsomCells.pdf Start of diagrams: http://comstud.com/cells.pdf - Chris On Oct 2, 2012, at 11:09 PM, Endre Karlson endre.karl...@gmail.com wrote: What is a cell contra Aggregate or Zone? Den 3. okt. 2012 07:46 skrev Chris Behrens cbehr...@codestud.com følgende: Yup… looking forward to seeing those contributions! I'm working on getting the branch updated right now. - Chris On Oct 2, 2012, at 10:13 PM, Sam Morrison sorri...@gmail.com wrote: OK great, will be good to get this into master. I have some stuff relating to key pairs, security groups that I'd like to contribute. Also we are looking at the ability for you to specify the cell when booting an instance. Cheers, Sam On 02/10/2012, at 1:06 PM, Chris Behrens cbehr...@codestud.com wrote: Yup, it's done. I just have to deal with some conflicts with our internal branch and my public one.. On Oct 1, 2012, at 7:47 PM, Sam Morrison sorri...@gmail.com wrote: On 02/10/2012, at 12:33 PM, Chris Behrens cbehr...@codestud.com wrote: Thanks, Tom! I have changes to push up that add rpc versioning, etc. Maybe I can get those up tomorrow. Great! I was going to start looking into it but will hold off if you've already done it. Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
Ok. This took a lot longer to resolve than I expected, but here we go: https://github.com/comstud/nova/tree/cells_service This is rebased against trunk and contains a bunch of new things since the last branch: Random fixes for things that trunk broke with cells (deleting instances for one) RPC versioning (Thanks to Brian Elliott!) Split Replies and Bandwidth Updates into their own queues to better deal with them A number of admin API extensions modified to support cells (Thanks to Dragon, Alex Meade, Brian Lamar, Matt Sherborne, et al) Snapshots/backups query glance in API cell (Thanks to Iccha) Handle quotas in API cell (Thanks to Johannes for fixes!) Things are rapidly getting more kludgy because of changes in trunk that don't have any consideration for cells (because cells is not in trunk!). I'm hoping we can get this into an acceptable shape such that we can get it merged ASAP. - Chris On Oct 2, 2012, at 10:13 PM, Sam Morrison sorri...@gmail.com wrote: OK great, will be good to get this into master. I have some stuff relating to key pairs, security groups that I'd like to contribute. Also we are looking at the ability for you to specify the cell when booting an instance. Cheers, Sam On 02/10/2012, at 1:06 PM, Chris Behrens cbehr...@codestud.com wrote: Yup, it's done. I just have to deal with some conflicts with our internal branch and my public one.. On Oct 1, 2012, at 7:47 PM, Sam Morrison sorri...@gmail.com wrote: On 02/10/2012, at 12:33 PM, Chris Behrens cbehr...@codestud.com wrote: Thanks, Tom! I have changes to push up that add rpc versioning, etc. Maybe I can get those up tomorrow. Great! I was going to start looking into it but will hold off if you've already done it. Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
Yup… looking forward to seeing those contributions! I'm working on getting the branch updated right now. - Chris On Oct 2, 2012, at 10:13 PM, Sam Morrison sorri...@gmail.com wrote: OK great, will be good to get this into master. I have some stuff relating to key pairs, security groups that I'd like to contribute. Also we are looking at the ability for you to specify the cell when booting an instance. Cheers, Sam On 02/10/2012, at 1:06 PM, Chris Behrens cbehr...@codestud.com wrote: Yup, it's done. I just have to deal with some conflicts with our internal branch and my public one.. On Oct 1, 2012, at 7:47 PM, Sam Morrison sorri...@gmail.com wrote: On 02/10/2012, at 12:33 PM, Chris Behrens cbehr...@codestud.com wrote: Thanks, Tom! I have changes to push up that add rpc versioning, etc. Maybe I can get those up tomorrow. Great! I was going to start looking into it but will hold off if you've already done it. Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
Thanks, Tom! I have changes to push up that add rpc versioning, etc. Maybe I can get those up tomorrow. On Oct 1, 2012, at 7:08 PM, Tom Fifield fifie...@unimelb.edu.au wrote: Hi all, As Chris is a rather busy guy, I've taken the liberty of putting up a blueprint and wiki page for Nova Compute Cells. https://blueprints.launchpad.net/nova/+spec/nova-compute-cells http://wiki.openstack.org/blueprint-nova-compute-cells NeCTAR's got to have this feature working well before year-end, and recently Sam Morrison and the team at the University of Melbourne have been working to try and iron out some of the kinks (security group and access key propagation) from Chris' current branch. The plan is to move forward and try and update the code from the comstud repo and the local changes here to get them into master asap. Regards, Tom ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
Yup, it's done. I just have to deal with some conflicts with our internal branch and my public one.. On Oct 1, 2012, at 7:47 PM, Sam Morrison sorri...@gmail.com wrote: On 02/10/2012, at 12:33 PM, Chris Behrens cbehr...@codestud.com wrote: Thanks, Tom! I have changes to push up that add rpc versioning, etc. Maybe I can get those up tomorrow. Great! I was going to start looking into it but will hold off if you've already done it. Cheers, Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
I agree it's disappointing that it's not in trunk yet… and it needs to get in there ASAP. I'm definitely happy to hear that others have been doing a bit of hacking on it, wrt security groups and so forth! I just got back from 2 weeks of vacation and I need to get my branch rebased. There's a couple of things broken because of RPC versioning bumped to 2.0, etc, as well… In any case, it's definitely my plan to get this into trunk extremely soon after Grizzly opens. - Chris On Sep 16, 2012, at 9:32 PM, balaji patnala patnala...@gmail.com wrote: Thanks for thumbs-up on the patch. Sure, we will also start using it. By the way do we have any docs / Info for this feature. On Mon, Sep 17, 2012 at 7:33 AM, Sam Morrison sorri...@gmail.com wrote: We have been using this patch in testing for a few months now. It works pretty well with launching instances etc. It is feature incomplete at the moment though. We have developed syncing of security groups and private keys down do child cells. Also done the live migration add ons for cells. It would be good to get this in master as soon as Folsom gets branched off as we'd like to contribute these back. Cheers, Sam On 16/09/2012, at 5:07 PM, balaji patnala patnala...@gmail.com wrote: Atleast we should have continued with ZONES until CELLS were part of up-stream releases. I was waiting for this feature eagerly to explore more on this. Can we try with the patch listed below. Only one concern with patches are we dont get much help from stack users/testers.. This is little disappointing... On Fri, Sep 14, 2012 at 10:35 PM, Joshua Harlow harlo...@yahoo-inc.com wrote: Ya, is there a session on it at the summit. I'd at least like to talk about it and what it could be in the end. Or maybe we can 'freestyle' that session :-P On 9/14/12 8:36 AM, Russell Bryant rbry...@redhat.com wrote: On 09/14/2012 11:08 AM, Joe Topjian wrote: We didnt find any information related to CELLS [which is planned to replace ZONES] in the latest Folsom pre-release. Can any body give us information on this. Unfortunately, cells was unable to make feature freeze. It should be in Grizzly. Sorry for the delay :/ This is very disappointing. I was looking forward to cells as well. When was this decided and was the decision announced somewhere else? I'd like to know so I can monitor for other announcements like this. The patch is here: https://review.openstack.org/#/c/10707/ It was proposed for inclusion on August 2nd, which was just a couple of weeks before the feature freeze. There were enough significant things that came up in discussion that it just wasn't going to make it in for Folsom. In addition to working through the technical details, it also desperately needs some documentation (ideally both architectural, as well as how to use it). It seems like we should be able to get this all wrapped up for Grizzly, though. -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] TC candidacy
Hi all, I'd like to announce my candidacy for a seat on the OpenStack Technical Committee. - General background - I have over 15 years of experience designing and building distributed systems. I am currently a Senior Software Developer at Rackspace, where I have been for a little over 2 years now. Most of my time at Rackspace has been spent working on OpenStack as both a developer and a technical leader. My first week at Rackspace was spent at the very first OpenStack Design Summit in Austin where the project was announced. Prior to working at Rackspace, I held various roles over 14 years at Concentric Network Corporation/XO Communications including Senior Software Architect and eventually Director of Engineering. My main focus there was on an award winning web/email hosting platform which we'd built to be extremely scalable and fault tolerant. While my name is not on this patent, I was heavily involved with the development and design that led to US6611861. - Why am I interested? - I have strong feelings for OpenStack and I want to help take it to the next level. I have a lot of technical knowledge and experience building scalable distributed systems. Having finished helping launch a large deployment of OpenStack at Rackspace on 8/1, I have the time and support to start making a larger contribution to the community. During most of my past experience, I haven't had the luxury of having access to a lot extremely fast hardware, so it's been important to make software as performant as possible. I've also had to put lots of effort into having 0 downtime, meaning code can be updated seamlessly without dropping clients. I've also been one to lead host and software security efforts so I have a lot of strong feelings in this area. I am extremely interested in using this experience to make OpenStack perform well, be secure, be more easily pluggable, and easy to use! - Relevant Private Source Experience - Unfortunately XO was not very big on open source, so there's a lot of innovative work that I have not been able to publish. But, I've worked on such things as developing multithreaded DNS and MX (mail) servers from scratch. The latter work is notable as it allowed scaling to tens of thousands of simultaneous socket connections to a single mail server, allowing us to be very efficient hardware wise. - Open Source Experience - My initial contributions to open source software date back to 1994, working on EFNet's IRCD daemon, submitting performance improvement patches and features to help EFNet grow to over 100,000 simultaneous clients. Starting in 1998, I rewrote the daemon to make it scale even further, also adding in support for dynamic upgradeability (code upgrades without dropping clients). - OpenStack - As I mentioned above, I was at the very first design summit, so I've been involved with the project from the beginning. I started the initial work for nova-scheduler shortly after the project was opened. I also implemented the RPC support for kombu, making sure to properly support reconnecting and so forth which didn't work quite so well with the carrot code. I've contributed a number of improvements designed to make nova-api more performant. I've worked on the filter scheduler as well as designing and implementing the first version of the Zones replacement that we named 'Cells'. Cells is being used today in production at Rackspace, but it does need a bit of work to fit into trunk (like documentation :). - Summary - I feel my years of experience contributing to and leading large scale technical projects along with my knowledge of the OpenStack projects will provide a good foundation for technical leadership. Thanks, - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
I found time to update the branch with the latest code tonight: https://github.com/comstud/nova/tree/cells_service I put a review up here as a WIP also: https://review.openstack.org/#/c/10707/ I reviewed what's changed since the last update… and it was essentially: Rebase against master… resolving things that had moved to openstack-common Push bandwidth usage updates to top level API cell. Push instance metadata updates to top level API cell (though I don't think they would change in a child cell) instance system metadata syncing with instance updates Better delete instance handling Removed broken near/far filter which could have potential DoS issues, until it can be redone Looks like I might have lost some code cleanups I had done previously… and I'll restore those asap. The cells code as it is in the above branch is what's now running in a production environment and working… but there's still some edge cases of issues and doesn't support things like security groups and host aggregates. I'll give an update tomorrow about trying to land this in folsom… but I'll probably be posting it at the new dev list: openstack-...@lists.openstack.org - Chris On Aug 1, 2012, at 9:03 PM, Chris Behrens wrote: Ah, hit send early from my phone. There's a few additions I have in a private branch along with it being up2date with trunk. Will get that into the public branch and get the update out tomorrow! On Aug 1, 2012, at 9:01 PM, Chris Behrens cbehr...@codestud.com wrote: I'll push up the latest tomorrow, promise! And I'll give an update at that time. Sorry, been crazy times lately preparing for Rackspace's release today. We are live with cells, and I'm extremely anxious to start getting it into trunk. There's been a few additions not in the branch on github. - Chris On Aug 1, 2012, at 8:19 PM, Russell Sim russell@gmail.com wrote: Hey, We have been experimenting with the cells branch and I'm hoping I can get an update. The branch on github hasn't been updated for a couple of months and we are starting to hack on it but we are hesitant because we are aware that there are uncommitted changes. Cheers, Russell ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
I'll push up the latest tomorrow, promise! And I'll give an update at that time. Sorry, been crazy times lately preparing for Rackspace's release today. We are live with cells, and I'm extremely anxious to start getting it into trunk. There's been a few additions not in the branch on github. - Chris On Aug 1, 2012, at 8:19 PM, Russell Sim russell@gmail.com wrote: Hey, We have been experimenting with the cells branch and I'm hoping I can get an update. The branch on github hasn't been updated for a couple of months and we are starting to hack on it but we are hesitant because we are aware that there are uncommitted changes. Cheers, Russell ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cells Status
Ah, hit send early from my phone. There's a few additions I have in a private branch along with it being up2date with trunk. Will get that into the public branch and get the update out tomorrow! On Aug 1, 2012, at 9:01 PM, Chris Behrens cbehr...@codestud.com wrote: I'll push up the latest tomorrow, promise! And I'll give an update at that time. Sorry, been crazy times lately preparing for Rackspace's release today. We are live with cells, and I'm extremely anxious to start getting it into trunk. There's been a few additions not in the branch on github. - Chris On Aug 1, 2012, at 8:19 PM, Russell Sim russell@gmail.com wrote: Hey, We have been experimenting with the cells branch and I'm hoping I can get an update. The branch on github hasn't been updated for a couple of months and we are starting to hack on it but we are hesitant because we are aware that there are uncommitted changes. Cheers, Russell ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance stuck in deleting state with error
You may still have to reset the instance's task_state to NULL in the DB (instances table) to delete ones already in this state. - Chris On Jul 30, 2012, at 1:48 PM, Kevin L. Mitchell wrote: On Mon, 2012-07-30 at 13:03 -0400, Jonathan Proulx wrote: I have an instance that has been in this state for a couple days: | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | deleting| | OS-EXT-STS:vm_state | error | If you're using the Xen driver on trunk, I recently cleaned up a few bugs that might have lead to this problem. Make sure you're updated and try again… -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Capacity based scheduling: What updated free_ram_mb in Folsom
Capacity based scheduling currently adds up all instance usage in host_manager.py There's another review marked as WIP that will be ready when retries go in that cleans that and the capacity tracking up. Look for it from Brian Elliott. On Jul 13, 2012, at 6:38 AM, Day, Phil philip@hp.com wrote: Hi Folks, I was reviewing a code change to add generic retries for build failures ( https://review.openstack.org/#/c/9540/2 ), and wanted to be sure that it wouldn’t invalidate the capacity accounting used by the scheduler. However I've been sitting here for a while working through the Folsom scheduler code trying to understand how the capacity based scheduling now works, and I’m sure I’m missing something obvious but I just can’t work out where the free_ram_mb value in the compute_node table gets updated. I can see the database api method to update the values, compute_node_utilization_update(), it doesn’t look as if anything in the code ever calls that ? From when I last looked at this / various discussions here and at the design summits I thought the approach was that: - The scheduler would make a call (rather than a cast) to the compute manger, which would then do some verification work, update the DB table whilst in the context of that call, and then start a thread to complete the spawn. The need to go all the way to the compute node as a call was to avoid race conditions from multiple schedulers. (the change I’m looking at is part of a blueprint to avoid such a race, so maybe I imagined the change from cast to call ?) - On a delete, the capacity_notifer (which had to be configured into the list_notifier) would detect the delete message, and decrement the database values. But now I look through the code it looks as if the scheduler is still doing a cast (scheduler/driver), and although I can see the database api call to update the values, compute_node_utilization_update(), it doesn’t look as if anything in the code ever calls that ? The ram_filter scheduler seems to use the free_ram_mb value, and that value seems to come from the host_manager in the scheduler which is read from the Database, but I can't for the life of me work out where these values are updated in the Database. The capacity_notifier, which used to decrement values on a VM deletion only (according to the comments the increment was done in the scheduler) seems to have now disappeared altogether in the move of the notifier to openstack/common ? So I’m sure I’m missing some other even more cunning plan on how to keep the values current, but I can’t for the life of me work out what it is – can someone fill me in please ? Thanks, Phil ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Cells
Partially developed. This probably isn't much use, but I'll throw it out there: http://comstud.com/cells.pdf ATM the messy code speaks for itself here: https://github.com/comstud/nova/tree/cells_service The basic architecture is: Top level cell with API service has DB, rabbit, and the nova-cells service. API's compute_api_class is overridden to use a new class that shoves every action on an instance into the nova-cells service, telling it which cell to route the request to based on instance['cell_name']. The nova-cells service routes the request to correct cell as requested… 1 hop at a time to the nova-cells service in each child. (Each child runs this new nova-cells service also) If nova-cells service gets a message destined for itself, it'll call the appropriate compute_api call in the child. DB updates are hooked in the child and pushed up to parent cells. New instance creation is slightly different. API will create the DB entry up front… and pass the uuid and all of the same data to the nova-cells service, which will pick a cell for the instance. When it is decided to use the 'current cell' in some child, it will create the DB entry there as well… push a notification upward… and cast the message over to the host scheduler (current scheduler). And the build continues as normal from there (host is picked, and message is casted to the host, etc). There's some code to sync instances in case of lost DB updates.. but there's improvements to make yet.. Sorry… that's very quick. I'm going to be AFK for a couple days.. - Chris On Jul 12, 2012, at 10:39 AM, Michael J Fork wrote: Outside of the Etherpad (http://etherpad.openstack.org/FolsomComputeCells) and presentation referenced there (http://comstud.com/FolsomCells.pdf), are there additional details available on the architecture / implementation of Cells? Thanks. Michael - Michael Fork Cloud Architect, Emerging Solutions IBM Systems Technology Group ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Cells
Sorry about this. I've had other priorities at Rackspace lately, but I have a functioning implementation that I can hope to start to merge ASAP. I'm on vacation for a couple days, so I can provide a better update on Monday. On Jul 12, 2012, at 4:29 PM, Jaesuk Ahn bluejay@gmail.com wrote: +1 here. I am also highly interested in an update on cells implementation since the last design summit. I have not seen any blueprints nor implementation progress update from the community. We have been reviewing all the cell related docs and info. and have been trying to make potential reference use cases. It will be super helpful if someone directs me to any kinds of update regarding cell. Thanks in advance. -- Jaesuk Ahn, Ph.D. Team Leader | Cloud OS Dev. Team Cloud Infrastructure Department KT (Korea Telecom) T. +82-10-9888-0328 | F. +82-303-0993-5340 Active member on OpenStack Korea Community On Fri, Jul 13, 2012 at 4:47 AM, Nathanael Burton nathanael.i.bur...@gmail.com wrote: That's a good question. I'm also interested in an update on cells. How is progress on cells going? Is there a blueprint for it? Is it targeted to a folsom milestone? Thanks, Nate On Thu, Jul 12, 2012 at 1:39 PM, Michael J Fork mjf...@us.ibm.com wrote: Outside of the Etherpad (http://etherpad.openstack.org/FolsomComputeCells) and presentation referenced there (http://comstud.com/FolsomCells.pdf), are there additional details available on the architecture / implementation of Cells? Thanks. Michael - Michael Fork Cloud Architect, Emerging Solutions IBM Systems Technology Group ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Nova] resource free -vs- allocated utilization?
Hi Jon, There's actually a review up right now proposing to add an OS API extension to be able to give some of this data: https://review.openstack.org/#/c/9544/ I don't know how you may be looking to query it, but it's not too difficult to get it directly from the instances table in the database, either. - Chris On Jul 11, 2012, at 4:50 PM, Jonathan Proulx wrote: is there an easy way to see free -vs- allocated nova resources? for example that 300 of 1000 cpu cores are in use (and similar with memory and storage). Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OVF vs. bare container formats for qcow2 images
That's correct. If there's an .ovf, it's currently ignored. It's also not generated when we upload images (snapshots). I'm sure there's plans in someone's head to implement that support at some point.. but right now we expect the .vhds to have specific names in the tar file. - Chris On Jul 8, 2012, at 6:01 PM, Lorin Hochstein wrote: Paul: I assume you're using XenServer? When I grepped through the code, there appears to be a XenServer plugin for glance for supporting tar balls that could be OVAs, but don't necessarily contain the OVF file (?). Here's the code: https://github.com/openstack/nova/blob/stable/essex/plugins/xenserver/xenapi/etc/xapi.d/plugins/glance#L365 Take care, Lorin -- Lorin Hochstein Lead Architect - Cloud Services Nimbis Services, Inc. www.nimbisservices.comhttps://www.nimbisservices.com/ On Jul 5, 2012, at 11:22 AM, Paul Voccio wrote: Lorin, We've been using OVA packages since the beginning. I believe there is a flag in glance for this. Thanks, ~pvo Paul Voccio paul.voc...@rackspace.commailto:paul.voc...@rackspace.com 770-335-2143 (c) pvo on #openstack ಠ_ಠ On Jul 3, 2012, at 9:07 PM, Lorin Hochstein wrote: On Jun 29, 2012, at 9:53 PM, Adam Young wrote: On 04/01/2012 11:15 AM, Lorin Hochstein wrote: On Mar 29, 2012, at 12:40 PM, Daniel P. Berrange wrote: On Wed, Mar 28, 2012 at 04:41:28PM -0400, Lorin Hochstein wrote: All: Given that I have a qcow2 image from somewhere (e.g., downloaded it from a uec-images.ubuntu.comhttp://uec-images.ubuntu.com/, created one from a raw image using qemu-img) that i want to add to glance: 1. How can I tell whether it's an ovf or bare container format? You are mixing up terminology here. Disk image formats are things like raw, qcow2, vmdk, etc. OVF refers to the format of a metadata file provided alongside the disk image, which describes various requirements for running the image. The two are not tied together at all, merely complementary to each other. Thanks, that clears things up. I was confused by this language, which sounded to me like the metadata was embedded in the disk image file: http://glance.openstack.org/formats.html The container format refers to whether the virtual machine image is in a file format that also contains metadata about the actual virtual machine. In addition, the docs have examples like this, which clearly aren't meaningful: http://glance.openstack.org/glance.html#important-information-about-uploading-images Just to add to the confusion the OVF can contain both the metadata file and the disk image file in a single archived file. An OVF package consists of several files, placed in one directory. A one-file alternative is the OVA package, which is a TAR file with the OVF directory inside. http://en.wikipedia.org/wiki/Open_Virtualization_Format#Technical_description Does anybody know if OpenStack (nova+glance) currently supports OVA packages? Take care, Lorin -- Lorin Hochstein Lead Architect - Cloud Services Nimbis Services, Inc. www.nimbisservices.comhttps://www.nimbisservices.com/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova and asynchronous instance launching
There wasn't a blueprint, but you can see the change here: https://review.openstack.org/#/c/7542/ Bandwidth is updated in a DB table outside of notifications. Notifications just pulls the last data received and sends it. With rapid state changes, I would expect that bandidth_usage would mostly not be different in the messages… unless a bandwidth update in the background happens to sneak in during the middle of the events. In any case… these state change events are noted by 'compute.instance.update'. For actions like 'rebuild', you'll get an 'exists' message when the action starts… but then you'll also see some instance.update events as the states switch. At least this is how I understand it. Besides the code, your best resource for information about notification payloads, etc is this: http://wiki.openstack.org/SystemUsageData - Chris On Jul 2, 2012, at 4:38 AM, Day, Phil wrote: Hi Chris, Thanks for the pointer on the new notification on state change stuff, I'd missed that change. Is there a blueprint or some such which describes the change ? In particular I'm trying to understand how the bandwidth_usage values fit in here. It seems that during a VM creation there would normally be a number of fairly rapid state changes, so re-calculating the bandwidth_usage figures might be quiet expensive jut to log a change in task_state from say Networking to Block Device Mapping. I was kind of expecting that to be more part of the compute.exists messages than the update. Do we have something that catalogues the various notification messages and their payloads ? Thanks, Phil -Original Message- From: Chris Behrens [mailto:cbehr...@codestud.com] Sent: 02 July 2012 00:14 To: Day, Phil Cc: Jay Pipes; Huang Zhiteng; openstack@lists.launchpad.net Subject: Re: [Openstack] Nova and asynchronous instance launching On Jul 1, 2012, at 3:04 PM, Day, Phil philip@hp.com wrote: Rather than adding debug statements could we please add additional notification events (for example a notification event whenever task_state changes) This has been in trunk for a month or maybe a little longer. FYI - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova and asynchronous instance launching
On Jul 1, 2012, at 3:04 PM, Day, Phil philip@hp.com wrote: Rather than adding debug statements could we please add additional notification events (for example a notification event whenever task_state changes) This has been in trunk for a month or maybe a little longer. FYI - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova and asynchronous instance launching
There's only 1 rpc call unless you're running cactus or something. All schedulers have a loop...not API. min-count is unfortunately special cased right now to be a single call vs cast, though. I was going to fix that real soon. Problem is scheduler creating the DB records vs API in this case. I can expand on this when I'm not replying from a phone. :) There's some other things that would be nice to do here with the API but the call can change to a cast with no API behavior change (except for speeding up the response :) - Chris On Jun 27, 2012, at 12:53 PM, Devin Carlen de...@openstack.org wrote: We filed a blueprint for this yesterday: https://blueprints.launchpad.net/nova/+spec/launch-instances-async Currently if a user attempts to create a lot of instances with a single API call (using min_count) the request will hang for a long time while all RPC calls are completed. For a large number of instances this can take a very long time. The API should return immediately and asynchronously make RPC calls. We are looking for creative ways to work around this problem, but in the meantime I'd like to hear from folks on what they think the preferred solution would be. Devin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins
It had some problems… now it just seems backed up… trying to catch up. On Jun 27, 2012, at 12:04 PM, Gary Kotton wrote: Hi, Is anyone aware of a problem with Jenkins? Thanks Gary ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Lossage in nova test suite?
Ok, I take that back. I do see one issue. It looks like each test run is leaving 2 stuck runner.py's for me… even if the tests complete successfully. On Jun 4, 2012, at 4:57 PM, Chris Behrens wrote: The only thing I notice is about a 50% increase in the unit test run time very recently… I don't know when that started.. maybe today. Not seeing the 100s of python processes. Hm! - Chris On Jun 4, 2012, at 4:00 PM, Kevin L. Mitchell wrote: Today I've noticed some significant problems with nova's test suite leaving literally hundreds of python processes out. I'm guessing that this has to do with the unit tests for the multiprocess patch, which was just approved. This could be causing problems with jenkins, too… Anybody have any other insights? -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Lossage in nova test suite?
Sorry, replying too soon before fully investigating. They *are* forking over and over again repeatedly as fast as possible… making them hard to kill. They fork a child and the parent exits immediately.. and that repeats endlessly. My VM is too slow right now that even a tight pkill loop doesn't hit them before they've forked again. http://paste.openstack.org/show/18338/ Only recourse may be to reboot my VM. On Jun 4, 2012, at 4:57 PM, Chris Behrens wrote: The only thing I notice is about a 50% increase in the unit test run time very recently… I don't know when that started.. maybe today. Not seeing the 100s of python processes. Hm! - Chris On Jun 4, 2012, at 4:00 PM, Kevin L. Mitchell wrote: Today I've noticed some significant problems with nova's test suite leaving literally hundreds of python processes out. I'm guessing that this has to do with the unit tests for the multiprocess patch, which was just approved. This could be causing problems with jenkins, too… Anybody have any other insights? -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] hidden / phasing out instance_types/flavors
Doug, That's the behavior I'd like to see and think it makes the most sense. It's really a requirement if we want a great cells implementation. instance_types table should only be used at the top level API cell. The data contained in the table is passed in the messaging and stored with the newly built, re-built, or re-sized instance. I brought this up a couple days ago internally at Rackspace while this current patch was being developed and I think we were going to start to look at that next…. assuming everyone loves the idea. :) - Chris On Jun 1, 2012, at 9:54 AM, Doug Davis wrote: Just wondering, is there any reason flavors are not limited to just create-time? Meaning, use it to create a new instance and then copy all of the flavor data into the new instance's data. This breaks the relationship between the instance and the flavor, allow each to be changed independently - or even deleted. Doing this would mean you wouldn't need to add a disabled flag at all - just delete the flavor if you don't want anyone to use it. This would also allow for an easier modification of existing instances - just modify the instance's property that needs to change w/o creating a whole new flavor (avoids the proliferation of flavors too). thanks -Doug __ STSM | Standards Architect | IBM Software Group (919) 254-6905 | IBM 444-6905 | d...@us.ibm.com The more I'm around some people, the more I like my dog. Matthew Sherborne matt.sherbo...@rackspace.com Sent by: openstack-bounces+dug=us.ibm@lists.launchpad.net 06/01/2012 10:41 AM To openstack@lists.launchpad.net cc Subject [Openstack] hidden / phasing out instance_types/flavors Hi Openstack community, We recently uploaded this change: https://review.openstack.org/#/c/8007/ It adds a 'disabled' field to the 'instance_type' or 'flavor' concept. The usage scenario we had in mind was to phase out a flavor that's already in use; people shouldn't be able to build new instances from that flavor, nor should customers see it in the list of available flavors. But when they view an existing instance with that flavor type, they should still be able to see the name of it at least. But should you change your mind later and wish to re-enable it, it's easy to just flip the flag. We'd appreciate feedback on the added field and the use of the namespace in the core code. (Line 56 here: https://review.openstack.org/#/c/8007/1/nova/api/openstack/compute/views/flavors.py ) The reasoning behind this is: * If we did it as an extension, it would greatly complicate the code. The code is much simpler being right in the core code. * We can't just add a field to the API quickly, so we need to use the namespace. * The hope is that eventually it would be accepted into the main API anyway, then the coding would be just removing the namespace. Many thanks in for reading. All feedback appreciated. Kind Regards, Matthew Sherborne___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] 'nova flavor-list' fails with ERROR: string indices must be integers, not str, but 'nova-manage flavor list' succeeds.
That's the traceback from novaclient. If you're getting a 503, there's likely
a traceback in the nova-api service logs.
- Chris
On May 9, 2012, at 5:38 PM, James R Penick pen...@yahoo-inc.com wrote:
Sorry, forgot to include that:
bash-4.1$ nova —debug image-list
connect: (127.0.0.1, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 127.0.0.1:5000\r\nAccept-Encoding:
identity\r\nContent-Length: 101\r\ncontent-type: application/json\r\naccept:
application/json\r\nuser-agent: python-novaclient\r\n\r\n'
send: '{auth: {tenantName: vmops, passwordCredentials: {username:
penick, password: tacos}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Vary: X-Auth-Token
header: Content-Length: 1903
header: Date: Thu, 10 May 2012 00:37:02 GMT
connect: (208.67.66.91, 8774)
send: u'GET /v2/c9d7f45d980d494fab3d69d9fc57547c/images/detail
HTTP/1.1\r\nHost: 208.67.66.91:8774\r\nx-auth-project-id:
vmops\r\nx-auth-token: 3261ef74e6494561830949780838\r\naccept-encoding:
compress, gzip\r\naccept: application/json\r\nuser-agent:
python-novaclient\r\n\r\n'
reply: 'HTTP/1.1 503 Service Unavailable\r\n'
header: Content-Length: 100
header: Content-Type: text/plain; charset=UTF-8
header: Date: Thu, 10 May 2012 00:37:02 GMT
DEBUG (shell:415) string indices must be integers, not str
Traceback (most recent call last):
File /usr/lib/python2.6/site-packages/novaclient/shell.py, line 412, in
main
OpenStackComputeShell().main(sys.argv[1:])
File /usr/lib/python2.6/site-packages/novaclient/shell.py, line 363, in
main
args.func(self.cs, args)
File /usr/lib/python2.6/site-packages/novaclient/v1_1/shell.py, line 350,
in do_image_list
image_list = cs.images.list()
File /usr/lib/python2.6/site-packages/novaclient/v1_1/images.py, line 47,
in list
return self._list(/images/detail, images)
File /usr/lib/python2.6/site-packages/novaclient/base.py, line 80, in
_list
data = body[response_key]
TypeError: string indices must be integers, not str
ERROR: string indices must be integers, not str
bash-4.1$
From: Vishvananda Ishaya vishvana...@gmail.com
To: James R Penick pen...@yahoo-inc.com
Cc: openstack@lists.launchpad.net openstack@lists.launchpad.net
Subject: Re: [Openstack] 'nova flavor-list' fails with ERROR: string indices
must be integers, not str, but 'nova-manage flavor list' succeeds.
Is there a traceback from nova-api?
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Moving nova.rpc into openstack.common
On Apr 3, 2012, at 8:00 AM, Russell Bryant wrote: […] 3) nova.local This looks like a good candidate for moving to openstack-common and switching all usage of nova.local to be openstack.common.local. Yeah, this one might be kind of weird, because it's used to set the 'nova context', and I'm not sure it actually applies to Openstack generically. […] 5) nova.context I haven't looked at this one in detail, yet. We'll have to sort out what to do with RequestContext. I see in the message from Swaminathan Venkataraman that both openstack-common and nova have RequestContext, but there's more code in the nova version. I suppose we should look at making the openstack-common version sufficient for nova and then switch nova to it. Hm. I wonder if we need some sort of serialize/deserialize callbacks or something so that nova can encode/decode the context that's passed… and not have any of this context stuff be a part of openstack-common. Just a random thought without looking or thinking about it too much. :) […] - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] proposal for Russell Bryant to be added to Nova Core
+1 from me On Apr 27, 2012, at 8:09 AM, Dan Prince dpri...@redhat.com wrote: Russell Bryant wrote the Nova Qpid rpc implementation and is a member of the Nova security team. He has been helping chipping away at reviews and contributing to discussions for some time now. I'd like to seem him Nova core so he can help out w/ reviews... definitely the RPC ones. Dan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Regular XenAPI Meeting
+1 and good on the time here On Apr 23, 2012, at 10:21 AM, John Garbutt john.garb...@citrix.com wrote: Hi, Are people keen for a XenAPI virt layer meetup on IRC every month? I have added a suggested time to the wiki, as a starting point: Monthly, second Wednesday at 17:00 UTC Does that seem a reasonable time for those that want to attend? It can be more frequent if we find that useful. I don’t want to detract from the other meetings, but it would be good to keep all of us working on the XenAPI layer in regular contact. Cheers, John ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] New Gerrit version (and server)
I'm noticing that core reviewers aren't tagged as a +2 anymore in the general review box. Instead, it just shows a check. (Except for my own reviews, which show as a +2). I have to look at the review comments to see if others voted +2 now. - Chris On Apr 12, 2012, at 4:23 PM, James E. Blair wrote: Hi, We've just upgraded Gerrit to version 2.3. There are a lot of changes behind the scenes that we've been looking forward to (like being able to store data in innodb rather than myisam tables for extra data longevity). And there are a few visible changes that may be of interest to OpenStack developers. One new addition in 2.3 is draft changes. The idea behind a draft change in Gerrit is that it is a change that is not ready for merging, or even general code review, but you would like to share it with some people to get early comments. If you upload a change as a draft, by default, no one else can see it. You must explicitly add each person you would like to share it with as a reviewer. Reviewers you add can leave comments, but can not vote at this stage. You can continue to upload new patchsets to the change as it evolves, and once it is ready for general review, you can click the Publish button. It will then become a normal change in Gerrit that everyone can see, including the earlier reviews from the draft stage. This is a one way transition; once a draft is published, it can't be made a draft again. If you're using git-review from source or the latest version from PyPI (version 1.16, released today), you can easily upload a draft change by adding the -D option (eg, git review -D). Earlier versions of git-review also have the -D option, but the git ref that Gerrit uses to indicate a change should be a draft was changed between the 2.3 release candidate and the final release; so if using -D results in an error, you may need to upgrade. You may notice some changes to the diff view. Notably, the header which contained all of the possible viewing options has been split up into several parts; you can switch between them by selecting options that show up under the menu at the top. I recommend setting Retain Header On File Switch under the Preferences section, as it is a nicer experience when changing files. Another notable new feature is the ability to add a group to the list of reviewers for a change. Just type in the name of the group and click Add Reviewer and all of the individuals in the group will be added to the list of reviewers (and will see the change on their review requests list. Finally, we've modified some of our local OpenStack style changes so that it is easier for us to track upstream changes in layout. It should mean a little more consistency throughout the interface, though we weren't able to keep the alternating row colors on the main table without a disproportionate amount of effort. Do note that you can click on a line in a table, and it will be highlighted to improve legibility. We've tried to give this as much testing as possible before moving it into production. If you encounter any issues, please let us know on IRC (mtaylor, jeblair, LinuxJedi), via email at openstack-ci-adm...@lists.launchpad.net, or you can file a bug at: https://bugs.launchpad.net/openstack-ci/ Thanks, Jim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Limit flavors to specific hosts
I have some plans for being able to set arbitrary capabilities for hosts via nova.conf that you can use to build scheduler filters. Right now, there are capabilities, but I believe we're only creating these from hypervisor stats. You can filter on those today. What I'm planning on adding is a way to specify additional keyname/value pairs in nova.conf to supplement the capabilities we build from hypervisor stats. You could set things like this in your nova.conf: --host_capabilities=instance_type_ids=1,2,3;keyX;keyY=something etc. Since capabilities are already passed to scheduler rules, you could add some basic filters that do: if 'instance_type_ids' in capabilities and instance_type.id not in capabilities['instance_type_ids']: return False Since host_capabilities are just arbitrary keyname/value pairs, you can pretty much add anything you want to --host_capabilities and then write some matching scheduler filter rules. That's the basic idea, anyway. The exact same behavior will apply to 'cells' and the cells scheduler as well. (Except you'll have cells_capabilities= somewhere (prob nova.conf for the cells service). - Chris On Apr 2, 2012, at 10:36 AM, Day, Phil wrote: Hi Folks, I’m looking for a capability to limit some flavours to some hosts. I want the mapping to be as flexible as possible, and work within a zone/cell (I don’t want to add zones just to get this mapping).For example I want to express something like: Host_1 supports flavours A, C Host_2 supports flavours A, B Host_3 supports flavours A, B, C Host_4 supports flavours D Ideally there would be some form of grouping to sets of flavours: Flavour_A is part of Flavour_Sets 1, 2, 3 Flavour_B is part of Flavour_Sets 2, 3 Flavour_C is part of Flavour_Sets 1, 3, 4 Host_1 supports flavour Set 1 Host_2 supports flavour Set 2 Host_3 supports flavour Set 3 Host_4 supports flavour Set 4 From the Essex design summit I thought that host aggregates was going to give this sort of capability, but having looked through the code that seems to be quite tightly coupled with specific hypervisor functionality, whereas this is purely a scheduler feature. I can see that I could define flavour group membership through the instanace_type_extra_specs, but not how to then associate these with specific hosts. I know I’m a tad behind some of the recent changes – so before suggesting a design summit session on this I thought I’d ask - is there something that already does this type of mapping ? Cheers, Phil ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Moving nova.rpc into openstack.common
Seems like a sensible plan. Carrot can go now. I marked it deprecated so we can remove in folsom. I can take care of this today, even. On Apr 2, 2012, at 3:02 PM, Duncan McGreggor dun...@dreamhost.com wrote: +1 Thanks for exploring this, Russell! Next step: getting a common REST API abstraction in place that all the projects can use... ;-) d On Mon, Apr 2, 2012 at 4:26 PM, Russell Bryant rbry...@redhat.com wrote: Greetings, There was a thread on this list a little while ago about moving the notification drivers that are in nova and glance into openstack.common since they provide very similar functionality, but have implementations that have diverged over time. Is anyone actively working on this? If so, please let me know. For the message queue notifiers, nova uses nova.rpc to do the heavy lifting. Glance has notifiers written against the messaging libs directly. I think it makes sense to take the nova approach. This would mean moving nova.rpc into openstack.common before the notifiers can get moved. I have started looking at moving nova.rpc to openstack.common.rpc. My plan is: 1) Write a series of patches that reduces coupling in nova.rpc on the rest of nova. 2) Submit changes needed to support this decoupling to openstack.common. 3) Once nova.rpc is sufficiently decoupled, move it to openstack.common. While doing the above, I want to aim for the least amount of disruption to the rpc interface as is practical. While we're at it, is it time to drop nova.rpc.impl_carrot? It is marked as deprecated in Essex. Is there any reason anyone would still use impl_carrot over impl_kombu? Thoughts? Thanks, -- Russell Bryant ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Distributed rate-limiting
My issue with using the URL is someone could easily DoS any tenant. Maybe you said that below. I only have a brief moment to scan email ATM. :) On Mar 29, 2012, at 5:26 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Thu, 2012-03-29 at 22:58 +0100, Day, Phil wrote: - As you get the tenant id from the context I assume this module has to come after the authentication in the pipeline. Yes, I have made that assumption. It seems reasonable, given that the existing rate-limit middleware is right after authentication as well. Have you thought about using the tenant_id in the URL instead ? (I'm thinking of the case where you want rate limit requests into the authentication system as well as Nova itself). No, I haven't. I don't trust the user, which is where the tenant_id in the URL comes from. I do trust the auth system, which is why I want to use the tenant ID from the context. (And yes, you could argue that authz would prevent their access to other tenants anyway, but why make nova have to check authz if rate limiting would stop them in the first place?) As for rate limiting requests into the authentication system, I'd suggest using a Limit subclass which uses the remote IP address in place of a tenant ID, at least for the user endpoint. I don't think we want any rate limiting at all on the service side of Keystone; our current architecture means that Keystone is going to be hit a *lot*: at least once for each request that hits Nova, and more in certain cases (i.e., instance boot, where we'll have to hit quantum and glance as well). - Does this work for EC2 as well as OSAPI ? Actually, it didn't occur to me to test, given that I don't really use the EC2 API. I don't think there's anything in the basic architecture which would be incompatible with EC2; the only possible sticking point that occurs to me is the URL construction in nova_limits:NovaClassLimit.route(): if the URL specified is prefixed with '/v1.1/' or '/v2/', the version identifier is dropped (otherwise the route wouldn't match). That would be easy to work around; simply extend NovaClassLimit and override route() to do the appropriate transformation for EC2. Any EC2 experts want to weigh in? -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] error on nova-compute on domU on xen
pip install xenapi - Chris On Mar 15, 2012, at 10:00 AM, Eduardo Nunes wrote: got this error on log, 2012-03-15 16:52:33,809 ERROR nova.compute.manager [-] Unable to load the virtualization driver: No module named XenAPI my nova-compute version diablo 2011.3 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Removal of VSA Code
+1 I'd like to see a way for plugins to be able to modify the DB schema. There's no easy way for a plugin to be able to add a column to a core nova table, for instance. In any case, I'm +1 on removing the vsa code for Essex. On Mar 15, 2012, at 10:08 AM, Ghe Rivero wrote: I think is time (after essex release) to rethink the way plugins are integrated into mainline code. This problem, an outdated plugin, it's not new (Hyper-V), and with the increasing numbers of them (storage like Zadara, Nexenta... network with Nicira, BigSwitch, Citrix...) we need a policy to deal with this. From the vendor point of view, it's not easy to follow the development release cycle that OpenStack has now days, with so many changes day after day. Maybe plugins should be out of OpenStack code, and let vendors to publish them after the release. Ghe Rivero On Thu, Mar 15, 2012 at 5:53 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Thu, 2012-03-15 at 09:02 -0700, Vladimir Popovski wrote: I was not aware of any issue with VSA code in diablo/stable (or at least major issues). I'll point out that the code we're concerned about is the code in trunk, not the code in diablo/stable. There have been substantial changes to the code since diablo was released, which has resulted in bitrot in the VSA code and the attendant breakages to which Vish is referring. -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Ghe Rivero OpenStack Distribution Engineer www.stackops.com | ghe.riv...@stackops.com | +34 625 63 45 23 | skype:ghe.rivero ADVERTENCIA LEGAL Le informamos, como destinatario de este mensaje, que el correo electrónico y las comunicaciones por medio de Internet no permiten asegurar ni garantizar la confidencialidad de los mensajes transmitidos, así como tampoco su integridad o su correcta recepción, por lo que STACKOPS TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias. Si no consintiese en la utilización del correo electrónico o de las comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro conocimiento de manera inmediata. Este mensaje va dirigido, de manera exclusiva, a su destinatario y contiene información confidencial y sujeta al secreto profesional, cuya divulgación no está permitida por la ley. En caso de haber recibido este mensaje por error, le rogamos que, de forma inmediata, nos lo comunique mediante correo electrónico remitido a nuestra atención y proceda a su eliminación, así como a la de cualquier documento adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o utilización de este mensaje, o de cualquier documento adjunto al mismo, cualquiera que fuera su finalidad, están prohibidas por la ley. * PRIVILEGED AND CONFIDENTIAL We hereby inform you, as addressee of this message, that e-mail and Internet do not guarantee the confidentiality, nor the completeness or proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L. does not assume any liability for those circumstances. Should you not agree to the use of e-mail or to communications via Internet, you are kindly requested to notify us immediately. This message is intended exclusively for the person to whom it is addressed and contains privileged and confidential information protected from disclosure by law. If you are not the addressee indicated in this message, you should immediately delete it and any attachments and notify the sender by reply e-mail. In such case, you are hereby notified that any dissemination, distribution, copying or use of this message or any attachments, for any purpose, is strictly prohibited by law. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] eventlet weirdness
On Mar 4, 2012, at 8:56 PM, Gabe Westmaas gabe.westm...@rackspace.com I agree with this paragraph whole heartedly! I would definitely like to see this separation not only for the reasons you list above (performance, all installations behaving the same way) but also because I think it gives us a lot more power to help handle seamless upgrades - another topic I'm sure we will be discussing at the conference. And it makes the compute cells stuff plug in a LOT more cleanly. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] eventlet weirdness
It's not just you On Mar 2, 2012, at 10:35 AM, Joshua Harlow wrote: Does anyone else feel that the following seems really “dirty”, or is it just me. “adding a few sleep(0) calls in various places in the Nova codebase (as was recently added in the _sync_power_states() periodic task) is an easy and simple win with pretty much no ill side-effects. :)” Dirty in that it feels like there is something wrong from a design point of view. Sprinkling “sleep(0)” seems like its a band-aid on a larger problem imho. But that’s just my gut feeling. :-( On 3/2/12 8:26 AM, Armando Migliaccio armando.migliac...@eu.citrix.com wrote: I knew you'd say that :P There you go: https://bugs.launchpad.net/nova/+bug/944145 Cheers, Armando -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: 02 March 2012 16:22 To: Armando Migliaccio Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] eventlet weirdness On 03/02/2012 10:52 AM, Armando Migliaccio wrote: I'd be cautious to say that no ill side-effects were introduced. I found a race condition right in the middle of sync_power_states, which I assume was exposed by breaking the task deliberately. Such a party-pooper! ;) Got a link to the bug report for me? Thanks! -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] eventlet weirdness
On Mar 1, 2012, at 12:27 PM, Adam Young wrote: On 03/01/2012 02:45 PM, Yun Mao wrote: There are plenty eventlet discussion recently but I'll stick my question to this thread, although it's pretty much a separate question. :) How is MySQL access handled in eventlet? Presumably it's external C library so it's not going to be monkey patched. Does that make every db access call a blocking call? Thanks, Nope, it goes through a thread pool. Actually, it doesn't use a thread pool right now... so it does block, unless something has changed recently that I'm not aware of. We were using the eventpool dbpool code, but we had to remove it at diablo release time due to issues. Correct me if this is wrong. I'm not sure it's ever been completely revisited, but this is definitely a huge issue for scaling. It's been on my list for a while to take a look at. - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] eventlet weirdness
On Mar 1, 2012, at 12:36 PM, Vishvananda Ishaya wrote: Yes it does. We actually tried to use a pool at diablo release and it was very broken. There was discussion about moving over to a pure-python mysql library, but it hasn't been tried yet. I know some people have tried this... and the performance is... not great. - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Basic networking/configuration woes
I'd assume FlatDHCPManager works much like FlatManager, but maybe I'm wrong. I use FlatManager and I always end up having to modify the fixed_ips table manually after running nova-manage because I think I'm trying to do something similar as you. I have a /23... and I want to give nova a /25 out of it. Though I'm giving nova a /25, it's still really a /23. I use nova-manage to add my /23 and then I edit the fixed_ips table and mark a lot of addresses as 'reserved'... or just remove them altogether. (When I try to specify the /25 to nova-manage, it doesn't go so well) As far as 169.254... you can reach that without any address assigned. Your NIC should receive a link local address when there's no other IP assigned which is in the 169.254.* range. Not sure if that helped much :) - Chris On Feb 23, 2012, at 3:12 PM, Justin Santa Barbara wrote: I'm trying to use OpenStack in what I think to be the typical non-public-cloud deployment, and my experience is not what it could/should be. I'm hoping someone can point me to the right way, or we can figure out what needs to change. My wishlist: * I want my instances to be on my network e.g. 10.0.0.0/16 * As Nova can't pull IPs from my DHCP server, I'm willing to allocate it a sub-range, e.g. 10.200.0.0/16 First decision: Choosing a networking mode: * I don't want / need VLANs * If I use FlatDHCPManager, I can't do the subrange stuff - it seems that this mode assumes it controls the entire address range. * So it's FlatManager. It works, but now I don't have DHCP, so I just have to inject info into the instance. Next decision: How to inject info (at least the IP address): * Supposedly the 'right way' is to use cloud-init. It looks like I'd still need DHCP before I can reach 169.254..., and I don't have that. It looks like cloud-init can't do network configuration even if nova passed the information in. And I'd be locked into cloud-init images - no Windows, no Debian etc. * The next best way is config_drive. It looks like I'd have to bundle my own image. Maybe I could use cloud-init, maybe with an OVF formatted config_drive, but even then I couldn't configure networking (?) * So now I'm back to file injection. That just works. So now I'm using FlatManager and file injection; and yet I feel this is the dodgy back alley of OpenStack, and I should be in the well-lit nice area. I worry that things like file injection and FlatManager are less favored and may be deprecated in future. But every time I try to do things right I just waste a lot of time and make no progress. Yet I feel I didn't really have a choice here. How are other people making this work? What is the right way? Justin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Basic networking/configuration woes
On Feb 23, 2012, at 3:55 PM, Justin Santa Barbara wrote: Thanks for chipping in. I have contributed a patch (which has merged) which should allow you to stop editing the SQL: https://review.openstack.org/#change,3816 With that, you should be able to pass the full range, with an additional argument specifying the subset that nova controls: e.g.-fixed_cidr=10.200.0.0/16 Oh cool.. that'll save me some pain. :) When I boot my VM, I think it gets a real address from my DHCP server (because the VM can reach the DHCP server), but not the address nova assigned it! I believe the nova iptables rules mean that the machine can't then do TCP/IP, but even if I am wrong/could overcome that, I don't think cloud-init could then configure the correct address. If you're going to go the cloud-init route... you wouldn't need DHCP, right? There should be iptables rules to allow you to talk to the metadata service over 169.254.* (And linux should give you a default link-local address that allows you to talk to the MD service magically) Do you have a non-nova DHCP server running as well? - Chris Justin On Thu, Feb 23, 2012 at 3:38 PM, Chris Behrens cbehr...@codestud.com wrote: I'd assume FlatDHCPManager works much like FlatManager, but maybe I'm wrong. I use FlatManager and I always end up having to modify the fixed_ips table manually after running nova-manage because I think I'm trying to do something similar as you. I have a /23... and I want to give nova a /25 out of it. Though I'm giving nova a /25, it's still really a /23. I use nova-manage to add my /23 and then I edit the fixed_ips table and mark a lot of addresses as 'reserved'... or just remove them altogether. (When I try to specify the /25 to nova-manage, it doesn't go so well) As far as 169.254... you can reach that without any address assigned. Your NIC should receive a link local address when there's no other IP assigned which is in the 169.254.* range. Not sure if that helped much :) - Chris On Feb 23, 2012, at 3:12 PM, Justin Santa Barbara wrote: I'm trying to use OpenStack in what I think to be the typical non-public-cloud deployment, and my experience is not what it could/should be. I'm hoping someone can point me to the right way, or we can figure out what needs to change. My wishlist: * I want my instances to be on my network e.g. 10.0.0.0/16 * As Nova can't pull IPs from my DHCP server, I'm willing to allocate it a sub-range, e.g. 10.200.0.0/16 First decision: Choosing a networking mode: * I don't want / need VLANs * If I use FlatDHCPManager, I can't do the subrange stuff - it seems that this mode assumes it controls the entire address range. * So it's FlatManager. It works, but now I don't have DHCP, so I just have to inject info into the instance. Next decision: How to inject info (at least the IP address): * Supposedly the 'right way' is to use cloud-init. It looks like I'd still need DHCP before I can reach 169.254..., and I don't have that. It looks like cloud-init can't do network configuration even if nova passed the information in. And I'd be locked into cloud-init images - no Windows, no Debian etc. * The next best way is config_drive. It looks like I'd have to bundle my own image. Maybe I could use cloud-init, maybe with an OVF formatted config_drive, but even then I couldn't configure networking (?) * So now I'm back to file injection. That just works. So now I'm using FlatManager and file injection; and yet I feel this is the dodgy back alley of OpenStack, and I should be in the well-lit nice area. I worry that things like file injection and FlatManager are less favored and may be deprecated in future. But every time I try to do things right I just waste a lot of time and make no progress. Yet I feel I didn't really have a choice here. How are other people making this work? What is the right way? Justin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Remove Zones code - FFE
Sector? On Feb 17, 2012, at 9:06 AM, Ed Leafe wrote: On Feb 15, 2012, at 10:07 AM, Armando Migliaccio wrote: I think you touched the crucial point here: what is exposed to the user and what not. Reading: http://wiki.openstack.org/MultiClusterZones#Design one would think that zones is a concept exposed to end users. You're saying otherwise; Is it just my misunderstanding or the wiki page is out of sync with the latest developments? If zones are not going to be exposed to the users, what will? Just availability zones? Zones *could* be exposed, but that is not intrinsic to their design. Availability zones could be designated as a particular level of nesting of the overall zone design, such as a particular region, and users could specify the AZ they want their instance to be provisioned in. But a region might have several data centers, each of which could be a zone, and individual DCs could have several zones within them based on the physical layout of the building, or networking capacity, or because of incremental build out, or for any number of other reasons, none of which are relevant to a user. The term zone was adopted at a time when we weren't really focusing on mimicking the AWS Availability Zone concept, and in hindsight, it was a poor choice. So we should learn from that mistake and make sure we don't choose a replacement term that already has a common usage, such as shards segments or clusters. -- Ed Leafe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Remove Zones code - FFE
On Feb 18, 2012, at 11:08 AM, Nathanael Burton wrote: Sectors remind me too much of disks. Right.. I mentioned it because I would think it'd be rather difficult to confuse 'Nova sectors' with sectors on a disk. How about? Layers, Slices, Fragments, Knots... Of those, I'd probably prefer 'Knots'. - Chris On Feb 18, 2012 1:57 PM, Chris Behrens cbehr...@codestud.com wrote: Sector? On Feb 17, 2012, at 9:06 AM, Ed Leafe wrote: On Feb 15, 2012, at 10:07 AM, Armando Migliaccio wrote: I think you touched the crucial point here: what is exposed to the user and what not. Reading: http://wiki.openstack.org/MultiClusterZones#Design one would think that zones is a concept exposed to end users. You're saying otherwise; Is it just my misunderstanding or the wiki page is out of sync with the latest developments? If zones are not going to be exposed to the users, what will? Just availability zones? Zones *could* be exposed, but that is not intrinsic to their design. Availability zones could be designated as a particular level of nesting of the overall zone design, such as a particular region, and users could specify the AZ they want their instance to be provisioned in. But a region might have several data centers, each of which could be a zone, and individual DCs could have several zones within them based on the physical layout of the building, or networking capacity, or because of incremental build out, or for any number of other reasons, none of which are relevant to a user. The term zone was adopted at a time when we weren't really focusing on mimicking the AWS Availability Zone concept, and in hindsight, it was a poor choice. So we should learn from that mistake and make sure we don't choose a replacement term that already has a common usage, such as shards segments or clusters. -- Ed Leafe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Remove Zones code - FFE
On Feb 18, 2012, at 11:36 AM, Ed Leafe wrote: I still prefer 'cell'. The parallel to single celled / multi-cellular life forms makes sense, and there is really no overloading of the word in the world of computers. I like 'cell' too. It makes sense. But 'cell' reminds me too much of batteries could be a response here if we're going to say that sectors remind me too much of disks. :) Batteries may less relate to computers, but there's: memory cell, 'Cell' processor. The question is.. is something like cell or sector or whatever really going to be confusing in the context of Openstack/Nova. 'Zone' is certainly an issue because of 'availability zone', so I think it's obviously not a choice. Anyway, I think 'cell' could be a good choice. At some point, I'd like to collect the ideas and put them up in a poll for vote. I suspect that's the only way we'll come to a fair conclusion. - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] gerrit: basing a change on another uncommitted change
Hm. It shouldn't have uploaded a new patch set for 3990 if you did not modify the commit in 3990. However, it does appear that you modified 3990's commit. This method was added: test_get_rrd_server() in nova/tests/test_xenapi.py You can see it here: https://review.openstack.org/#patch,unified,3990,7,nova/tests/test_xenapi.py And click on the arrow for 'patch history' in upper left. Then select the differences between patch set 6 and patch set 7. - Chris On Feb 16, 2012, at 5:09 PM, Renuka Apte wrote: Hello, What is the right way to base a change on another uncommitted change? For example, I have https://review.openstack.org/#change,4267 based on top of https://review.openstack.org/#change,3990. I checked out 3990. Created a topic branch with git checkout -b name. I committed my own change locally and then did git review. This created the right dependency between the changes, but additionally, since git review does a rebase, it uploaded a new patch set to 3990. That was not what I expected (sorry Vish), and could be pretty harmful if 3990 had received all its reviews and was ready to be merged. How can I avoid this? Thanks, Renuka. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] removing zones for essex
On Feb 14, 2012, at 2:31 PM, Chris Behrens wrote: [...] With that, we have 2 branches up for review to remove the current zones code: novaclient: https://review.openstack.org/4163 nova: https://review.openstack.org/4062 Oops... should have been: https://review.openstack.org/#change,3629 - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Propose to make Monsyne Dragon a nova core developer
+1 On Feb 9, 2012, at 9:11 AM, Matt Dietz wrote: Any more hits on this? I fear that this was lost in the torrent of email, and I really do think Dragon deserves to be a core developer for all the hard work he's been doing. From: Paul Voccio openst...@substation9.com Date: Tue, 7 Feb 2012 12:38:15 -0600 To: Matt Dietz matt.di...@rackspace.com Cc: openstack@lists.launchpad.net openstack@lists.launchpad.net Subject: Re: [Openstack] Propose to make Monsyne Dragon a nova core developer +1 On Mon, Feb 6, 2012 at 4:48 PM, Matt Dietz matt.di...@rackspace.com wrote: Hey guys, Dragon has really stepped up lately on reviewing patches into Nova, and has a ton of knowledge around Nova proper, so I propose he be added to Nova core. I think he'd be a great addition to the team. Matt ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Remove Zones code - FFE
Well, I can actually say with confidence that the replacement would be stable by essex release. In fact, I expect the first commit to be a completely working solution that solves a number of problems with the original implementation. I don't think there's any issue getting something committed by the 15th if there's not much bickering on the review. The code is dead simple (currently a 500 line diff) and requires almost no modification of nova core. The only modifications to nova core are: Specify a different compute API class to use Modify rpc code to allow some kwargs to Connection __init__ so you can specify a specific rabbit server to use to send a message to a zone Add 2 new rpc methods: cast_to_zone and call_to_zone (which use the above change) Add a few zone_api.update_instance() calls in some places in compute so that we can push instance updates to top level zone. Migration for zones DB table to add rpc credential information. Besides that, the rest of the code is standalone. There's absolutely no concern that it'll make non-zones less stable. The few zone_api.update_instance() type calls would be no-ops when zones is turned off. As far as meeting a 2/15 date, here's what I have left: 1) clean up some things in my compute API subclass 2) finish writing tests (Vish: I was joking when I asked if you meant 15th of March ;) After what I would plan to push for the first commit, there's some refactoring desired to better share functionality between the zone scheduler and the host scheduler, but it'd be nothing required for Essex if we want to wait on that. And there's a desire to fold this new 'zones' nova service I'm creating into the current scheduler as potentially just a different scheduler driver. There's 1 thing that would be lost in what I'd propose: zone scheduling would initially be random zone selection. Either way, I think I'll throw something up for review by mid next week to see what thoughts are. I'd like to at least share what this looks like at the moment. If we still decide to wait for F1, cool. - Chris On Feb 2, 2012, at 1:27 AM, Vishvananda Ishaya wrote: I talked with chris a bit offline, and I'm a little concerned that we will be pushing too hard to try and get this into a working state by Essex. I think even if we to slam it in we will be faced with bugs that will make the essex version potentially as broken as the current zones code is. It is probably much more reasonable to target F1 as a delivery date for this feature. Alejandro, is your team ok with deploying milestone releases? I know it would take a lot of pressure off of Chris, Sandy, et. al. since they are trying to meet some pretty hard delivery dates as it is. Vish On Feb 1, 2012, at 3:44 PM, Alejandro Comisario wrote: Sounds pretty good Vish. Since we are mostly deployers, and the ones who are gonna try the new code from day zero, whats good for Sandy, its good for us. Alejandro. On 02/01/2012 06:57 PM, Vishvananda Ishaya wrote: Thanks for the feedback. It is good to get input from one of the largest openstack installs! So it sounds like the existing code is pretty broken. Based on this feedback I would like to propose the following: 1) cut out zones code (meaning merge the existing branch) 2) grant an FFe for the new rpc based zone code as long is it can be merged without heavily modifying core. This means: a) it should be deployable with the feature disabled b) it should only include minor modifications to core components c) if a major change is needed to distributed_scheduler (for example), consider leaving the existing version in, and copying the code to a new file (distributed_scheduler_v2) and doing the modifications there. That way we can minimize chances of breakage d) it needs to be merged by the 15th Does that seem reasonable? Vish On Feb 1, 2012, at 1:42 PM, Alejandro Comisario wrote: Hi guys. Its true that we are trying to make multizones work, actually we did, but we get into an instance were listing all vms from the parent zone ( where is has to go thru all the child zones ) is buggy ( if not impossible by now ). So, if there is a new zone architecture that actually works ( we want to stop using our own deployer to do that ) or has a chance to be fully working when 2012-1 is out, (we would prefer not to wait till Folsom) we are totally into it ! since by now, we were actually waiting for this new Zones code to come out to try again. Alejandro. On 02/01/2012 06:17 PM, Nathanael Burton wrote: +1 On Feb 1, 2012 4:13 PM, Vishvananda Ishaya vishvana...@gmail.com wrote: I would prefer that if it can be done super-super fast. :) Vish On Feb 1, 2012, at 1:04 PM, Chris Behrens wrote: I wonder if we can use some of the architecture of the new code and move the current implementation to that model. It'd preserve the existing functionality, set us up
Re: [Openstack] Fast cloning - FF-Exception request
I just chatted with Paul. We're okay with this if we can have a flag to allow turning off the caching part of things, since both cow and caching are behavior changes for the Xen side. There's already a flag for 'cow', so that's covered, but there is not one for caching in the current branch up for review. I understand that they should default to 'on' to match libvirt functionality, but I think we need to have a way to have Xen operate how it did before since it has been stable. I'm definitely looking forward to these features, but would like to minimize the impact in case it is unstable or people simply want the current behavior. (I posted the same comments on the review) - Chris On Jan 31, 2012, at 11:02 AM, Vishvananda Ishaya wrote: I'm ok with this going in, but I would like to leave it up to the guys who are using Xen on a daily basis. So if Chris Behrens and Paul Voccio are ok with it, I think an FFE is fine. Vish On Jan 31, 2012, at 10:31 AM, Devdeep Singh wrote: Hi, I am asking for a Feature-Freeze exception for the blueprinthttps://blueprints.launchpad.net/nova/+spec/fast-cloning-for-xenserver. I appreciate that it is now late for getting this feature in, however we are ready to get this feature early in E4. We have got a few reviews in the queue, and we are close to completing them. It is my understanding that E4 was not for big changes. However, this change only touches the xenapi code for instance creation. I have tested these change extensively to make sure it doesn’t break anything. Moreover, it will help close the gap between KVM and XenAPI. Thanks, Devdeep ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Supporting start/stop compute api from OpenStack API
Vish: Looks like it's only in ec2 api. Tomoe: The support will need to be added in an extension, since it's not in the current API spec. Ie, it'll need to go under nova/api/openstack/compute/contrib in the tree, not directly in compute/servers.py. - Chris On Jan 17, 2012, at 9:21 PM, Vishvananda Ishaya wrote: You can propose code without an approved blueprint. Are you sure there isn't already a server action for stop/start? Vish On Jan 17, 2012, at 7:46 PM, Tomoe Sugihara wrote: Hi, I have put up a blueprint (https://blueprints.launchpad.net/nova/+spec/start-stop-methods-support-in-os-servers-api) for supporting start/stop compute api, which should work well for boot-from-volume from OpenStack API, and I'm happy to contribute code for Essex release. Could someone tell me what the next step would be? I thought the blueprint should be approved to push the code in, but I couldn't find how to get an approval from here: http://wiki.openstack.org/HowToContribute Comment, guidance appreciated. Cheers, Tomoe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova and trusted computing
On Jan 12, 2012, at 12:00 PM, Yang, Fred wrote: Vish, Mark and I have worked a while and yet to have a get a better approach. Let me describe key features need to be addressed and a possible approach for base discussion Nova scheduler's select_and_run can dispatch N instances directly which makes trusted computing a little bit challenge. That functionality needs to go away for the time being because it presents multiple problems. It's not in the OS API spec... and I'd heard that the functionality added to the EC2 api for it was also not really to spec, but I've not checked on that. Have you taken a look at: https://review.openstack.org/#change,2552 to see if it makes your life any easier? There's more changes coming. One of the goals is to make things a bit more pluggable and eventually even combine all of the current schedulers. - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] reconnecting to RabbitMQ
On Dec 28, 2011, at 5:36 AM, Ask Solem wrote: [...] amqplib does not support heartbeats so there may be cases where the client/server does not properly detect that the socket has been closed, this usually only a problem with some particular intermediate I received a private reply from Roman which seems to indicate this is what is happening. The OS is seeing the tcp connections as ESTABLISHED still, even though they are dead. Not much an application can do there Sounds like enabling tcp keepalive solved the problem for him. Doing heartbeats in the app itself is another way, as you mention. - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] reconnecting to RabbitMQ
Hi Roman, Are you using diablo or trunk (both of which use kombu by default and should support reconnecting)? If so, can you 'egrep' for the following strings in logs and provide the output? You can reply to me directly so we're not spamming this list. Unable to connect to AMQP server Connected to AMQP server Re-established AMQP queues AMQP server on.*unreachable 'Failed to consume Failed to publish Thanks, - Chris On Dec 24, 2011, at 1:56 PM, Roman Sokolkov wrote: Hello! If for some reason rabbitmq has been down some time or connection was aborted, nova couldn't reconnect automatically and I need to restart all nova services manually. Additionally after restarting rabbit, If I make some call to API. For example, nova floating-ip-list only API reconnects automatically. Other nova services staying disconnected. Are there mechanisms to monitor establising of connections? And it difficult to add them? -- Best regards, Roman Sokolkov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] using objects returned from DB layer
Yes, that was what I was saying. :) On Dec 15, 2011, at 12:06 AM, Soren Hansen wrote: 2011/12/15 Jesse Andrews anotherje...@gmail.com: I agree except I though the preference was for instance_uuid = instance['uuid'] not instance_uuid = instance.uuid (use dict's and don't assume sqlalchemy) Yes... That's what Chris is saying, isn't it? -- Soren Hansen| http://linux2go.dk/ Ubuntu Developer| http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] using objects returned from DB layer
Agree with both #1s as a start. And I wouldn't try to 'rip off the band-aid'
either.
I was bringing this up initially as I want to enforce *something* when
reviewing, but if we want to start 'fixing' things, we can start hitting small
sections of code to limit conflicts. The 'enforce_model' thing might be a bit
extreme right now. We may be able to get to a place where we feel we've got
everything... then dictify it all at that time and see what raises in tests.,
fixing those last things. Hm... although, I guess I really do like the ability
to turn something off if it's broken. :)
- Chris
On Dec 15, 2011, at 12:29 AM, Rick Harris wrote:
++ on moving to a consistent dict-style syntax.
We could attempt to rip the Band-Aid off here by:
1. Rejecting any new patches which use the dot-notation
2. Trying to switch out to using dot-notation access all at once in one big
'fix-it-up' patch.
I'm not convinced 2) is possible. Separating model objects from other kinds
of objects will be time-consuming and error prone. Given the scope of this
issue, I'm not sure this could be done without a real risk of prolonged,
recurring breakage.
Instead, I'd advocate a step-wise approach:
1. Reject any new patches which use the dot-notation
2. Add code which helps us identify and fix dot-notation usage throughout the
codebase.
The basic idea here is that we'd add a flag to fail loudly when a object is
accessed using non-dict methods.
This could be done by adding a decorator to the sqlalchemy/db/api functions
such that instead of returning SQLAlchemy objects directly they instead
return objects of type AttrAccessibleDict.
An AttrAccessibleDict would be defined something like:
class AttrAccessibleDict(dict)
def __getattr__(self, attr):
if FLAGS.enforce_model_dict_access:
raise ModelObjectAccessError('must use dict-style access on model
objects')
else:
return self[attr]
def dictify_model(f):
def wrapper(*args, **kwargs):
rv = f(*args, **kwargs)
attr_dict = convert_to_attr_accessible_dict(rv)
return attr_dict
return wrapper
@dictify_model
def instance_get(…..):
pass
3. Make a concerted effort to fix the code over a period of a few weeks.
Developers could set --enforce_model_dict_access to True and fix up as many
cases as they can find. When they're tired of fixing cases or when more
pressing things come along for the moment, they can temporarily set
--enforce_model_dict_access back to False to that everything goes back to
humming along smoothly.
4. Once the dot-notation has been excised, we can switch over to actually
returning real Python dictionaries instead of AttrAccessibleDict objects.
5. Final step would be removing the cleanup code.
Once everything is returning dictionaries, we would remove the decorator
entirely and any other cleanup code we added along the way.
Feels like this approach would be a bit roundabout, but each step feels safe,
and it seem like it would get us to where we want to be in the course of a
few weeks (perhaps a couple of months, worst case).
Thoughts?
-Rick
On Dec 15, 2011, at 1:10 AM, Chris Behrens wrote:
I've seen a number of patches lately that have code like this:
instance = db.instance_get(...)
instance_uuid = instance.uuid
instead of:
instance_uuid = instance['uuid']
There's a mix of usage throughout the code, and I know some people are just
matching the surrounding code. But, in a number of cases, I've asked for
these to be corrected to the latter, on assumption that the DB layer will be
returning dictionaries at some point vs the models. It also pushes the code
towards consistent usage. But I might be the only Nova Core member looking
at this and/or maybe my assumption is wrong.
So, I ask here: Should Nova Core make an effort to reject patches with the
former format? Or did I miss any DB layer plans where the former format is
now preferred?
- Chris
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] using objects returned from DB layer
Hm. I've just been going off of this doc string in db/api.py: Functions in this module are imported into the nova.db namespace. Call these functions from nova.db namespace, not the nova.db.api namespace. All functions in this module return objects that implement a dictionary-like interface. Currently, many of these objects are sqlalchemy objects that implement a dictionary interface. However, a future goal is to have all ofthese objects be simple dictionaries. I didn't put any thought into it, but I can see our own objects being better. I would just want to make sure they are easily serializable so they can be sent across AMQP. I admit that I haven't paid attention to any of the DB discussions. I guess that's why I brought up my question in the first place. ;) - Chris On Dec 15, 2011, at 10:52 AM, Aaron Lee wrote: On Dec 15, 2011, at 10:54 AM, Johannes Erdfelt wrote: What kinds of things? I'm just trying to better understand what is object-oriented about the data returned from a database? What methods would we want to use? JE Any direct access to the data within a record should be encapsulated within the model object. For example, changing an instance's progress should be a method on Instance, not a method buried in vmops. ~ Aaron ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] using objects returned from DB layer
I've seen a number of patches lately that have code like this: instance = db.instance_get(...) instance_uuid = instance.uuid instead of: instance_uuid = instance['uuid'] There's a mix of usage throughout the code, and I know some people are just matching the surrounding code. But, in a number of cases, I've asked for these to be corrected to the latter, on assumption that the DB layer will be returning dictionaries at some point vs the models. It also pushes the code towards consistent usage. But I might be the only Nova Core member looking at this and/or maybe my assumption is wrong. So, I ask here: Should Nova Core make an effort to reject patches with the former format? Or did I miss any DB layer plans where the former format is now preferred? - Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
I need to catch up a bit with this thread, but I wanted to mention I have a huge patch coming that refactors almost all of the scheduler tests into true unit tests. I'd started this for other reasons and I hope it jives with the plans here. But if anyone is looking at the scheduler tests, we should sync up. - Chris On Nov 30, 2011, at 1:07 PM, Duncan McGreggor dun...@dreamhost.com wrote: On Tue, Nov 29, 2011 at 12:21 PM, Soren Hansen so...@linux2go.dk wrote: It's been a bit over a week since I started this thread. So far we've agreed that running the test suite is too slow, mostly because there are too many things in there that aren't unit tests. We've also discussed my fake db implementation at length. I think we've generally agreed that it isn't completely insane, so that's moving along nicely. Duncan has taken the first steps needed to split the test suite into unit tests and everything else: https://review.openstack.org/#change,1879 Just one more core +1 needed. Will someone beat me to it? Only time will tell :) Thanks, Duncan! Anything else around unit testing anyone wants to get into The Great Big Plan[tm]? Actually, yeah... one more thing :-) Jay and I were chatting about organization of infrastructure last night/this morning (on the review comments for the branch I submitted). He said that I should raise a concern I expressed for wider discussion: right now, tests are all piled into the tests directory. Below are my thoughts on this. I think such an approach is just fine for smaller projects; there's not a lot there, and it's all pretty easy to find. For large projects, this seems like not such a good idea for the following reasons: * tests are kept separate from the code they relate to * there are often odd test module file naming practices required (e.g., nova/a/api.py and nova/b/api.py both needing test cases in nova/tests/) * there's no standard exercised for whether a subpackage gets a single test case module or whether it gets a test case subpackage * test modules tend to be very long (and thus hard to navigate) due to the awkwardness of naming modules when all the code lives together * it makes it harder for newcomers to find code; when they live together, it's a no-brainer OpenStack is definitely not a small project, and as our test coverage becomes more complete, these issues will have increased impact. I would like to clean all of this up :-) And I'm volunteering to do the work! Here's the sort of thing I envision, using nova.volume as an example: * create nova/volume/tests * move all scheduler-related tests (there are several) from nova/tests into nova/volume/tests * break out tests on a per-module basis (e.g., nova/volume/driver.py would get the test module nova/volume/tests/test_driver.py, etc.) * for modules that have already been broken out at a more fine-grained level, keep (smaller test case modules are nice!) * only nova/*.py files will have a test case module in nova/tests * bonus: update the test runner to print the full dotted path so it's immediately (and visually) clear where one has to go to address any failures Given approval, this work would be done in its own blueprint. All this work would be done in small chunks (probably one branch per module) so that it will be easy to review and adjust the approach as needed. Thoughts? d -- Soren Hansen| http://linux2go.dk/ Ubuntu Developer| http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
It'll be a couple days yet. I was refactoring a few things in the scheduler and while re-doing some tests, I ended up going down this rabbit hole of re-doing all of the tests. It's turned into a 6500 line diff so far... :) which is a bit much for just the refactoring that I need to get in first. So, I'm currently splitting these out into a couple of different reviews. - Chris On Nov 30, 2011, at 1:53 PM, Duncan McGreggor wrote: On 30 Nov 2011 - 19:26, Chris Behrens wrote: I need to catch up a bit with this thread, but I wanted to mention I have a huge patch coming that refactors almost all of the scheduler tests into true unit tests. Nice! I'd started this for other reasons and I hope it jives with the plans here. But if anyone is looking at the scheduler tests, we should sync up. I was going to actually use the scheduler as the example when I sent this email out, but I switched to something a bit cleaner instead... so this is great news! Can't wait to see it :-) d - Chris On Nov 30, 2011, at 1:07 PM, Duncan McGreggor dun...@dreamhost.com wrote: On Tue, Nov 29, 2011 at 12:21 PM, Soren Hansen so...@linux2go.dk wrote: It's been a bit over a week since I started this thread. So far we've agreed that running the test suite is too slow, mostly because there are too many things in there that aren't unit tests. We've also discussed my fake db implementation at length. I think we've generally agreed that it isn't completely insane, so that's moving along nicely. Duncan has taken the first steps needed to split the test suite into unit tests and everything else: https://review.openstack.org/#change,1879 Just one more core +1 needed. Will someone beat me to it? Only time will tell :) Thanks, Duncan! Anything else around unit testing anyone wants to get into The Great Big Plan[tm]? Actually, yeah... one more thing :-) Jay and I were chatting about organization of infrastructure last night/this morning (on the review comments for the branch I submitted). He said that I should raise a concern I expressed for wider discussion: right now, tests are all piled into the tests directory. Below are my thoughts on this. I think such an approach is just fine for smaller projects; there's not a lot there, and it's all pretty easy to find. For large projects, this seems like not such a good idea for the following reasons: * tests are kept separate from the code they relate to * there are often odd test module file naming practices required (e.g., nova/a/api.py and nova/b/api.py both needing test cases in nova/tests/) * there's no standard exercised for whether a subpackage gets a single test case module or whether it gets a test case subpackage * test modules tend to be very long (and thus hard to navigate) due to the awkwardness of naming modules when all the code lives together * it makes it harder for newcomers to find code; when they live together, it's a no-brainer OpenStack is definitely not a small project, and as our test coverage becomes more complete, these issues will have increased impact. I would like to clean all of this up :-) And I'm volunteering to do the work! Here's the sort of thing I envision, using nova.volume as an example: * create nova/volume/tests * move all scheduler-related tests (there are several) from nova/tests into nova/volume/tests * break out tests on a per-module basis (e.g., nova/volume/driver.py would get the test module nova/volume/tests/test_driver.py, etc.) * for modules that have already been broken out at a more fine-grained level, keep (smaller test case modules are nice!) * only nova/*.py files will have a test case module in nova/tests * bonus: update the test runner to print the full dotted path so it's immediately (and visually) clear where one has to go to address any failures Given approval, this work would be done in its own blueprint. All this work would be done in small chunks (probably one branch per module) so that it will be easy to review and adjust the approach as needed. Thoughts? d -- Soren Hansen| http://linux2go.dk/ Ubuntu Developer| http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
+1 on the thoughts here. Exactly what I meant by my reply. Not sure what the
interface should look like for #2, but we must be able to do it somehow.
- Chris
On Nov 29, 2011, at 11:55 AM, Vishvananda Ishaya wrote:
e) is the right solution imho. The only reason joinedloads slipped in is for
efficiency reasons.
In an ideal world the solution would be:
1) (explicitness) Every object or list of related objects is retrieved with
an explicit call:
instance = db.instance_get(id)
ifaces = db.interfaces_get_by_instance(id)
for iface in ifaces:
ip = db.fixed_ip_get_by_interface(iface['id'])
2) (efficiency) Queries are perfectly efficient and all joins that will be
used are made at once.
So the above would be a single db query that joins all instances ifaces and
ips.
Unless we're doing source code analysis to generate our queries, then we're
probably going
to have to make some tradeoffs to get as much efficiency and explicitness as
possible.
Brainstorming, perhaps we could use a hinting/caching mechanism that the
backend could support.
something like db.interfaces_get_by_instance(id, hint='fixed_ip'), which
states that you are about to make
another db request to get the fixed ips, so the backend could prejoin and
cache the results. Then the next
request could be: db.fixed_ip_get_by_interface(iface['id'], cached=True) or
some such.
I would like to move towards 1) but I think we really have to solve 2) or we
will be smashing the database with too many queries.
Vish
On Nov 29, 2011, at 7:20 AM, Soren Hansen wrote:
Hi, guys.
Gosh, this turned out to be long. Sorry about that.
I'm adding some tests for the DB api, and I've stumbled across something
that we should probably discuss.
First of all, most (if not all) of the various *_create methods we have,
return quite amputated objects. Any attempt to access related objects
will fail with the much too familiar:
DetachedInstanceError: Parent instance Whatever at 0x4f5c8d0 is not
bound to a Session; lazy load operation of attribute 'other_things'
cannot proceed
Also, with the SQLAlchemy driver, this test would pass:
network = db.network_create(ctxt, {})
network = db.network_get(ctxt, network['id'])
instance = db.instance_create(ctxt, {})
self.assertEquals(len(network['virtual_interfaces']), 0)
db.virtual_interface_create(ctxt, {'network_id': network['id'],
'instance_id': instance['id']}
self.assertEquals(len(network['virtual_interfaces']), 0)
network = db.network_get(ctxt, network['id'])
self.assertEquals(len(network['virtual_interfaces']), 1)
I create a network, pull it out again (as per my comment above), verify
that it has no virtual_interfaces related to it, create a virtual
interface in this network, and check the network's virtual_interfaces
key and finds that it still has length 0. Reloading the network now
reveals the new virtual interface.
SQLAlchemy does support looking these things up on the fly. In fact,
AFAIK, this is its default behaviour. We just override it with
joinedload options, because we don't use scoped sessions.
My fake db driver looks stuff like this up on the fly (so the
assertEquals after the virtual_interface_create will fail with that db
driver).
So my question is this: Should this be
a) looked up on the fly,
b) looked up on first key access and then cached,
c) looked up when the parent object is loaded and then never again,
d) or up to the driver author?
Or should we do away with this stuff altogether? I.e. no more looking up
related objects by way of __getitem__ lookups, and instead only allow
lookups through db methods. So, instead of
network['virtual_interfaces'], you'd always do
db.virtual_interfaces_get_by_network(ctxt, network['id']). Let's call
this option e).
I'm pretty undecided myself. If we go with option e) it becomes clear to
consumers of the DB api when they're pulling out fresh stuff from the DB
and when they're reusing potentially old results. Explicit is better
than implicit, but it'll take quite a bit of work to change this.
If we go with one of options a) through d), my order of preference is
(from most to least preferred): a), d), c), b).
There's value in having a right way and a wrong way to do this. If it's
either-or, it makes testing (as in validation) more awkward. I'd say
it's always possible to do on-the-fly lookups. Overriding __getitem__
and fetching fresh results is pretty simple, and, as mentioned earlier,
I believe this is SQLAlchemy's default behaviour (somebody please
correct me if I'm wrong). Forcing an arbitrary ORM to replicate the
behaviour of b) and c) could be incredibly awkward, and c) is also
complicated because there might be reference loops involved. Also,
reviewing correct use of something where the need for reloads depends on
previous use of your db objects (which might itself be conditional or
happen
Re: [Openstack] Proposal to add Kevin Mitchell to nova-core
Yep... +1 On Nov 9, 2011, at 5:59 AM, Brian Waldon wrote: Vek has absolutely stepped up and started doing quite few reviews, so I'd like to nominate him to be added to nova-core. Waldon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal to add Johannes Erdfelt to nova-core
+1 -- For a while now, I've been going to review things, finding Johannes's name already in the review list quite often. I also agree with his reviews. :) - Chris On Nov 9, 2011, at 6:02 AM, Brian Waldon wrote: I'd like to nominate Johannes for nova-core, as he has definitely been doing a good number of reviews lately. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OSAPI equivalent of euca-get-console-output ?
For OSAPI: There's POST /version/project/servers/server_id/consoles to create a console Use GET to get consoles for that server_id Then you can use: GET /version/project/servers/server_id/consoles/console_id to get the detailed info. I don't think there's support for this in nova-client. (Look at nova/api/openstack/consoles.py) - Chris On Oct 21, 2011, at 9:30 AM, Day, Phil wrote: Hi Folks, The title says it all really – is there an OSAPI / nova-client equivalent to the EC2 command to get the console output of a VM ?(I can’t see anything in the code or extensions which calls the relevant compute.api method) If there’s nothing at the moment are there any plans for adding in (seems like is should be a core server action rather that an extension) ? Thanks, Phil ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposed amendment to HACKING
+1 On Oct 21, 2011, at 10:37 AM, Jay Pipes wrote: ++ I also believe the requirements to: a) Have a single-line 80 characters brief description AND a long description b) Start the description on the same line as the beginning are silly. I think this: This is a description, long or short, of what the thing does. It doesn't have to have any particular length or lack of length :param some_param: Description of param Is more readable than: This is a description, artificially limited to 80 characters for som This is a longer description that doesn't really need to be here. :param some_param: Description of param -jay On Fri, Oct 21, 2011 at 1:29 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: Our HACKING includes this requirement for multiline docstrings: [...] After you have finished your descriptions add an extra newline and close the quotations. I propose that we remove this line and requirement from HACKING. We don't have to adjust all the docstrings to remove an extraneous space, but we should not require that it be here. Below is my defense: This requirement has never made sense to me, and there does seem to be some disagreement within nova about following this. So, for more insight, I went and read PEP 257, and discovered this rationale: The BDFL [3] recommends inserting a blank line between the last paragraph in a multi-line docstring and its closing quotes, placing the closing quotes on a line by themselves. This way, Emacs' fill-paragraph command can be used on it. (The indicated footnote is just an explanation of the term BDFL.) So, the first point I want to make is that this was a recommendation of PEP 257, rather than a requirement. The second point is that it was due to a limitation of the fill-paragraph command of a single editor. And the third point is that the editor in question, which happens to be my editor of choice, no longer has this limitation, at least in recent versions--fill-paragraph on docstrings terminated by triple-quotes ignores the triple-quotes. Therefore, I believe the recommendation no longer has any substance behind it, and so it should no longer be required of HACKING-conforming code for nova. -- Kevin L. Mitchell kevin.mitch...@rackspace.com This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OSAPI equivalent of euca-get-console-output ?
Ah, I see. consoles.py is for getting info about VNC (or similar) console...ie: create console, get info on how to connect to the vnc console, etc. It doesn't appear compute_api.get_console_output() is exposed in OS API right now, which is what you want. Those compute_api console methods are also currently not implemented for 'xenapi'. - Chris On Oct 21, 2011, at 12:35 PM, Jake Dahn wrote: Chris, What is the output of the detailed info call? I'm actually working on an extension to get tailable console output, and I didn't see the request you mentioned anywhere in the api code. Correct me if I'm wrong, but - consoles.py talks to Consoles.API when i think to get the actual output of things we need to talk to Compute.API if we want to get the actual output of a console and not just info about it. On Oct 21, 2011, at 11:25 AM, Chris Behrens wrote: For OSAPI: There's POST /version/project/servers/server_id/consoles to create a console Use GET to get consoles for that server_id Then you can use: GET /version/project/servers/server_id/consoles/console_id to get the detailed info. I don't think there's support for this in nova-client. (Look at nova/api/openstack/consoles.py) - Chris On Oct 21, 2011, at 9:30 AM, Day, Phil wrote: Hi Folks, The title says it all really – is there an OSAPI / nova-client equivalent to the EC2 command to get the console output of a VM ?(I can’t see anything in the code or extensions which calls the relevant compute.api method) If there’s nothing at the moment are there any plans for adding in (seems like is should be a core server action rather that an extension) ? Thanks, Phil ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OpenStack API v1.0 Removal from Nova
+3 On Oct 11, 2011, at 1:59 PM, Brian Waldon brian.wal...@rackspace.com wrote: I would like to propose we remove our implementation of OSAPI v1.0 from Nova for the following reasons: 1) Our implementation is incomplete, and there are no (visible) plans to complete it. Shared IP Groups and Backup Schedules have been unimplemented since I started on the project. 2) The v1.1 spec is not backwards-compatible with v1.0. I would like minor versions to be backwards compatible as we move forward, so I see v1.0 as something we need to just get rid of. 3) As we are trying to complete the v1.1 implementation, we are running into problems created by having to support v1.0 (specifically image and server uuids). 4) Our implementation of v1.0 and v1.1 within the same modules have caused us to compromise code quality. Working on the controllers (specifically servers) is a royal pain. I've already done the work of removing v1.0 from the code, and here's the review of my branch: https://review.openstack.org/840. I think it makes a lot of sense for the community to focus on making our v1.1 implementation solid, rather than constantly worrying about how we are affecting v1.0. If this is something we don't want to do, I would love to hear why :) Waldon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova DB Connection Pooling
Not sure the 'data that are shared' is the right wording below, but I think I get the point. The one thing that jumps out to me as a current 'issue' is the fact that the instance_types table must be kept in sync. I can't think of anything else at the moment, but there might be something. And the comment (not quoted below) about 'X' dashboards *I think* is invalid. I don't know much about it.. does it talk more than to API? If not, just point it at top level zone. I'm really not a fan of distributed DB architectures if they can be avoided. They add a lot of unneeded complexity and a lot more potential for breakage. The instance type issue can be solved differently. That said, it can be an interesting discussion topic. :) - Chris On Sep 26, 2011, at 8:53 PM, Ed Leafe wrote: On Sep 26, 2011, at 10:14 PM, Joshua Harlow wrote: It seems like it would be good to talk about this during the conference, since it seems sort of odd to have pieces of data that are shared across zones along with pieces of data that are not shared across zones. What pieces of data are shared across zones? -- Ed Leafe This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance formats supported?
There's currently no parsing of manifest.ovf in an .ova. nova assumes that an 'image.vhd' exists within the .ova archive if disk_format is vhd.. - Chris On Sep 23, 2011, at 9:49 AM, Jay Pipes wrote: On Wed, Sep 21, 2011 at 4:02 PM, Joshua Harlow harlo...@yahoo-inc.com wrote: It would be really cool if there were openstack examples for each container/image format. Agreed. This is something we can work on better in the Essex release series indeed. I don’t currently understand how the container format connects into the actual image (and the result with nova-compute). Since it seems like say when u upload an iso, ovf doesn’t make sense (so would that be the bare type?) Or if u do the following “disk_format=vhd container_format=ovf type=raw” windows.ova why is it needed to specify the disk_format? Isn’t that part of the ova structure? Also, will these varying types of formats work with VM’s used by openstack. Is there a conversion that occurs when a VM is started to the right format (for say kvm? or for say xen? since I doubt either takes the ova format directly...). I'd like to ask Paul Voccio or Josh Kearney to answer the above question, since it's not something I'm too familiar with... Paul, Josh? Thanks, jay On 9/21/11 12:56 PM, Jay Pipes jaypi...@gmail.com wrote: On Wed, Sep 21, 2011 at 3:44 PM, Joshua Harlow harlo...@yahoo-inc.com wrote: Hi all, I was wondering how these different types of formats supported by glance @ http://glance.openstack.org/formats.html will be workable with nova-compute. Just an FYI, Glance does not (currently) do any inspection to ensure that what the user is registering in Glance is in fact the disk or container format that the user says it is. It seems like most examples I see are using precreated ami/ari/aki images. I haven’t seen a lot of examples using say an iso, or a ovf container. Has anyone been successful with those when creating a vm? XenServer supports OVF images, and I believe Rackspace teams have tested with OVAs. You can see some basic discussion here: http://wiki.openstack.org/XenServerDevelopment#Instance_Management Cheers, jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal to add Chris Behrens to nova-core
Thanks! On Sep 22, 2011, at 8:59 AM, Soren Hansen wrote: I just added Chris to nova-core. Welcome to the team, Chris! I'll add you to the review rotation next time that schedule gets updated. -- Soren Hansen| http://linux2go.dk/ Ubuntu Developer| http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] A possible alternative to Gerrit ...
Sure, I agree with the below. I tend to think the PPB is the place for the decision for the reasons you state below (though that's more than 3-4 people tops). But whether it's the PPB or some other small group of people, I'd want to see everybody have a chance to provide enough feedback for that small group to be able to make a well-informed decision. Obviously some sort of time limit has to be set, but there should be enough time and communication that the resulting decision doesn't come out of no where. That is a blanket statement of my opinion and would be my thoughts no matter how previous events transpired. I say this for future decisions. At the moment, Gerrit is what was chosen and I'm just interested in seeing if we can alleviate some of the pain my team is having working with it. - Chris On Sep 8, 2011, at 12:11 AM, Soren Hansen wrote: If this thread has anything clear to me at all, it's that adding *more* people to this discussion isn't going to bring us any closer to an agreement. Here's a thought: How about we appoint (formally, informally, whatever, it's beside the point) someone (3-4 people tops) to come up with a set of tools and the rest of us just shut up, use the tools, write some cool software and don't waste another full development cycle arguing about stuff we'll never agree on anyway? Yes, I will probably waste a good 10-20 minutes retraining my muscle memory to not type bzr, but rather darcs, hg, git or whatever this group comes up with. The alternative is to argue about this for months, wait for something to get built based on this discussion, start using it, wait a couple of weeks until the next mutiny, lather, rinse, repeat. Seriously. Who -- apart from our competitors -- gains anything at all from this? -- Soren Hansen| http://linux2go.dk/ Ubuntu Developer| http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] A possible alternative to Gerrit ...
On Sep 7, 2011, at 9:21 AM, Jay Pipes wrote: The problem is that instead of spending time coding on features and bugs for Nova, Glance, Swift and Keystone, a bunch of devs are instead spending time working on an alternate solution to what has already been decided by the PPB, discussed publicly, and coded on for months by my team. It was not discussed publicly, unless I've somehow not received a ton of email. There was no announcement that I can find to the list that 'Gerrit' was decided by the PPB. There was no discussion about how we might want to modify it to fit our needs. The development all happened mostly in the private as far as I can tell. Not a lot of room for complaints until it was implemented for a repo that a fair amount of people use. So, now there's complaints. Frankly, this whole situation hearkens back to when everyone except a few folks on the Titan team was griping about Launchpad and Bazaar, saying that they couldn't get anything done with these tools, except for some on the Titan team which just adapted and ended up being very productive using LP and Bazaar. If you guys would give Gerrit a chance and try to work with the toolset, instead of constantly going off and trying to make everything just GitHub, I think we'd actually be able to get more done. Just my two cents. You can probably tell I am quite frustrated with this debate going on *yet again*. I understand your frustration. I would be too if it were my team that had done months of development on it. But I feel some of it could have been avoided if there actually public discussion up front. That said, Sandy just provided a list of annoyances. Johannes has provided some as well. I haven't seen any comments, yet, on if any of those can be addressed. - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] A possible alternative to Gerrit ...
On Sep 7, 2011, at 11:46 AM, Jay Pipes wrote: On Wed, Sep 7, 2011 at 2:19 PM, Chris Behrens chris.behr...@rackspace.com wrote: On Sep 7, 2011, at 9:21 AM, Jay Pipes wrote: The problem is that instead of spending time coding on features and bugs for Nova, Glance, Swift and Keystone, a bunch of devs are instead spending time working on an alternate solution to what has already been decided by the PPB, discussed publicly, and coded on for months by my team. It was not discussed publicly, unless I've somehow not received a ton of email. There was no announcement that I can find to the list that 'Gerrit' was decided by the PPB. False. https://lists.launchpad.net/openstack/msg03493.html http://eavesdrop.openstack.org/meetings/openstack-meeting/2011/openstack-meeting.2011-08-09-20.00.log.html Thanks. I apparently missed the emails about the voting taking place.. I was actually trying to dig further back than 1 month ago. I knew there were discussions last month. In fact, I remember your 'Ready to move' email from 8/1 which was apparently before the vote. This actually doesn't make me feel any better, because it feels like a decision was made even prior to the PPB voting. You claimed 3 months of development. That means at least 2 months of development happened prior to the PPB voting. The CI repos were also moved prior to the PPB voting. During none of the 2 months of development can I find any references to Gerrit on _this_ list. Not until Monty's note on 7/18 about CI repos moved. There was no discussion about how we might want to modify it to fit our needs. False. There was lots of discussion on it. On this list there was not 'lots' prior to 1 month ago. Some conversations about GitHub, project autonomy, and migration going back to June: http://www.mail-archive.com/openstack-poc@lists.launchpad.net/msg00176.html https://lists.launchpad.net/openstack-poc/msg00150.html http://osdir.com/ml/openstack-cloud-computing/2011-08/msg00077.html Sorry, I don't read openstack-poc. Maybe I should. I don't feel like that's 'public discussion', though. The development all happened mostly in the private as far as I can tell. False. Not a single thing about the CI environment or the automated patch queue management has been in private. It is all on GitHub, in public repos, and documented publically: https://github.com/openstack/openstack-ci http://ci.openstack.org http://wiki.openstack.org/GerritWorkflow Sorry, that's not what I meant. I realize that's what I said, but I really didn't mean that all of the code wasn't open. :) I meant that a lot of us knew nothing about this Gerrit development until well into the process, giving us only a choice to voice complaints very late in the process... and apparently mostly not until around the time of the actual PPB vote. That said, Sandy just provided a list of annoyances. Johannes has provided some as well. I haven't seen any comments, yet, on if any of those can be addressed. Refresh your email. LP's mailing lists are slow today. There's been responses to all lists of annoyances, Chris. Thanks. I see them. It's not that I didn't think there would be responses. I'm just trying to keep us on track to trying to resolve the issues while I still complain that *I* feel we should have had some more/earlier/better communication on this list. :P - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] A possible alternative to Gerrit ...
Jay, On Sep 4, 2011, at 10:52 AM, Jay Pipes wrote: I actually didn't plan on responding all that much on this conversation. We had months of discussion and debate about this, weeks upon weeks of discussion in the PPB about project autonomy and tooling, and the decision has been made. I find it a bit unfortunate that all the people saying Gerrit is terrible and that we should just use GitHub haven't done a single review or change request in any of the projects that are currently using the Gerrit/Git toolset that has been decided will be used for core OpenStack projects. [...] I'm not sure this is true. A few people that I've seen speaking up have used it. I've been wanting to comment for a while, but I don't have a lot of ground to stand on, because I'm in the boat that you describe. I've not done a single review or change request with Gerrit yet. That said, I've seen a bit from those who have and I'm very much the opposite of excited about moving nova to it at some point, at least how it stands now. But leaving aside whether I like it or dislike it, what really bothers me is that there was discussion about moving things to github. And, I was 'ok' with that decision to do so despite preferring bzr and LP. My 'ok' was based on knowing how git, github pull requests, reviews, and so forth work. Now I feel like we're moving things to something to which I (and the community) never agreed. I never saw any discussion about Gerrit on the mailing list as far as is everyone cool with this? The first mention of it that I can find was July 18th regarding moving the CI repos. Doesn't seem like we were given much of an option. That really irks me. Above you say that has been decided will be used for the core OpenStack projects. So, I have to ask: 'Who decided?'. I must have missed something. And I want to be clear that I'm not meaning to put down anyone's efforts here. I'm positive a lot of hard work was put into the transition, and I do appreciate it. - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] libvirt vs. Xen driver handling of local storage
On Sep 2, 2011, at 8:07 AM, Paul Voccio wrote: On Fri, Sep 2, 2011 at 8:01 AM, Soren Hansen so...@linux2go.dk wrote: [...] The potential for filesystem bugs that could bring the host down gives me the heebie jeebies. I really, really don't want to mount people's filesystems. Can you explain a bit more here? I would like to understand your concerns. I would advocate mounting in a utility VM if you mean to protect from mounting instance with malicious data. We may have to do this to expand partitions or resize down for Windows. Mounting someone's filesystem should not be necessary if we have certain restrictions on the management. I.e., we could say we will only resize the last filesystem in the partition table. That would avoid needing to know the filesystem layout in the image (looking at /etc/fstab or updating it). Not sure that's a desired restriction, however. That said, we'd still need to attach the VM disk somewhere and run fs resize utils... and it might still be best to do this in a utility VM. - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] libvirt vs. Xen driver handling of local storage
Yeah, I think that can be rather fair for Unix. It's just that as you pointed out... Windows is a huge pain. Need to make sure there's enough space on C: and I think there are still a lot of things that stupidly rely on being installed on C: On Sep 2, 2011, at 10:32 AM, Paul Voccio wrote: My first thought was to do a singled fixed disk and never resize that disk at all. If you need space, you have to use a volume service. Ultimately, I don't think this the right approach either, but it solves the initial use case of needing more storage space. On Fri, Sep 2, 2011 at 11:34 AM, Chris Behrens chris.behr...@rackspace.com wrote: On Sep 2, 2011, at 8:07 AM, Paul Voccio wrote: On Fri, Sep 2, 2011 at 8:01 AM, Soren Hansen so...@linux2go.dk wrote: [...] The potential for filesystem bugs that could bring the host down gives me the heebie jeebies. I really, really don't want to mount people's filesystems. Can you explain a bit more here? I would like to understand your concerns. I would advocate mounting in a utility VM if you mean to protect from mounting instance with malicious data. We may have to do this to expand partitions or resize down for Windows. Mounting someone's filesystem should not be necessary if we have certain restrictions on the management. I.e., we could say we will only resize the last filesystem in the partition table. That would avoid needing to know the filesystem layout in the image (looking at /etc/fstab or updating it). Not sure that's a desired restriction, however. That said, we'd still need to attach the VM disk somewhere and run fs resize utils... and it might still be best to do this in a utility VM. - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] libvirt vs. Xen driver handling of local storage
Vish, I think Rackspace ozone/titan has some upcoming work to do for the resizing for xenserver that might close some of the gap. I think we need some options (flags) if we are to synchronize libvirt/xen. At some point, Rackspace also needs an API extension to support a couple different ways of handling resizes. Until we get there, we at least need an option to keep the xenserver code working as-is for now. I assume others need the current libvirt implementation to stay as well. That said, I think it's probably not too difficult to do the 'libvirt way' for Xen, but I don't know about it making diablo. Adding support into libvirt to do the 'xen way' should be easier, I'd think. But I'm the opposite of you, Vish. I don't know the libvirt layer as well. :) If we can FLAG the way it works... and make these options work in both libvirt/xen, I think we can all remain happy. - Chris On Aug 31, 2011, at 11:45 AM, Vishvananda Ishaya wrote: Hey guys, We have a very annoying discrepancy between how local space is used in the xen driver vs the libvirt driver. I think it is vital that this is rectified before the Diablo release. We already have a few functional gaps between the drivers, but the fact that disks are partitioned completely differently between the two is very confusing to users. Bug is here: https://bugs.launchpad.net/nova/+bug/834189 The libvirt driver: * downloads the image from glance * resizes the image to 10G if it is 10G (in the case of a separate kernel and ramdisk image it extends the filesystem as well. In the case of a whole-disk image it just resizes the file because it doesn't know enough to change the filesystem) * attaches a second disk the size of local_gb to the image (when using block device mapping through the ec2 api, more swap/ephemeral disks can be attached as volumes as well) The XenServer driver (I'm less familiar with this code so please correct me if i am wrong here): * downloads the image from glance * creates a vdi from the base image * resizes the vdi to the size of local_gb The first method of resize to 10G and having separate local_gb is essentially the strategy taken by aws. Drawbacks of the first method: 1) The actual space used by the image is local_gb + 10G (or more if the base image is larger than 10G) which is inconsistent. 2) The guest has to deal with the annoyance of not having one large filesystem. It is easier for the user if they can just use all the space that they have without thinking about it. Drawbacks of the second method: 1) Limits cloud images to a particular format. We can't always guarantee that we can resize the image properly. We need to decide on a common strategy and use it for both hypervisors. Vish ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] New nova service proposal
I was wondering if some of this could be solved by simply using rpc.call vs rpc.cast so that we get appropriate responses, even if they are exceptions. - Chris On Aug 26, 2011, at 2:23 PM, Brian Lamar wrote: Hey Ed, I absolutely agree that we need to be confident that all requests will be handled by the system eventually. However, I'm unsure of the need for a new service to be created to handle error cases. I'm not saying that we can solve every case through our current architecture, but with some subtle tweaks I think it can be made much more reliable. I feel like if we look at every place where errors *can* occur we can find solutions not involving an external service. Are there any particular bugs/situations that are happening which aren't listed as bugs in Launchpad? Long story short I'd rather not create an external service which attempts to clean up after poor exception handling / bad logic, but there might be some cases I'm not considering. Brian -Original Message- From: Ed Leafe e...@leafe.com Sent: Friday, August 26, 2011 3:22pm To: openstack@lists.launchpad.net Subject: [Openstack] New nova service proposal Sorry I haven't come up with a snazzy name for it yet, but what I have in mind is a new service that is essential for my employer (Rackspace), and might be important for other OpenStack deployments. This new service would be completely optional, of course - only those for whom it is relevant would run it. Let me start by stating the problem: when a customer requests that we create instances for them, nova casts those requests into the queue, where they are eventually acted upon. That usually works great, but in cases where the instance creation fails, we need to detect that failure and re-issue the create request with a different host. This is currently not possible with the asynchronous design of the compute-scheduler interactions. So what I envision is a service that scans a list of recent requests' reservation IDs, and follows up to see if the request was successful or not, and takes action if needed. The blueprint for this can be found at https://blueprints.launchpad.net/nova/+spec/instance-creation-assurance, with an Etherpad created for ongoing idea exchange at http://etherpad.openstack.org/instance-creation-assurance -- Ed Leafe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [SPAM] How to connect br100 to xen bridge
Depending on what you're doing for networking, you should be able to shove this into nova.conf: --flat_network_bridge=xenbr0 to avoid modifying the DB.. - Chris On Aug 17, 2011, at 5:34 PM, Wayne A. Walls wrote: Greetings, Josh! Just update the bridge in the db, I've done that in the past to get things aligned with Xen. Seems to do the trick... Cheers, Wayne Sent from my iPhone On Aug 17, 2011, at 7:17 PM, Joshua Harlow harlo...@yahoo-inc.com wrote: Hi all, Just trying to continue with my xen+openstack integration. Is there a recommended way to get openstack to connect to the xen bridge. Apparently the xen bridge is @ xenbr0 but openstack likes to use br100. Is it just a simple DB update to make this happen, or can this be changed in the nova-network config? Has anyone had any experience with this? Thanks ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Physical host identification
I see the v1.1 API spec talks about a 'hostId' item returned when you list your
instances (section 4.1.1 in the spec). These should be the same thing, IMO.
I think you're right, though. I don't believe we have any sort of 'hostId'
today, since hosts just become available by attaching to AMQP.
- Chris
On Jul 15, 2011, at 1:16 PM, Glen Campbell wrote:
I understand that we're all familiar with virtualization and its benefits.
However, in the Real World, those of us who run clouds often need to work
with physical devices. I've proposed a blueprint and spec for a /hosts admin
API resource that would return information on physical hosts. However, I
don't believe that there's any way for us to actually identify a specific
server (I'm actually hoping I'm mistaken about this, because that would make
my life easier).
So, to get information about a specific host, you'd use /host/{id} — but what
should go in the {id} slot?
We'd also like to include this data elsewhere; for example, in error
messages, it might help to know the physical device on which a server is
created.
signature[4].png
This email may include confidential information. If you received it in error,
please delete it.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
This email may include confidential information. If you received it in error,
please delete it.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Physical host identification
I think it's sensitive because one could figure out how many hosts a SP has
globally... which a SP might not necessarily want to reveal.
- Chris
On Jul 15, 2011, at 3:34 PM, karim.allah.ah...@gmail.com wrote:
On Fri, Jul 15, 2011 at 11:31 PM, Chris Behrens chris.behr...@rackspace.com
wrote:
Nevermind. Just found a comment in the API spec that says hostID is unique
per account, not globally. Hmmm...
This is weird ! I can't find anything in the code that says so !! hostID is
just a hashed version of the 'host' which is set as the 'hostname' of the
physical machine and this isn't user sensitive. So, It's supposed to be a
global thing !
Can somebody explain how this is a user sensitive ?
On Jul 15, 2011, at 2:27 PM, Chris Behrens wrote:
I see the v1.1 API spec talks about a 'hostId' item returned when you list
your instances (section 4.1.1 in the spec). These should be the same
thing, IMO.
I think you're right, though. I don't believe we have any sort of 'hostId'
today, since hosts just become available by attaching to AMQP.
- Chris
On Jul 15, 2011, at 1:16 PM, Glen Campbell wrote:
I understand that we're all familiar with virtualization and its benefits.
However, in the Real World, those of us who run clouds often need to work
with physical devices. I've proposed a blueprint and spec for a /hosts
admin API resource that would return information on physical hosts.
However, I don't believe that there's any way for us to actually identify
a specific server (I'm actually hoping I'm mistaken about this, because
that would make my life easier).
So, to get information about a specific host, you'd use /host/{id} — but
what should go in the {id} slot?
We'd also like to include this data elsewhere; for example, in error
messages, it might help to know the physical device on which a server is
created.
signature[4].png
This email may include confidential information. If you received it in
error, please delete it.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
This email may include confidential information. If you received it in error,
please delete it.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
--
Karim Allah Ahmed.
LinkedIn
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
This email may include confidential information. If you received it in error,
please delete it.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cross-zone instance identifiers in EC2 API - Is it worth the effort?
On Jul 11, 2011, at 12:01 PM, Ed Leafe wrote: On Jul 11, 2011, at 2:04 PM, Eric Day wrote: How is nova-account-instance uuid any different than: ---- Where // (or some subset of them) are reserved/regulated? Nothing, if -- is a full UUID. If we compare to swift, the account prefix is a UUID too. The account prefix could be fixed for a session or passed in to every request depending on how things are decided. sigh It's a shame that the ipv6 proposal was never more fully considered. That would handle the uniqueness, with the added benefit of providing simple zone routing via DNS, with the exact same 128-bit/32 char size. I don't I remember that proposal, but that's such a neat idea. Was anything discussed at all in Santa Clara regarding encoding zone information in the instance identifier? I apparently missed the instance identifier discussion somehow. - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cross-zone instance identifiers in EC2 API - Is it worth the effort?
On Jul 11, 2011, at 12:37 PM, Ed Leafe wrote: On Jul 11, 2011, at 3:24 PM, Chris Behrens wrote: It's a shame that the ipv6 proposal was never more fully considered. That would handle the uniqueness, with the added benefit of providing simple zone routing via DNS, with the exact same 128-bit/32 char size. I don't I remember that proposal, but that's such a neat idea. Was anything discussed at all in Santa Clara regarding encoding zone information in the instance identifier? I apparently missed the instance identifier discussion somehow. At the end of the instance referencing discussion, Van Lindbergh brought up the idea. We discussed it with several people both in the conference room and over lunch. I believe that the main objections were that not everyone would have an IPv6 creation scheme, whereas UUID generators are ubiquitous. The other was a vague concern Ya, I was guessing that would be a concern. Doesn't seem like a huge deal, since everyone should be moving towards ipv6 only anyway. I could see some objections raised if you didn't want any sort of public network interface at all... but you could still assign an unused address. about revealing internal structure, but since the information revealed would be the exact same info in the instance's public network info, it didn't strike me as a serious concern. Right. As long as there is a public network interface, people are going to be able to figure out at least some part of 'internal structure'. - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cross-zone instance identifiers in EC2 API - Is it worth the effort?
If you're referring to encoding zone information, yes it would. I was trying to ask more generally as well. IPv6 would be a very good solution, IMO. - Chris On Jul 11, 2011, at 12:47 PM, Sandy Walsh wrote: Won't an IPv6 address do that by it's very nature? -S From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of Chris Behrens [chris.behr...@rackspace.com] Sent: Monday, July 11, 2011 4:24 PM To: Ed Leafe Cc: openstack@lists.launchpad.net; Chris Behrens Subject: Re: [Openstack] Cross-zone instance identifiers in EC2 API - Is it worth the effort? On Jul 11, 2011, at 12:01 PM, Ed Leafe wrote: On Jul 11, 2011, at 2:04 PM, Eric Day wrote: How is nova-account-instance uuid any different than: ---- Where // (or some subset of them) are reserved/regulated? Nothing, if -- is a full UUID. If we compare to swift, the account prefix is a UUID too. The account prefix could be fixed for a session or passed in to every request depending on how things are decided. sigh It's a shame that the ipv6 proposal was never more fully considered. That would handle the uniqueness, with the added benefit of providing simple zone routing via DNS, with the exact same 128-bit/32 char size. I don't I remember that proposal, but that's such a neat idea. Was anything discussed at all in Santa Clara regarding encoding zone information in the instance identifier? I apparently missed the instance identifier discussion somehow. - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cross-zone instance identifiers in EC2 API - Is it worth the effort?
Yeah, I'm not sure *how much* of the zone information would be sensitive, though. Ie, is it okay to expose a unique identifier and nothing more? Or do we want to expose _nothing_? - Chris On Jul 8, 2011, at 2:28 PM, Sandy Walsh wrote: Isn't there a concern of leaking internal Zone information to the outside world (particularly in the Service Provider model)? If so, we're back to the mapping table. And, when multi-instance boot commands are more common (provision me 10 servers vs. 1), then more people will be searching by Reservation Id, Project Id or Owner Id (I suspect). So, how long will this be a problem for? Do the same quirks apply to EC2 Reservation ID's as Instance ID's? -S From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of Chris Behrens [chris.behr...@rackspace.com] Sent: Friday, July 08, 2011 5:43 PM To: George Reese Cc: openstack@lists.launchpad.net; Ed Leafe; Chris Behrens Subject: Re: [Openstack] Cross-zone instance identifiers in EC2 API - Is it worth the effort? On Jul 8, 2011, at 5:11 AM, George Reese wrote: I would just like to re-iterate that I think the entire UUID approach is flawed and issues like this are one of the key reasons why. The only problem I'm aware of is that developers using the EC2 API are not adhering to the spec. If everyone treated them as strings, as they are supposed to be, then we wouldn't have to have this discussion. That said, I do have a particular problem with the current UUID implementation in that I wish some sort of unique zone identifier were a part of it. Accompany that with some other changes to zones and we could have more efficient zone routing. A side effect of that would be that it is less work to come up with an ID that, if truncated, would also be unique to EC2. Taking Ed Leafe's approach, you could remove the recursive zone checks. I'm not sure I'd vote for that route, though. It implies we kludge UUID generation just for EC2 in the heart of nova, which I think is completely wrong. I'm pretty much with Vish on everything he's said so far in this thread. - Chris This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp This email may include confidential information. If you received it in error, please delete it. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Injecting user data into instances
Hm, inject_file is supposed to be tied into instance creation in the API. If it's not, we have some code missing from API. Ed Leafe did this work IIRC, as we use this here at Rackspace. Wonder if some code got dropped at some point or if it was just never completely finished. - Chris On Jun 8, 2011, at 8:27 PM, Mark Washenberger wrote: I don't know much about Cloudpipe and VPN, so I hope I don't hijack the thread. However, regarding inject_file Another interesting situation is with inject_file compute APIs … on API level there is no even file/contents fields, only def inject_file(self, context, instance_id): but they exist on compute.manager level: def inject_file(self, context, instance_id, path, file_contents): I believe that the nova.compute.api inject_file function is there erroneously. From my brief look, it does not appear to be called anywhere and it does not appear that it would work correctly. With the current feature set it seems like the only time that you can inject a file is during other api actions similar to instance creation. I take it this is a feature you would like to be able to use when the instance has been running for some time? Thanks ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Global deployment of Glance
Each zone should definitely have glance instances, IMO. At least two per zone for redundancy and networking reasons in large OpenStack installations. There's some work to do to support this, though. - Chris On May 17, 2011, at 8:59 AM, Glen Campbell wrote: If we are going to deploy Glance to support a global deployment of Nova, would it make sense to have replicas in different regions for better performance? Or, to put it another way, is there a recommended way to keep multiple Glance installations in sync? Users doing snapshots/backups, etc., would presumably get better performance if Glance was local, but how would we keep the base/shared images in sync? signature[4].png Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Global deployment of Glance
Ignoring how it is actually implemented, I think we do want copies of base images in reach region. We don't want any sort of outage in one region to adversely affect another region. - Chris On May 17, 2011, at 9:36 AM, Jay Pipes wrote: On Tue, May 17, 2011 at 11:59 AM, Glen Campbell glen.campb...@rackspace.com wrote: If we are going to deploy Glance to support a global deployment of Nova, would it make sense to have replicas in different regions for better performance? Or, to put it another way, is there a recommended way to keep multiple Glance installations in sync? Hi Glen! I think a better idea than having multiple copies of an image in different regions is to do two things: a) Use a proxy caching server like Varnish or Squid to cache pieces or all of an image in various zones b) Use a highly-available storage system like Swift behind the global Glance server For a) we need to complete the HTTP 1.1 Cache headers blueprint (https://blueprints.launchpad.net/glance/+spec/caching) and for b) you would simply use the Swift backend, configured appropriately for a large Swift cluster. Users doing snapshots/backups, etc., would presumably get better performance if Glance was local, but how would we keep the base/shared images in sync? This is actually something that Rick H and Chris McG are working on. The basic strategy that they came up with was to add a parent ID attribute to the image and for any snapshot image, simply refer to the base image as the snapshot image's parent. The glance client would check for a parent_id that wasn't null and continue streaming the image while it found a parent URI/ID. For example, let's say you have a golden image with the URI: http://glance.example.com/12345. A user creates an instance with this image and some time later, decides to do a snapshot or backup of their running instance. The snapshotting code in the virtualization layer produces what is essentially a differential snapshot, containing only the differing bits of the existing image with the base golden image. This snapshot (typically much smaller than the original image) could be stored in the local (zone-local) Glance server with a call to POST /images. When pushing this snapshot image to the local Glance server, we would set the parent ID to http://glance.example.com/12345. Let's say at some later time, the user wanted to restore from this backup. The virtualization layer that implemented the restore call would need to stream the backup image from the local Glance server. In doing so, it would use the glance client class' get_image() method. When calling this method, the glance client would first return the snapshot image piece. Noticing the image had a parent ID, it would continue to stream the golden image from the global image Glance server in-line, essentially enabling us to store only the small diff of the snapshot locally while streaming the bulk of the image master from the global Glance server. I'll let Rick elaborate on the above and correct any mistakes I made in my description. :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Code Reviews
I'm not nova-core, but this makes a heck of a lot of sense to me, too. On May 11, 2011, at 1:16 PM, Jay Pipes wrote: ++ on your suggestions. On Wed, May 11, 2011 at 3:38 PM, Vishvananda Ishaya vishvana...@gmail.com wrote: Hello Everyone, We have quite a large backlog of merge proposals here: https://code.launchpad.net/~rlane/nova/lp773690/+merge/59565 I've been attempting to go through them to find some high priority ones to review. It seems like people are being pretty active in reviewing branches, but there are a lot old branches that haven't been touched in a while. So first I have a general request that anyone who has old branches in for review: please update your branches or mark them Work In Progress to remove them from the review queue. I'd also like to propose a change to our process that will make the ready to review branches easier to find. I'd like for nova-core to set branches to WIP if there are two significant needs fixings or or needs information. That way everyone doesn't have to sort through branches that have already been reviewed but are waiting on updates. We may need to use our judgement here, so if a large branch has a needs fixing for a minor typo or some such, you could leave it under needs review so it gets viewed by more people. Here is an example where i think this policy will be useful: You see a branch that already has a 'Needs Fixing: this needs a failing test. If you look at the branch and reach the same conclusion, you can mark it Needs Fixing: I agree, needs a test like xxx and then set the branch to Work In Progress. When the author has added the test or needs to make more comments, he can set it back to Needs Review. I think this will generally keep the review board a little cleaner and also each branch will end up with a couple of people that are queued to review once the changes have come in. Does this seem acceptable to everyone? If I don't here any major dissents, I will add this info to the wiki and we can put it into practice. Vish ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance IDs and Multiple Zones
It's early here, but I think it's closer to 200 zones? :) On Mar 24, 2011, at 5:16 AM, Ed Leafe e...@leafe.com wrote: On Mar 23, 2011, at 9:41 PM, Justin Santa Barbara wrote: The type of a server @id in CloudServers is xsd:int, which is a 32-bit signed integer: http://docs.rackspacecloud.com/servers/api/v1.0/xsd/server.xsd So if you have 1 billion integers per zone, you only get 2 zones. You can have 4 if you're willing to go negative, but surely it's too early in the campaign. Yes, you're correct. That always trips me up: why would anyone pick a signed integer for a PK? OK, so I'll slice the ranges down to the current Rackspace practice of 10 million. That will allow for around 2000 zones. -- Ed Leafe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance IDs and Multiple Zones
You have a fundamental misunderstanding of my fundamental understanding of how inter-zone communication works. :) I understand how it works. I'm asking about an admin API that has privileges for actions for all VMs. As an ISP, I want to disable a particular VM because it's being 'bad'. If someone has injected a collision, I would be sending an action to more than 1 VM, not only the intended target. I don't see how collisions can be made to work at all. And yes, we're talking about spoofing (or really, purposefully colliding a known UUID). I haven't seen any mention to anything else (although I may have missed it). I'm certainly really not worried about machine generated UUIDs colliding, myself. But what we're also talking about here is efficient routing. Is it necessary? No. Would it scale? Yes. A zone name or ID needs to be part of the identifier. I prefer the DNS name idea, although prefixing UUIDs or reserving bits in a UUID could also work. - Chris On Mar 23, 2011, at 9:01 AM, Ed Leafe wrote: On Mar 23, 2011, at 11:28 AM, Chris Behrens wrote: How would the admin API know which ID to work with if there are collisions? Eric's point is that we'd not know where to route the request. This reflects a fundamental misunderstanding of the way inter-zone communication works. There is no direct routing. Instead, a zone knows about its instances and its child zones. If the zone receives a request for some action involving a particular instance, it checks if it has that instance among its compute nodes; if not, it forwards the request to each of its child zones. That is repeated until the leaf zones are reached, and most of those will respond with something akin to a 404, indicating that they didn't handle the request. The zone that does have the requested instance, though, will carry out the action and return the result of that action. The child zone responses are then aggregated. If all indicate 404, the zone returns the same. If one child responds that it has handled the request, that response is returned. This repeats back up the zone tree until the zone that originally received the request has heard from all of its child zones (or they timed out). If there were to be a collision (i.e., two leaf nodes handling the request), there are only two possibilities: either the authenticated user has rights to those nodes, or they do not. If they do not, nothing will happen beyond an authorization failure message. If they do have rights to both instances, then the action will happen to both instances. Since the context of this discussion is deliberate spoofing, my response would be serves them right. :) So it seems that spoofing should have no effect, assuming that our authentication/authorization system is sound. If it isn't, then we have bigger issues than just ID spoofing, since I could write a program to send API delete requests for random instance IDs - no spoofing required. Without spoofing, let's be realistic: the chance of duplicate uuid values colliding is much, much smaller than the chance of a meteorite smashing into our data centers. From Wikipedia: In other words, only after generating 1 billion UUIDs every second for the next 100 years, the probability of creating just one duplicate would be about 50%. I believe that that is well beyond our scalability goals, so we can effectively ignore the impact of non-spoofed collisions. -- Ed Leafe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Instance IDs and Multiple Zones
I think Dragon got it right. We need a zone identifier prefix on the IDs. I think we need to get away from numbers. I don't see any reason why they need to be numbers. But, even if they did, you can pick very large numbers and reserve some bits for zone ID. - Chris On Mar 22, 2011, at 10:48 AM, Justin Santa Barbara wrote: I think _if_ we want to stick with straight numbers, the following are the 'traditional' choices: 1) Skipping - so zone1 would allocate numbers 1,3,5, zone2 numbers 2,4,6. Requires that you know in advance how many zones there are. 2) Prefixing - so zone0 would get 0xxx, zone1 1xx. 3) Central allocation - each zone would request an ID from a central pool. This might not be a bad thing, if you do want to have a quick lookup table of ID - zone. Doesn't work if the zones aren't under the same administrative control. 4) Block allocation - a refinement of #3, where you get a bunch of IDs. Effectively amortizes the cost of the RPC. Probably not worth the effort here. (If you want central allocation without a shared database, that's also possible, but requires some trickier protocols.) However, I agree with Monsyne: numeric IDs have got to go. Suppose I'm a customer of Rackspace CloudServers once it is running on OpenStack, and I also have a private cloud that the new Rackspace Cloud Business unit has built for me. I like both, and then I want to do cloud bursting in between them, by putting an aggregating zone in front of them. I think at that stage, we're screwed unless we figure this out now. And this scenario only has one provider (Rackspace) involved! We can square the circle however - if we want numbers, let's use UUIDs - they're 128 bit numbers, and won't in practice collide. I'd still prefer strings though... Justin On Tue, Mar 22, 2011 at 9:40 AM, Ed Leafe e...@leafe.com wrote: I want to get some input from all of you on what you think is the best way to approach this problem: the RS API requires that every instance have a unique ID, and we are currently creating these IDs by use of an auto-increment field in the instances table. The introduction of zones complicates this, as each zone has its own database. The two obvious solutions are a) a single, shared database and b) using a UUID instead of an integer for the ID. Both of these approaches have been discussed and rejected, so let's not bring them back up now. Given integer IDs and separate databases, the only obvious choice is partitioning the numeric space so that each zone starts its auto-incrementing at a different point, with enough room between starting ranges to ensure that they would never overlap. This would require some assumptions be made about the maximum number of instances that would ever be created in a single zone in order to determine how much numeric space that zone would need. I'm looking to get some feedback on what would seem to be reasonable guesses to these partition sizes. The other concern is more aesthetic than technical: we can make the numeric spaces big enough to avoid overlap, but then we'll have very large ID values; e.g., 10 or more digits for an instance. Computers won't care, but people might, so I thought I'd at least bring up this potential objection. -- Ed Leafe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Queue Service, next steps
I am very concerned about seeing a lot of I'd love to learn Erlang on something we probably don't want to screw up. At the same time, I know there's a number of us very experienced with C/C++ (even with respect to concurrency/scalability). - Chris On Feb 21, 2011, at 2:40 AM, Thierry Carrez wrote: Quick comment on the languages from a dev community perspective: The Nova/Swift community is Python-based, so you won't directly reuse it, whether you use C++ or Erlang. For new contributors, you either target very experienced C++ developers (to get concurrency/scalability right), or moderately-experienced Erlang devs. Both probably come in short supplies... but you might find more people willing to learn Erlang than people willing to perfect their C++ skills (me for example). I've a slight preference for Erlang because it will probably be more difficult to do it wrong, and my gut feeling is that Python devs might be more convertible to Erlang than to C++. Regards, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Queue Service, next steps
I have been meaning to reply with the exact same thing. On Feb 18, 2011, at 10:12 AM, Jay Pipes jaypi...@gmail.com wrote: Personally, I'd prefer C++ since that's what I'm used to, but I'd be open to learning Erlang, too. Been wanting to learn it for a while now. -jay On Fri, Feb 18, 2011 at 1:11 AM, Eric Day e...@oddments.org wrote: Duck farm? :) Are you two concerned about building a developer community around a project in Erlang? I'm all for going that route if other folks are comfortable with it. I also have some concern around the speed of Erlang. It's great, especially if you know what primitives can be expensive and how to tame beam, but it will always be slower (sometimes significantly) that C/C++ for some tasks. -Eric On Thu, Feb 17, 2011 at 03:26:09PM -0800, Joshua McKenty wrote: +1 for Erlang, as long as we have a duck farm. On Thu, Feb 17, 2011 at 2:25 PM, Devin Carlen devin.car...@gmail.com wrote: I'll put in a +1 for Erlang as an OpenStack supported platform. We'd be able to write a stable queue with a much more concise code base, and this is project would be a great fit for Erlang. Devin On Feb 17, 2011, at 2:21 PM, Eric Day wrote: Thanks to everyone who gave great feedback on the first queue service thread. I've updated the wiki page to include the suggestions. http://wiki.openstack.org/QueueService With a decent vision of what we want to build, the next step is figuring out how. In a previous thread it was suggested that the preferred languages for OpenStack projects are Python, C, and C++. Since there is an emphasis on speed and efficiency for the queue service, I suggest we use C++. I expect this service to be CPU bound and would benefit being able to leverage multiple cores efficiently (within the same process), so I don't think Python is a good fit. I think C++ is a better fit than C due to the need for modular interfaces. While this can obviously be done in C, C++ APIs are more concise and much less error prone. The OO style will also make it easier for Python developers who also want to learn and assist with C++ projects. Erlang is not on the preferred lists, but I would also put it out there as an option. While it may be a great fit for a project like this, I worry it won't attract the developer resources since Erlang isn't really a first-class language yet. If we decide to take the C++ path, I propose using a modular application framework I've been working on over the past year (mostly in my spare time). It provides a simple module programming interface with dependency tracking (kind of like Linux kernel modules). It already provides a multi-threaded event module (currently based on libevent, but this is pluggable) with simple networking abstractions built on top of it. We should be able to dive in an start writing the HTTP protocol module and queue processing modules. You can check out the current project at: https://launchpad.net/scalestack http://scalestack.org/ The intention of using a framework like this is so we can easily reuse the other modules (auth, HTTP, logging, ...) for other OpenStack services in the future. Much like we use Eventlet, WSGI, etc. (and eventually openstack-common) for Python, we could prefer using the modules in Scale Stack for lower level projects. Thoughts? -Eric ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of
Re: [Openstack] xen server agent code in nova?
Jay, It sounds like you understand it. :) The guest agent will be required to configure the VM's network settings and root password, etc. I share your thoughts. It feels like it should be a part of nova to me as well. - Chris On Feb 10, 2011, at 7:53 AM, Jay Pipes wrote: If this guest agent code is going to be required for certain features in Nova's Cactus release (at least, XenServer features), then shouldn't it be in Nova? Or am I not understanding what the guest agent code does? -jay On Thu, Feb 10, 2011 at 3:57 AM, Thierry Carrez thie...@openstack.org wrote: Paul Voccio wrote: So the question I want to pose to the community is if they are interested in the code, where should it go and how should we move forward on extending it? I think it's always interesting to publish the code. I'm not convinced of the value of shipping it inside Nova though. I'm not in the POC so it's not my job to define what should be considered Openstack core and what should not, but compatible guest agents IMO typically warrant their own project... Regards, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
