[Openstack] Huge spikes in get_instance_nw_info in grizzly

2013-07-17 Thread Sam Morrison
Hi, We have recently upgraded to grizzly and we have started seeing huge spikes on our network queue. Our network queue sits on about 1 message per second usually but we get spikes of 200m/s every so often. These spikes are all due to the message get_instance_nw_info. We're pretty sure this

[Openstack] Running multiple nova-cells

2013-07-15 Thread Sam Morrison
Hi, We have a reasonably large deployment with nova cells and we are having issues consuming messages fast enough at the top level due to all the instance updates etc. We've done some tuning to some of the periodic tasks but users can trigger hundreds of messages to be sent upwards. It looks

Re: [Openstack] cinder api list volumes very slow and inefficient.

2013-07-08 Thread Sam Morrison
On 09/07/2013, at 2:53 AM, John Griffith john.griff...@solidfire.com wrote: On Sun, Jul 7, 2013 at 5:43 PM, Sam Morrison sorri...@gmail.com wrote: Hi, I'm wondering if anyone is using cinder and is having issue with the list volumes API call. We have a grizzly install and we're

[Openstack] cinder api list volumes very slow and inefficient.

2013-07-07 Thread Sam Morrison
Hi, I'm wondering if anyone is using cinder and is having issue with the list volumes API call. We have a grizzly install and we're at about 160 active volumes in cinder at the moment. To do cinder list with a tenant that has about 140 volumes it takes about 6 seconds. API calls on a

Re: [Openstack] Keystone, pki tokens and memcache

2013-06-17 Thread Sam Morrison
On 18/06/2013, at 1:18 AM, Adam Young ayo...@redhat.com wrote: On 06/17/2013 12:27 AM, Sam Morrison wrote: I'm currently looking into Grizzly and have been having some issues getting PKI tokens to work. If I have memcache as the token backend keystone issues uuid based tokens, if I have

[Openstack] Keystone, pki tokens and memcache

2013-06-16 Thread Sam Morrison
I'm currently looking into Grizzly and have been having some issues getting PKI tokens to work. If I have memcache as the token backend keystone issues uuid based tokens, if I have sql as the backend then it issues PKI tokens. Does this mean you can't use memcache backend if you want to use

Re: [Openstack] Share Glance between cells or regions

2013-05-19 Thread Sam Morrison
Hi Jay, We are doing something similar. We have a single glance registry which is backed by galera DB replication. Then we have multiple glance-apis around the place. Currently they are all backed onto the same swift but I'd like to have it so each glance-api can talk to it's own swift. The

Re: [Openstack] Share Glance between cells or regions

2013-05-19 Thread Sam Morrison
resolve in DNS to the local swift in each region or you could use anycast. Original message From: Sam Morrison sorri...@gmail.com Date: 05/19/2013 8:39 PM (GMT-05:00) To: Jay Pipes jaypi...@gmail.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Share Glance

[Openstack] ask.openstack invalid certificate error

2013-04-23 Thread Sam Morrison
Hi guys, Looks like the web server for ask.openstack.org isn't sending the necessary intermediate certification authority files to the client and hence getting a cert not valid (when using firefox at least) ask.openstack.org uses an invalid security certificate. The certificate is not

Re: [Openstack] Ubuntu archive no longer resolves on UEC images

2013-03-13 Thread Sam Morrison
On 13/03/2013, at 9:21 PM, Daviey Walker dave.wal...@canonical.com wrote: On 13 March 2013 03:04, Sam Morrison sorri...@gmail.com wrote: Hi, When booting up a UEC image on our cloud cloud-init writes the apt sources file with: http://availability-zone.clouds.archive.ubuntu.com/ubuntu

[Openstack] Ubuntu archive no longer resolves on UEC images

2013-03-12 Thread Sam Morrison
Hi, When booting up a UEC image on our cloud cloud-init writes the apt sources file with: http://availability-zone.clouds.archive.ubuntu.com/ubuntu Today all of a sudden this doesn't resolve to anything. I note that http://availability-zone.cloud.archive.ubuntu.com does (cloud not clouds)

Re: [Openstack] AggregateInstanceExtraSpecs very slow?

2013-02-26 Thread Sam Morrison
, but maybe you can test that this at least makes the scheduling faster: http://paste.openstack.org/show/32534/ That seems to generate a much more acceptable query. - Chris On Feb 25, 2013, at 9:40 PM, Sam Morrison sorri...@gmail.com wrote: On 26/02/2013, at 4:31 PM, Chris Behrens

Re: [Openstack] AggregateInstanceExtraSpecs very slow?

2013-02-25 Thread Sam Morrison
Hi Joe, On 26/02/2013, at 11:19 AM, Joe Gordon j...@cloudscaling.com wrote: On Sun, Feb 24, 2013 at 3:31 PM, Sam Morrison sorri...@gmail.com wrote: I have been playing with the AggregateInstanceExtraSpecs filter and can't get it to work. In our staging environment it works fine with 4

Re: [Openstack] AggregateInstanceExtraSpecs very slow?

2013-02-25 Thread Sam Morrison
Hi Joe, On 26/02/2013, at 1:39 PM, Joe Gordon j...@cloudscaling.com wrote: On Mon, Feb 25, 2013 at 6:14 PM, Sam Morrison sorri...@gmail.com wrote: Hi Joe, On 26/02/2013, at 11:19 AM, Joe Gordon j...@cloudscaling.com wrote: On Sun, Feb 24, 2013 at 3:31 PM, Sam Morrison sorri

Re: [Openstack] AggregateInstanceExtraSpecs very slow?

2013-02-25 Thread Sam Morrison
On 26/02/2013, at 2:15 PM, Chris Behrens cbehr...@codestud.com wrote: On Feb 25, 2013, at 6:39 PM, Joe Gordon j...@cloudscaling.com wrote: It looks like the scheduler issues are related to the rabbitmq issues. host 'qh2-rcc77' ... is disabled or has not been heard from in a while

Re: [Openstack] AggregateInstanceExtraSpecs very slow?

2013-02-25 Thread Sam Morrison
On 26/02/2013, at 4:31 PM, Chris Behrens cbehr...@codestud.com wrote: After thinking more, it does seem like we're doing something wrong if the query itself is returning 300k rows. :) I can take a better look at it in front of the computer later if no one beats me to it. Yeah I think it's

[Openstack] AggregateInstanceExtraSpecs very slow?

2013-02-24 Thread Sam Morrison
I have been playing with the AggregateInstanceExtraSpecs filter and can't get it to work. In our staging environment it works fine with 4 compute nodes, I have 2 aggregates to split them into 2. When I try to do the same in our production environment which has 80 compute nodes (splitting them

[Openstack] Horizon and open connections

2013-01-30 Thread Sam Morrison
We have horizon running based on the Ubuntu Folsom Cloud Archive packages. What I notice is that after a while we have thousands of connections in the CLOSE_WAIT state to keystone and our nova api servers. The host also uses up all it's available memory (2GB) After a restart of apache all the

Re: [Openstack] New build dependency on keyring

2012-12-12 Thread Sam Morrison
My question is what does this extra dependancy give us apart from extra complexity? I can't see any enhancement in security with this method? Cheers, Sam On 13/12/2012, at 4:44 AM, Ken Thomas k...@yahoo-inc.com wrote: Greetings all! I'm look into using keyring as a way to (optionally)

Re: [Openstack] New build dependency on keyring

2012-12-12 Thread Sam Morrison
client. It seems like we could have *less* complexity if it were a hard dependency instead of having the code check if the import worked or not. Ken On 12/12/2012 2:46 PM, Sam Morrison wrote: My question is what does this extra dependancy give us apart from extra complexity? I can't see

[Openstack] Duplication of code in nova and cinder

2012-12-11 Thread Sam Morrison
I attempted to create a volume from an image in cinder and was getting this strange error, turns out it was because I had my glance servers specified as https://glanceserver:9292 In cinder the version of images/glance.py is older than the one in nova and is missing the ssl support additions.

[Openstack] NetApp cinder driver

2012-12-11 Thread Sam Morrison
Hi, We are currently evaluating some NetApp kit to be used for cinder volume storage. I have been able to get it working mostly but have got some issues around snapshots and some other minor things. Is there anyone from NetApp that could help me with them? It looks as if some of the responses

[Openstack] Resource claiming twice bug

2012-11-21 Thread Sam Morrison
Hi, I was wondering if someone could have a look at https://bugs.launchpad.net/nova/+bug/1055069 It's been marked as low but it my opinion it should be critical, we can't launch large instances because of it. Cheers, Sam ___ Mailing list:

[Openstack] Base images removed in upgrade essex - folsom and other stories

2012-11-13 Thread Sam Morrison
We upgraded our cloud from Essex - Folsom yesterday and had a major loss of data that I thought I'd share. With Essex the flag remove_used_base_images had a default of False, with Folsom it was changed to True. We hadn't explicitly set this so we had whatever the default was. After the

Re: [Openstack] Glance talking to swift https really slow

2012-10-08 Thread Sam Morrison
, 8 Oct 2012, Sam Morrison wrote: OK we changed to use nginx as opposed to pound and got a huge increase in speed. With taking glance out of it and talking directly to swift an image took 1m38s to download with pound in front. With nginx this took just 5s not sure what is wrong with pound

Re: [Openstack] [folsom] Swift-backed glance fails x-glance-api-copy-from

2012-10-07 Thread Sam Morrison
I have a bug for this at https://bugs.launchpad.net/ubuntu/+source/glance/+bug/1057322 Are you using the cloud archive packages? I'm assuming this is a version compatibility for glance/eventlet etc. Haven't had a chance to test this on quantal. Cheers, Sam On 08/10/2012, at 12:24 PM, Pete

[Openstack] Glance talking to swift https really slow

2012-10-07 Thread Sam Morrison
Hi, We've recently enabled ssl on our swift-proxy and now glance is incredibly slow. I've read some stuff about disabling zlib compression etc. Has anyone got a good fix for this? We're using Ubuntu 12.04 Cheers, Sam ___ Mailing list:

Re: [Openstack] Glance talking to swift https really slow

2012-10-07 Thread Sam Morrison
for pound or nginx. Sam On 08/10/2012, at 2:50 PM, Sam Morrison sorri...@gmail.com wrote: Hi, We've recently enabled ssl on our swift-proxy and now glance is incredibly slow. I've read some stuff about disabling zlib compression etc. Has anyone got a good fix for this? We're using Ubuntu

Re: [Openstack] Cells Status

2012-10-02 Thread Sam Morrison
...@codestud.com wrote: Yup, it's done. I just have to deal with some conflicts with our internal branch and my public one.. On Oct 1, 2012, at 7:47 PM, Sam Morrison sorri...@gmail.com wrote: On 02/10/2012, at 12:33 PM, Chris Behrens cbehr...@codestud.com wrote: Thanks, Tom! I have changes

Re: [Openstack] Cells Status

2012-10-01 Thread Sam Morrison
On 02/10/2012, at 12:33 PM, Chris Behrens cbehr...@codestud.com wrote: Thanks, Tom! I have changes to push up that add rpc versioning, etc. Maybe I can get those up tomorrow. Great! I was going to start looking into it but will hold off if you've already done it. Cheers, Sam

Re: [Openstack] Ubuntu Cloud Archive information

2012-09-26 Thread Sam Morrison
a launchpad error. Thanks. Sam On 25/09/2012, at 11:38 AM, Chuck Short chuck.sh...@canonical.com wrote: Hi On 12-09-24 07:39 PM, Sam Morrison wrote: Hi, I've started using the Ubuntu Cloud Archive packages for Folsom in Precise. Haven't been able to find out much information about them so I'm

[Openstack] Ubuntu Cloud Archive information

2012-09-24 Thread Sam Morrison
Hi, I've started using the Ubuntu Cloud Archive packages for Folsom in Precise. Haven't been able to find out much information about them so I'm asking here. I've found the packages have quite a few bugs eg.[1]. So trying to figure out where to submit bugs for these and also where the sources

Re: [Openstack] Ubuntu Cloud Archive information

2012-09-24 Thread Sam Morrison
On 25/09/2012, at 11:38 AM, Chuck Short chuck.sh...@canonical.com wrote: You are doing it in the right place, please submit any bugs that you find in launchpad. OK great, so I just treat them as normal ubuntu packages. Doe anyone know anything about these packages? What do you want

Re: [Openstack] Glance API now needs a DB?

2012-09-21 Thread Sam Morrison
you're stuck either configuring the sql params in glance-api or disabling the v2 API altogether (enable_v2_api=False). Waldon On Sep 20, 2012, at 9:28 PM, Sam Morrison wrote: Have just started playing with Folsom and when configuring glance-api it now requires a database. Just

[Openstack] Glance API now needs a DB?

2012-09-20 Thread Sam Morrison
Have just started playing with Folsom and when configuring glance-api it now requires a database. Just wondering if this is meant to be a separate DB to glance-registry or is it meant to point to the same one? Couldn't find any information about this in the docs. Thanks, Sam

Re: [Openstack] Cells Status

2012-09-16 Thread Sam Morrison
We have been using this patch in testing for a few months now. It works pretty well with launching instances etc. It is feature incomplete at the moment though. We have developed syncing of security groups and private keys down do child cells. Also done the live migration add ons for cells. It

Re: [Openstack] Change user password (not admin)

2012-06-06 Thread Sam Morrison
Hi, There has been a first attempt at this in keystone. See https://review.openstack.org/#/c/7437/ And bug: https://bugs.launchpad.net/keystone/+bug/996922 It needs more work to make it secure though. Cheers, Sam On Thu, Jun 7, 2012 at 7:13 AM, Kiall Mac Innes ki...@managedit.ie wrote: On

Re: [Openstack] No JSON object could be decoded

2012-05-31 Thread Sam Morrison
Hi Imen, If you take out the pipe to python you should see the correct error. Eg do: curl -d '{auth: {tenantName: service, passwordCredentialsusername: swift, password: swiftpass}}}' -H Content-type: application/json http://192.168.1.68:35357/v2.0/tokens Cheers, Sam On 31/05/12 19:53,

Re: [Openstack] swift3 middleware split.

2012-05-21 Thread Sam Morrison
Would also be good to start thinking about how these are packaged up and added to ubuntu/epel etc. archives. Will the people who do the deb/rpm packaging for swift also be doing these plugins? Or are they entirely separate in that sense too? Cheers, Sam On Mon, May 21, 2012 at 10:55 PM,

Re: [Openstack] s3 interface in Essex

2012-04-27 Thread Sam Morrison
On Thu, Apr 26, 2012 at 2:16 PM, Sam Morrison sorri...@gmail.com wrote: Hi, I'm trying to get the s3 interface in Essex working. I'm using the packages provided in Ubuntu 12.04. my pipeline order is: pipeline = catch_errors healthcheck cache swift3 s3token authtoken keystone proxy-server

Re: [Openstack] Unable to download images using Glance+Keystone+Swift

2012-04-26 Thread Sam Morrison
Hi Ross, I had the same issue. Could upload images to swift but not download them getting a 404. I needed to apply the patch outlined in this bug to fix it: https://bugs.launchpad.net/glance/+bug/979745 Cheers, Sam On Fri, Apr 27, 2012 at 9:53 AM, Lillie Ross-CDSR11

[Openstack] s3 interface in Essex

2012-04-25 Thread Sam Morrison
Hi, I'm trying to get the s3 interface in Essex working. I'm using the packages provided in Ubuntu 12.04. my pipeline order is: pipeline = catch_errors healthcheck cache swift3 s3token authtoken keystone proxy-server [filter:s3token] paste.filter_factory =