Re: [Openstack] Caching strategies in Nova ...

[Yun Mao]

Got it. Thanks,

If I read your number correctly, there are 10 db api calls, with total
time 0.388 seconds.

This is certainly not lightning fast. But it's not really slow, given
that the user is expecting to have the VM created in more than 10
seconds. <0.5 s latency is tolerable. If most of the time is spent in
network to db, then I'd say when we scale up a lot in compute/vm
numbers, the latency won't increase much.

One thing to note is that right now the DB APIs are all blocking
calls. So it could be tricky to get the performance number right when
measuring multiple concurrent requests.

Yun

On Fri, Mar 23, 2012 at 6:47 PM, Mark Washenberger
 wrote:
> Yun,
>
> I was working with a very small but fairly realistic setup. In this
> case I had only 3 Xen hosts, no more than 10 nova vms up at a time.
> And the environment was very nearly "fresh" so I believe the db
> tables were as small as they could be. I believe the utilization
> across the board in my setup was very low, and indeed the numbers
> were very consistent (I ran a large number of times, but didn't
> save all of the data :-(). Also, there were only 2 compute nodes
> running, but as the workflow only had rpc casts, I'm not sure that
> really mattered very much.
>
> The profile I gave was for vm creation. But I also ran tests for
> deletion, listing, and showing vms in the OS API.
>
> Networks were static throughout the process. Volumes were absent.
>
> "Yun Mao"  said:
>
>> Hi Mark,
>>
>> what workload and what setup do you have while you are profiling? e.g.
>> how many compute nodes do you have, how many VMs do you have, are you
>> creating/destroying/migrating VMs, volumes, networks?
>>
>> Thanks,
>>
>> Yun
>>
>> On Fri, Mar 23, 2012 at 4:26 PM, Mark Washenberger
>>  wrote:
>>>
>>>
>>> "Johannes Erdfelt"  said:
>>>

 MySQL isn't exactly slow and Nova doesn't have particularly large
 tables. It looks like the slowness is coming from the network and how
 many queries are being made.

 Avoiding joins would mean even more queries, which looks like it would
 slow it down even further.

>>>
>>> This is exactly what I saw in my profiling. More complex queries did
>>> still seem to take longer than less complex ones, but it was a second
>>> order effect compared to the overall volume of queries.
>>>
>>> I'm not sure that network was the culprit though, since my ping
>>> roundtrip time was small relative to the wall time I measured for each
>>> nova.db.api call.
>>>
>>>
>>> ___
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to     : openstack@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [OpenStack] Xen Hypervisor

[Todd Deshane]

On Fri, Mar 23, 2012 at 1:45 PM, Alexandre Leites  wrote:
>
> After all this things, i got Xen working, but i have a error with bridge
> now, as trace below:
>
> 2012-03-23 16:30:00,116 DEBUG nova.virt.xenapi [-] Updating host stats from
> (pid=23556) update_status
> /usr/lib/python2.7/dist-packages/nova/virt/xenapi_conn.py:488
> 2012-03-23 16:30:00,988 WARNING nova.virt.xenapi [-] Task
> [Async.host.call_plugin] OpaqueRef:d7a9f0df-0c7c-a760-6b76-3e985c747b1d
> status: failure    ['XENAPI_PLUGIN_FAILURE', 'host_data', 'KeyError',
> "'host_uuid'"]
> 2012-03-23 16:30:00,992 WARNING nova.compute.manager [-] Error during
> report_driver_status(): ['XENAPI_PLUGIN_FAILURE', 'host_data', 'KeyError',
> "'host_uuid'"]
> 2
>

Try tracing it through as suggested by:
https://lists.launchpad.net/openstack/msg08599.html

Hope that helps.

Thanks,
Todd

-- 
Todd Deshane
http://www.linkedin.com/in/deshantm
http://blog.xen.org/
http://wiki.xen.org/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] MySQL connection gone away handling in OpenStack projects

[Adam Gandelman]

On 03/22/2012 09:48 AM, Vishvananda Ishaya wrote:
This looks like a much better solution than MySQLPingListener.  It 
would be good to get this into common / nova, especially if we can 
verify that it works with postgres as well.


Vish


+1 for making this the standard method of initializing databases across 
projects.  A quick test of adding a wrapped call to _ENGINE.connect() in 
the main try/except block of configure_db seems to totally solve the 
issue of services starting at boot before a database is reachable, 
reported @ https://bugs.launchpad.net/ubuntu/+source/nova/+bug/959426


-Adam
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

Great suggestions guys ... we'll give some thought on how the community
can share and compare performance measurements in a consistent way.

-S

On 03/23/2012 07:26 PM, Joe Gordon wrote:
> +1
> 
> Documenting these findings would be nice too.
> 
> 
> best,
> Joe
> 
> On Fri, Mar 23, 2012 at 2:15 PM, Justin Santa Barbara
> mailto:jus...@fathomdb.com>> wrote:
> 
> This is great: hard numbers are exactly what we need.  I would love
> to see a statement-by-statement SQL log with timings from someone
> that has a performance issue.  I'm happy to look into any DB
> problems that demonstrates.
> 
> The nova database is small enough that it should always be in-memory
> (if you're running a million VMs, I don't think asking for one
> gigabyte of RAM on your DB is unreasonable!)
> 
> If it isn't hitting disk, PostgreSQL or MySQL with InnoDB can serve
> 10k 'indexed' requests per second through SQL on a low-end (<$1000)
> box.  With tuning you can get 10x that.  Using one of the SQL bypass
> engines (e.g. MySQL HandlerSocket) can supposedly give you 10x
> again.  Throwing money at the problem in the form of multi-processor
> boxes (or disks if you're I/O bound) can probably get you 10x again.
> 
> However, if you put a DB on a remote host, you'll have to wait for a
> network round-trip per query.  If your ORM is doing a 1+N query, the
> total read time will be slow.  If your DB is doing a sync on every
> write, writes will be slow.  If the DB isn't tuned with a sensible
> amount of cache (at least as big as the DB size), it will be
> slow(er).  Each of these has a very simple fix for OpenStack.
> 
> Relational databases have very efficient caching mechanisms built
> in.  Any out-of-process cache will have a hard time beating it.
>  Let's make sure the bottleneck is the DB, and not (for example)
> RabbitMQ, before we go off a huge rearchitecture.
> 
> Justin
> 
> 
> 
> 
> On Thu, Mar 22, 2012 at 7:53 PM, Mark Washenberger
>  > wrote:
> 
> Working on this independently, I created a branch with some simple
> performance logging around the nova-api, and individually around
> glance, nova.db, and nova.rpc calls. (Sorry, I only have a local
> copy and its on a different computer right now, and probably needs
> a rebase. I will rebase and publish it on GitHub tomorrow.)
> 
> With this logging, I could get some simple profiling that I found
> very useful. Here is a GH project with the analysis code as well
> as some nova-api logs I was using as input.
> 
> https://github.com/markwash/nova-perflog
> 
> With these tools, you can get a wall-time profile for individual
> requests. For example, looking at one server create request (and
> you can run this directly from the checkout as the logs are saved
> there):
> 
> markw@poledra:perflogs$ cat nova-api.vanilla.1.5.10.log | python
> profile-request.py req-3cc0fe84-e736-4441-a8d6-ef605558f37f
> keycountavg
> nova.api.openstack.wsgi.POST   1  0.657
> nova.db.api.instance_update1  0.191
> nova.image.show1  0.179
> nova.db.api.instance_add_security_group1  0.082
> nova.rpc.cast  1  0.059
> nova.db.api.instance_get_all_by_filters1  0.034
> nova.db.api.security_group_get_by_name 2  0.029
> nova.db.api.instance_create1  0.011
> nova.db.api.quota_get_all_by_project   3  0.003
> nova.db.api.instance_data_get_for_project  1  0.003
> 
> key  count  total
> nova.api.openstack.wsgi  1  0.657
> nova.db.api 10  0.388
> nova.image   1  0.179
> nova.rpc 1  0.059
> 
> All times are in seconds. The nova.rpc time is probably high
> since this was the first call since server restart, so the
> connection handshake is probably included. This is also probably
> 1.5 months stale.
> 
> The conclusion I reached from this profiling is that we just plain
> overuse the db (and we might do the same in glance). For example,
> whenever we do updates, we actually re-retrieve the item from the
> database, update its dictionary, and save it. This is double the
> cost it needs to be. We also handle updates for data across tables
> inefficiently, where they could be handled in single database round
> trip.
> 
> In particular, in the case of server listings, extensions are just
> rough on performance. Most extensions hit the database again
> at leas

Re: [Openstack] Caching strategies in Nova ...

[Mark Washenberger]

Yun,

I was working with a very small but fairly realistic setup. In this
case I had only 3 Xen hosts, no more than 10 nova vms up at a time.
And the environment was very nearly "fresh" so I believe the db 
tables were as small as they could be. I believe the utilization
across the board in my setup was very low, and indeed the numbers
were very consistent (I ran a large number of times, but didn't
save all of the data :-(). Also, there were only 2 compute nodes
running, but as the workflow only had rpc casts, I'm not sure that
really mattered very much.

The profile I gave was for vm creation. But I also ran tests for
deletion, listing, and showing vms in the OS API.

Networks were static throughout the process. Volumes were absent.

"Yun Mao"  said:

> Hi Mark,
> 
> what workload and what setup do you have while you are profiling? e.g.
> how many compute nodes do you have, how many VMs do you have, are you
> creating/destroying/migrating VMs, volumes, networks?
> 
> Thanks,
> 
> Yun
> 
> On Fri, Mar 23, 2012 at 4:26 PM, Mark Washenberger
>  wrote:
>>
>>
>> "Johannes Erdfelt"  said:
>>
>>>
>>> MySQL isn't exactly slow and Nova doesn't have particularly large
>>> tables. It looks like the slowness is coming from the network and how
>>> many queries are being made.
>>>
>>> Avoiding joins would mean even more queries, which looks like it would
>>> slow it down even further.
>>>
>>
>> This is exactly what I saw in my profiling. More complex queries did
>> still seem to take longer than less complex ones, but it was a second
>> order effect compared to the overall volume of queries.
>>
>> I'm not sure that network was the culprit though, since my ping
>> roundtrip time was small relative to the wall time I measured for each
>> nova.db.api call.
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Mark Washenberger]

Hmm. . it was definitely different xen virtual machines on either the
same hypervisor or one that was adjacent to it in an L2 sense. On a
similar environment I have set up now, I notice that the ping time
from one vm to another on the same hypervisor is not noticeably less
than the ping time to a vm on a different hypervisor. Not sure why that
is the case! In any case it is trivial. . ~3 ms for first ping, ~0.3 ms
for subsequent pings. 

"Sandy Walsh"  said:

> Was the db on a separate server or loopback?
> 
> On 03/23/2012 05:26 PM, Mark Washenberger wrote:
>>
>>
>> "Johannes Erdfelt"  said:
>>
>>>
>>> MySQL isn't exactly slow and Nova doesn't have particularly large
>>> tables. It looks like the slowness is coming from the network and how
>>> many queries are being made.
>>>
>>> Avoiding joins would mean even more queries, which looks like it would
>>> slow it down even further.
>>>
>>
>> This is exactly what I saw in my profiling. More complex queries did
>> still seem to take longer than less complex ones, but it was a second
>> order effect compared to the overall volume of queries.
>>
>> I'm not sure that network was the culprit though, since my ping
>> roundtrip time was small relative to the wall time I measured for each
>> nova.db.api call.
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Joe Gordon]

+1

Documenting these findings would be nice too.


best,
Joe

On Fri, Mar 23, 2012 at 2:15 PM, Justin Santa Barbara
wrote:

> This is great: hard numbers are exactly what we need.  I would love to see
> a statement-by-statement SQL log with timings from someone that has a
> performance issue.  I'm happy to look into any DB problems that
> demonstrates.
>
> The nova database is small enough that it should always be in-memory (if
> you're running a million VMs, I don't think asking for one gigabyte of RAM
> on your DB is unreasonable!)
>
> If it isn't hitting disk, PostgreSQL or MySQL with InnoDB can serve 10k
> 'indexed' requests per second through SQL on a low-end (<$1000) box.  With
> tuning you can get 10x that.  Using one of the SQL bypass engines (e.g.
> MySQL HandlerSocket) can supposedly give you 10x again.  Throwing money at
> the problem in the form of multi-processor boxes (or disks if you're I/O
> bound) can probably get you 10x again.
>
> However, if you put a DB on a remote host, you'll have to wait for a
> network round-trip per query.  If your ORM is doing a 1+N query, the total
> read time will be slow.  If your DB is doing a sync on every write, writes
> will be slow.  If the DB isn't tuned with a sensible amount of cache (at
> least as big as the DB size), it will be slow(er).  Each of these has a
> very simple fix for OpenStack.
>
> Relational databases have very efficient caching mechanisms built in.  Any
> out-of-process cache will have a hard time beating it.  Let's make sure the
> bottleneck is the DB, and not (for example) RabbitMQ, before we go off a
> huge rearchitecture.
>
> Justin
>
>
>
>
> On Thu, Mar 22, 2012 at 7:53 PM, Mark Washenberger <
> mark.washenber...@rackspace.com> wrote:
>
>> Working on this independently, I created a branch with some simple
>> performance logging around the nova-api, and individually around
>> glance, nova.db, and nova.rpc calls. (Sorry, I only have a local
>> copy and its on a different computer right now, and probably needs
>> a rebase. I will rebase and publish it on GitHub tomorrow.)
>>
>> With this logging, I could get some simple profiling that I found
>> very useful. Here is a GH project with the analysis code as well
>> as some nova-api logs I was using as input.
>>
>> https://github.com/markwash/nova-perflog
>>
>> With these tools, you can get a wall-time profile for individual
>> requests. For example, looking at one server create request (and
>> you can run this directly from the checkout as the logs are saved
>> there):
>>
>> markw@poledra:perflogs$ cat nova-api.vanilla.1.5.10.log | python
>> profile-request.py req-3cc0fe84-e736-4441-a8d6-ef605558f37f
>> keycountavg
>> nova.api.openstack.wsgi.POST   1  0.657
>> nova.db.api.instance_update1  0.191
>> nova.image.show1  0.179
>> nova.db.api.instance_add_security_group1  0.082
>> nova.rpc.cast  1  0.059
>> nova.db.api.instance_get_all_by_filters1  0.034
>> nova.db.api.security_group_get_by_name 2  0.029
>> nova.db.api.instance_create1  0.011
>> nova.db.api.quota_get_all_by_project   3  0.003
>> nova.db.api.instance_data_get_for_project  1  0.003
>>
>> key  count  total
>> nova.api.openstack.wsgi  1  0.657
>> nova.db.api 10  0.388
>> nova.image   1  0.179
>> nova.rpc 1  0.059
>>
>> All times are in seconds. The nova.rpc time is probably high
>> since this was the first call since server restart, so the
>> connection handshake is probably included. This is also probably
>> 1.5 months stale.
>>
>> The conclusion I reached from this profiling is that we just plain
>> overuse the db (and we might do the same in glance). For example,
>> whenever we do updates, we actually re-retrieve the item from the
>> database, update its dictionary, and save it. This is double the
>> cost it needs to be. We also handle updates for data across tables
>> inefficiently, where they could be handled in single database round
>> trip.
>>
>> In particular, in the case of server listings, extensions are just
>> rough on performance. Most extensions hit the database again
>> at least once. This isn't really so bad, but it clearly is an area
>> where we should improve, since these are the most frequent api
>> queries.
>>
>> I just see a ton of specific performance problems that are easier
>> to address one by one, rather than diving into a general (albeit
>> obvious) solution such as caching.
>>
>>
>> "Sandy Walsh"  said:
>>
>> > We're doing tests to find out where the bottlenecks are, caching is the
>> > most obvious solution, but there may be others. Tools like memcache do a
>> > really good job of sharing memory across servers so we don't have to
>> > reinvent the wheel or hit the db at all.
>> >
>> > In addition to looking into caching technolog

Re: [Openstack] Enabling remote participation at Folsom Design Summit

[Stefano Maffulli]

On Thu, 2012-03-22 at 16:08 -0700, Stefano Maffulli wrote:
> Hardware is the first thing we need to procure before this project takes
> off.

And we have closed the deal: habemus hardware, kindly provided by the
nice folks at zareason.com. 

During the first week of April we'll meet and configure the machines to
use icecast... unless somebody finds some time to fiddle with the
software that powers http://streamti.me/ (it was used at Pycon --code on
github https://github.com/timsvideo). And we can always use Cisco's
webex, too. Plenty of options, I love it.

Get ready for an install and config fest, old style, in a few days.
It'll be fun.

Thanks everybody
stef


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] OpenStack Community Newsletter –March 23, 2012

[Stefano Maffulli]

OpenStack Community Newsletter –March 23, 2012

A great community story:


Over the course of just a week I’ve gone from despair and
frustration to almost wanting to shout from the rooftops the
achievements that have been made in OpenStack and my own
deployments and testing


Read the rest of it on
https://lists.launchpad.net/openstack/msg08950.html

HIGHLIGHTS


  * Submit session proposals for OpenStack Folsom Design Summit
https://lists.launchpad.net/openstack/msg08674.html
  * OpenStack Essex release candidates are ready: go test them while
supplies last
  * Shout out to Andrew Hutchings for the new Jenkins and Gerrit
styles

http://www.linuxjedi.co.uk/2012/03/new-gerrit-and-jenkins-styles-are-live.html


EVENTS


  * Australian OpenStack Users Group MeetUp Mar 26, 2012 – Sidney,
Australia http://aosug.openstack.org.au/events/49679462/
  * Lights, Camera, Stackers! Mar 29, 2012 – Los Angeles, CA
http://www.meetup.com/OpenStack-LA/events/54125632/
  * OpenStack Spring 2012 Design Summit Apr 16 – 18 and Conference
Apr 19-20 – San Francisco, California
http://openstack.org/conference/
  * OpenStack Israel 2012 May 30, 2012 – Tel Aviv-Yafo, Israel
http://www.meetup.com/IGTCloud/events/53864922/
  * Open Cloud and Public Administration Jun 19, 2012 – Bern,
Switzerland Call for papers/sponsors
http://www.ch-open.ch/index.php?id=1034


OTHER NEWS


 1. New user groups in Canberra
http://www.meetup.com/Canberra-OpenStack-Users-Group/ and India
http://www.meetup.com/Indian-OpenStack-User-Group
  * OpenStack Wiki Recent Changes –
http://wiki.openstack.org/RecentChanges
  * How To Update Corporate CLA
http://wiki.openstack.org/HowToUpdateCorporateCLA
  * Governance/Foundation/Funding
http://wiki.openstack.org/Governance/Foundation/Funding
  * Nova Orchestration Spec For Folsom
http://wiki.openstack.org/NovaOrchestrationSpecForFolsom
  * XenServer http://wiki.openstack.org/XenServer
  * QuantumPackages
http://wiki.openstack.org/QuantumPackages
  * Project meeting summary

http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-03-20-21.02.html


COMMUNITY STATISTICS


  *  Activity on the main branch of OpenStack repositories, lines of
code added and removed per developer during week 7 of 2012 (from
Mon Mar 5 00:00:00 UTC 2012 to Mon March 12 00:00:00 UTC 2012)




Changes to Glance project – week 11 2012
Changes to Glance project – week 11 2012 
Changes to Horizon project – week 11 2012
Changes to Horizon project – week 11 2012 
Changes to Keystone project – week 11 2012
Changes to Keystone project – week 11 2012 




Changes to Manuals project – week 11 2012
Changes to Manuals project – week 11 2012 
Changes to Nova project – week 11 2012
Changes to Nova project – week 11 2012 
Changes to Quantum project – week 11 2012
Changes to Quantum project – week 11 2012 




Changes to Swift project – week 11 2012
Changes to Swift project – week 11 2012 
Changes to Tempest project – week 11 2012
Changes to Tempest project – week 11 2012 



This weekly newsletter is a way for the community to learn about all the
various activities occurring on a weekly basis. If you would like to add
content to a weekly update or have an idea about this newsletter, please
leave a comment.

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Yun Mao]

Hi Mark,

what workload and what setup do you have while you are profiling? e.g.
how many compute nodes do you have, how many VMs do you have, are you
creating/destroying/migrating VMs, volumes, networks?

Thanks,

Yun

On Fri, Mar 23, 2012 at 4:26 PM, Mark Washenberger
 wrote:
>
>
> "Johannes Erdfelt"  said:
>
>>
>> MySQL isn't exactly slow and Nova doesn't have particularly large
>> tables. It looks like the slowness is coming from the network and how
>> many queries are being made.
>>
>> Avoiding joins would mean even more queries, which looks like it would
>> slow it down even further.
>>
>
> This is exactly what I saw in my profiling. More complex queries did
> still seem to take longer than less complex ones, but it was a second
> order effect compared to the overall volume of queries.
>
> I'm not sure that network was the culprit though, since my ping
> roundtrip time was small relative to the wall time I measured for each
> nova.db.api call.
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Being pedantic about pedanticism: HACKING styleguide

[Maru Newby]

Hi Andy,

On 2012-03-22, at 10:00 PM, Andy Smith wrote:

> The rule is there because it makes it obvious where you are using objects 
> from (and they aren't in the current namespace), prevents that "where is this 
> defined -scan around the file- oh, it is being imported from this other 
> thing, let's go look there" pattern, instead you see very obviously that it 
> isn't defined in the file and usually has a unique enough name already that 
> you know exactly where to look.
> 
> It is something pulled from the google style guide.


I won't argue with the positive result of the rule - the avoidance of circular 
imports - but I would not want to endure an increase in code verbosity solely 
to make it clear where a symbol was defined.  Modern editors expose powerful 
regex search and jump-to-definition functions that make it easy to determine 
where a symbol comes from.

Cheers,


Maru
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Justin Santa Barbara]

This is great: hard numbers are exactly what we need.  I would love to see
a statement-by-statement SQL log with timings from someone that has a
performance issue.  I'm happy to look into any DB problems that
demonstrates.

The nova database is small enough that it should always be in-memory (if
you're running a million VMs, I don't think asking for one gigabyte of RAM
on your DB is unreasonable!)

If it isn't hitting disk, PostgreSQL or MySQL with InnoDB can serve 10k
'indexed' requests per second through SQL on a low-end (<$1000) box.  With
tuning you can get 10x that.  Using one of the SQL bypass engines (e.g.
MySQL HandlerSocket) can supposedly give you 10x again.  Throwing money at
the problem in the form of multi-processor boxes (or disks if you're I/O
bound) can probably get you 10x again.

However, if you put a DB on a remote host, you'll have to wait for a
network round-trip per query.  If your ORM is doing a 1+N query, the total
read time will be slow.  If your DB is doing a sync on every write, writes
will be slow.  If the DB isn't tuned with a sensible amount of cache (at
least as big as the DB size), it will be slow(er).  Each of these has a
very simple fix for OpenStack.

Relational databases have very efficient caching mechanisms built in.  Any
out-of-process cache will have a hard time beating it.  Let's make sure the
bottleneck is the DB, and not (for example) RabbitMQ, before we go off a
huge rearchitecture.

Justin



On Thu, Mar 22, 2012 at 7:53 PM, Mark Washenberger <
mark.washenber...@rackspace.com> wrote:

> Working on this independently, I created a branch with some simple
> performance logging around the nova-api, and individually around
> glance, nova.db, and nova.rpc calls. (Sorry, I only have a local
> copy and its on a different computer right now, and probably needs
> a rebase. I will rebase and publish it on GitHub tomorrow.)
>
> With this logging, I could get some simple profiling that I found
> very useful. Here is a GH project with the analysis code as well
> as some nova-api logs I was using as input.
>
> https://github.com/markwash/nova-perflog
>
> With these tools, you can get a wall-time profile for individual
> requests. For example, looking at one server create request (and
> you can run this directly from the checkout as the logs are saved
> there):
>
> markw@poledra:perflogs$ cat nova-api.vanilla.1.5.10.log | python
> profile-request.py req-3cc0fe84-e736-4441-a8d6-ef605558f37f
> keycountavg
> nova.api.openstack.wsgi.POST   1  0.657
> nova.db.api.instance_update1  0.191
> nova.image.show1  0.179
> nova.db.api.instance_add_security_group1  0.082
> nova.rpc.cast  1  0.059
> nova.db.api.instance_get_all_by_filters1  0.034
> nova.db.api.security_group_get_by_name 2  0.029
> nova.db.api.instance_create1  0.011
> nova.db.api.quota_get_all_by_project   3  0.003
> nova.db.api.instance_data_get_for_project  1  0.003
>
> key  count  total
> nova.api.openstack.wsgi  1  0.657
> nova.db.api 10  0.388
> nova.image   1  0.179
> nova.rpc 1  0.059
>
> All times are in seconds. The nova.rpc time is probably high
> since this was the first call since server restart, so the
> connection handshake is probably included. This is also probably
> 1.5 months stale.
>
> The conclusion I reached from this profiling is that we just plain
> overuse the db (and we might do the same in glance). For example,
> whenever we do updates, we actually re-retrieve the item from the
> database, update its dictionary, and save it. This is double the
> cost it needs to be. We also handle updates for data across tables
> inefficiently, where they could be handled in single database round
> trip.
>
> In particular, in the case of server listings, extensions are just
> rough on performance. Most extensions hit the database again
> at least once. This isn't really so bad, but it clearly is an area
> where we should improve, since these are the most frequent api
> queries.
>
> I just see a ton of specific performance problems that are easier
> to address one by one, rather than diving into a general (albeit
> obvious) solution such as caching.
>
>
> "Sandy Walsh"  said:
>
> > We're doing tests to find out where the bottlenecks are, caching is the
> > most obvious solution, but there may be others. Tools like memcache do a
> > really good job of sharing memory across servers so we don't have to
> > reinvent the wheel or hit the db at all.
> >
> > In addition to looking into caching technologies/approaches we're gluing
> > together some tools for finding those bottlenecks. Our first step will
> > be finding them, then squashing them ... however.
> >
> > -S
> >
> > On 03/22/2012 06:25 PM, Mark Washenberger wrote:
> >> What problems are caching s

[Openstack] Keystone 2012.1 RC1 available

[Thierry Carrez]

Hello everyone,

Last but not least, the tarball for the first release candidate for
OpenStack Identity (Keystone) 2012.1 is now available at:

https://launchpad.net/keystone/essex/essex-rc1

Unless release-critical issues are found that warrant a release
candidate respin, this RC1 will be formally released as the 2012.1
("Essex") final version. You are therefore strongly encouraged to test
and validate it.

You should test the tarballs above, but you also can directly use the
milestone-proposed branch at:

https://github.com/openstack/keystone/tree/milestone-proposed

If you find an issue that could be considered release-critical, please
file it at:

https://bugs.launchpad.net/keystone/+filebug

and tag it "essex-rc-potential" to bring it to Joe's attention.

Note that the "master" branch of Keystone is now open for Folsom
development, feature freeze restrictions no longer apply.

-- 
Thierry Carrez (ttx)
Release Manager, OpenStack

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Nova Live Migration Error

[Joe Breu]

Hi Andrew,

Do you have a shared /var/lib/nova between the two instances?  If you do not 
then you won't be able to do a migration without using --block_migrate.


---
Joseph Breu
Deployment Engineer
Rackspace Cloud Builders
210-312-3508

On Mar 23, 2012, at 1:23 PM, Andrew Weiss wrote:

Hey guys,

Not sure if this is a bug or something wrong on my end. Attempting to conduct a 
live migration between two compute nodes I have running.  Using Essex release 
rc1.  When I execute the following command: 'nova live-migration  
' I get an HTTP 400 failure. Below is an error that I found 
in my nova-scheduler.log file:

ERROR: Live migration of instance 7a938bdb-9970-41fd-acc2-d27fa3d13088 to host 
computehost1 failed (HTTP 400)
(nova.rpc.amqp): TRACE:   File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 222, in 
schedule_live_migration
(nova.rpc.amqp): TRACE: disk_over_commit)
(nova.rpc.amqp): TRACE:   File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 322, in 
_live_migration_common_check
(nova.rpc.amqp): TRACE: self.mounted_on_same_shared_storage(context, 
instance_ref, dest)
(nova.rpc.amqp): TRACE:   File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 521, in 
mounted_on_same_shared_storage
(nova.rpc.amqp): TRACE: raise exception.FileNotFound(file_path=filename)
(nova.rpc.amqp): TRACE: FileNotFound: File tmpQ9Ig08 could not be found.
(nova.rpc.amqp): TRACE:
2012-03-23 14:18:06 ERROR nova.rpc.amqp 
[req-aca97b5b-f456-4b9a-8561-ff12bd66aad9 10daac9302ba42eb9f4b592955680407 
e57faa938486428a9ba6d77402f5279c] Returning exception File tmpQ9Ig08 could not 
be found. to caller
2012-03-23 14:18:06 ERROR nova.rpc.amqp 
[req-aca97b5b-f456-4b9a-8561-ff12bd66aad9 10daac9302ba42eb9f4b592955680407 
e57faa938486428a9ba6d77402f5279c] ['Traceback (most recent call last):\n', '  
File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in 
_process_data\nrval = node_func(context=ctxt, **node_args)\n', '  File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 97, in 
_schedule\ncontext, ex, *args, **kwargs)\n', '  File 
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
self.gen.next()\n', '  File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 92, in 
_schedule\nreturn driver_method(*args, **kwargs)\n', '  File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 222, in 
schedule_live_migration\ndisk_over_commit)\n', '  File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 322, in 
_live_migration_common_check\nself.mounted_on_same_shared_storage(context, 
instance_ref, dest)\n', '  File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 521, in 
mounted_on_same_shared_storage\nraise 
exception.FileNotFound(file_path=filename)\n', 'FileNotFound: File tmpQ9Ig08 
could not be found.\n']

I can't seem to figure out what file is is trying to find. Just looking to get 
some guidance on this.

Thanks,

Andrew Weiss
wei...@purdue.edu 
LinkedIn | http://www.linkedin.com/in/weissa

___
Mailing list: https://launchpad.net/~openstack
Post to : 
openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Rick Jones]

On 03/23/2012 01:26 PM, Mark Washenberger wrote:



"Johannes Erdfelt"  said:



MySQL isn't exactly slow and Nova doesn't have particularly large
tables. It looks like the slowness is coming from the network and how
many queries are being made.

Avoiding joins would mean even more queries, which looks like it would
slow it down even further.



This is exactly what I saw in my profiling. More complex queries did
still seem to take longer than less complex ones, but it was a second
order effect compared to the overall volume of queries.

I'm not sure that network was the culprit though, since my ping
roundtrip time was small relative to the wall time I measured for each
nova.db.api call.


How much data would the queries return, and how long between queries? 
One "networking" thing that might come into play would be slow start 
after idle - if the query returns are > INITCWND (either 3 or 10 
segments depending on which kernel) and they are separated by at least 
one RTO (or is it RTT?) then they will hit slow start each time.  Now, 
the extent to which that matters is a function of how large the return 
is, and it is only adding RTTs so it wouldn't be minutes, but it could 
add up a bit I suppose.


rick jones

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Being pedantic about pedanticism: HACKING styleguide

[Andy Smith]

On Fri, Mar 23, 2012 at 1:17 PM, Mark McLoughlin  wrote:

> On Thu, 2012-03-22 at 22:00 -0700, Andy Smith wrote:
>
> > It is something pulled from the google style guide.
>
> You know what else is in the Google style guide? "Avoid global
> variables" :-)
>
>
> http://google-styleguide.googlecode.com/svn/trunk/pyguide.html?showone=Global_variables#Global_variables
>
> Mark.
>
>
And yet they use a global config object in most of their code, as do
facebook and digg (at quick survey of those people sitting near me).

They also have a variety of things in there that we don't follow because it
is easier to convince people of PEP8 than of the google style guide, but
there are plenty of other good ideas in there that we could absorb, too.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

Was the db on a separate server or loopback?

On 03/23/2012 05:26 PM, Mark Washenberger wrote:
> 
> 
> "Johannes Erdfelt"  said:
> 
>>
>> MySQL isn't exactly slow and Nova doesn't have particularly large
>> tables. It looks like the slowness is coming from the network and how
>> many queries are being made.
>>
>> Avoiding joins would mean even more queries, which looks like it would
>> slow it down even further.
>>
> 
> This is exactly what I saw in my profiling. More complex queries did
> still seem to take longer than less complex ones, but it was a second
> order effect compared to the overall volume of queries. 
> 
> I'm not sure that network was the culprit though, since my ping
> roundtrip time was small relative to the wall time I measured for each
> nova.db.api call.
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Monsyne Dragon]

On Mar 23, 2012, at 10:20 AM, Kevin L. Mitchell wrote:

> On Fri, 2012-03-23 at 13:43 +, Gabe Westmaas wrote:
>> However, I kind of expect that many users
>> will still poll even if they know they won't get new data until X
>> time. 
> 
> I wish there was some kind of way for us to issue push notifications to
> the client, i.e., have the client register some sort of callback and
> what piece of data / state change they're interested in, then nova would
> call that callback when the condition occurred.  It probably wouldn't
> stop polling, but we could ratchet down rate limits to encourage users
> to use the callback mechanism.

Actually, that is (one) of the things the notifications system was designed to 
accommodate.   If you use attach a feed generator (like Yagi) to the 
notification queues, plus a PubSubHubub hub, folks can subscribe to events by 
event type.  (Other pubsub strategies would work too, like XMPP pubsub) 


> Of course, then there's the problem of, what if the user is behind a
> firewall or some sort of NAT... :/

 PSH pushes to a web callback supplied by the client. Presumably they could run 
the callback receiver somewhere else, or some thru some proxy. 


> -- 
> Kevin L. Mitchell 
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

--
Monsyne M. Dragon
OpenStack/Nova 
cell 210-441-0965
work x 5014190


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Mark Washenberger]

"Johannes Erdfelt"  said:

> 
> MySQL isn't exactly slow and Nova doesn't have particularly large
> tables. It looks like the slowness is coming from the network and how
> many queries are being made.
> 
> Avoiding joins would mean even more queries, which looks like it would
> slow it down even further.
> 

This is exactly what I saw in my profiling. More complex queries did
still seem to take longer than less complex ones, but it was a second
order effect compared to the overall volume of queries. 

I'm not sure that network was the culprit though, since my ping
roundtrip time was small relative to the wall time I measured for each
nova.db.api call.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Being pedantic about pedanticism: HACKING styleguide

[Mark McLoughlin]

On Thu, 2012-03-22 at 22:00 -0700, Andy Smith wrote:

> It is something pulled from the google style guide.

You know what else is in the Google style guide? "Avoid global
variables" :-)

http://google-styleguide.googlecode.com/svn/trunk/pyguide.html?showone=Global_variables#Global_variables

Mark.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Being pedantic about pedanticism: HACKING styleguide

[Mark McLoughlin]

On Fri, 2012-03-23 at 11:18 -0500, Andrew Bogott wrote:
> 1)  I can back out my new style-guide from openstack-common

openstack-common still needs a HACKING file for itself :)

Cheers,
Mark.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

Ugh (reply vs reply-all again)

On 03/23/2012 02:58 PM, Joshua Harlow wrote:
> Right,
> 
> Lets fix the problem, not add a patch that hides the problem.
> 
> U can’t put lipstick on a pig, haha. Its still a pig...

When stuff is expensive to compute, caching is the only option (yes?).
Whether that lives in memcache, a db or in a dict. Tuning sql queries
will only get us so far. I think creating custom view tables is a
laborious and error prone tact ... additionally you get developers that
start to depend on the view tables as gospel.

Or am I missing something here?

-S


> On 3/22/12 8:02 PM, "Mark Washenberger"
>  wrote:
> 
> This is precisely my concern.
> 
> It must be brought up that with Rackspace Cloud Servers, nearly
> all client codes routinely submit requests with a query parameter
> "cache-busting=" just to get around problems with
> cache invalidation. And woe to the client that does not.
> 
> I get the feeling that once trust like this is lost, a project has
> a hard time regaining it. I'm not saying that we can avoid
> inconsistency entirely. Rather, I believe we will have to embrace
> some eventual-consistency models to enable the performance and
> scale we will ultimately attain. But I just get the feeling that
> generic caches are really only appropriate for write-once or at
> least write-rarely data. So personally I would rule out external
> caches entirely and try to be very judicious in selecting internal
> caches as well.

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

You can. The sanctioned approach is to use Yagi with a feed into
something like PubSubHubBub that lives on the public interweeb.

It's just an optional component.

-S

On 03/23/2012 12:20 PM, Kevin L. Mitchell wrote:
> On Fri, 2012-03-23 at 13:43 +, Gabe Westmaas wrote:
>> However, I kind of expect that many users
>> will still poll even if they know they won't get new data until X
>> time. 
> 
> I wish there was some kind of way for us to issue push notifications to
> the client, i.e., have the client register some sort of callback and
> what piece of data / state change they're interested in, then nova would
> call that callback when the condition occurred.  It probably wouldn't
> stop polling, but we could ratchet down rate limits to encourage users
> to use the callback mechanism.
> 
> Of course, then there's the problem of, what if the user is behind a
> firewall or some sort of NAT... :/

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] KVM crash.

[Guilherme Birk]

I'm having problems with KVM on a single node installation. My problem is like 
these ones: 

https://bugzilla.kernel.org/show_bug.cgi?id=42703
http://www.spinics.net/lists/kvm/msg67635.html

The KVM normally crashes when I'm doing a load test with a large number of 
connections on the VM's.

This appears to not happen using a dual node installation, with the 
nova-compute running with a dedicated machine.
  ___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Object expirer in Swift SAIO

[Pete Zaitcev]

Hello:

Is running swift-object-expirer necessary in SAIO? Currently, there is
no instruction to set one up, and so swift-init bypasses it, only reports
that it's missing. Things seem to work fine without. I found a bunch of
tombstones, but they are all zero length.

Curious,
-- Pete

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Johannes Erdfelt]

On Fri, Mar 23, 2012, Debo Dutta (dedutta)  wrote:
> +1 to DBs being slow. But what if we used a combo of memcache and db. Or
> use couch/mongo. 
> 
> Comparision:
> http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis 
> 
> Anyone has experience on large deployments to see the kind of db traffic
> we need to optimize for?
> 
> Another thing could be to avoid joins and then do sharding. 

Seems like that's the opposite of what we want.

MySQL isn't exactly slow and Nova doesn't have particularly large
tables. It looks like the slowness is coming from the network and how
many queries are being made.

Avoiding joins would mean even more queries, which looks like it would
slow it down even further.

JE


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Nova Live Migration Error

[Andrew Weiss]

Hey guys, 


Not sure if this is a bug or something wrong on my end. Attempting to conduct a 
live migration between two compute nodes I have running. Using Essex release 
rc1. When I execute the following command: 'nova live-migration  
' I get an HTTP 400 failure. Below is an error that I found 
in my nova-scheduler.log file: 



ERROR: Live migration of instance 7a938bdb-9970-41fd-acc2-d27fa3d13088 to host 
computehost1 failed (HTTP 400) 
(nova.rpc.amqp): TRACE: File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 222, in 
schedule_live_migration 
(nova.rpc.amqp): TRACE: disk_over_commit) 
(nova.rpc.amqp): TRACE: File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 322, in 
_live_migration_common_check 
(nova.rpc.amqp): TRACE: self.mounted_on_same_shared_storage(context, 
instance_ref, dest) 
(nova.rpc.amqp): TRACE: File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 521, in 
mounted_on_same_shared_storage 
(nova.rpc.amqp): TRACE: raise exception.FileNotFound(file_path=filename) 
(nova.rpc.amqp): TRACE: FileNotFound: File tmpQ9Ig08 could not be found. 
(nova.rpc.amqp): TRACE: 
2012-03-23 14:18:06 ERROR nova.rpc.amqp 
[req-aca97b5b-f456-4b9a-8561-ff12bd66aad9 10daac9302ba42eb9f4b592955680407 
e57faa938486428a9ba6d77402f5279c] Returning exception File tmpQ9Ig08 could not 
be found. to caller 
2012-03-23 14:18:06 ERROR nova.rpc.amqp 
[req-aca97b5b-f456-4b9a-8561-ff12bd66aad9 10daac9302ba42eb9f4b592955680407 
e57faa938486428a9ba6d77402f5279c] ['Traceback (most recent call last):\n', ' 
File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in 
_process_data\n rval = node_func(context=ctxt, **node_args)\n', ' File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 97, in 
_schedule\n context, ex, *args, **kwargs)\n', ' File 
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', 
' File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 92, 
in _schedule\n return driver_method(*args, **kwargs)\n', ' File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 222, in 
schedule_live_migration\n disk_over_commit)\n', ' File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 322, in 
_live_migration_common_check\n self.mounted_on_same_shared_storage(context, 
instance_ref, dest)\n', ' File 
"/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 521, in 
mounted_on_same_shared_storage\n raise 
exception.FileNotFound(file_path=filename)\n', 'FileNotFound: File tmpQ9Ig08 
could not be found.\n'] 


I can't seem to figure out what file is is trying to find. Just looking to get 
some guidance on this. 


Thanks, 
Andrew Weiss 
wei...@purdue.edu 
LinkedIn | http://www.linkedin.com/in/weissa 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [devstack] unknown url type: git+https in quantum pip dependencies

[Deepak Garg]

This same issue is being discussed in the netstack ML and a couple of
solutions are mentioned

https://lists.launchpad.net/netstack/msg00927.html


Cheers,
Deepak


On Fri, Mar 23, 2012 at 11:20 PM, Nicolas de BONFILS
 wrote:
> Hi stackers,
>
> I use devstack to install, and got the following error :
>>
>> + cd /home/nicolas/openstack/quantum
>> + sudo python setup.py develop
>> running develop
>> Checking .pth file support in /usr/local/lib/python2.7/dist-packages/
>> /usr/bin/python -E -c pass
>> TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support
>> .pth files
>> running egg_info
>> creating quantum.egg-info
>> writing requirements to quantum.egg-info/requires.txt
>> writing quantum.egg-info/PKG-INFO
>> writing eager_resources to quantum.egg-info/eager_resources.txt
>> writing top-level names to quantum.egg-info/top_level.txt
>> writing dependency_links to quantum.egg-info/dependency_links.txt
>> writing entry points to quantum.egg-info/entry_points.txt
>> writing manifest file 'quantum.egg-info/SOURCES.txt'
>> reading manifest file 'quantum.egg-info/SOURCES.txt'
>> reading manifest template 'MANIFEST.in'
>> writing manifest file 'quantum.egg-info/SOURCES.txt'
>> running build_ext
>> Creating /usr/local/lib/python2.7/dist-packages/quantum.egg-link (link to
>> .)
>> quantum 2012.2 is already the active version in easy-install.pth
>> Installing quantum-linuxbridge-agent script to /usr/local/bin
>> Installing quantum-openvswitch-agent script to /usr/local/bin
>> Installing quantum-ryu-agent script to /usr/local/bin
>> Installing quantum-server script to /usr/local/bin
>>
>> Installed /home/nicolas/openstack/quantum
>> Processing dependencies for quantum==2012.2
>> Searching for python-quantumclient
>> Best match: python-quantumclient [unknown version]
>> Downloading
>> git+https://review.openstack.org/p/openstack/python-quantumclient#egg=python-quantumclient
>> error: Download error for
>> git+https://review.openstack.org/p/openstack/python-quantumclient: unknown
>> url type: git+https
>> ++ failed
>> ++ local r=1
>> ++ set +o xtrace
>
>
> Its seems that the git+https protocol is not recognize. I tried to google it
> and find no reference. Do I need to install git review to download those
> files ?
>
> Thanks for the help
>
> ---
> Nicolas
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>



-- 

Deepak Garg,
Data Center and Cloud Div.
Citrix R&D, India
Skype-id: deepakgarg.iit

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [devstack] unknown url type: git+https in quantum pipdependencies

[Nicolas de BONFILS]

Thanks a lot, it works ! :D

I looked earlier at this thread on launchpad, but it was before the
answers, and I forgot to look again :-S

---
Nicolas



On Fri, Mar 23, 2012 at 19:08, Rohit Agarwalla (roagarwa) <
roaga...@cisco.com> wrote:

>  https://answers.launchpad.net/devstack/+question/191325
>
> ** **
>
> Workarounds suggested. Just tried and it worked.
>
> ** **
>
> Thanks
>
> Rohit
>
> ** **
>
> *From:* openstack-bounces+roagarwa=cisco@lists.launchpad.net [mailto:
> openstack-bounces+roagarwa=cisco@lists.launchpad.net] *On Behalf Of 
> *Nicolas
> de BONFILS
> *Sent:* Friday, March 23, 2012 10:50 AM
> *To:* openstack@lists.launchpad.net
> *Subject:* [Openstack] [devstack] unknown url type: git+https in quantum
> pipdependencies
>
> ** **
>
> Hi stackers,
>
> I use devstack to install, and got the following error :
>
> + cd /home/nicolas/openstack/quantum
> + sudo python setup.py develop
> running develop
> Checking .pth file support in /usr/local/lib/python2.7/dist-packages/
> /usr/bin/python -E -c pass
> TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support
> .pth files
> running egg_info
> creating quantum.egg-info
> writing requirements to quantum.egg-info/requires.txt
> writing quantum.egg-info/PKG-INFO
> writing eager_resources to quantum.egg-info/eager_resources.txt
> writing top-level names to quantum.egg-info/top_level.txt
> writing dependency_links to quantum.egg-info/dependency_links.txt
> writing entry points to quantum.egg-info/entry_points.txt
> writing manifest file 'quantum.egg-info/SOURCES.txt'
> reading manifest file 'quantum.egg-info/SOURCES.txt'
> reading manifest template 'MANIFEST.in'
> writing manifest file 'quantum.egg-info/SOURCES.txt'
> running build_ext
> Creating /usr/local/lib/python2.7/dist-packages/quantum.egg-link (link to
> .)
> quantum 2012.2 is already the active version in easy-install.pth
> Installing quantum-linuxbridge-agent script to /usr/local/bin
> Installing quantum-openvswitch-agent script to /usr/local/bin
> Installing quantum-ryu-agent script to /usr/local/bin
> Installing quantum-server script to /usr/local/bin
>
> Installed /home/nicolas/openstack/quantum
> Processing dependencies for quantum==2012.2
> Searching for python-quantumclient
> Best match: python-quantumclient [unknown version]
> Downloading git+
> https://review.openstack.org/p/openstack/python-quantumclient#egg=python-quantumclient
> error: Download error for git+
> https://review.openstack.org/p/openstack/python-quantumclient: unknown
> url type: git+https
> ++ failed
> ++ local r=1
> ++ set +o xtrace
> 
>
>
> Its seems that the git+https protocol is not recognize. I tried to google
> it and find no reference. Do I need to install git review to download those
> files ?
>
> Thanks for the help
>
> ---
>
> Nicolas
>
> ** **
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Debo Dutta (dedutta)]

+1 to DBs being slow. But what if we used a combo of memcache and db. Or
use couch/mongo. 

Comparision:
http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis 

Anyone has experience on large deployments to see the kind of db traffic
we need to optimize for?

Another thing could be to avoid joins and then do sharding. 

debo

-Original Message-
From: openstack-bounces+dedutta=cisco@lists.launchpad.net
[mailto:openstack-bounces+dedutta=cisco@lists.launchpad.net] On
Behalf Of Brian Lamar
Sent: Friday, March 23, 2012 10:51 AM
To: openstack@lists.launchpad.net
Subject: Re: [Openstack] Caching strategies in Nova ...


On Mar 23, 2012, at 11:22 AM, Kevin L. Mitchell wrote:

> On Fri, 2012-03-23 at 08:55 -0300, Sandy Walsh wrote:
>> I don't doubt for a second the db is the culprit for many of our
woes.
>> 
>> The thing I like about internal caching using established tools is
>> that
>> it works for db issues too without having to resort to custom tables.
>> SQL query optimization, I'm sure, will go equally far. 
> 
> For that matter, I wouldn't be surprised if there were things we could
> do to nova's DB to speed things up.  For instance, what if we
supported
> non-SQL data stores?

Any database is going to be slow if you're talking to it more than
necessary. Even if we replaced MySQL with the latest and greatest
web-scale noSQL database out there we'd still be slow. I'd love to see a
combination effort of improving the flexibility of the DB layer as well
as improvements surrounding the sheer number of calls to the database.

> -- 
> Kevin L. Mitchell 
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [devstack] unknown url type: git+https in quantum pipdependencies

[Rohit Agarwalla (roagarwa)]

https://answers.launchpad.net/devstack/+question/191325

 

Workarounds suggested. Just tried and it worked.

 

Thanks

Rohit

 

From: openstack-bounces+roagarwa=cisco@lists.launchpad.net 
[mailto:openstack-bounces+roagarwa=cisco@lists.launchpad.net] On Behalf Of 
Nicolas de BONFILS
Sent: Friday, March 23, 2012 10:50 AM
To: openstack@lists.launchpad.net
Subject: [Openstack] [devstack] unknown url type: git+https in quantum 
pipdependencies

 

Hi stackers,

I use devstack to install, and got the following error :

+ cd /home/nicolas/openstack/quantum
+ sudo python setup.py develop
running develop
Checking .pth file support in /usr/local/lib/python2.7/dist-packages/
/usr/bin/python -E -c pass
TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support 
.pth files
running egg_info
creating quantum.egg-info
writing requirements to quantum.egg-info/requires.txt
writing quantum.egg-info/PKG-INFO
writing eager_resources to quantum.egg-info/eager_resources.txt
writing top-level names to quantum.egg-info/top_level.txt
writing dependency_links to quantum.egg-info/dependency_links.txt
writing entry points to quantum.egg-info/entry_points.txt
writing manifest file 'quantum.egg-info/SOURCES.txt'
reading manifest file 'quantum.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'quantum.egg-info/SOURCES.txt'
running build_ext
Creating /usr/local/lib/python2.7/dist-packages/quantum.egg-link (link 
to .)
quantum 2012.2 is already the active version in easy-install.pth
Installing quantum-linuxbridge-agent script to /usr/local/bin
Installing quantum-openvswitch-agent script to /usr/local/bin
Installing quantum-ryu-agent script to /usr/local/bin
Installing quantum-server script to /usr/local/bin

Installed /home/nicolas/openstack/quantum
Processing dependencies for quantum==2012.2
Searching for python-quantumclient
Best match: python-quantumclient [unknown version]
Downloading 
git+https://review.openstack.org/p/openstack/python-quantumclient#egg=python-quantumclient
error: Download error for 
git+https://review.openstack.org/p/openstack/python-quantumclient: unknown url 
type: git+https
++ failed
++ local r=1
++ set +o xtrace



Its seems that the git+https protocol is not recognize. I tried to google it 
and find no reference. Do I need to install git review to download those files ?

Thanks for the help

---

Nicolas

 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Joshua Harlow]

+ 100

On 3/23/12 10:50 AM, "Brian Lamar"  wrote:



On Mar 23, 2012, at 11:22 AM, Kevin L. Mitchell wrote:

> On Fri, 2012-03-23 at 08:55 -0300, Sandy Walsh wrote:
>> I don't doubt for a second the db is the culprit for many of our woes.
>>
>> The thing I like about internal caching using established tools is
>> that
>> it works for db issues too without having to resort to custom tables.
>> SQL query optimization, I'm sure, will go equally far.
>
> For that matter, I wouldn't be surprised if there were things we could
> do to nova's DB to speed things up.  For instance, what if we supported
> non-SQL data stores?

Any database is going to be slow if you're talking to it more than necessary. 
Even if we replaced MySQL with the latest and greatest web-scale noSQL database 
out there we'd still be slow. I'd love to see a combination effort of improving 
the flexibility of the DB layer as well as improvements surrounding the sheer 
number of calls to the database.

> --
> Kevin L. Mitchell 
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Being pedantic about pedanticism: HACKING styleguide

[Andy Smith]

>
>
> I am happy to take on and implement any of these approaches, up to and
> including scheduling an in-person Code Summit Battledome in order to
> resolve differences of opinion about docstring whitespace.  But, my
> preference for a unified style-guide is a mild one, and I'll probably just
> opt for solution 1 unless others express that unification is worth the
> trouble.


May the odds be ever in your favor.



>
>
> -Andrew
>
>
>
> __**_
> Mailing list: 
> https://launchpad.net/~**openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : 
> https://launchpad.net/~**openstack
> More help   : 
> https://help.launchpad.net/**ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Joshua Harlow]

Right,

Lets fix the problem, not add a patch that hides the problem.

U can't put lipstick on a pig, haha. Its still a pig...

On 3/22/12 8:02 PM, "Mark Washenberger"  wrote:

This is precisely my concern.

It must be brought up that with Rackspace Cloud Servers, nearly
all client codes routinely submit requests with a query parameter
"cache-busting=" just to get around problems with
cache invalidation. And woe to the client that does not.

I get the feeling that once trust like this is lost, a project has
a hard time regaining it. I'm not saying that we can avoid
inconsistency entirely. Rather, I believe we will have to embrace
some eventual-consistency models to enable the performance and
scale we will ultimately attain. But I just get the feeling that
generic caches are really only appropriate for write-once or at
least write-rarely data. So personally I would rule out external
caches entirely and try to be very judicious in selecting internal
caches as well.

"Joshua Harlow"  said:

> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> Just from experience.
>
> They do a great job. But the killer thing about caching is how u do the cache
> invalidation.
>
> Just caching stuff is easy-peasy, making sure it is invalidated on all 
> servers in
> all conditions, not so easy...
>
> On 3/22/12 4:26 PM, "Sandy Walsh"  wrote:
>
> We're doing tests to find out where the bottlenecks are, caching is the
> most obvious solution, but there may be others. Tools like memcache do a
> really good job of sharing memory across servers so we don't have to
> reinvent the wheel or hit the db at all.
>
> In addition to looking into caching technologies/approaches we're gluing
> together some tools for finding those bottlenecks. Our first step will
> be finding them, then squashing them ... however.
>
> -S
>
> On 03/22/2012 06:25 PM, Mark Washenberger wrote:
>> What problems are caching strategies supposed to solve?
>>
>> On the nova compute side, it seems like streamlining db access and
>> api-view tables would solve any performance problems caching would
>> address, while keeping the stale data management problem small.
>>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>



___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] [devstack] unknown url type: git+https in quantum pip dependencies

[Nicolas de BONFILS]

Hi stackers,

I use devstack to install, and got the following error :

> + cd /home/nicolas/openstack/quantum
> + sudo python setup.py develop
> running develop
> Checking .pth file support in /usr/local/lib/python2.7/dist-packages/
> /usr/bin/python -E -c pass
> TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support
> .pth files
> running egg_info
> creating quantum.egg-info
> writing requirements to quantum.egg-info/requires.txt
> writing quantum.egg-info/PKG-INFO
> writing eager_resources to quantum.egg-info/eager_resources.txt
> writing top-level names to quantum.egg-info/top_level.txt
> writing dependency_links to quantum.egg-info/dependency_links.txt
> writing entry points to quantum.egg-info/entry_points.txt
> writing manifest file 'quantum.egg-info/SOURCES.txt'
> reading manifest file 'quantum.egg-info/SOURCES.txt'
> reading manifest template 'MANIFEST.in'
> writing manifest file 'quantum.egg-info/SOURCES.txt'
> running build_ext
> Creating /usr/local/lib/python2.7/dist-packages/quantum.egg-link (link to
> .)
> quantum 2012.2 is already the active version in easy-install.pth
> Installing quantum-linuxbridge-agent script to /usr/local/bin
> Installing quantum-openvswitch-agent script to /usr/local/bin
> Installing quantum-ryu-agent script to /usr/local/bin
> Installing quantum-server script to /usr/local/bin
>
> Installed /home/nicolas/openstack/quantum
> Processing dependencies for quantum==2012.2
> Searching for python-quantumclient
> Best match: python-quantumclient [unknown version]
> Downloading git+
> https://review.openstack.org/p/openstack/python-quantumclient#egg=python-quantumclient
> error: Download error for git+
> https://review.openstack.org/p/openstack/python-quantumclient: unknown
> url type: git+https
> ++ failed
> ++ local r=1
> ++ set +o xtrace
>

Its seems that the git+https protocol is not recognize. I tried to google
it and find no reference. Do I need to install git review to download those
files ?

Thanks for the help

---
Nicolas
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Brian Lamar]

On Mar 23, 2012, at 11:22 AM, Kevin L. Mitchell wrote:

> On Fri, 2012-03-23 at 08:55 -0300, Sandy Walsh wrote:
>> I don't doubt for a second the db is the culprit for many of our woes.
>> 
>> The thing I like about internal caching using established tools is
>> that
>> it works for db issues too without having to resort to custom tables.
>> SQL query optimization, I'm sure, will go equally far. 
> 
> For that matter, I wouldn't be surprised if there were things we could
> do to nova's DB to speed things up.  For instance, what if we supported
> non-SQL data stores?

Any database is going to be slow if you're talking to it more than necessary. 
Even if we replaced MySQL with the latest and greatest web-scale noSQL database 
out there we'd still be slow. I'd love to see a combination effort of improving 
the flexibility of the DB layer as well as improvements surrounding the sheer 
number of calls to the database.

> -- 
> Kevin L. Mitchell 
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [OpenStack] Xen Hypervisor

[Alexandre Leites]

Hi folks,

Sorry for late reply, i was trying to install this without following any 
ready-to-use scripts ( but i used one :( ) to understand how things are made. 
So i installed XCP 1.4.90 from DVD and configured it from installation screen.

Execute the following commands on dom0
--- Dom 0 Extra Config 
cd /etc/xapi.d/plugins/
wget -q 
https://raw.github.com/openstack/nova/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/xenhost
wget -q 
https://raw.github.com/openstack/nova/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/xenstore.py
wget -q 
https://raw.github.com/openstack/nova/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/pluginlib_nova.py
wget -q 
https://raw.github.com/openstack/nova/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/migration
wget -q 
https://raw.github.com/openstack/nova/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/glance
wget -q 
https://raw.github.com/openstack/nova/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/agent
chmod 777 *
service xapi restart
---

After that i downloaded XenCenter and created a VM from Ubuntu Server 11.10 
CD... (all from below is on guest)

After this, i did updated all the system with command apt-get update && apt-get 
upgrade -y && apt-get dist-upgrade -y and rebooted the machine. When you do 
this, you'll boot on newer kernel (3.0.0-16-server, check with uname -a 
command)... and unsinstall the old kernel (apt-get purge 
linux-image-3.0.0-12-server).

After this, i did installed the virtual kernel (3.0.0-16-virtual) and rebooted 
the machine. Check if you rebooted in this kernel via uname -a and unsinstall 
the old kernel.

After this, execute on dom0 again:
--- Dom 0 Extra Config 
cd ~
wget http://94.212.78.134/res/xenserver/makepv.sh
chmod +x makepv.sh
./makepv.sh YOUR-XEN-GUEST-NAME
---

After this, you can do
apt-get install -y cracklib-runtime curl wget ssh openssh-server tcpdump 
ethtool python-pip git vim-nox sudo
and
pip install xenapi
wget http://images.ansolabs.com/xen/xe-guest-utilities_5.6.100-651_amd64.deb -O 
xe-guest-utilities_5.6.100-651_amd64.deb
dpkg -i xe-guest-utilities_5.6.100-651_amd64.deb
update-rc.d -f xe-linux-distribution remove
update-rc.d xe-linux-distribution defaults

mkdir -p /usr/share/cracklib
echo a | cracklib-packer
pwconv

echo root:password | chpasswd

rm -f /etc/localtime
groupadd libvirtd
useradd stack -s /bin/bash -d /opt/stack -G libvirtd
echo stack:password | chpasswd
echo "stack ALL=(ALL) NOPASSWD: ALL" >> etc/sudoers
mkdir -p /opt/stack
chown -R stack /opt/stack

After all this, you can install nova-compute , copy nova.conf from your 
controller and configure the following vars:
--connection_type=xenapi
--xenapi_connection_username=root
--xenapi_connection_password=password
--xenapi_connection_url=http://<>

and restart your nova-compute service.

You can test your XenAPI configuration using sudo nova-manage shell python and 
after pasting this
import XenAPI
import nova.virt.xenapi_conn
nova.virt.xenapi_conn.XenAPI = XenAPI
x = 
nova.virt.xenapi_conn.XenAPIConnection("http://<>","root","password")
x.list_instances()

--

After all this things, i got Xen working, but i have a error with bridge now, 
as trace below:

2012-03-23 16:30:00,116 DEBUG nova.virt.xenapi [-] Updating host stats from 
(pid=23556) update_status 
/usr/lib/python2.7/dist-packages/nova/virt/xenapi_conn.py:488
2012-03-23 16:30:00,988 WARNING nova.virt.xenapi [-] Task 
[Async.host.call_plugin] OpaqueRef:d7a9f0df-0c7c-a760-6b76-3e985c747b1d status: 
failure['XENAPI_PLUGIN_FAILURE', 'host_data', 'KeyError', "'host_uuid'"]
2012-03-23 16:30:00,992 WARNING nova.compute.manager [-] Error during 
report_driver_status(): ['XENAPI_PLUGIN_FAILURE', 'host_data', 'KeyError', 
"'host_uuid'"]
2

And yes, my permissions are 777 to all plugins.

> From: todd.desh...@xen.org
> Date: Wed, 21 Mar 2012 11:04:19 -0400
> To: openstack@lists.launchpad.net
> Subject: Re: [Openstack] [OpenStack] Xen Hypervisor
> 
> Just realized my reply was accidentally not sent to the list.
> 
> On Tue, Mar 20, 2012 at 2:03 PM, Todd Deshane  wrote:
> > Please post specific error messages.
> >
> > Some general suggestions inline.
> >
> > On Tue, Mar 20, 2012 at 12:56 PM, Alexandre Leites  
> > wrote:
> >> Hi folks,
> >>
> >> First let me say that i'm trying to install xen hypervisor and integrate it
> >> with OpenStack for more than one week. I'm studying OpenStack for a company
> >> and this company doesn't allow us to use ready scripts (Why? they want to 
> >> be
> >> different from the whole world).
> >>
> >> I have used some links for references:
> >> https://github.com/openstack-dev/devstack/blob/master/tools/xen/README.md
> >> http://wiki.openstack.org/XenAPI
> >> http://wiki.openstack.org/XenServer/DevStack
> >> http://wiki.openstack.org/XenServer/Install
> >> http://wiki.openstack.org/XenServerDevelopment
> >> http://wiki.openstack.org/XenXCPAndXenServer
> >

Re: [Openstack] Can't delete instances with "error" status.

[Day, Phil]

Here’s the way we’ve approached this:


-  A user can always send a delete request for a VM in any state (this 
is the only action that is always allowed).

-  Once a VM has a task_state of “Deleting” (set in the API server) the 
only action they can perform is delete

o   Hence at this point we can stop billing for it, and the user shouldn’t have 
it counted in their quota

-  A common reason for VMs getting stuck in Deleting is that the 
compute manager is restarted (or fails) �C so we have added code to the 
computer manager start-up to check for instances with a task_state of deleting 
and delete them (this needs to be able to cope with various exceptions if the 
delete was part way thought).   Since the manage is restarting we can be sure 
that the eventlet that was handling the delete isn’t doing it anymore ;-)

So from the user perspective we honour their request to delete VMs, make sure 
they can’t change their mind, and try to cleanup eventually as part of 
compute_manager restart.


-  By the same logic we also reset the “Image_Snapshot” and 
“Image_backup” task_state, as we know they aren’t true anymore.

-  It would be possible to also handle other task_states such as 
“rebuilding” and “rebooting”, but we haven’t tried that.

Phil

From: openstack-bounces+philip.day=hp@lists.launchpad.net 
[mailto:openstack-bounces+philip.day=hp@lists.launchpad.net] On Behalf Of 
Guilherme Birk
Sent: 22 March 2012 13:24
To: Openstack Mail List
Subject: Re: [Openstack] Can't delete instances with "error" status.

Gabe, responding to your question "Do you know how to reliably reproduce an 
instance in ERROR state that cannot be deleted?":
In my case, I'm updating the status of the VM to "error" directly on the 
database. This is just for testing. But even when my VM is running and working 
fine, when I update it to "error" in the database I can't delete it.
> From: gabe.westm...@rackspace.com
> To: cp16...@gmail.com
> Date: Thu, 22 Mar 2012 06:49:15 +
> CC: openstack@lists.launchpad.net
> Subject: Re: [Openstack] Can't delete instances with "error" status.
>
> There are definitely lots of cases where deleting an instance in error state 
> works fine, and I’d like to know about the cases where it doesn’t. They do 
> count against quota as well, so that’s a problem!
>
> I can see value in keeping an instance around �C if the operations team has 
> configured it to do so. However, it seems like the end user has asked for it 
> to be deleted and to them it should appear to be deleted. Johannes had an 
> idea a while ago to allow the operations team to specify a project ID that 
> deleted servers that match certain criteria should be moved to. If the delete 
> finishes up fine, then its no problem, the delete is done, the customer is 
> happy and the ops account is empty. If it fails for some reason, there is 
> manual cleanup to be done, but that should be on the operator of the 
> deployment, not the user. I think its critical for anything like this to be 
> configurable, as public clouds and private clouds have different privacy and 
> retention concerns, I would guess.
>
> Do we have cases where we can reliably reproduce this issue? If its happening 
> every time on some deployments there is a very serious problem!
>
> Gabe
>
>
> From: Craig Vyvial 
> [mailto:cp16...@gmail.com]
> Sent: Thursday, March 22, 2012 2:20 AM
> To: Ga be Westmaas
> Cc: Yong Sheng Gong; Openstack Mail List
> Subject: Re: [Openstack] Can't delete instances with "error" status.
>
> My understanding is that you would not always want a user to delete an 
> instance in an error state. So an operations person can figure out what went 
> wrong. I think the instances that are in error state do not count against the 
> quota but i agree that they clutter up the API calls to list servers.
>
> I have noticed this with my team and written code around this case to force 
> the instance into an 'active' state before sending nova the delete call if 
> the instance was in an 'error' or 'suspended' state.
>
> -Craig
>
> On Thu, Mar 22, 2012 at 1:02 AM, Gabe Westmaas 
> mailto:gabe.westm...@rackspace.com>> wrote:
> Instances in deleted status can normally be deleted, but there is definitely 
> a bug to file here somewhere �C possibly more than one.  A common reason I 
> have seen is that the node the instance lives on is no longer operating 
> correctly, so the compute manager never gets the delete request, so it 
> doesn’t finish.  If we can narrow the cases where this happens, we can file 
> bugs and decide how to resolve them �C although there may be some additional 
> work beyond just a developer picking up the bug and working on it to decide 
> what should happen!
>
> Do you know how to reliably reproduce an instance in ERROR state that cannot 
> be deleted?
>
> Gabe
>

Re: [Openstack] Keystone auth_token confusion in Swift vs Glance; Persistent tokens or not ?

[Joseph Heck]

Florian - 

Would love to have some docs patches!

-joe

On Mar 23, 2012, at 6:04 AM, Florian Daniel Otel wrote:

> Thanks Jay for the in-depth explanation, it actually does make sense :) 
> 
> Sorry for the late reply -- I had to to change my Keystone, Swift and Glance 
> setup and digest the changes. Still not completely out of the woods yet
> 
> Thanks Chmouel for the the patch [1]  for the Glance and Swift config files. 
> It's pretty much what I had in mind as well. However, shouldn't  
> "keystone.conf" itself be changed accordingly  ? I can take a stab at it and 
> submit a proposal if it helps...
> 
> Also, wrt to the docs [2]. AFAICT the patch covers only 
> "doc/sources/configuringservices.rst" , but it needs fixing in other places 
> too (e.g. "doc/sources/middleware_architecture.rst").  Same thing there --  I 
> can try to make a patch and submit it (I need to get the hang of it first...)
> 
> Regards,
> 
> Florian
> 
> [1] 
> https://github.com/openstack-dev/devstack/commit/bd07d61c4545c52d39b9c957ff9e4423525ca452
> [2] https://review.openstack.org/#change,5190
> 
> 
> 
> On Tue, Mar 20, 2012 at 10:09 PM, Jay Pipes  wrote:
> Hi Florian, I appreciate your post and sympathize with your 
> frustration/confusion. I'll do my best below to help un-confuse :)
> 
> 
> On 03/20/2012 12:15 PM, Florian Daniel Otel wrote:
> Hello all,
> 
> I need some help with inconsistencies and -- in my mind -- confusing
> instructions wrt "auth_token" Keytone middleware.
> 
> So far I have Keystone and Swift  w/ Keystone auth working well together
> (ussing Essex-4 milestone release  of Keystone resp v1.4.6 of Swift).
> 
> What I am now  trying is to get  Glance on Swift with Keystone Auth  but
> I'm faced with conflicting info that I cannot make sense of
> 
> 1) Naming inconsistencies -- "token_auth" , "tokenauth" (Keystone resp
> Swift) vs "authtoken"  (Glance)
> 
> The existing Keystone and Swift docs (e.g. this one:
> http://keystone.openstack.org/configuringservices.html#configuring-swift-to-use-keystone*)
> *use as names "token_auth" (Keystone) resp "tokenauth" (Swift)  wheareas
> 
> in Glance docs the same midldleware (i.e.
> "keystone/middleware/auh_token.py") is referred to as "authtoken"  (see
> e.g.  here http://glance.openstack.org/authentication.html )
> 
> While this may be only pedantic IMHO it would help if things would be
> called the same way in different places.  Or, if I'm confusing things,
> someone point me out the differences to which is what and why.
> 
> Completely agreed, and Chmouel has submitted two patches to address these 
> inconsistencies.
> 
> 
> 2) "auth_token":  Persistent tokens or user/pass ?
> 
> As per Jay Pipes comment here:
> https://bugs.launchpad.net/glance/+bug/953989/comments/2  (from
> 2012-03-13)  the concept of long-lived (i.e. persistent) tokens are no
> longer supported by the Keystone "auth_token" middleware  -- and that is
> listed as why that bug was invalid.
> 
> Actually, no not quite. The bug was marked Invalid because the source of the 
> issue was three things that Kevin (the bug reporter) had failed to do in his 
> setup -- two of which had to do with the authentication and one having to do 
> with an incorrect endpoint template placeholder.
> 
> Anyway, let me try and explain what the deal is with the "long-lived token 
> deprecation" that is mentioned in the bug report.
> 
> Previously in Keystone there was the concept of a "service token" -- 
> otherwise known as a long-lived token. These special tokens would be read 
> from a configuration file or paste-ini file by the auth_token middleware that 
> sat, for instance, in the Glance API server pipeline. Importantly, these 
> special tokens were **not required to be attached to any user or tenant**.
> 
> The auth_token middleware is responsible for authenticating -- via Keystone 
> -- incoming regular user requests on an API endpoint. For example, if I call 
> GET http://example.com:9292/v1/images to get a list of images from a Glance 
> API server, and that server was configured with the auth_token middleware, 
> then the middleware looks for certain HTTP headers in the incoming request.
> 
> If the HTTP_X_AUTH_TOKEN header is present in the incoming request,
> then the middleware would need to *validate* this authentication token with 
> Keystone. This validation call (POST /v2.0/tokens/{$token_id} [1]) is a 
> *privileged* call in the Keystone Service Admin API, and as such, in order to 
> make the validation request, the HTTP request to Keystone itself needs, 
> itself, to supply an HTTP_X_AUTH_TOKEN that is "scoped" to some entity that 
> can make this privileged validation call. However, as mentioned before, the 
> long-lived "service token" was not scoped to any user/tenant, and these 
> "unscoped" tokens were accepted by the validate token API call as privileged 
> tokens...
> 
> OK, so recent changes removed this inconsistency in the scoping of tokens. 
> There are no longer any tokens that are u

Re: [Openstack] Melange RC1 now available

[Jay Pipes]

Great work, Troy and team!

On 03/23/2012 11:50 AM, Troy Toman wrote:

Melange 2012.1 RC1 is now available at:

https://launchpad.net/melange/essex/essex-rc1

The only updates are bug fixes at this point as new work is being
directed towards a planned merge within the Quantum project in the
Folsom timeframe. Please report any issues or concerns with this release
soon. If none are found, we will release this as a final version by
April 5th.

Troy Toman

Melange PTL


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Being pedantic about pedanticism: HACKING styleguide

[Jay Pipes]

On 03/23/2012 12:18 PM, Andrew Bogott wrote:

OK, so I see three ways of going forward here.

1) I can back out my new style-guide from openstack-common, and add my
new encoding rule to each project guide.


This seems good to me.

-jay

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] keystone, euc2ools and nova-manage issue

[Vijay]

Hello,
I have integrated keystone, nova/openstack and horizon.
I have created user/tenant/ etc according to 
http://docs.openstack.org/diablo/openstack-compute/install/content/identity-configure-keystone.html
 
I was able to upload an image and launch it. I was able to ssh and ping to the 
vm through fixed ip.
 
Question 1: 
I was able to allocate a floating ip address,
However, I have not been able to associate floating ip address to the instance:
I am getting error:
ApiError: (ip_address) is not allocated.
I see in the logs that Project_id is always " none".
Is there a way to associate project_id that euca uses (in nova.projects table) 
with keystone tenant?
Or is there any method other than euca-associate-address to use?
 
Question 2:
When I try to increase the  floating_ips quota, I get the following error:
sudo nova-manage project quota floating_ips 100
Command failed, please check log for more info
 
In the logs, I see the following error:
AttributeError: 'NoneType' object has no attribute 'lower'
 
Any clue  or lead is appreciated.
 
Thanks,
-vijay___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Being pedantic about pedanticism: HACKING styleguide

[Andrew Bogott]

OK, so I see three ways of going forward here.

1)  I can back out my new style-guide from openstack-common, and add my 
new encoding rule to each project guide.


2)  We can modify each project-specific guide so that it's 
supplemental.  "Follow the common style guide, but with the following 
exceptions..."


3)  We can continue to dicker about rules in hopes of coming to a 
universal consensus.  I'm not clear that there's even actual 
disagreement about style, only about whether or not given rules are are 
aren't too niggling to put in the styleguide.  That doesn't strike me as 
an uncrossable rift, but I've already shown my naivete in these regards.


I am happy to take on and implement any of these approaches, up to and 
including scheduling an in-person Code Summit Battledome in order to 
resolve differences of opinion about docstring whitespace.  But, my 
preference for a unified style-guide is a mild one, and I'll probably 
just opt for solution 1 unless others express that unification is worth 
the trouble.


-Andrew


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Melange RC1 now available

[Troy Toman]

Melange 2012.1 RC1 is now available at:

https://launchpad.net/melange/essex/essex-rc1

The only updates are bug fixes at this point as new work is being directed 
towards a planned merge within the Quantum project in the Folsom timeframe. 
Please report any issues or concerns with this release soon. If none are found, 
we will release this as a final version by April 5th.

Troy Toman

Melange PTL
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Russell Bryant]

On 03/23/2012 11:36 AM, Johannes Erdfelt wrote:
> On Fri, Mar 23, 2012, Kevin L. Mitchell  wrote:
>> On Fri, 2012-03-23 at 13:43 +, Gabe Westmaas wrote:
>>> However, I kind of expect that many users
>>> will still poll even if they know they won't get new data until X
>>> time. 
>>
>> I wish there was some kind of way for us to issue push notifications to
>> the client, i.e., have the client register some sort of callback and
>> what piece of data / state change they're interested in, then nova would
>> call that callback when the condition occurred.  It probably wouldn't
>> stop polling, but we could ratchet down rate limits to encourage users
>> to use the callback mechanism.
>>
>> Of course, then there's the problem of, what if the user is behind a
>> firewall or some sort of NAT... :/
> 
> Long polling is always an option.

Or WebSockets.

-- 
Russell Bryant

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Johannes Erdfelt]

On Fri, Mar 23, 2012, Kevin L. Mitchell  wrote:
> On Fri, 2012-03-23 at 13:43 +, Gabe Westmaas wrote:
> > However, I kind of expect that many users
> > will still poll even if they know they won't get new data until X
> > time. 
> 
> I wish there was some kind of way for us to issue push notifications to
> the client, i.e., have the client register some sort of callback and
> what piece of data / state change they're interested in, then nova would
> call that callback when the condition occurred.  It probably wouldn't
> stop polling, but we could ratchet down rate limits to encourage users
> to use the callback mechanism.
> 
> Of course, then there's the problem of, what if the user is behind a
> firewall or some sort of NAT... :/

Long polling is always an option.

JE


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Kevin L. Mitchell]

On Fri, 2012-03-23 at 08:55 -0300, Sandy Walsh wrote:
> I don't doubt for a second the db is the culprit for many of our woes.
> 
> The thing I like about internal caching using established tools is
> that
> it works for db issues too without having to resort to custom tables.
> SQL query optimization, I'm sure, will go equally far. 

For that matter, I wouldn't be surprised if there were things we could
do to nova's DB to speed things up.  For instance, what if we supported
non-SQL data stores?
-- 
Kevin L. Mitchell 


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Kevin L. Mitchell]

On Fri, 2012-03-23 at 13:43 +, Gabe Westmaas wrote:
> However, I kind of expect that many users
> will still poll even if they know they won't get new data until X
> time. 

I wish there was some kind of way for us to issue push notifications to
the client, i.e., have the client register some sort of callback and
what piece of data / state change they're interested in, then nova would
call that callback when the condition occurred.  It probably wouldn't
stop polling, but we could ratchet down rate limits to encourage users
to use the callback mechanism.

Of course, then there's the problem of, what if the user is behind a
firewall or some sort of NAT... :/
-- 
Kevin L. Mitchell 


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Mark Washenberger]

Alas, I let my patch get too stale to rebase properly. However, it is a
fairly "dumb" approach I took that can be demonstrated just from the
patch. And in any case, I think the approach you're taking, profiling
based on Tach, is going to be better in the long run and more share-able
in the community.

+ 1 gazillion to getting good metrics!

"Sandy Walsh"  said:

> (resent to list as I realized I just did a Reply)
> 
> Cool! This is great stuff. Look forward to seeing the branch.
> 
> I started working on a similar tool that takes the data collected from
> Tach and fetches the data from Graphite to look at the performance
> issues (no changes to nova trunk requires since Tach is awesome).
> 
> It's a shell of an idea yet, but the basics work:
> https://github.com/ohthree/novaprof
> 
> But if there is something already existing, I'm happy to kill it off.
> 
> I don't doubt for a second the db is the culprit for many of our woes.
> 
> The thing I like about internal caching using established tools is that
> it works for db issues too without having to resort to custom tables.
> SQL query optimization, I'm sure, will go equally far.
> 
> Thanks again for the great feedback ... keep it comin'!
> 
> -S
> 
> 
> On 03/22/2012 11:53 PM, Mark Washenberger wrote:
>> Working on this independently, I created a branch with some simple
>> performance logging around the nova-api, and individually around
>> glance, nova.db, and nova.rpc calls. (Sorry, I only have a local
>> copy and its on a different computer right now, and probably needs
>> a rebase. I will rebase and publish it on GitHub tomorrow.)
>>
>> With this logging, I could get some simple profiling that I found
>> very useful. Here is a GH project with the analysis code as well
>> as some nova-api logs I was using as input.
>>
>> https://github.com/markwash/nova-perflog
>>
>> With these tools, you can get a wall-time profile for individual
>> requests. For example, looking at one server create request (and
>> you can run this directly from the checkout as the logs are saved
>> there):
>>
>> markw@poledra:perflogs$ cat nova-api.vanilla.1.5.10.log | python
>> profile-request.py req-3cc0fe84-e736-4441-a8d6-ef605558f37f
>> keycountavg
>> nova.api.openstack.wsgi.POST   1  0.657
>> nova.db.api.instance_update1  0.191
>> nova.image.show1  0.179
>> nova.db.api.instance_add_security_group1  0.082
>> nova.rpc.cast  1  0.059
>> nova.db.api.instance_get_all_by_filters1  0.034
>> nova.db.api.security_group_get_by_name 2  0.029
>> nova.db.api.instance_create1  0.011
>> nova.db.api.quota_get_all_by_project   3  0.003
>> nova.db.api.instance_data_get_for_project  1  0.003
>>
>> key  count  total
>> nova.api.openstack.wsgi  1  0.657
>> nova.db.api 10  0.388
>> nova.image   1  0.179
>> nova.rpc 1  0.059
>>
>> All times are in seconds. The nova.rpc time is probably high
>> since this was the first call since server restart, so the
>> connection handshake is probably included. This is also probably
>> 1.5 months stale.
>>
>> The conclusion I reached from this profiling is that we just plain
>> overuse the db (and we might do the same in glance). For example,
>> whenever we do updates, we actually re-retrieve the item from the
>> database, update its dictionary, and save it. This is double the
>> cost it needs to be. We also handle updates for data across tables
>> inefficiently, where they could be handled in single database round
>> trip.
>>
>> In particular, in the case of server listings, extensions are just
>> rough on performance. Most extensions hit the database again
>> at least once. This isn't really so bad, but it clearly is an area
>> where we should improve, since these are the most frequent api
>> queries.
>>
>> I just see a ton of specific performance problems that are easier
>> to address one by one, rather than diving into a general (albeit
>> obvious) solution such as caching.
>>
>>
>> "Sandy Walsh"  said:
>>
>>> We're doing tests to find out where the bottlenecks are, caching is the
>>> most obvious solution, but there may be others. Tools like memcache do a
>>> really good job of sharing memory across servers so we don't have to
>>> reinvent the wheel or hit the db at all.
>>>
>>> In addition to looking into caching technologies/approaches we're gluing
>>> together some tools for finding those bottlenecks. Our first step will
>>> be finding them, then squashing them ... however.
>>>
>>> -S
>>>
>>> On 03/22/2012 06:25 PM, Mark Washenberger wrote:
 What problems are caching strategies supposed to solve?

 On the nova compute side, it seems like streamlining db access and
 api-view tables would solve any performance problems caching would
 address, while keeping the stale dat

Re: [Openstack] Caching strategies in Nova ...

[Gabe Westmaas]

On 3/23/12 8:56 AM, "Sandy Walsh"  wrote:

>
>
>On 03/23/2012 09:44 AM, Gabe Westmaas wrote:
>> I'd prefer to just set a different expectation for the user.  Rather
>>than worrying about state change and invalidation, lets just set the
>>expectation that the system as a whole is eventually consistent.  I
>>would love to prevent any cache busting strategies or expectations as
>>well as anything that requires something other than time based data
>>refreshing.  We can all agree, I hope, that there is some level of
>>eventual consistency even without caching in our current system.  The
>>fact is that db updates are not instantaneous with other changes in the
>>system; see snapshotting, instance creation, etc.
>
>I think that's completely valid. The in-process caching schemes are
>really just implementation techniques. The end-result (of view tables vs
>key/value in-memory dicts vs whatever) is the same.
Agreed! As long as the interface doesn't imply one implementation over
another (see below).

>
>
>> What I'd like to see is additional fields included in the API response
>>that how old this particular piece of data is.  This way the consumer
>>can decide if they need to be concerned about the fact that this state
>>hasn't changed, and it allows operators to tune their system to whatever
>>their deployments can handle.  If we are exploring caching, I think that
>>gives us the advantage of not a lot of extra code that worries about
>>invalidation, allowing deployers to not use caching at all if its
>>unneeded, and paves the way for view tables in large deployments which I
>>think is important when we are thinking about this on a large scale.
>
>My fear is clients will simply start to poll the system until new data
>magically appears. An alternative might be, rather than say how old the
>data is, how long until the cache expires?
Definitely a valid concern.  However, I kind of expect that many users
will still poll even if they know they won't get new data until X time.
In addition, I think if we say how old the data is, it still implies too
much knowledge unless we go with a strict caching system.  I'd love for us
to leave the ability for us to update that data asynchronously, and
hopefully really quickly, except in the cases where the system is under
unexpected load.  Basically, if we give them that information, and we miss
it, that¹s a call in to support, not to say they won't call in if it takes
too long to update, of course.

Also, if its hitting a cache or something optimized for GETs, hopefully we
can handle lots of polling by adding more API nodes.

Gabe

>
>
>> 
>> Gabe
>> 
>>> -Original Message-
>>> From: openstack-
>>> bounces+gabe.westmaas=rackspace@lists.launchpad.net
>>> [mailto:openstack-
>>> bounces+gabe.westmaas=rackspace@lists.launchpad.net] On Behalf Of
>>> Sandy Walsh
>>> Sent: Friday, March 23, 2012 7:58 AM
>>> To: Joshua Harlow
>>> Cc: openstack
>>> Subject: Re: [Openstack] Caching strategies in Nova ...
>>>
>>> Was reading up some more on cache invalidation schemes last night. The
>>> best practice approach seems to be using a sequence ID in the key. When
>>> you want to invalidate a large set of keys, just bump the sequence id.
>>>
>>> This could easily be handled with a notifier that listens to instance
>>>state
>>> changes.
>>>
>>> Thoughts?
>>>
>>>
>>> On 03/22/2012 09:28 PM, Joshua Harlow wrote:
 Just from experience.

 They do a great job. But the killer thing about caching is how u do
 the cache invalidation.

 Just caching stuff is easy-peasy, making sure it is invalidated on all
 servers in all conditions, not so easy...

 On 3/22/12 4:26 PM, "Sandy Walsh"  wrote:

 We're doing tests to find out where the bottlenecks are, caching
is the
 most obvious solution, but there may be others. Tools like
memcache do
>>> a
 really good job of sharing memory across servers so we don't have
to
 reinvent the wheel or hit the db at all.

 In addition to looking into caching technologies/approaches we're
gluing
 together some tools for finding those bottlenecks. Our first step
will
 be finding them, then squashing them ... however.

 -S

 On 03/22/2012 06:25 PM, Mark Washenberger wrote:
 > What problems are caching strategies supposed to solve?
 >
 > On the nova compute side, it seems like streamlining db access
and
 > api-view tables would solve any performance problems caching
would
 > address, while keeping the stale data management problem small.
 >

 ___
 Mailing list: https://launchpad.net/~openstack
 Post to : openstack@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~openstack
 More help   : https://help.launchpad.net/ListHelp

>>>
>>> 

Re: [Openstack] Enabling remote participation at Folsom Design Summit

[Deepak Garg]

Sorry,
a. can't invest office hrs.
b. can't make BT call

Stefano, are we going to have video recordings ?

Thanks,
Deepak


On Fri, Mar 23, 2012 at 9:54 AM, hitesh wadekar
 wrote:
> Hello,
>
> As we are located in India, Its very hard for us to come folosom design
> summit, but we don;t want to miss this opportunity if you are going to
> provide us as a online conference.
>
> Many big firms from US use to take a call from us on webex, but that will be
> good for 2 to 3 hrs. and I am expecting our design summit will be going
>  through a day.
>
> Can we get some free calling bridge number from big firm along with webex?
> so that, we can easily connect that conference bridge whenever required,
> also it will be easy to ask questions, suggestions etc..(Summit moderator
> can take this initiative...)
>
> Looking forward some solutions for this, so that we won't miss this
> opportunity  !!!
>
> Deepak, What is your openion on this?
>
> Thanks,
> Hitesh Wadekar
>
> On Fri, Mar 23, 2012 at 4:38 AM, Stefano Maffulli 
> wrote:
>>
>> On Wed, 2012-03-21 at 15:47 -0700, Debo Dutta (dedutta) wrote:
>> > SW side: Another option would be to run webex meetings or even google
>> > hangout!
>>
>> On the software side we have *plenty* of options. Webex, Justin.tv,
>> ustream, g+, the great system that powered live streaming from pycon
>> http://streamti.me/ ... lots of options, free as in freedom and free as
>> in beer. Software is not a problem.
>>
>> Hardware is the first thing we need to procure before this project takes
>> off.
>>
>> I started talking with a hardware company in the bay area, friendly to
>> Linux. Hopefully we'll be able to pull it off. I'd love to have
>> alternatives though, should this deal not close.
>>
>> Thanks,
>> stef
>>
>>
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>
>



-- 

Deepak Garg,
Data Center and Cloud Div.
Citrix R&D, India
Skype-id: deepakgarg.iit

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Keystone auth_token confusion in Swift vs Glance; Persistent tokens or not ?

[Florian Daniel Otel]

Thanks Jay for the in-depth explanation, it actually does make sense :)

Sorry for the late reply -- I had to to change my Keystone, Swift and
Glance setup and digest the changes. Still not completely out of the woods
yet

Thanks Chmouel for the the patch [1]  for the Glance and Swift config
files. It's pretty much what I had in mind as well. However, shouldn't
"keystone.conf" itself be changed accordingly  ? I can take a stab at it
and submit a proposal if it helps...

Also, wrt to the docs [2]. AFAICT the patch covers only
"doc/sources/configuringservices.rst" , but it needs fixing in other places
too (e.g. "doc/sources/middleware_architecture.rst").  Same thing there --
I can try to make a patch and submit it (I need to get the hang of it
first...)

Regards,

Florian

[1]
https://github.com/openstack-dev/devstack/commit/bd07d61c4545c52d39b9c957ff9e4423525ca452
[2] https://review.openstack.org/#change,5190



On Tue, Mar 20, 2012 at 10:09 PM, Jay Pipes  wrote:

> Hi Florian, I appreciate your post and sympathize with your
> frustration/confusion. I'll do my best below to help un-confuse :)
>
>
> On 03/20/2012 12:15 PM, Florian Daniel Otel wrote:
>
>> Hello all,
>>
>> I need some help with inconsistencies and -- in my mind -- confusing
>> instructions wrt "auth_token" Keytone middleware.
>>
>> So far I have Keystone and Swift  w/ Keystone auth working well together
>> (ussing Essex-4 milestone release  of Keystone resp v1.4.6 of Swift).
>>
>> What I am now  trying is to get  Glance on Swift with Keystone Auth  but
>> I'm faced with conflicting info that I cannot make sense of
>>
>> 1) Naming inconsistencies -- "token_auth" , "tokenauth" (Keystone resp
>> Swift) vs "authtoken"  (Glance)
>>
>> The existing Keystone and Swift docs (e.g. this one:
>> http://keystone.openstack.org/**configuringservices.html#**
>> configuring-swift-to-use-**keystone*
>> )
>> *use as names "token_auth" (Keystone) resp "tokenauth" (Swift)  wheareas
>>
>> in Glance docs the same midldleware (i.e.
>> "keystone/middleware/auh_**token.py") is referred to as "authtoken"  (see
>> e.g.  here 
>> http://glance.openstack.org/**authentication.html)
>>
>> While this may be only pedantic IMHO it would help if things would be
>> called the same way in different places.  Or, if I'm confusing things,
>> someone point me out the differences to which is what and why.
>>
>
> Completely agreed, and Chmouel has submitted two patches to address these
> inconsistencies.
>
>
>  2) "auth_token":  Persistent tokens or user/pass ?
>>
>> As per Jay Pipes comment here:
>> https://bugs.launchpad.net/**glance/+bug/953989/comments/2
>>  (from
>> 2012-03-13)  the concept of long-lived (i.e. persistent) tokens are no
>> longer supported by the Keystone "auth_token" middleware  -- and that is
>> listed as why that bug was invalid.
>>
>
> Actually, no not quite. The bug was marked Invalid because the source of
> the issue was three things that Kevin (the bug reporter) had failed to do
> in his setup -- two of which had to do with the authentication and one
> having to do with an incorrect endpoint template placeholder.
>
> Anyway, let me try and explain what the deal is with the "long-lived token
> deprecation" that is mentioned in the bug report.
>
> Previously in Keystone there was the concept of a "service token" --
> otherwise known as a long-lived token. These special tokens would be read
> from a configuration file or paste-ini file by the auth_token middleware
> that sat, for instance, in the Glance API server pipeline. Importantly,
> these special tokens were **not required to be attached to any user or
> tenant**.
>
> The auth_token middleware is responsible for authenticating -- via
> Keystone -- incoming regular user requests on an API endpoint. For example,
> if I call GET 
> http://example.com:9292/v1/**imagesto get 
> a list of images from a Glance API server, and that server was
> configured with the auth_token middleware, then the middleware looks for
> certain HTTP headers in the incoming request.
>
> If the HTTP_X_AUTH_TOKEN header is present in the incoming request,
> then the middleware would need to *validate* this authentication token
> with Keystone. This validation call (POST /v2.0/tokens/{$token_id} [1]) is
> a *privileged* call in the Keystone Service Admin API, and as such, in
> order to make the validation request, the HTTP request to Keystone itself
> needs, itself, to supply an HTTP_X_AUTH_TOKEN that is "scoped" to some
> entity that can make this privileged validation call. However, as mentioned
> before, the long-lived "service token" was not scoped to any user/tenant,
> and these "unscoped" tokens were accepted by the validate token API call as
> privileged tokens...
>
> OK, so recent changes remove

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

On 03/23/2012 09:44 AM, Gabe Westmaas wrote:
> I'd prefer to just set a different expectation for the user.  Rather than 
> worrying about state change and invalidation, lets just set the expectation 
> that the system as a whole is eventually consistent.  I would love to prevent 
> any cache busting strategies or expectations as well as anything that 
> requires something other than time based data refreshing.  We can all agree, 
> I hope, that there is some level of eventual consistency even without caching 
> in our current system.  The fact is that db updates are not instantaneous 
> with other changes in the system; see snapshotting, instance creation, etc.  

I think that's completely valid. The in-process caching schemes are
really just implementation techniques. The end-result (of view tables vs
key/value in-memory dicts vs whatever) is the same.


> What I'd like to see is additional fields included in the API response that 
> how old this particular piece of data is.  This way the consumer can decide 
> if they need to be concerned about the fact that this state hasn't changed, 
> and it allows operators to tune their system to whatever their deployments 
> can handle.  If we are exploring caching, I think that gives us the advantage 
> of not a lot of extra code that worries about invalidation, allowing 
> deployers to not use caching at all if its unneeded, and paves the way for 
> view tables in large deployments which I think is important when we are 
> thinking about this on a large scale.

My fear is clients will simply start to poll the system until new data
magically appears. An alternative might be, rather than say how old the
data is, how long until the cache expires?


> 
> Gabe
> 
>> -Original Message-
>> From: openstack-
>> bounces+gabe.westmaas=rackspace@lists.launchpad.net
>> [mailto:openstack-
>> bounces+gabe.westmaas=rackspace@lists.launchpad.net] On Behalf Of
>> Sandy Walsh
>> Sent: Friday, March 23, 2012 7:58 AM
>> To: Joshua Harlow
>> Cc: openstack
>> Subject: Re: [Openstack] Caching strategies in Nova ...
>>
>> Was reading up some more on cache invalidation schemes last night. The
>> best practice approach seems to be using a sequence ID in the key. When
>> you want to invalidate a large set of keys, just bump the sequence id.
>>
>> This could easily be handled with a notifier that listens to instance state
>> changes.
>>
>> Thoughts?
>>
>>
>> On 03/22/2012 09:28 PM, Joshua Harlow wrote:
>>> Just from experience.
>>>
>>> They do a great job. But the killer thing about caching is how u do
>>> the cache invalidation.
>>>
>>> Just caching stuff is easy-peasy, making sure it is invalidated on all
>>> servers in all conditions, not so easy...
>>>
>>> On 3/22/12 4:26 PM, "Sandy Walsh"  wrote:
>>>
>>> We're doing tests to find out where the bottlenecks are, caching is the
>>> most obvious solution, but there may be others. Tools like memcache do
>> a
>>> really good job of sharing memory across servers so we don't have to
>>> reinvent the wheel or hit the db at all.
>>>
>>> In addition to looking into caching technologies/approaches we're gluing
>>> together some tools for finding those bottlenecks. Our first step will
>>> be finding them, then squashing them ... however.
>>>
>>> -S
>>>
>>> On 03/22/2012 06:25 PM, Mark Washenberger wrote:
>>> > What problems are caching strategies supposed to solve?
>>> >
>>> > On the nova compute side, it seems like streamlining db access and
>>> > api-view tables would solve any performance problems caching would
>>> > address, while keeping the stale data management problem small.
>>> >
>>>
>>> ___
>>> Mailing list: https://launchpad.net/~openstack
>>> Post to : openstack@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~openstack
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Gabe Westmaas]

I'd prefer to just set a different expectation for the user.  Rather than 
worrying about state change and invalidation, lets just set the expectation 
that the system as a whole is eventually consistent.  I would love to prevent 
any cache busting strategies or expectations as well as anything that requires 
something other than time based data refreshing.  We can all agree, I hope, 
that there is some level of eventual consistency even without caching in our 
current system.  The fact is that db updates are not instantaneous with other 
changes in the system; see snapshotting, instance creation, etc.  

What I'd like to see is additional fields included in the API response that how 
old this particular piece of data is.  This way the consumer can decide if they 
need to be concerned about the fact that this state hasn't changed, and it 
allows operators to tune their system to whatever their deployments can handle. 
 If we are exploring caching, I think that gives us the advantage of not a lot 
of extra code that worries about invalidation, allowing deployers to not use 
caching at all if its unneeded, and paves the way for view tables in large 
deployments which I think is important when we are thinking about this on a 
large scale.

Gabe

> -Original Message-
> From: openstack-
> bounces+gabe.westmaas=rackspace@lists.launchpad.net
> [mailto:openstack-
> bounces+gabe.westmaas=rackspace@lists.launchpad.net] On Behalf Of
> Sandy Walsh
> Sent: Friday, March 23, 2012 7:58 AM
> To: Joshua Harlow
> Cc: openstack
> Subject: Re: [Openstack] Caching strategies in Nova ...
> 
> Was reading up some more on cache invalidation schemes last night. The
> best practice approach seems to be using a sequence ID in the key. When
> you want to invalidate a large set of keys, just bump the sequence id.
> 
> This could easily be handled with a notifier that listens to instance state
> changes.
> 
> Thoughts?
> 
> 
> On 03/22/2012 09:28 PM, Joshua Harlow wrote:
> > Just from experience.
> >
> > They do a great job. But the killer thing about caching is how u do
> > the cache invalidation.
> >
> > Just caching stuff is easy-peasy, making sure it is invalidated on all
> > servers in all conditions, not so easy...
> >
> > On 3/22/12 4:26 PM, "Sandy Walsh"  wrote:
> >
> > We're doing tests to find out where the bottlenecks are, caching is the
> > most obvious solution, but there may be others. Tools like memcache do
> a
> > really good job of sharing memory across servers so we don't have to
> > reinvent the wheel or hit the db at all.
> >
> > In addition to looking into caching technologies/approaches we're gluing
> > together some tools for finding those bottlenecks. Our first step will
> > be finding them, then squashing them ... however.
> >
> > -S
> >
> > On 03/22/2012 06:25 PM, Mark Washenberger wrote:
> > > What problems are caching strategies supposed to solve?
> > >
> > > On the nova compute side, it seems like streamlining db access and
> > > api-view tables would solve any performance problems caching would
> > > address, while keeping the stale data management problem small.
> > >
> >
> > ___
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> >
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

Was reading up some more on cache invalidation schemes last night. The
best practice approach seems to be using a sequence ID in the key. When
you want to invalidate a large set of keys, just bump the sequence id.

This could easily be handled with a notifier that listens to instance
state changes.

Thoughts?


On 03/22/2012 09:28 PM, Joshua Harlow wrote:
> Just from experience.
> 
> They do a great job. But the killer thing about caching is how u do the
> cache invalidation.
> 
> Just caching stuff is easy-peasy, making sure it is invalidated on all
> servers in all conditions, not so easy...
> 
> On 3/22/12 4:26 PM, "Sandy Walsh"  wrote:
> 
> We're doing tests to find out where the bottlenecks are, caching is the
> most obvious solution, but there may be others. Tools like memcache do a
> really good job of sharing memory across servers so we don't have to
> reinvent the wheel or hit the db at all.
> 
> In addition to looking into caching technologies/approaches we're gluing
> together some tools for finding those bottlenecks. Our first step will
> be finding them, then squashing them ... however.
> 
> -S
> 
> On 03/22/2012 06:25 PM, Mark Washenberger wrote:
> > What problems are caching strategies supposed to solve?
> >
> > On the nova compute side, it seems like streamlining db access and
> > api-view tables would solve any performance problems caching would
> > address, while keeping the stale data management problem small.
> >
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

(resent to list as I realized I just did a Reply)

Cool! This is great stuff. Look forward to seeing the branch.

I started working on a similar tool that takes the data collected from
Tach and fetches the data from Graphite to look at the performance
issues (no changes to nova trunk requires since Tach is awesome).

It's a shell of an idea yet, but the basics work:
https://github.com/ohthree/novaprof

But if there is something already existing, I'm happy to kill it off.

I don't doubt for a second the db is the culprit for many of our woes.

The thing I like about internal caching using established tools is that
it works for db issues too without having to resort to custom tables.
SQL query optimization, I'm sure, will go equally far.

Thanks again for the great feedback ... keep it comin'!

-S


On 03/22/2012 11:53 PM, Mark Washenberger wrote:
> Working on this independently, I created a branch with some simple
> performance logging around the nova-api, and individually around 
> glance, nova.db, and nova.rpc calls. (Sorry, I only have a local
> copy and its on a different computer right now, and probably needs
> a rebase. I will rebase and publish it on GitHub tomorrow.) 
> 
> With this logging, I could get some simple profiling that I found
> very useful. Here is a GH project with the analysis code as well
> as some nova-api logs I was using as input. 
> 
> https://github.com/markwash/nova-perflog
> 
> With these tools, you can get a wall-time profile for individual
> requests. For example, looking at one server create request (and
> you can run this directly from the checkout as the logs are saved
> there):
> 
> markw@poledra:perflogs$ cat nova-api.vanilla.1.5.10.log | python 
> profile-request.py req-3cc0fe84-e736-4441-a8d6-ef605558f37f
> keycountavg
> nova.api.openstack.wsgi.POST   1  0.657
> nova.db.api.instance_update1  0.191
> nova.image.show1  0.179
> nova.db.api.instance_add_security_group1  0.082
> nova.rpc.cast  1  0.059
> nova.db.api.instance_get_all_by_filters1  0.034
> nova.db.api.security_group_get_by_name 2  0.029
> nova.db.api.instance_create1  0.011
> nova.db.api.quota_get_all_by_project   3  0.003
> nova.db.api.instance_data_get_for_project  1  0.003
> 
> key  count  total
> nova.api.openstack.wsgi  1  0.657
> nova.db.api 10  0.388
> nova.image   1  0.179
> nova.rpc 1  0.059
> 
> All times are in seconds. The nova.rpc time is probably high
> since this was the first call since server restart, so the
> connection handshake is probably included. This is also probably
> 1.5 months stale.
> 
> The conclusion I reached from this profiling is that we just plain
> overuse the db (and we might do the same in glance). For example,
> whenever we do updates, we actually re-retrieve the item from the
> database, update its dictionary, and save it. This is double the
> cost it needs to be. We also handle updates for data across tables
> inefficiently, where they could be handled in single database round
> trip.
> 
> In particular, in the case of server listings, extensions are just
> rough on performance. Most extensions hit the database again
> at least once. This isn't really so bad, but it clearly is an area
> where we should improve, since these are the most frequent api
> queries.
> 
> I just see a ton of specific performance problems that are easier
> to address one by one, rather than diving into a general (albeit
> obvious) solution such as caching.
> 
> 
> "Sandy Walsh"  said:
> 
>> We're doing tests to find out where the bottlenecks are, caching is the
>> most obvious solution, but there may be others. Tools like memcache do a
>> really good job of sharing memory across servers so we don't have to
>> reinvent the wheel or hit the db at all.
>>
>> In addition to looking into caching technologies/approaches we're gluing
>> together some tools for finding those bottlenecks. Our first step will
>> be finding them, then squashing them ... however.
>>
>> -S
>>
>> On 03/22/2012 06:25 PM, Mark Washenberger wrote:
>>> What problems are caching strategies supposed to solve?
>>>
>>> On the nova compute side, it seems like streamlining db access and
>>> api-view tables would solve any performance problems caching would
>>> address, while keeping the stale data management problem small.
>>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
> 
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.ne

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

Yup, makes sense. Thanks for the feedback. I agree that the external
caches are troublesome and we'll likely be focusing on the internal
ones. Whether that manifests itself as a memcache-like implementation or
another db view is unknown.

The other thing about in-process caching I like is the ability to have
it in a common (nova-common?) library where we can easily compute
hit/miss ratios and adjust accordingly.

-S


On 03/23/2012 12:02 AM, Mark Washenberger wrote:
> This is precisely my concern.
> 
> It must be brought up that with Rackspace Cloud Servers, nearly
> all client codes routinely submit requests with a query parameter 
> "cache-busting=" just to get around problems with
> cache invalidation. And woe to the client that does not.
> 
> I get the feeling that once trust like this is lost, a project has
> a hard time regaining it. I'm not saying that we can avoid
> inconsistency entirely. Rather, I believe we will have to embrace
> some eventual-consistency models to enable the performance and
> scale we will ultimately attain. But I just get the feeling that
> generic caches are really only appropriate for write-once or at
> least write-rarely data. So personally I would rule out external
> caches entirely and try to be very judicious in selecting internal
> caches as well.
> 
> "Joshua Harlow"  said:
> 
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>> Just from experience.
>>
>> They do a great job. But the killer thing about caching is how u do the cache
>> invalidation.
>>
>> Just caching stuff is easy-peasy, making sure it is invalidated on all 
>> servers in
>> all conditions, not so easy...
>>
>> On 3/22/12 4:26 PM, "Sandy Walsh"  wrote:
>>
>> We're doing tests to find out where the bottlenecks are, caching is the
>> most obvious solution, but there may be others. Tools like memcache do a
>> really good job of sharing memory across servers so we don't have to
>> reinvent the wheel or hit the db at all.
>>
>> In addition to looking into caching technologies/approaches we're gluing
>> together some tools for finding those bottlenecks. Our first step will
>> be finding them, then squashing them ... however.
>>
>> -S
>>
>> On 03/22/2012 06:25 PM, Mark Washenberger wrote:
>>> What problems are caching strategies supposed to solve?
>>>
>>> On the nova compute side, it seems like streamlining db access and
>>> api-view tables would solve any performance problems caching would
>>> address, while keeping the stale data management problem small.
>>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
> 
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Sandy Walsh]

Thanks ... that's good feedback and we were discussing cache
invalidation issues today.

Any tips or suggestions?

-S



On 03/22/2012 09:28 PM, Joshua Harlow wrote:
> Just from experience.
> 
> They do a great job. But the killer thing about caching is how u do the
> cache invalidation.
> 
> Just caching stuff is easy-peasy, making sure it is invalidated on all
> servers in all conditions, not so easy...
> 
> On 3/22/12 4:26 PM, "Sandy Walsh"  wrote:
> 
> We're doing tests to find out where the bottlenecks are, caching is the
> most obvious solution, but there may be others. Tools like memcache do a
> really good job of sharing memory across servers so we don't have to
> reinvent the wheel or hit the db at all.
> 
> In addition to looking into caching technologies/approaches we're gluing
> together some tools for finding those bottlenecks. Our first step will
> be finding them, then squashing them ... however.
> 
> -S
> 
> On 03/22/2012 06:25 PM, Mark Washenberger wrote:
> > What problems are caching strategies supposed to solve?
> >
> > On the nova compute side, it seems like streamlining db access and
> > api-view tables would solve any performance problems caching would
> > address, while keeping the stale data management problem small.
> >
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Specify an owner when adding an image in Glance

[Eoghan Glynn]

> Done.
> 
> https://bugs.launchpad.net/glance/+bug/962998


Thanks, fixed here:  https://review.openstack.org/5727

Cheers,
Eoghan

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Specify an owner when adding an image in Glance

[Haefliger, Juerg]

Done.

https://bugs.launchpad.net/glance/+bug/962998

Thanks
...Juerg


> -Original Message-
> From: Eoghan Glynn [mailto:egl...@redhat.com]
> Sent: Friday, March 23, 2012 12:16 PM
> To: Haefliger, Juerg
> Cc: openstack@lists.launchpad.net
> Subject: Re: [Openstack] Specify an owner when adding an image in
> Glance
> 
> 
> Hi Juerg,
> 
> That's because 'owner' is not supported as an explicit parameter to
> 'glance add'.
> 
> So as a result the CLI treats it a generic image property, and passes
> this to the API service via the header:
> 
>   x-image-meta-property-owner: 2
> 
> The 'x-image-meta-property-' prefix is used to distinguish additional
> image properties, separate to the first class image attributes.
> 
> Please raise a bug against the glance CLI so that 'owner' is supported
> as an explicit field for the add command (as it currently is for the
> update command).
> 
> Cheers,
> Eoghan
> 
> > Is there a particular reason why an owner can't be specified when
> > adding an image? I.e., the following:
> >
> > $ glance add name=testing owner=99 < testing
> >
> > results in:
> >
> > URI: http://jabba:9292/v1/images/22
> > Id: 22
> > Public: No
> > Name: testing
> > Status: active
> > Size: 36614
> > Disk format: None
> > Container format: None
> > Minimum Ram Required (MB): 0
> > Minimum Disk Required (GB): 0
> > Owner: 2
> > Property 'owner': 99
> >
> > whereas I expect it to be:
> >
> > URI: http://jabba:9292/v1/images/22
> > Id: 22
> > Public: No
> > Name: testing
> > Status: active
> > Size: 36614
> > Disk format: None
> > Container format: None
> > Minimum Ram Required (MB): 0
> > Minimum Disk Required (GB): 0
> > Owner: 99
> >
> >
> > Regards
> > ...Juerg
> >
> >
> > ___
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> >
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Specify an owner when adding an image in Glance

[Eoghan Glynn]

Hi Juerg,

That's because 'owner' is not supported as an explicit parameter to 'glance 
add'.

So as a result the CLI treats it a generic image property, and passes this to 
the
API service via the header:

  x-image-meta-property-owner: 2

The 'x-image-meta-property-' prefix is used to distinguish additional image
properties, separate to the first class image attributes.

Please raise a bug against the glance CLI so that 'owner' is supported as an
explicit field for the add command (as it currently is for the update command).

Cheers,
Eoghan

> Is there a particular reason why an owner can't be specified when
> adding an image? I.e., the following:
> 
> $ glance add name=testing owner=99 < testing
> 
> results in:
> 
> URI: http://jabba:9292/v1/images/22
> Id: 22
> Public: No
> Name: testing
> Status: active
> Size: 36614
> Disk format: None
> Container format: None
> Minimum Ram Required (MB): 0
> Minimum Disk Required (GB): 0
> Owner: 2
> Property 'owner': 99
> 
> whereas I expect it to be:
> 
> URI: http://jabba:9292/v1/images/22
> Id: 22
> Public: No
> Name: testing
> Status: active
> Size: 36614
> Disk format: None
> Container format: None
> Minimum Ram Required (MB): 0
> Minimum Disk Required (GB): 0
> Owner: 99
> 
> 
> Regards
> ...Juerg
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Specify an owner when adding an image in Glance

[Haefliger, Juerg]

Hi,

Is there a particular reason why an owner can't be specified when adding an 
image? I.e., the following:

$ glance add name=testing owner=99 < testing

results in:

URI: http://jabba:9292/v1/images/22
Id: 22
Public: No
Name: testing
Status: active
Size: 36614
Disk format: None
Container format: None
Minimum Ram Required (MB): 0
Minimum Disk Required (GB): 0
Owner: 2
Property 'owner': 99

whereas I expect it to be:

URI: http://jabba:9292/v1/images/22
Id: 22
Public: No
Name: testing
Status: active
Size: 36614
Disk format: None
Container format: None
Minimum Ram Required (MB): 0
Minimum Disk Required (GB): 0
Owner: 99


Regards
...Juerg


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Network Connectivity issue

[Nicolas Odermatt]

Thank you very much for your answer!

There was indeed a missing route. I added the following routes:

Client: sudo route add -net 192.168.163.128 netmask 255.255.255.128 gw
192.168.163.20 dev eth2

Instanz:sudo route add -net 192.168.163.0 netmask 255.255.255.128 gw
10.0.0.1 dev eth0

And after that it worked like a charm ;)

2012/3/22 Diego Parrilla Santamaría 

> Hi Nicolas, thanks for using our distro!
>
> Just to summarize:
> - Single node
> - Management network: 192.168.163.0/25 = 192.168.163.1 to
> 192.168.163.126. Let's assume gateway = 192.168.163.1
> - Service Network: 192.168.163.128/25 = 192.168.163.129 to
> 192.168.163.254. gateway should be = 192.168.163.129
>
> If you wan to ping from the service network to a host in the management
> network (and it's not the single node) you need a route to that network. Or
> you can configure your system to use the Management Network as the public
> network. Then you will reach your hosts.
>
> If you are using FlatDHCP, may be you can configure the smart installer to
> use as the service network a subset (not a subnet) of the management
> network. It's a dirty hack (I prefer to split networks always) we did in
> the past...
>
> For example if you want to setup a little lab in your lapto with the
> distro you can use this settings:
> - Single node
> - Management network: 192.168.10.0/24 (this is my network, use yours)
> - Service Network: 10.0.0.0/24
> - Public Network (floating range): 192.168.10.64/28 (and of course I
> don't have any host in this subnet)
>
> With this configuration floating IPs will reach hosts in the management
> network.
>
> Cheers!
> Diego
>
>
> --
> Diego Parrilla
> *CEO*
> *www.stackops.com | * diego.parri...@stackops.com** | +34 649 94 43 29 |
> skype:diegoparrilla*
> * 
> **
>
>
>
> On Thu, Mar 22, 2012 at 3:47 PM, Nicolas Odermatt wrote:
>
>> Hello guys, I am dealing with a strange phenomenon my StackOps
>> environment.
>>
>> The thing is like this:
>> I am running a single-node stackops deployment in a little /25 network.
>> The installation and the few configurations I had to do afterwards went
>> just fine, no problems so far. I was also able to upload an image to glance
>> and to start an instance using euca2ools. I authorized the necessary port
>> for ssh, allowed icmp and then connected to the instance with ssh.
>> Afterwards I checked the instance's connection to the internet, which was
>> up and running. Now to my so called "strange phenomenon": Although I am
>> able to ping my stackops-node, public ip adresses (e.g 8.8.8.8) and in
>> spite the fact that name resolving works as well, I can't ping a Client
>> machine in my local area network. My stackops-node has the ip-address
>> 192.168.163.20/25, the instance has the ip-address(192.168.163.130/25)
>> and the client has the ip-address 192.168.163.5/25.
>>
>> I can't think of an approach to this problem, as I have enabled ssh and
>> icmp and the connection between Client and stackops-node exists.
>> Is it possible that I might have forgott to add a certain rule?
>>
>> Thanks for your ideas!
>>
>> Best regards.
>> Nicolas
>> --
>> Freundliche Grüsse,
>> Nicolas Odermatt
>>
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>


-- 
Freundliche Grüsse,
Nicolas Odermatt
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp