Re: [Openstack] ZFS/ZVol + iscsi for volume

[Justin Santa Barbara]

As Diego pointed out, this should all work already.  You just point your
nova-volume at your Solaris-like box, and it runs all the commands for you
(over SSH).  I wrote the original way-back-when as a stepping-stone to
support for HP SANs (as I had much easier access to Solaris than real
SANs), and it sounds like Diego fixed it up in Diablo, so it hasn't totally
bit-rotted.

Have you tried this?  Did you encounter any problems?  Is there something
extra you need?

Justin





On Mon, Jun 11, 2012 at 12:06 PM, Nicolas de BONFILS <
openst...@nicolas2bonfils.com> wrote:

> **
>
> In fact, my question is more about how to integrate it within nova
> workflow. I know how do each step (zfs pool, iscsi export, iscsi attach in
> vm) but this way is not register by nova db because it's not nova commands.
> I was looking for people who have replace the LVM creation and iscsiadm
> export part with zfs (pool+export) .
>
> --- Nicolas
>
> On 11-06-2012 17:05, Alberto Molina Coballes wrote:
>
> What about Debian/KFreeBSD for nova-volume?
>
> Debian testing (wheezy) with FreeBSD kernel provides support for both
> OpenStack (nova-volume) and ZFS. We are deploying a little cloud for
> testing purposes and it will probably be one of the alternatives we will
> test for storage.
>
> Cheers
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Encrypted virtual machines

[Justin Santa Barbara]

I think one of us is misunderstanding the model.  My understanding is that
we produce software that we trust, and then prove to the caller that we're
running that software.  All optimizations remain possible.

Check out section 6.1 of the paper!


On Thu, Apr 26, 2012 at 3:24 PM, Matt Joyce  wrote:

> Functionally if the scheduler doesn't know what it's passing to the
> CPU or into paging memory a lot of optimization possibilities go out
> the window.  If it does know one can infer a great deal about your
> datasets protected or not.
>
> -Matt
>
> On Thu, Apr 26, 2012 at 3:08 PM, Justin Santa Barbara
>  wrote:
> > I think that Intel's trusted cloud work is trying to solve that exact
> > compute host problem.  It may already have the framework to do so even if
> > the software hasn't caught up (i.e. if we still have some work to do!)
> >
> > It relies on a TPM chip, all code is measured before being run, and then
> > there's a protocol to prove that a system is running that code (remote
> > attestation).  If you change the software stack by introducing a sniffer,
> > you change the hash.  So we'd need a stack with no root-access /
> back-doors.
> >  Once a back-door becomes known, the hash should no longer be trusted.
> >
> > I'm by no means an expert (I'm still learning about it), but I believe
> it is
> > possible, having read this
> > paper:
> http://www.research.ibm.com/trl/projects/watc/FredericStumpfPaper.pdf
> >
> > I'm sure there are still exploits (hardware RAM taps?), and we rely on a
> > total code audit, but we can raise the bar a long way.
> >
> > Anyone from Intel / familiar with Intel's trusted cloud work want to
> explain
> > better than I can?
> >
> > Justin
> >
> >
> >
> >
> > On Thu, Apr 26, 2012 at 1:44 PM, Matt Joyce 
> wrote:
> >>
> >> As far as storage is concerned, certainly a cloud storage environment
> >> could be leveraged to store pre-encrypted data in such a way that
> >> would make it difficult bordering on impossible to seize or access
> >> without the consent of the owner.
> >>
> >> As far as compute hosts are concerned, it is a whole different matter.
> >>
> >> For the foreseeable future ( barring the invention of new widely
> >> distributed in CPU technology ) .  Anyone with ring 0 execution access
> >> on a system ( ie root / sudo ) will be able to pull data from a
> >> running instance pretty much no matter what you do.
> >>
> >> You can certainly raise the bar on difficulty there, but the
> >> fundamental path of sniffing schedulers / paging memory / etc will be
> >> there for a fairly long time.  Even trusted computing wouldn't be
> >> applicable to protecting a vm's scheduler from the hypervisors owner.
> >>
> >> So, I think functionally it should be assumed that a provider will be
> >> able to access anything that you access on a hosted VM.  As far as a
> >> trust relationship goes in elastic computing, there must be an
> >> implicit trust of the cloud provider.  And as with any trust
> >> relationship there is always going to be an element of risk.
> >>
> >> -Matt
> >>
> >> On Thu, Apr 26, 2012 at 9:53 AM, Daniel P. Berrange <
> berra...@redhat.com>
> >> wrote:
> >> > On Thu, Apr 26, 2012 at 09:05:41AM -0700, Matt Joyce wrote:
> >> >> From a security stand point I am curious what you see the benefit as?
> >> >
> >> > Consider that you might have separate people in your data center
> >> > managing the virtualization hosts, vs the storage hosts vs the
> >> > network. As it standards today any of those groups of people can
> >> > compromise data stored in a VM disk image (assuming a network based
> >> > filesystem).
> >> >
> >> > First you encrypt the disk image, so that a person with access
> >> > to the storage hosts, or network sniffing can't read any data. Then
> >> > you have a central key server that only gives out the decryption key
> >> > to Nova compute nodes when they have been explicitly authorized to
> >> > run an instance of that VM.
> >> >
> >> > So now people with access to the storage hosts cannot compromise
> >> > any data. People with access to the virtualization hosts can only
> >> > compromise data if the host has been authorized to use that disk
> >> > image
> >> >
> >> > You 

Re: [Openstack] Encrypted virtual machines

[Justin Santa Barbara]

I think that Intel's trusted cloud work is trying to solve that exact
compute host problem.  It may already have the framework to do so even if
the software hasn't caught up (i.e. if we still have some work to do!)

It relies on a TPM chip, all code is measured before being run, and then
there's a protocol to prove that a system is running that code (remote
attestation).  If you change the software stack by introducing a sniffer,
you change the hash.  So we'd need a stack with no root-access /
back-doors.  Once a back-door becomes known, the hash should no longer be
trusted.

I'm by no means an expert (I'm still learning about it), but I believe it
is possible, having read this paper:
http://www.research.ibm.com/trl/projects/watc/FredericStumpfPaper.pdf

I'm sure there are still exploits (hardware RAM taps?), and we rely on a
total code audit, but we can raise the bar a long way.

Anyone from Intel / familiar with Intel's trusted cloud work want to
explain better than I can?

Justin




On Thu, Apr 26, 2012 at 1:44 PM, Matt Joyce  wrote:

> As far as storage is concerned, certainly a cloud storage environment
> could be leveraged to store pre-encrypted data in such a way that
> would make it difficult bordering on impossible to seize or access
> without the consent of the owner.
>
> As far as compute hosts are concerned, it is a whole different matter.
>
> For the foreseeable future ( barring the invention of new widely
> distributed in CPU technology ) .  Anyone with ring 0 execution access
> on a system ( ie root / sudo ) will be able to pull data from a
> running instance pretty much no matter what you do.
>
> You can certainly raise the bar on difficulty there, but the
> fundamental path of sniffing schedulers / paging memory / etc will be
> there for a fairly long time.  Even trusted computing wouldn't be
> applicable to protecting a vm's scheduler from the hypervisors owner.
>
> So, I think functionally it should be assumed that a provider will be
> able to access anything that you access on a hosted VM.  As far as a
> trust relationship goes in elastic computing, there must be an
> implicit trust of the cloud provider.  And as with any trust
> relationship there is always going to be an element of risk.
>
> -Matt
>
> On Thu, Apr 26, 2012 at 9:53 AM, Daniel P. Berrange 
> wrote:
> > On Thu, Apr 26, 2012 at 09:05:41AM -0700, Matt Joyce wrote:
> >> From a security stand point I am curious what you see the benefit as?
> >
> > Consider that you might have separate people in your data center
> > managing the virtualization hosts, vs the storage hosts vs the
> > network. As it standards today any of those groups of people can
> > compromise data stored in a VM disk image (assuming a network based
> > filesystem).
> >
> > First you encrypt the disk image, so that a person with access
> > to the storage hosts, or network sniffing can't read any data. Then
> > you have a central key server that only gives out the decryption key
> > to Nova compute nodes when they have been explicitly authorized to
> > run an instance of that VM.
> >
> > So now people with access to the storage hosts cannot compromise
> > any data. People with access to the virtualization hosts can only
> > compromise data if the host has been authorized to use that disk
> > image
> >
> > You would need to compromise the precise host the VM disk is being
> > used on, or compromise the key server or the management service
> > that schedules VMs (thus authorizing key usage on a node).
> >
> > NB this is better than relying on the guest OS to do encryption,
> > since you can do stricter decryption key management from the
> > host side.
> >
> >> On Thu, Apr 26, 2012 at 8:53 AM, Michael Grosser <
> d...@seetheprogress.net> wrote:
> >> > Hey,
> >> >
> >> > I'm following the openstack development for some time now and I was
> >> > wondering if there was a solution to spin up encrypted virtual
> machines by
> >> > default and if it would be a huge performance blow.
> >> >
> >> > Any ideas?
> >
> > I would like to extend the libvirt driver in Nova to make use of the
> qcow2
> > encryption capabilities between libvirt & QEMU which I describe here:
> >
> >
> http://berrange.com/posts/2009/12/02/using-qcow2-disk-encryption-with-libvirt-in-fedora-12/
> >
> > Regards,
> > Daniel
> > --
> > |: http://berrange.com  -o-
> http://www.flickr.com/photos/dberrange/ :|
> > |: http://libvirt.org  -o-
> http://virt-manager.org :|
> > |: http://autobuild.org   -o-
> http://search.cpan.org/~danberr/ :|
> > |: http://entangle-photo.org   -o-
> http://live.gnome.org/gtk-vnc :|
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.la

Re: [Openstack] Encrypted virtual machines

[Justin Santa Barbara]

On Thu, Apr 26, 2012 at 9:05 AM, Matt Joyce  wrote:

> >From a security stand point I am curious what you see the benefit as?


I think that long-term there is the potential to have a cloud where you
don't have to trust the cloud provider (e.g. Intel Trusted Compute).
 However, there are a huge number of steps that need to happen first, so I
don't know that encrypting the qcow disk image would get you very much
today.

However, you could encrypt your filesystem (inside the disk image), and
have it prompt for a password on boot.  Then you could go in via VNC
(today) and unlock your disk image.

Your cloud provider can still grab memory etc.  But I think that's the best
you can do today.  One day we may be able to automate something similar,
yet still have it be secure.

Virtualized I/O performance is poor compared to CPU performance, so I guess
you wouldn't even notice the hit!  But this is pure speculation,


A little plug - one of the pieces of the big picture is figuring out how to
store secrets; at the design summit I proposed storing them securely in
Keystone; I just wrote up the (first draft?) of the blueprint:
https://blueprints.launchpad.net/nova/+spec/secure-secret-storage

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] How does everyone build OpenStack disk images?

[Justin Santa Barbara]

How does everyone build OpenStack disk images?  The official documentation
describes a manual process (boot VM with ISO), which is sub-optimal in
terms of repeatability / automation / etc.  I'm hoping we can do better!

I posted how I do it on my blog, here:
http://blog.justinsb.com/blog/2012/04/25/creating-an-openstack-image/

Please let me know the many ways in which I'm doing it wrong :-)

I'm thinking we can have a discussion here, and then I can then compile the
responses into a wiki page and/or a nice script...

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [OpenStack][Nova] Minimum required code coverage per file

[Justin Santa Barbara]

One concern I have is this: suppose we find that a code block is
unnecessary, or can be refactored more compactly, but it has test coverage.
 Then removing it would make the % coverage fall.

We want to remove the code, but we'd have to add unrelated tests to the
same merge because otherwise the test coverage % would fall?

I think we can certainly enhance the metrics, but I do have concerns over
strict gating (particularly per file, where the problem is more likely to
occur than per-project)

Maybe the gate could be that line count of uncovered lines must not
increase, unless the new % coverage > 80%.

Or we could simply have a gate bypass.

Justin

On Wed, Apr 25, 2012 at 2:45 PM, Monty Taylor  wrote:

> Hey - funny story - in responding to Justin I re-read the original email
> and realized it was asking for a static low number, which we _can_ do -
> at least project-wide. We can't do per-file yet, nor can we fail on a
> downward inflection... and I've emailed Justin about that.
>
> If we have consensus on gating on project-wide threshold, I can
> certainly add adding that to the gate to the todo list. (If we decide to
> do that, I'd really like to make that be openstack-wide rather than just
> nova... although I imagine it might take a few weeks to come to
> consensus on what the project-wide low number should be.
>
> Current numbers on project-wide lines numbers:
>
> nova: 79%
> glance: 75%
> keystone: 81%
> swift: 80%
> horizon: 91%
>
> Perhaps we get nova and glance up to 80 and then set the threshold for 80?
>
> Also, turns out we're not running this on the client libs...
>
> Monty
>
> On 04/25/2012 03:53 PM, Justin Santa Barbara wrote:
> > If you let me know in a bit more detail what you're looking for, I can
> > probably whip something up.  Email me direct?
> >
> > Justin
> >
> >
> > On Wed, Apr 25, 2012 at 6:59 AM, Monty Taylor  > <mailto:mord...@inaugust.com>> wrote:
> >
> >
> >
> > On 04/24/2012 10:08 PM, Lorin Hochstein wrote:
> > >
> > > On Apr 24, 2012, at 4:11 PM, Joe Gordon wrote:
> > >
> > >> Hi All,
> > >>
> > >> I would like to propose a minimum required code coverage level per
> > >> file in Nova.  Say 80%.  This would mean that any new feature/file
> > >> should only be accepted if it has over 80% code coverage.
>  Exceptions
> > >> to this rule would be allowed for code that is covered by skipped
> > >> tests (as long as 80% is reached when the tests are not skipped).
> > >>
> > >
> > > I like the idea of looking at code coverage numbers. For any
> > particular
> > > merge proposal, I'd also like to know whether it increases or
> > decreases
> > > the overall code coverage of the project. I don't think we should
> gate
> > > on this, but it would be helpful for a reviewer to see that,
> > especially
> > > for larger proposals.
> >
> > Yup... Nati requested this a couple of summits ago - main issue is
> that
> > while we run code coverage and use the jenkins code coverage plugin
> to
> > track the coverage numbers, the plugin doesn't fully support this
> > particular kind of report.
> >
> > HOWEVER - if any of our fine java friends out there want to chat
> with me
> > about adding support to the jenkins code coverage plugin to track and
> > report this, I will be thrilled to put it in as a piece of reported
> > information.
> >
> > >> With 193 python files in nova/tests, Nova unit tests produce 85%
> > >> overall code coverage (calculated with ./run_test.sh -c [1]).
> >  But 23%
> > >> of files (125 files) have lower then 80% code coverage (30 tests
> > >> skipped on my machine).  Getting all files to hit the 80% code
> > >> coverage mark should be one of the goals for Folsom.
> > >>
> > >
> > > I would really like to see a visualization of the code coverage
> > > distribution, in order to help spot the outliers.
> > >
> > >
> > > Along these lines, there's been a lot of work in the software
> > > engineering research community about predicting which parts of the
> > code
> > > are most likely to contain bugs ("fault prone" is a good keyword
> > to find
> > > this stuff, e.g.: http://scholar.google.com/scholar?q=fault+prone,
> big
&g

Re: [Openstack] [OpenStack][Nova] Minimum required code coverage per file

[Justin Santa Barbara]

If you let me know in a bit more detail what you're looking for, I can
probably whip something up.  Email me direct?

Justin


On Wed, Apr 25, 2012 at 6:59 AM, Monty Taylor  wrote:

>
>
> On 04/24/2012 10:08 PM, Lorin Hochstein wrote:
> >
> > On Apr 24, 2012, at 4:11 PM, Joe Gordon wrote:
> >
> >> Hi All,
> >>
> >> I would like to propose a minimum required code coverage level per
> >> file in Nova.  Say 80%.  This would mean that any new feature/file
> >> should only be accepted if it has over 80% code coverage.  Exceptions
> >> to this rule would be allowed for code that is covered by skipped
> >> tests (as long as 80% is reached when the tests are not skipped).
> >>
> >
> > I like the idea of looking at code coverage numbers. For any particular
> > merge proposal, I'd also like to know whether it increases or decreases
> > the overall code coverage of the project. I don't think we should gate
> > on this, but it would be helpful for a reviewer to see that, especially
> > for larger proposals.
>
> Yup... Nati requested this a couple of summits ago - main issue is that
> while we run code coverage and use the jenkins code coverage plugin to
> track the coverage numbers, the plugin doesn't fully support this
> particular kind of report.
>
> HOWEVER - if any of our fine java friends out there want to chat with me
> about adding support to the jenkins code coverage plugin to track and
> report this, I will be thrilled to put it in as a piece of reported
> information.
>
> >> With 193 python files in nova/tests, Nova unit tests produce 85%
> >> overall code coverage (calculated with ./run_test.sh -c [1]).  But 23%
> >> of files (125 files) have lower then 80% code coverage (30 tests
> >> skipped on my machine).  Getting all files to hit the 80% code
> >> coverage mark should be one of the goals for Folsom.
> >>
> >
> > I would really like to see a visualization of the code coverage
> > distribution, in order to help spot the outliers.
> >
> >
> > Along these lines, there's been a lot of work in the software
> > engineering research community about predicting which parts of the code
> > are most likely to contain bugs ("fault prone" is a good keyword to find
> > this stuff, e.g.: http://scholar.google.com/scholar?q=fault+prone, big
> > names include Nachi Nagappan at MS Research and Elaine Weyuker, formerly
> > of AT&T Research). I would *love* to see some academic researchers try
> > to apply those techniques to OpenStack to help guide QA activities by
> > identifying which parts of the code should get more rigorous  testing
> > and review.
>
> ++
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] File injection and disk resize not working

[Justin Santa Barbara]

I believe config-drive is the way to fix injection problems.  I use it and
it works great.  (I just wish there was a way to use it from Horizon...)

We'll never be able to inject into every image; also some people don't want
us messing with their images.

Config drive is effectively injection into a disk we create, so we pick a
format that we know how to inject into.  It also seems to meet the needs of
people that would prefer that we don't mess with their images.

Have you considered using config drive instead of primary disk injection?

Justin


On Wed, Apr 25, 2012 at 5:13 AM, Alvaro Lopez  wrote:

> Dear.
>
> I've run into troubles with the resize of partitions and file injection;
> using libvirt and raw images.
>
> According to https://bugs.launchpad.net/nova/+bug/929005 file injection
> into images only works in the following situations:
>  1.- whole disk image -> inject to partition 1.
>  2.- separate kernel and ramdisk -> inject without partitions.
>
> In situation 1 (i.e. using whole disk images, with one partition
> inside) the file injection works fine, the disk is resized to the
> actual size, but the partition is not extended. This is because the
> 'extend' function in 'nova/virt/disk/api.py' executes 'qemu-img',
> 'e2fsck' and 'resize2fs' in a row, assuming that there are no
> partitions at all.
>
> In situation 2 (i.e. using whole disk images) if there is no kernel
> and ramdisk (but the file is a self-contained disk with an appropriate
> kernel) file injection does not work, but the resize works fine (since
> there are no partitions and the fsck and resize work as expected).
>
> Shouldn't the file injection be extended so as to make it work in any
> situation? That is, add injection without partitions and without
> kernel/ramdisk.
>
> Shouldn't the resize of disk take into account that it might be a disk
> with a (single) partition inside? This would imply dealing with the
> partitions (that is, delete and recreate them to the appropiate size).
>
> Regards,
> --
> Álvaro López García  al...@ifca.unican.es
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Public image repository via a synchronization script

[Justin Santa Barbara]

I think it would be very convenient to have public image repositories and
an easy way to use them, so someone installing OpenStack can easily get
images into Glance with minimal work.

At the design summit, there were a number of talks which all seemed to
hinge on how to get an image into glance from a public store.  I'd like to
make a proposal to get the ball rolling, which we can collectively pick
apart to figure out the best way.

   1. A public image repository lives on an HTTP / FTP / ... site and can
   be dumb. (Glance could one day be the server here as well, but this should
   also work on a CDN or simple webserver etc).
   2. Each image is stored as a (possibly compressed) file.
   3. Each image has a .properties file alongside it.
   4. There is a 'directory' file which lists the property files.  An
   md5sum of the directory is easy, and the hashes be useful generally.
   5. If there are different sets of images, we have multiple directory
   files.  They can point to the same images.
   6. We don't try to solve security initially.  But this could be as
   simple as signing the directory files, if they had hashes of all the other
   files (e.g. md5sum).

Then we have a client-side utility which takes a property file URL,
download it, and copies the associated image into Glance if it doesn't yet
exist.  It can verify checksums, and it uploads any image properties which
are present.

Finally, a simple bash script can download the directory file and call the
utility for each image.

This script could be run as a cron job if desired, to download new images
e.g. as Canonical publishes them.

---

An example 'properties' file might look like this:

org.openstack.sync__1__image=debian_squeeze_20120425.qcow2.gz
org.openstack.sync__1__expand=gzip
org.openstack.sync__1__size=627965952
org.openstack.sync__1__checksum=224237ee1b9a341ac7d79bcebda0580d
disk_format=qcow2
container_format=bare
is_public=True
org.openstack__1__os_distro=debian.org
org.openstack__1__os_version=6.0.4
org.openstack__1__architecture=x64
org.openstack__1__updated_at=20120425

A few things to note:

   - The org.openstack.sync__ prefix is used for controlling the mirror
   script.  It specifies the path to the image, the uncompressed size, the
   compression technique, the checksum etc.
   - The image can be compressed using e.g. gzip.  My 8GB raw image became
   a 600MB qcow2 image, which became a 220MB gzip image.  A bzip2 image was
   200MB (10% smaller), but bzip2 does not have as easy decompression in many
   languages.
   - Properties which aren't in the org.openstack.sync__ prefix become
   image properties in glance
   - This example is using the common image properties, which I've just got
   around to writing up: http://wiki.openstack.org/CommonImageProperties


---

I've implemented this in the Java bindings, and it works great:

https://github.com/platformlayer/openstack-java/blob/master/bin/image-sync
https://github.com/platformlayer/openstack-java/blob/master/openstack-cli/src/main/java/org/openstack/client/cli/commands/MirrorImage.java

Called with...

bin/image-sync http://images.platformlayer.org/ --prefix
platformlayerImages/

This then syncs the images, prefixing the names to avoid collisions.  And
yes, there is a real image up there - check out
http://images.platformlayer.org/md5sum ...


Justin

---

Justin Santa Barbara
Founder, FathomDB
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Canonical AWSOME

[Justin Santa Barbara]

Thanks for the pointer.  I found the etherpad:
http://etherpad.openstack.org/VersioningNovaRPCAPIs

Is there a blueprint that came / is coming out of that?

I think the data representation is orthogonal e.g. in theory, we could even
use XML schemas:
PyDict --> XML --> XML Schema Validation --> Warn / Throw
   \-> JSON -> Rabbit

As it sounds like we are using JSON, it makes most sense to use JSON
schemas.  But doing so doesn't preclude us from using e.g. a binary format
in future.

I'd imagine that the RPC mechanism would simply select the relevant schema
based on a few attributes of the dict (likely the queue & method name).  So
this would remain transparent to the caller.

A basic RPC mechanisms might validate and warn against a JSON schema,
another might use the schema to build a compact binary representation.  But
I think we can achieve this without changing nova.



On Tue, Apr 24, 2012 at 8:39 AM, Eric Windisch wrote:

>
> Actually, I think JSON schema for our message-bus messages might be a
> really good idea (tm).  Violations could just be warnings until we get
> things locked down... maybe I should propose a blueprint? (Although I have
> enough of a blueprint backlog as it is...)
>
>
> This was discussed at the summit in (I believe) the API versioning talk.
> There is a strong bias toward JSON inside RPC messages. However, this is
> actually transparent to Nova as the RPC implementations do the decoding and
> encoding. It is also hard to test and trigger such warnings as, as far as
> Nova knows, all the interfaces pass python data types, not JSON.  Some RPC
> mechanisms could transparently serialize.  As long as there is an
> abstraction layer, it should be oblivious to the serialization and we
> should not care too strongly.
>
> There was a patch a few weeks ago that has enforced using a common RPC
> exception serializer using JSON, which I'm not sure is, or is not, a good
> thing. I haven't yet updated the ZeroMQ driver to use this, but am in the
> process of making these changes as I update for Folsom.
>
> All that said, I do intend that the ZeroMQ driver will use JSON when it
> lands in Folsom.   (it currently Pickles, but only because there was a bug
> in Essex at one point, breaking  JSON serialization)
>
> --
> Eric Windisch
>
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Canonical AWSOME

[Justin Santa Barbara]

>
> If EC2 API is limiting what we can do, that's not going to change just
> because you move the EC2 API implementation into a proxy in front of the
> OpenStack API. The only difference is that it's suddenly the AWSOME
> development team's problem.
>

I think it's actually the EC2 API caller's problem.  It's easy to map a
subset to a superset, but you can't cover the superset with the subset.
 Suppose Quantum defines e.g. Layer 2 security groups, which I think has no
parallel in EC2.  If you want to use L2 security groups, you'd have to use
the OpenStack API.

A nice AWSOME developer might treat it as their problem, but it's not
really.

> With one native API, we can focus all our energies on making sure that API
> > works.  Then, knowing that the native API works, we can build other APIs
> on
> > top through simple translation layers, and they will work also.  Other
> APIs
> > can be built on top in the same way (e.g. OCCI)
>
> Sorry, I'm having trouble here.. Are you suggesting that having two
> sibling frontend API's talking to a shared backend API is poor software
> engineering, but layering similar purposed API's on top of each other
> like this is good software engineering?


Obviously not when you put it like that :-)

The difference that I see is whether the 'service' layer is documented and
kept stable (i.e. subject to API evolution rules).  Right now, we do that
work for the OpenStack API only.

I think it's great if someone is going to document and stabilize our
internal APIs.  However, unless you're willing to do it personally, I don't
think anyone should assume it'll be done.

So my viewpoint is that, while it would be great to have interface
contracts on all our internal interfaces, I'll believe it when I see it.
 Until then, we have one stable interface, and so using that stable
interface as the service layer seems like a sensible approach.

I can see us using ProtocolBuffers or JSON schema for messages that go over
the message bus, maybe even in Folsom.  I can't see us locking down the
other code interfaces before we give up on this Python thing and port to
Java, i.e. it's not going to happen.

Actually, I think JSON schema for our message-bus messages might be a
really good idea (tm).  Violations could just be warnings until we get
things locked down... maybe I should propose a blueprint? (Although I have
enough of a blueprint backlog as it is...)

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Canonical AWSOME

[Justin Santa Barbara]

On Mon, Apr 23, 2012 at 12:31 PM, Eric Windisch wrote:

> There seemed to be a strong agreement at the summit regarding the need for
> contracts on those "private" apis. This is because those APIs are no longer
> really private, they're shared amongst incubated projects.  Furthermore, it
> seems this may be required to support version heterogeneity during upgrades
> with versioned RPC calls.
>
> The incubated projects could use REST APIs, but we're talking of
> introducing artificial scaling and reliability constraints to do that.  It
> seems far better to me, if we can have contracts on those internal APIs and
> use them between the incubated projects.  There is a strong enough push to
> maintain these versions *anyway*.
>

Ah - thanks for explaining that.  I'm all for locking down these internal
interfaces.  I didn't realize people were willing to do so.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Canonical AWSOME

[Justin Santa Barbara]

I think the documented 'private' API should be the OpenStack API and should
be available to all callers (subject to permissions).  I only see downside
to having two APIs (public & internal).  If a proxy needs additional
functionality, odds are it probably belongs in the official OpenStack API
because advanced callers (e.g. PlatformLayer) might want the same power.

I think it makes most sense to expose our API via a REST/JSON interface
initially (as we do today); if we find performance issues we can expose it
via e.g. ZeroMQ/ProtocolBuffers in future.  But again, we should do so in a
way that benefits advanced callers as well as proxies.

Of course, this is heading dangerously towards a purely semantic argument -
what _is_ the OpenStack API? :-)



On Mon, Apr 23, 2012 at 11:01 AM, Eric Windisch wrote:

>  Creating a contract on the private API will allow the external APIs to
> be created and tested without needing a translation layer, even if
> contributory APIs were developed outside of the project (such as in AWSOME).
>
> It is clearly better, architecturally, if the EC2/OCCI apis can access the
> internal apis directly. The RPC and database can be made to scale in Nova,
> but a REST endpoint won't be as reliable or scale as well.
>
> --
> Eric Windisch
>
> On Monday, April 23, 2012 at 11:15 AM, Justin Santa Barbara wrote:
>
> What's the advantage of replacing the native EC2 compatibility layer with
> AWSOME from a user / operator point of view?
>
>
> Although I wasn't able to attend the design summit session, right now we
> have two "native" APIs, which means we have two paths into the system.
>  That is poor software engineering, because we must code and debug
> everything twice.  Some developers will naturally favor one API over the
> other, and so disparities happen.  Today, both APIs are effectively using
> an undocumented private API, which is problematic.  We also can't really
> extend the EC2 API, so it is holding us back as we extend OpenStack's
> capabilities past those of the legacy clouds.
>
> With one native API, we can focus all our energies on making sure that API
> works.  Then, knowing that the native API works, we can build other APIs on
> top through simple translation layers, and they will work also.  Other APIs
> can be built on top in the same way (e.g. OCCI)
>
> Which is a long way of saying the external approach will result in _all_
> APIs (OpenStack, EC2, OCCI etc) becoming more reliable, more secure and
> just more AWSOME.
>
> Justin
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Canonical AWSOME

[Justin Santa Barbara]

>
> What's the advantage of replacing the native EC2 compatibility layer with
> AWSOME from a user / operator point of view?
>

Although I wasn't able to attend the design summit session, right now we
have two "native" APIs, which means we have two paths into the system.
 That is poor software engineering, because we must code and debug
everything twice.  Some developers will naturally favor one API over the
other, and so disparities happen.  Today, both APIs are effectively using
an undocumented private API, which is problematic.  We also can't really
extend the EC2 API, so it is holding us back as we extend OpenStack's
capabilities past those of the legacy clouds.

With one native API, we can focus all our energies on making sure that API
works.  Then, knowing that the native API works, we can build other APIs on
top through simple translation layers, and they will work also.  Other APIs
can be built on top in the same way (e.g. OCCI)

Which is a long way of saying the external approach will result in _all_
APIs (OpenStack, EC2, OCCI etc) becoming more reliable, more secure and
just more AWSOME.

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] PlatformLayer adds support for Solr-as-a-Service

[Justin Santa Barbara]

Inspired by the launch of AWS CloudSearch on Thursday, I added
Solr-as-a-service to PlatformLayer so that we could have
search-as-a-service on Openstack as well:
http://news.ycombinator.com/item?id=3841826

I think it's a nice demonstration of what "everything as a service" means!

My ecosystem talk on PlatformLayer wasn't accepted, so please find me at
the Design Summit if you're interested in finding out more / collaborating.

And of course, the most important question is: which service should be next?

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Just JSON, and extensibility

[Justin Santa Barbara]

>
>
>>- Each (known) extension has its own strongly-typed model object.
>>
>>
> Does that mean that an extension cannot add properties to an existing
> object (such as adding a new attribute an Image), or just that all of those
> properties will be an a nested object (such as
> Image.my_extension.my_attribute)?
>

Good point - I should clarify...  I'm not using the standard Java XML
binding code here, because they would require a wrapping object.  It's
using custom code to do the binding whenever data from a different
namespace is encountered, although it uses the standard JAXB annotations to
guide it.  From the user's point of view, "it just works"; the
implementation is not as nice :-)

Because we're not wrapping our extensions in their own elements, I didn't
know how to get the standard tooling to work with it anyway.  Interesting
data-point for the "standard tools" argument...


> Now, how you would do something that awesome in Python, I don't know ;-)
>>
>
> Probably the same way. I would use setuptools entry points to map
> namespace prefixes to the class, so the client could load the handler
> automatically when it encounters data it does not recognize.
>

Yes - I should have said "dictionary" rather than "registry"!  Are you
doing this in python-novaclient?  Awesome!
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Just JSON, and extensibility

[Justin Santa Barbara]

It's easy when each new version is defined by a central body.

The problem we face is that we want to allow HP, Rackspace, Nexenta etc to
define their own extensions, without serializing through a central body.
 Some extensions may only apply to private clouds and never be shared
publicly.

This is a bit like ZFS feature flags, to use an example that should be near
and dear to your heart!



On Fri, Apr 13, 2012 at 2:24 PM, Caitlin Bestler <
caitlin.best...@nexenta.com> wrote:

>  Exactly what do you see as the required “non-linear extensibility”?
>
> ** **
>
> These are ultimately requests to a server. Each new extension is coded in
> that server.
>
> There is no value in a client making up its own extensions that are not
> understood by the server.
>
> ** **
>
> What is relevant is a server continuing to support clients that have not
> yet been updated to
> understand a new format.
>
> ** **
>
> As I stated in my first post, that problem was solved in ANSI C.
> Python/JSON is trivial.
>
> ** **
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Just JSON, and extensibility

[Justin Santa Barbara]

I don't think that works for non-linear extensibility...

I would be very happy if we could agree out how we're going to deal with
extensibility in JSON.  It is easy to support XML & JSON & any future
formats, and have them all be nice if there's willingness to do so, but
there's not, so let's drop it and focus on "stuff that matters"!

I believe we have a solution now for JSON extensibility (thanks Jorge),
it's not as "complete" as XML (read into that what you will) but it seems
to work fine.

My understanding is that the solution we have now is that any extension
goes into its own namespace; we assign a prefix to the namespace and have a
way to map that prefix to the full namespace.  (Similar to XML schemas).
 Currently prefixes are hard-coded, but we may not be able to keep doing
this forever (XML has pre-document prefixes to avoid the need for a central
authority).

I see 3 questions:
1) Is my summary correct?
2) Are there any problems with the solution?
3) Are there any other problems we're not addressing?


As one of the two authors of the Java binding, I can tell you how I plan on
dealing with extensions:


   - Map the JSON/XML/HPSTR to a strongly-typed model (so the
   representation is irrelevant to the consumer of the Java library).
   - Each (known) extension has its own strongly-typed model object.
   - These are stored in a registry.
   - When we come across an extension, we look it up in the registry and
   either bind or ignore it.
   - Every model object has an "Extensions" collection, which can be
   queried by type, to see if that extension data was present.


(Note: this has mostly been tested with the XML)

The nice thing about this is that a consumer of the library can write a
binding for an extension, and register it with the client library, and "it
just works".  So, even if you have a private extension, which you tell
nobody about and run only on your private cloud, you can use it with the
stock Java library.

Now, how you would do something that awesome in Python, I don't know ;-)

Justin



On Fri, Apr 13, 2012 at 11:53 AM, Caitlin Bestler <
caitlin.best...@nexenta.com> wrote:

> The argument that XML has better extensibility than JSON isn't very
> convincing to my ears.
>
> I'm an old war horse, and recall extending message formats in ANSI C so as
> to maintain backwards
> compatibility with existing clients (by versioning the struct name itself
> and always keeping the same
> fields up front).
>
> If this problem could be solved in ANSI C, the solutions available to
> Python with JSON can only be better.
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

I may disagree with a couple of the points along the way; but I agree with
the conclusion for OpenStack.

Thanks for writing it!

Justin

PS vim or emacs?



On Thu, Apr 12, 2012 at 12:58 PM, Mark Nottingham  wrote:

> A little fuel for the fire / entertainment before the summit:
>  http://www.mnot.net/blog/2012/04/13/json_or_xml_just_decide
>
> Cheers,
>
>
> On 10/04/2012, at 3:56 PM, Vishvananda Ishaya wrote:
>
> > On Apr 10, 2012, at 2:26 AM, Thierry Carrez wrote:
> >
> >> Jay Pipes wrote:
>  I take it you didn't attend the glorious JSON debate of a couple of
>  summits ago :-)
> >>>
> >>> Glorious it was indeed.
> >>
> >> I think the key quote was something like:
> >> "Please don't bastardize my JSON with your XML crap"
> >
> > According to my twitter, the actual quote was: "Don't bring your XML
> filth into my JSON"
> >
> > Vish
> >
> >
> >
> > ___
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
>
> --
> Mark Nottingham
> http://www.mnot.net/
>
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [SWIFT] Looking for an approach to attach an account or a container in operating system . Like a NAS of SAN driver.

[Justin Santa Barbara]

I made some patches to cloudfuse to get it to support Keystone auth:
http://blog.justinsb.com/blog/2012/03/29/openstack-storage-with-fuse/

I'd like to get this merged upstream, but haven't heard anything on my pull
request...  Redbo?  Bueller? :-)

I also saw an SFTP gateway floating about somewhere; that also seemed like
a good approach.

Should we make cloudfuse an openstack hosted project?

Frederik:  If you do have any issues that should be fixed, I'd love to know
what they are!

Justin



On Thu, Apr 12, 2012 at 10:03 AM, Frederik Van Hecke wrote:

> Hi Kuo,
>
> Here are some quick links:
>
> https://github.com/redbo/cloudfuse
> http://gladinet.blogspot.com/2010/10/openstack-windows-client.html
>
>
> I'm running cloudfuse on Ubuntu without much to complain about.
>
>
>
> Kind regards,
> Frederik Van Hecke
>
> *T:*  +32487733713
> *E:*  frede...@cluttr.be
> *W:* www.cluttr.be
>
>
>
> *This e-mail and any attachments thereto may contain information which is 
> confidential and/or protected by intellectual property rights and are 
> intended for the sole use of the recipient(s)named above. Any use of the 
> information contained herein (including, but not limited to, total or partial 
> reproduction, communication or distribution in any form) by persons other 
> than the designated recipient(s) is prohibited. If you have received this 
> e-mail in error, please notify the sender either by telephone or by e-mail 
> and delete the material from any computer. Thank you for your cooperation.*
>
>
>
>
> On Thu, Apr 12, 2012 at 17:57, Kuo Hugo  wrote:
>
>> I'm keeping in search and think about  the easiest way for users to
>> leverage their swift account.
>>
>> There has several client applications for accessing swift around . Either
>> Windows / Linux / Mac.
>>
>> But eventually , user still need to install a client for this purpose.
>>
>> What if user can attach their own account from swift-proxy (or something
>> else) directly via NFS or CIFS  or iscsi target will be much better.
>>
>> As I know , both linux and Windows has application to reach  this target.
>> Such as Gladinet Server / S3FS etc.
>>
>> How about provide such interfaces in Swift-proxy ?
>>
>> If there's an exist discussion ticket , please let me know it .
>>
>> Cheers
>>
>>
>> --
>> +Hugo Kuo+
>> tonyt...@gmail.com
>> + 886 935004793
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] minimal IaaS openstack installation FROM SOURCE on CentOS

[Justin Santa Barbara]

I have it running from source in a more-production-environment than
devstack, though on Debian...
https://github.com/justinsb/openstack-simple-config

Would you like to collaborate on fixing this up for CentOS & the Essex
release?  It was working a week or two before feature freeze, and post
Keystone redux, so it should be good, so it should mostly be CentOS
differences.

More production-ready = "won't wipe your data if you type the wrong thing
into devstack" :-)

I'm hoping to establish a baseline of a recommended way to package
OpenStack.  I'm leading a Design Summit session about this, I believe!

Justin





On Thu, Apr 12, 2012 at 10:09 AM, Vishvananda Ishaya
wrote:

> Devstack just gained support for Fedora, so you could try using it. You
> might have to make some modifications, but it is just a shell script so it
> should be easy to read.
>
> (From devstack.org) try:
>
> git clone git://github.com/openstack-dev/devstack.git
>
> cd devstack; ./stack.sh
>
>
> On Apr 12, 2012, at 6:31 AM, Mauch, Viktor (SCC) wrote:
>
> Hi,
> ** **
> I need to install a minimal/simple Openstack IaaS Deployment framework on
> CentOS 6.x from Source.
> ** **
> Is there anywhere a nice howto tutorial?
> ** **
> And yes: I know there are packeges, but I need it from source.
> ** **
> Cheers Viktor
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] profiling nova-api

[Justin Santa Barbara]

Both the profiling & the multi-process work are great - good stuff!

Jay: Won't going multi-process just make it harder to profile?  I think
it's actually a good thing to profile without the multi-process patch, find
& fix and the bottlenecks in single-request performance, and then use
multi-process for great concurrent-request performance?

Justin




On Thu, Apr 12, 2012 at 6:49 AM, Jay Pipes  wrote:

> Hi Yun!
>
> Thanks very much for sharing this information. Can I ask you to pull the
> code in this branch:
>
> https://review.openstack.org/#**change,5762
>
> And retry your test? On my test machines, the work that Huang Zhiteng has
> done to convert Nova's servers to use multiple operating system processes,
> each with its own greenthread service pool, resulted in a massive
> throughput improvement, turning a quickly-bottlenecked system into a server
> that could scale nearly linearly with the number of worker processes.
>
> Be sure to set osapi_workers to the number of cores your machine has...
> and also note that your code profiling technique is unlikely to be
> effective since cProfile wouldn't track the forked child worker processes'
> stacks, AFAIK. Still interested to see if the time to execute the 300 API
> calls is dramatically reduced, though.
>
> Looking forward to any results you might have.
>
> Best,
> -jay
>
>
> On 04/11/2012 04:48 PM, Yun Mao wrote:
>
>> Hi Stackers, I spent some time looking at nova-api today.
>>
>> Setup: everything-on-one-node devstack, essex trunk. I setup 1 user
>> with 10 tiny VMs.
>> Client: 3 python threads each doing a loop of "nova list" equivalent
>> for 100 times. So 300 API calls with concurrency=3.
>> how to profile: python -m cProfile -s time
>> /opt/stack/nova/bin/nova-api --flagfile=/etc/nova/nova.conf
>> --logdir=/var/log/nova --nodebug
>> The partial output is attached in the end.
>>
>> Observations:
>> * It takes about 60 seconds to finish. CPU of nova-api is around 70% to
>> 90%
>>
>> * Database access: Each "nova list" API call will issue 4 db APIs: 3
>> instance_get_all_by_filters(), 1
>> instance_fault_get_by_**instance_uuids(), so 1200 db API calls total
>> (note: not necessarily 1200 SQL statements, could be more). The 900
>> instance_get_all_by_filters() calls took 30.2 seconds (i.e. 0.03s
>> each)! The 300 instance_fault_get_by_**instance_uuids() calls only took
>> 1.129 seconds (0.004 each).
>>
>> You might think: MySQL sucks. Not so fast. Remember this is a tiny
>> database with only 10 VMs. Profile also shows that the actual
>> _mysql.connection.query() method only took 1.883 seconds in total. So,
>> we pretty much spend 29 seconds out of 60 seconds doing either
>> sqlalchemy stuff or our own wrapper. You can also see from the sheer
>> volume of sqlalchemy library calls involved.
>>
>> * the cfg.py library inefficiency. During 300 API calls,
>> common.cfg.ConfigOpts._get() is called 135005 times! and we paid 2.470
>> sec for that.
>>
>> Hopefully this is useful for whoever wants to improve the performance
>> of nova-api.
>>
>> Thanks,
>> Yun
>>
>> ===
>>
>>  23355694 function calls (22575841 primitive calls) in 77.874
>> seconds
>>
>>Ordered by: internal time
>>
>>ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>>   812   25.7250.032   25.7250.032 {method 'poll' of
>> 'select.epoll' objects}
>>  24081.8830.0011.8830.001 {method 'query' of
>> '_mysql.connection' objects}
>> 703801.6670.0007.1870.000
>> expression.py:2263(**corresponding_column)
>>1350051.2540.0002.4700.000 cfg.py:1058(_get)
>> 410271.0430.0001.9070.000 schema.py:542(__init__)
>> 388021.0080.0001.2190.000 __init__.py:451(format)
>>1622060.8210.0000.8210.000 util.py:883(values)
>>   15306660.7730.0000.7740.000 {isinstance}
>> 135046/1349600.7160.0001.9190.000 cfg.py:1107(_substitute)
>>  12050.7130.0011.3690.001 base.py:2106(__init__)
>>1836000.6900.0000.7960.000 interfaces.py:954(_reduce_*
>> *path)
>> 810020.6870.0002.4920.000 compiler.py:312(visit_label)
>> 388020.6500.0006.0870.000 log.py:227(format)
>>3192700.6220.0000.7480.000 attributes.py:164(__get__)
>> 890242/8842290.6080.0001.8850.000 {getattr}
>> 405000.6050.0003.1010.000 schema.py:955(_make_proxy)
>> 120783/1207380.6030.0000.6050.000 {method 'sub' of
>> '_sre.SRE_Pattern' objects}
>> 810000.6010.0002.1560.000
>> interfaces.py:677(create_row_**processor)
>> 630000.5900.0000.7070.000
>> times.py:44(DateTime_or_None)
>> 981020.5880.0000.8860.000
>> compiler.py:337(visit_column)
>>6580980.5800.0000.5810.000 {method 'intersection'
>> of 'set' object

[Openstack] Metadata and File Injection (code summit session?)

[Justin Santa Barbara]

Surely if you haven't got spoofing locked down on your cloud it's game over
anyway?

I think we do have another potential requirement here though: it would be
great to get a "machine" Keystone token securely.  I think it would also be
nice to be able to get the machine's automatically generated SSH key
fingerprint (although parsing the console log does work).

Then we could:

   - Make OpenStack API calls from machine -> cloud without uploading a
   user's OpenStack credentials
   - Use Keystone to authenticate any other service you want your VM to
   talk to
   - Make SSH calls to the machine securely, to run anything we want e.g.
   install the management software of your choice, mount a disk etc.
   - (For symmetry, we could sign calls from the machine using our SSH key
   as well, but I don't think this is useful in practice)

I guess I just don't understand the dislike of using SSH.  Nothing we write
is going to be any better.

In particular, a secure distributed fault tolerant key-value store that
nodes can read and write for any purpose, that we write from scratch as
part of nova, seems a little "ambitious".

Justin

On Tue, Apr 10, 2012 at 4:37 PM, Vishvananda Ishaya
wrote:

> On Apr 10, 2012, at 4:24 PM, Justin Santa Barbara wrote:
>
>  One advantage of a network metadata channel is it allows for
>> communication with cloud provider services without having to put a key into
>> the vm. In other words, the vm can be authenticated via its ipv6 address.
>>
>
> Did you have a use case in mind here?  It seems that Keystone could use
> the IPV6 address to authenticate an instance without having to upload
> credentials, which would indeed be useful (e.g. for auto-scaling), but I
> don't see why that needs any special metadata support (?)
>
>
> Arbitrarily allowing keystone to authenticate ipv6 would be vulnerable to
> spoofing. You need a channel direct from guest-host-keystone to be sure..
>  I think authentication is the main concern, because if auth is over a
> secure channel, then you can do all of the other communication over the
> regular internet. The vm could connect to the control domain for a service
> by subscribing to a message queue (for example) via a public ip.
>
> You could also secure the channel by having a private network attached to
> the vm and only putting the control domain for the service on the private
> network. Having keystone validate ipv6 only over that network might be an
> option.
>
> Vish
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Metadata and File Injection (code summit session?)

[Justin Santa Barbara]

>
> One advantage of a network metadata channel is it allows for communication
> with cloud provider services without having to put a key into the vm. In
> other words, the vm can be authenticated via its ipv6 address.
>

Did you have a use case in mind here?  It seems that Keystone could use the
IPV6 address to authenticate an instance without having to upload
credentials, which would indeed be useful (e.g. for auto-scaling), but I
don't see why that needs any special metadata support (?)
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Metadata and File Injection (code summit session?)

[Justin Santa Barbara]

>
> The immediate use case I have in mind is to support this:
> http://wiki.openstack.org/**PackageConfigForNova.
>   That design requires periodic checkins between an instance agent and a
> nova driver.  It certainly /could/ be implemented using ssh, but I
> originally wrote the design imagining there was a ready-made, standard
> communication service, and I still think it would be convenient.
>

My concern with that proposal is that it starts simple enough, but then
when you want to know e.g. "was the package installed successfully? is the
service healthy?" then you need more and more complexity i.e. you end up
with PlatformLayer, RedDwarf, Heat, Puppet, Chef or Juju.  So putting a
small piece of the required functionality into nova doesn't address your
actual use case, which is "I want configured machines, not just the stock
images".  It's probably easier to put that logic into your management
system of choice, so nova shouldn't do it.  Am I off base here?

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Metadata and File Injection (code summit session?)

[Justin Santa Barbara]

>
> Having the ability to read config data from a runtime changeable
> metadata server (rather then a config file on an injected disk) serves a
> use case I am interested in.  The only problem is horizontal scalability
> of the metadata server in this model which may not be a problem with
> some tinkering.
>

Can you please share that use case?  I'm especially interested in finding
use cases that would not better be better served by e.g. SSH or Zookeeper
or Corosync..
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Metadata and File Injection (code summit session?)

[Justin Santa Barbara]

I presume you're looking for more than SSH offers.  What are the missing
features you need?

On Tue, Apr 10, 2012 at 12:36 PM, Andrew Bogott wrote:

>  What I crave is a communication channel between nova and running
> instances.  There was discussion at some point about extending the metadata
> api to have this ability.
>
> Having a solid config drive standard seems like a good idea, but it won't
> get me run-time interaction, will it?
>
> -Andrew
>
>
> On 4/10/12 11:40 AM, Eric Windisch wrote:
>
> I maintain my stance from pre-Diablo, that the configuration drive should
> be exported as a virtual cdrom device with an ISO9660 filesystem. We can
> generate the filesystem without root access and the filesystem is
> well-supported.  Additionally, it lacks the patent-related issues
> associated with the other many-platform filesystems (i.e. FAT).
>
>  Also, doing the above happens to make the configuration-drive
> surprisingly similar to the optional sub-feature of OVF. I'm not sure what
> priority OVF is for Nova (it is a low priority for me), but it might be
> worth considering, especially since Glance seems to advertise some OVF
> support.
>
>  --
> Eric Windisch
>
>  On Tuesday, April 10, 2012 at 11:52 AM, Scott Moser wrote:
>
>   On Tue, 10 Apr 2012, Andrew Bogott wrote:
>
>   I'm reviving this ancient thread to ask: Will there be a code summit
> session
> about this? And/or are there plans to start developing a standard set of
> guest agents for Folsom?
>
>
>  http://summit.openstack.org/sessions/view/100
>
>
>  ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

Our XML support isn't good enough to be helpful.  The reality is that our
XML support is an afterthought, so we're not getting those extensibility
and validation benefits anyway.

People want APIs that work.  As a Java programmer, I'm perfectly capable of
talking to XML, JSON, ASCII or HPSTR.  The one thing I can't interface with
is a buggy implementation.

Everything gets hidden in a library anyway.  We now have a Java library; I
believe it is the only statically typed & only XML binding out there.  As
more and more bugs are discovered in the XML, it becomes less of an XML
binding and more of a JSON binding.

As a concrete example, the namespace for extensions changed late in Essex.
 Some of the public clouds still use the old namespace, some use the new
namespace.  The easiest fix was to move those extensions to JSON.  Security
groups have whitespace problems with XML => JSON.  Floating IPs don't work
with XML => JSON.  Key Pairs don't seem to work with XML => JSON.

In contrast, while Glance doesn't work with XML, it took me 30 seconds to
say "use JSON for all glance calls", as opposed to figuring it out one bug
at a time (hours of work).


On Tue, Apr 10, 2012 at 11:42 AM, Jorge Williams <
jorge.willi...@rackspace.com> wrote:

>  I'm also a strong supporter of XML. XML does a good job of lowering
> barriers for a key group of clients, specifically those that work with
> statically typed languages.  It offers key benefits in terms of
> extensibility and validation.  I'd hate to lose it.
>
>  -jOrGe W.
>
>  On Apr 10, 2012, at 12:57 PM, Justin Santa Barbara wrote:
>
>  It definitely has improved - thank you for all your work;  I didn't mean
> to put down anyone's work here.  It's simply a Sisyphean task.
>
> Either way, though, if I had the choice, I'd rip all of nova's XML
>> support out tomorrow…
>>
>
>  As a strong supporter of XML, who thinks JSON is for kids that haven't
> figured out that the Easter bunny isn't real yet :-)...  +1
>
>  Justin
>
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Metadata and File Injection (code summit session?)

[Justin Santa Barbara]

Config drive can support all EC2 functionality, I believe.

Images would need to be respun for OpenStack with config-drive, unless we
populated the config drive in a way that worked with cloud-init.  (Scott?)

Personally, I'd rather our effort went into producing great images for
OpenStack, than into compatibility with last-generation clouds.  Any idea
what the "important" EC2 images are?  Are there a handful of images that we
could duplicate and then just forget about EC2?


On Tue, Apr 10, 2012 at 10:30 AM, Joshua Harlow wrote:

>  Except for the fact that the config drive is non-EC2 right?
>
> That might blow it out of the water to start, as I know a lot of people
> want the EC2 equivalents/compat.
>
> But maybe if done right it shouldn’t matter (ie cloud-init could instead
> of calling out to urls could also call out to “local files” on a config
> drive).
>
> I just worry that config drive is openstack api only, afaik.
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

It definitely has improved - thank you for all your work;  I didn't mean to
put down anyone's work here.  It's simply a Sisyphean task.

Either way, though, if I had the choice, I'd rip all of nova's XML support
> out tomorrow…
>

As a strong supporter of XML, who thinks JSON is for kids that haven't
figured out that the Easter bunny isn't real yet :-)...  +1

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Agreeing a common set of Image Properties

[Justin Santa Barbara]

Signing would definitely be a great v2 feature.  For v1, we just need some
way to know that an image is provided by the cloud provider, and some idea
of what that image "is".

I believe every cloud is stuck respinning their own images, because we
haven't been able to agree a "golden image" standard.  So signing etc by
the distros is pointless until we get that figured out.

I trust the cloud providers today - I have no choice but to do so.  I think
you're trying to solve a much harder problem - how do I cope in a world
where I trust Canonical but not my cloud?  Once we have hardware trust of
clouds, then we'll have to up our game substantially on every front here.

On Tue, Apr 10, 2012 at 8:04 AM, Scott Moser  wrote:

> The data you're after might be useful to you, and might scratch an itch.
>
I will not discount that, but I would much prefer a bit of metadata
> associated with an image that was signed by an entity I trusted that
> identified the image as good.
>

I have to trust my cloud provider.  A single protected flag in metadata
saying "official cloud image" is no less secure than anything more
complicated at the moment (sadly)

OS distro, version_major, version_minor are even less important where you

don't care (or know) that your OS came from Canonical or RedHat, what you
> were really interested in is running "WhizBang! Fooberator" version 2.0.
>

Unless the distros stop changing config directory locations, or agree a
common init.d approach, then this simply isn't true.

Maybe you're talking about running pre-built appliances?  I'm talking about
not treating the machines as infallible black boxes (I think mine is the
more common use case, but irrespective, mine is definitely a valid use case)

I can see that some tagged info on the contents of the image would be
> useful for certain things, but specifically OS specific information is
> just not that important.
>

It's very important to me as a consumer of images.  How are you coding
image selection for launching instances on the public OpenStack clouds?
 I'm interested in any alternative.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Announcing project Heat

[Justin Santa Barbara]

Congratulations on your project!

Just to clarify, PlatformLayer can run anything as a service, not just
databases.  I added support for running memcache as a service last night,
and Zookeeper clusters on Sunday...  I think RedDwarf is also moving away
from just supporting DBs.

A crowded space indeed! :-)

Justin




On Tue, Apr 10, 2012 at 5:41 AM, Angus Salkeld  wrote:

> Hi
>
> I'd like to announce a new project that we have been working
> on, we have named it "Heat" (heat raises the clouds:).
>
> Our goal is to make it possible to manage multiple instance cloud
> applications with one template/specification file.
>
> Initially we want to implement as much as we can of the "AWS
> CloudFormation"
> API and later look at other API like TOSCA.
>
> We are attempting to write this in the style of an openstack project
> so as many people as possible can help out (come join in!).
>
> I know this is a busy topic area with several other interesting
> projects like Burrow, Donabe, Kanyun, Dough, Reddwarf and PlatformLayer.
>
> My hope is there we can form an exciting community at this layer and
> integrate where it makes sense.
>
> For instance many of the above projects implement (or could) some of
> the AWS resource types [1].
>
> Some things that could be done together:
>  - monitoring (Kanyun) and using the results to provide auto scaling.
>  - AWS::SQS::Queue (Burrow)
>  - AWS::RDS::DBInstance (Reddwarf/PlatformLayer)
>
>
> Unfortunately I won't be at the summit but Mark McLoughlin will give
> a lightening talk about it.
>
>
> Our project pages:
>  http://heat-api.org/
>  http://wiki.openstack.org/Heat
>  https://github.com/heat-api/heat
>
> Drop in at #heat on freenode or email our mailing list (
> http://lists.heat-api.org/mailman/listinfo).
>
> Regards
> Angus Salkeld
>
>
> [1]
> http://docs.amazonwebservices.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Metadata and File Injection (code summit session?)

[Justin Santa Barbara]

I think config drive is "injection done right"; it doesn't need to figure
out a partition because it creates its own.

Config drive + the init script I've contributed is rock-solid for me.  No
metadata service to go slow or need special configuration, no DHCP problems
etc.

If there's a proliferation of options, we should identify the important use
cases and make sure that we have a good approach for each of them.  Config
drive is very under-utilized, in my opinion.

Justin


On Tue, Apr 10, 2012 at 9:49 AM, Day, Phil  wrote:

> +1
>
> I was looking at this area earlier in terms of how the system works out
> what partition to inject keys/files/etc into, which feels like it should be
> specified by image metadata but currently defaults to partition 1.
>
> Made me wonder if we really need so many different ways for instances to
> get their metadata ?
>
> -Original Message-
> From: openstack-bounces+philip.day=hp@lists.launchpad.net [mailto:
> openstack-bounces+philip.day=hp@lists.launchpad.net] On Behalf Of
> Scott Moser
> Sent: 10 April 2012 16:52
> To: andrewbog...@gmail.com
> Cc: openstack@lists.launchpad.net
> Subject: Re: [Openstack] Metadata and File Injection (code summit session?)
>
> On Tue, 10 Apr 2012, Andrew Bogott wrote:
>
> > I'm reviving this ancient thread to ask:  Will there be a code summit
> > session about this?  And/or are there plans to start developing a
> > standard set of guest agents for Folsom?
>
> http://summit.openstack.org/sessions/view/100
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

I'd really rather we supported one format, if they're not going to be equal
citizens (i.e. both generated from a common model).

I wasted a lot of time with nova's XML "support"; I'm sure the Java binding
was the only project ever to try to use it; we'd have been able to proceed
much faster if we'd just stuck with JSON - we now have a horrible hybrid,
where JSON is used for some calls because the XML has/had bugs.

You prefer working from a JSON model; you're doing the work; so JSON it is.
 Please spend the time you would have spent on XML support on making glance
2.0 the best image server there is.

Justin


On Tue, Apr 10, 2012 at 7:47 AM, Jay Pipes  wrote:

> FWIW, Nova already has this kind of abstraction, with views and
> serializers... I wasn't planning on reinventing any wheels with the 2.0
> Images API implementation; just using what Nova had (and hopefully moving
> it to openstack-common before bringing the code into Glance).
>
> Best,
> -jay
>
>
> On 04/10/2012 06:51 AM, Doug Hellmann wrote:
>
>>
>>
>> On Mon, Apr 9, 2012 at 5:14 PM, Justin Santa Barbara
>> mailto:jus...@fathomdb.com>> wrote:
>>
>>When you're designing JSON considering only JSON, you'd
>>probably use {
>>
>>key1: value1 } - as you have done.  If you're designing
>>generically,
>>you'd probably use { key: key1, value: value1 }.
>>
>>
>>You mean we'd have to do dumb crap because XML doesn't have the
>>native concept of a list? ;)
>>
>>
>>XML has lists, as does Avro, ProtocolBuffers & Thrift.  XML supports
>>extensible lists, which is why the syntax is different.
>>
>>You'd *think* this would work. In practice, however, it
>>really doesn't. Neither does (good, valid) code generation...
>>
>>
>>Of course it works!  Every JAX-RS webserver does this.  You just
>>can't start with JSON first and expect everything to magically be OK.
>>
>>If you think it doesn't work, can you provide an example?
>>
>>You start with an abstract model, and then check what it looks like
>>in JSON, in XML, in ProtocolBuffers, in Avro, in Thrift, in HPSTR,
>>etc...  If you start with JSON, then of course it won't work.  If
>>we're going to treat XML as an afterthought, then I'd rather we just
>>didn't support XML at all (and yes, I absolutely mean that - it is
>>good that Glance is honest that they don't support XML.)
>>
>>
>> Kevin Dangoor and Christophe de Vienne have done some work on
>> abstracting the view of data inside and outside of the API with
>> TGWebServices [1] (a TurboGears add-on) and the more recent "Web
>> Services Made Easy" [2], which is framework agnostic. I have used TGWS
>> in the past to create an API using SOAP and JSON (it also supports
>> generic XML, but we didn't need that). I found that it worked well for
>> our purposes at the time.
>>
>> [1] http://code.google.com/p/tgws/
>> [2] http://packages.python.org/**WSME/ <http://packages.python.org/WSME/>
>>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

>
> Extensible lists are pointless. Lists have no attributes other than their
> length. I made this point a couple design summits ago... but whatever :)


Looks like the Sapir-Whorf hypothesis might be true after all ;-)

Let's dust off the pugil-sticks for the design summit..
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

>
> How about we discuss this further at the summit :-)
>

I think that's a sensible proposal.  We're not likely to reach a good
conclusion here.  I think my viewpoint is that even json-dressed-as-xml is
fine; no end-user gives two hoots what our JSON/XML/HPSTR looks like.  I'd
wager most users of the EC2 API have never even seen the data
representation!
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Agreeing a common set of Image Properties

[Justin Santa Barbara]

>
>
>> Are the major and minor numbers going to be sufficient versioning
>>> information? See for example PEP 386 for more detailed version strings (
>>> http://www.python.org/dev/peps/pep-0386/).
>>>
>>
>> For a distro, I believe yes.  Do you have a counter-example?
>>
>
> Not off the top of my head, but I really only use RedHat variants or
> Ubuntu variants, so I thought I would bring it up in case there are others
> that use different schemes. For example, how about pre-releases?
>

Good point (and ick!)  I like the PEP 386 / maven idea; non-official
releases get a suffix like "minor=0-SNAPSHOT" or "minor-0-beta1" or
"minor=0-20120401"?



> It seems suboptimal, but I guess it's reasonable to expect the caller of
> the API to be able to provide the distro-specific package name for the
> application or library they want included in the image.
>

I expect the most common use-case is that the caller wants e.g. Debian
Squeeze + a list of packages (openssh-server, mysql-server, nginx,
apache2).  It might see the following images:

#1 has packages: openssh-server
#2 has packages: openssh-server, mysql-server
#3 has packages: openssh-server, mysql-server, nginx, gnome-desktop

Now it could choose #1, in which case it know it needs to install mysql,
nginx + apache2.  It's always better to install #2, which avoids
downloading & installing mysql-server.

#3 may or may not be "better"; the caller may have to uninstall
gnome-desktop to reduce the attack surface area; that may be more expensive
than starting with #2.  I imagine most callers will actually choose an
image which has a strict subset of images; it's simpler.

That raises two issues: should openssh-server be listed (and what about
hypervisor specific packages)?  What do we do about meta-packages (like
gnome-desktop)?

The easy answer here is that the name of every installed package is listed
(as generated by the system package list tool e.g. dpkg).  But that would
be a big list...
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

>
> When you're designing JSON considering only JSON, you'd probably use {
>>
> key1: value1 } - as you have done.  If you're designing generically,
>> you'd probably use { key: key1, value: value1 }.
>>
>
> You mean we'd have to do dumb crap because XML doesn't have the native
> concept of a list? ;)


XML has lists, as does Avro, ProtocolBuffers & Thrift.  XML supports
extensible lists, which is why the syntax is different.



> You'd *think* this would work. In practice, however, it really doesn't.
>> Neither does (good, valid) code generation...
>
>
Of course it works!  Every JAX-RS webserver does this.  You just can't
start with JSON first and expect everything to magically be OK.

If you think it doesn't work, can you provide an example?

You start with an abstract model, and then check what it looks like in
JSON, in XML, in ProtocolBuffers, in Avro, in Thrift, in HPSTR, etc...  If
you start with JSON, then of course it won't work.  If we're going to treat
XML as an afterthought, then I'd rather we just didn't support XML at all
(and yes, I absolutely mean that - it is good that Glance is honest that
they don't support XML.)

Even ignoring XML, I can't help but think that not having a strict
delineation between user-provided data and the structure of your document
is a pretty risky idea.


In the 2.0 API we *are* specifying it in JSON. JSON Schema, specifically...
>

Are JSON schemas an April Fool's joke?  Once you introduce schemas, you
might as well just go with XML ;-)


> > I think the only thing you need to avoid is
>
>> "no changing-at-runtime keys"; I think this makes it compatible
>> with XML, Avro, ProtocolBuffers & Thrift.
>>
>
> That is antithetical to having dynamic, but discoverable, schemas. JSON
> Schema (and XSD, fwiw) provide things like additionalProperties and
>  for just this sort of thing. Making a schema entirely static is
> really only useful for generating (bad and soon-to-be-outdated) client code.
>

> Having dynamic and discoverable schemas enables clients to respond to
> backwards-compatible schema changes (like the addition of standard
> properties or the addition of "extra-standard" additionalProperties)
> without having to recompile a client or change any client code at all...


I couldn't disagree more: "what does it mean"?  There's the implicit
contract underlying the interface; the semantics that underpin the syntax.
 e.g. syntax: a glance image id is a string, semantics: the id is unique to
a glance installation and is used to refer to an image in REST calls.

xsd:any allows you to put elements _from another schema_ into your XML
document.  That foreign schema defines the semantics of those elements.
 It's schemas all the way down, giving semantics to your syntax.

If your additional properties in Glance are properly schematized, then
that's great.  But true cross-representation schemas are an open problem, I
believe, so you're really painting yourself into a corner (how do you
support XML output, if you let people upload arbitrary JSON schemas?)


Incidentally, I just heard about yet another new format - apparently
>> this one is going to be the standard for use with node.js
>> - Hypertext-Powered-State-**Transfer-Representation.
>>
>
> HPSTR? H(i)PST(e)R? Are you sure that wasn't an April Fool's Joke? :)
>

Are you sure it wasn't mine? ;-)
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Agreeing a common set of Image Properties

[Justin Santa Barbara]

I should probably clarify my terminology a little here, as I may have
mangled it:

   - I'm talking about additional/extension properties, not properties that
   are well-known to Glance
   - However, we agree to use a common set of properties to mean certain
   things.

In other words, we all agree to call Debian "openstack.org:distro=debian.org"
rather than "os_distro=Debian" on Cloud1, "distro=debian.org" on Cloud2,
"name=Debian Squeeze" on Cloud3 etc

3 main pieces of metadata: os:distro, os:version_major, os:version_minor
>>
>
> In the proposed Images API 2.0, we use JSON Schema's additionalProperties
> collection to allow for discoverable custom properties. For the properties
> you list above, you could either add those properties with the prefix you
> propose above (in additionalProperties), or argue for inclusion in the
> standard "properties" collection of the Image Entry schema. You can see
> both the properties and a sample of what additionalProperties are used for
> shown in the schema:
>
> https://docs.google.com/**document/d/1rb7ZVn0Du_**
> 5NZqUyQpqUZSmv7Qd66TMHYAtvsow7**LH4/edit#heading=h.**1nk6x0hs4enq


Ah, OK I understand the proposed  2.0 API a bit better now.

I'd like to solve this now though, not in 6 / 12 / 18 months.  It sounds
like my proposal is compatible with the new API as well, so no problem
there!


>  * Some clouds will automatically respin images e.g. weekly with the
>>latest updates.  This could also be exposed through metadata.
>>  os:updated_through= "20120301" ?
>>
>
> Possible, though a version identifier would also work. Or simply a query
> like:
> GET /images/sort_by=created_at&**sort_order=desc&limit=1&**
> property-os:distro=Debian
> (and you can do that in the existing API as well, BTW)
>

created_at != updated_through in general.   As a simple example, the cloud
provider might make a "clean" image (stock Debian 6.0.3) at the same time
as you make a totally up-to-date image.

Not sure what you mean by a version identifier?


>  * Some clouds will offer only bare images, some will provide a variety
>>e.g. bare, LAMP, Hadoop, etc.  Should we use the native package
>>names to indicate additional packages?  e.g.
>>os:packages="apache2,mysql,**php5" ?
>>
>
> IMHO, no. That would get overkill. In the new API, you could use tags,
> though. Or you could add a custom extension in the new API so that you
> could expose packages as a subresource of /images//.
>

It is important to differentiate a "minimal" image from a "loaded" image,
users will typically want a loaded image, code will typically want a bare
image; alternative suggestions are welcome.  If we're going to use tags in
2.0, shouldn't we use properties/metadata today?

 As a (programmatic) consumer of these images, my wishlist:
>>
>>  * A cloud will have to put on whatever drivers / agents they need to,
>>but ideally these should otherwise be clean images, with minimal
>>deviation from the stock install.  (Or 'clean' images should be
>>available alongside other images)  It would be great if I could be
>>launch a 'clean' image on any OpenStack cloud and have it behave
>>more or less the same; I shouldn't have to second guess any
>>additional tweaks.
>>
>
> No disagreement from me here, but I don't see how this relates to a common
> set of image properties? Could you elaborate?
>

Here's what I see on Cloud X (edited to protect the party, and I removed
the kernels/ramdisks):

417 Debian Squeeze 6.0.3 Server 64-bit 20120123ACTIVE
x_image_type=machine, image_location=local, image_state=available,
project_id=None, x_md_version=1, kernel_id=415, min_ram=0, ramdisk_id=416,
x_image_id=c89dee3bca7a62103f7d88d2a02f4dc8, owner=None,
x_image_builddate=20120123, architecture=amd64, min_disk=0,
x_image_version=1x1.1
414 CentOS 6.2 Server 64-bit 20120125  ACTIVE
x_image_type=machine, image_location=local, image_state=available,
project_id=None, x_md_version=1, kernel_id=412, min_ram=0, ramdisk_id=413,
x_image_id=f2fbb1bf37a13e7c5da897c7082684df, owner=None,
x_image_builddate=20120125, architecture=x86_64, min_disk=0,
x_image_version=1x1
229 Ubuntu Oneiric 11.10 Server 64-bit 20111212ACTIVE
image_location=local, image_state=available, kernel_id=228, min_ram=0,
min_disk=0, architecture=amd64, owner=None, project_id=None


227 Ubuntu Natty 11.04 Server 64-bit 20111212  ACTIVE
image_location=local, image_state=available, kernel_id=226, min_ram=0,
min_disk=0, architecture=amd64, owner=None, project_id=None


225 Ubuntu Maverick 10.10 Server 64-bit 20111212   ACTIVE
image_location=local, image_state=available, kernel_id=224, min_ram=0,
min_disk=0, architecture=amd64, owner=None, project_id=None


223 Ubuntu Lucid 10.04 LTS Server 64-bit 20111212  ACTIVE
image_location=local, image_state=available, kernel_id=222, 

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

>
> Justin, what does "design a model which works with XML" mean?


Simply avoiding the handful of things that are specific to JSON (or
specific to XML).  Nothing too onerous (no angle brackets)!


> I think this is only done in the image properties.
>>
>
> No, the image properties have been removed in the 2.0 Images API and
> replaced with the standard concept of "tags":
>
> https://docs.google.com/**document/d/1rb7ZVn0Du_**
> 5NZqUyQpqUZSmv7Qd66TMHYAtvsow7**LH4/edit#heading=h.**y10rq9pdt27y


I was looking at properties in "GET /schemas/image/access" and "GET
/schemas/access".  In general, the Glance 2.0 model is very close to being
XML-compatible.

 On the downside: The JSON isn't as JSON-ic as it could be.
>>
>
> In what ways?


When you're designing JSON considering only JSON, you'd probably use {
key1: value1 } - as you have done.  If you're designing generically, you'd
probably use { key: key1, value: value1 }.

 On the upside: You need never worry about XML again
>>
>
> We haven't worried about it at all up until now. And Glance has been
> perfectly fine without it. ;)
>

Well yes, I agree that our XML support generally is sufficiently buggy as
to be unusable, hence unused.  I don't actually care about XML, I just care
about having a well-specified API that works.

In my opinion, the way to do that is to specify a model, and we have JSON
and XML representations of that model.  If we design the model right, JSON
& XML both "just work" without any code (and less code = less bugs).
 Moreover, when someone introduces another data interchange format, we
again just write one piece of middleware, one documentation generator, and
we're done.  Doing this by hand for each format is simply busywork.

I don't care whether we specify the model as a Python class, Java class,
XML schema, JSON schema, or using stick figures with long arms pointing at
boxes.  You can even specify it in JSON if you stick to a "lowest common
denominator" subset.  I think the only thing you need to avoid is "no
changing-at-runtime keys"; I think this makes it compatible with XML, Avro,
ProtocolBuffers & Thrift.

I don't think it should be a big deal for the JSON format to avoid dynamic
keys, given how much you win whenever you have to support a new format.

Incidentally, I just heard about yet another new format - apparently this
one is going to be the standard for use with node.js
- Hypertext-Powered-State-Transfer-Representation.  Apparently JSON is too
complicated now - with its strict need for quote marks and multiple data
types and all that over engineering, so HPSTR just sends the values; you
just use convention over configuration to infer the keys.  I trust it's not
going to be a lot of work for OpenStack to support it.

> See you in a week!

Looking forward to it!
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Image API v2 Draft 4

[Justin Santa Barbara]

>
> APPENDIX B: Outstanding issues
>
> 4) Need to write xsds :(
>

This is easy if you design a model which works with XML.  If you have an
XML compatible model, you can generate an XSD and a JSON model from that.
 Also, it means you can just use common middleware to map XML to JSON,
rather than coding XML parsing functions.

I think the one thing you'd have to change is that XML won't allow you to
serialize a dictionary with elements as keys (e.g. { "key1": "value1",
"key2": "value2" }, you have to do this: [ { "key": "key1", "value":
"value1" }, { "key": "key2", "value": "value2" } ]

I think this is only done in the image properties.

On the downside: The JSON isn't as JSON-ic as it could be.
On the upside: You need never worry about XML again
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Agreeing a common set of Image Properties

[Justin Santa Barbara]

On Mon, Apr 9, 2012 at 8:18 AM, Doug Hellmann
wrote:

> I'm thinking of the os prefix as standing for OpenStack, not Operating
>> System.
>>
>
> I would never have guessed that from the context. Why OpenStack instead of
> Operating System?
>

We reserve this limited region of the namespace (openstack.org: or os:) for
keys that we agree to have general meaning.  In return for not using that
prefix for their own purposes, anyone uploading images knows that there
will no collisions, no matter what cloud that image lands on, and no matter
what additional values openstack.org defines in the future.

In general, I'd suggest that prefixing custom properties with your domain
name is "a good idea" for the long term, but that's a separate issue.

Sounds like you'd prefer openstack.org:version or openstack.org:os_version.
 Both work for me, I think I prefer the second.


> Are the major and minor numbers going to be sufficient versioning
> information? See for example PEP 386 for more detailed version strings (
> http://www.python.org/dev/peps/pep-0386/).
>

For a distro, I believe yes.  Do you have a counter-example?

  The package names may vary depending on the platform, so the names
> probably shouldn't be tied to the names of the packages themselves but to
> the projects. Or maybe that's just a taxonomy problem that should be left
> up to the taggers.
>

Absolutely.  The package name would be meaningful/comparable only after the
distro/major/minor combo.  I can't imagine any distro changing package
names within a release.  So we really do defer this issue to the distro.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Jcloud and openstack relation

[Justin Santa Barbara]

There is also a (friendly rival) OpenStack Java binding being developed:
https://github.com/platformlayer/openstack-java
https://github.com/woorea/openstack-java-sdk/

That library supports a direct-to-OpenStack Jenkins plugin which I
confidently predict will rapidly surpass any lowest-common-denominator
attempts (*):
https://github.com/platformlayer/openstack-jenkins

(*: This opinion may be biased)

If you do evaluate JClouds against the OpenStack Java bindings (or the
Jenkins plugin), please let me know of any areas where the OpenStack
bindings aren't better, and I will make sure to improve that area.

Monty: We should port the OpenStack CI team's work to the OpenStack Jenkins
plugin at the summit!

Justin



On Sun, Apr 8, 2012 at 6:37 PM, Monty Taylor  wrote:

> Hi!
>
> jclouds is an open source java library for connecting to clouds. As
> such, it supports connecting to OpenStack based clouds. The OpenStack CI
> team have been working with jclouds on both support for OpenStack as
> well as a plugin for jenkins so that we can have more direct control of
> build slaves on the clouds we use for that purpose.
>
> Does that help?
>
> Monty
>
> On 04/07/2012 07:02 PM, Frans Thamura wrote:
> > hi all
> >
> > can share to me, what is the relateion between openstack and jclouds.
> >
> > F
> >
> >
> > ___
> > Mailing list: https://launchpad.net/~openstack
> > Post to : openstack@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Agreeing a common set of Image Properties

[Justin Santa Barbara]

Thanks Padraig & Nathanael - virt-inspector is a great source of
inspiration.  Can we put the virt-inspector output into a glance property?
 Would all the clouds agree to do that?

I still would also like simpler metadata, as it just feels like too much
information and work required for what is a comparatively simple use case
("launch the official Debian Squeeze image" either "the latest one" or "a
known revision").

Most importantly, I would like to see a solution to this now, rather than
with the next release... If we agree these attributes now, we won't have to
solve the unification problem in 6 months.

Padraig: Can you explain the issue with "updated_through"?  I would have
thought that would define a unique set of package versions (if I'm using
the standard repos)?  I expect that most people will either want to pick
the oldest image (with a particular minor vision) or the newest image (with
the greatest minor version), so it might be that this simply doesn't matter
other than to define an ordering.  It might also be that the creation date
is adequate here.

I have no objection to whatever attributes we define today being populated
automatically using these cool tools in future!

Justin



2012/4/7 Nathanael Burton 

> Looks like Pádraig and I were thinking alike.
> On Apr 7, 2012 8:49 PM, "Pádraig Brady"  wrote:
>
>> On 04/07/2012 11:13 PM, Justin Santa Barbara wrote:
>> > Is there a (de-facto) standard for image metadata/properties?  I'd like
>> to be able to able to launch e.g. the Debian Squeeze image provided by the
>> cloud.  This is particularly important for clouds that don't allow image
>> upload, but likely this will remain useful because different clouds will
>> have different tweaks needed (e.g installing the right drivers based on the
>> hypervisor).
>> >
>> > I could try "smart"-parsing the names, but it seems like metadata is
>> the right way to do this, and I see no reason why any cloud would gain any
>> advantage from not adopting a common convention.  I know some clouds have
>> started implementing their own approaches, but I don't believe anyone is
>> locked into anything.
>> >
>> > In the interest of efficiency, I'm going to make a proposal for people
>> to attack:
>> >
>> > 3 main pieces of metadata: os:distro, os:version_major, os:version_minor
>> >
>> > I'm thinking of the os prefix as standing for OpenStack, not Operating
>> System.  I'd like to 'reserve' the os prefix for 'agreed' prefixes.  Maybe
>> this should be openstack.org:distro ?
>> >
>> > os:distro would be the primary domain name of the distro, i.e.
>> redhat.com <http://redhat.com>, ubuntu.com <http://ubuntu.com>,
>> debian.org <http://debian.org>, centos.org <http://centos.org> etc
>> >
>> > os:version_major would be the major version of the release:
>> > debian.org <http://debian.org>: 6.0, 5.0, 4.0, 3.1, 3.0, 2.2, 2.1, 2.0
>> > ubuntu.com <http://ubuntu.com>: 10.04, 10.10, 11.04, 11.10, 12.04
>> >
>> > Numbers seem more machine-friendly than codenames - 6.0, not squeeze -
>> humans will probably use the image names.
>> >
>> > os:version_minor would be the minor version of the release (because
>> it's a minor revision, hopefully we shouldn't normally have to check this,
>> although we might want to select the latest always).
>> >
>> > So os:distro="debian.org <http://debian.org>" os:version_major "6.0"
>> os:version_minor "3" would be an image of debian 6.0.3.
>> >
>> > Questions / ideas for other properties:
>> >
>> >   * Some clouds will automatically respin images e.g. weekly with the
>> latest updates.  This could also be exposed through metadata.
>>  os:updated_through= "20120301" ?
>> >   * Some clouds will offer only bare images, some will provide a
>> variety e.g. bare, LAMP, Hadoop, etc.  Should we use the native package
>> names to indicate additional packages?  e.g.
>> os:packages="apache2,mysql,php5" ?
>> >
>> > As a (programmatic) consumer of these images, my wishlist:
>> >
>> >   * A cloud will have to put on whatever drivers / agents they need to,
>> but ideally these should otherwise be clean images, with minimal deviation
>> from the stock install.  (Or 'clean' images should be available alongside
>> other images)  It would be great if I could be launch a 'clean' image on
>> any OpenStack cloud and have it be

Re: [Openstack] Agreeing a common set of Image Properties

[Justin Santa Barbara]

I'm talking here about publicly accessible attributes of images, that the
image owner would set, so that API callers could select the right image
with which to launch their instances.

e.g. I'm trying to launch a Zookeeper cluster made up of 5 instances on 5
different clouds; how can I figure out which image is "Debian Squeeze" on
each of AT&T, Rackspace, HP, IBM, Internap, Dreamhost etc.  I could code
image ids or logic for each of the clouds, but that's already annoying with
a handful of clouds and impractical with 100.  Now imagine also that the
images are being re-spun each week!

There's no protection against the image uploader setting the wrong values,
so the API user would definitely only want images from trusted providers.
 Is there an easy way to differentiate publicly shared images from
'official' cloud images?

I don't believe these attributes would be suitable for billing purposes.
 You could make it so that certain attributes can't be changed by the user
(e.g. billing codes), or that some values are system assigned (e.g. "this
is an official image"), but I think that's a future conversation / future
Glance feature.

I'm just thinking this set of attributes would be a win for everyone vs.
each cloud rolling their own, so we can just agree them without any need
for code!

Justin


On Sat, Apr 7, 2012 at 3:43 PM, Jamey Meredith  wrote:

>  So would these accessible via a system admin API only?   The problem we
> face as a
> Public cloud is that we need to know things about an image for billing
> purposes.   If a user can snapshot one of my windows images and change the
> distro field to Debian to avoid license fees, that wont work for us.  I
> need to know the distro that this image was originally based upon.
>
>  I also need to denote things like managed service level for an image
> that might be the same base distro as an unmanaged image.
>
> Sent from my iPhone
>
> On Apr 7, 2012, at 5:14 PM, "Justin Santa Barbara" 
> wrote:
>
>   Is there a (de-facto) standard for image metadata/properties?  I'd like
> to be able to able to launch e.g. the Debian Squeeze image provided by the
> cloud.  This is particularly important for clouds that don't allow image
> upload, but likely this will remain useful because different clouds will
> have different tweaks needed (e.g installing the right drivers based on the
> hypervisor).
>
>  I could try "smart"-parsing the names, but it seems like metadata is the
> right way to do this, and I see no reason why any cloud would gain any
> advantage from not adopting a common convention.  I know some clouds have
> started implementing their own approaches, but I don't believe anyone is
> locked into anything.
>
> In the interest of efficiency, I'm going to make a proposal for people to
> attack:
>
> 3 main pieces of metadata: os:distro, os:version_major, os:version_minor
>
> I'm thinking of the os prefix as standing for OpenStack, not Operating
> System.  I'd like to 'reserve' the os prefix for 'agreed' prefixes.  Maybe
> this should be openstack.org:distro ?
>
> os:distro would be the primary domain name of the distro, i.e. redhat.com,
> ubuntu.com, debian.org, centos.org etc
>
> os:version_major would be the major version of the release:
> debian.org: 6.0, 5.0, 4.0, 3.1, 3.0, 2.2, 2.1, 2.0
> ubuntu.com: 10.04, 10.10, 11.04, 11.10, 12.04
>
> Numbers seem more machine-friendly than codenames - 6.0, not squeeze -
> humans will probably use the image names.
>
> os:version_minor would be the minor version of the release (because it's a
> minor revision, hopefully we shouldn't normally have to check this,
> although we might want to select the latest always).
>
> So os:distro="debian.org" os:version_major "6.0" os:version_minor "3"
> would be an image of debian 6.0.3.
>
> Questions / ideas for other properties:
>
>- Some clouds will automatically respin images e.g. weekly with the
>latest updates.  This could also be exposed through metadata.
> os:updated_through= "20120301" ?
>- Some clouds will offer only bare images, some will provide a variety
>e.g. bare, LAMP, Hadoop, etc.  Should we use the native package names to
>indicate additional packages?  e.g. os:packages="apache2,mysql,php5" ?
>
>  As a (programmatic) consumer of these images, my wishlist:
>
>- A cloud will have to put on whatever drivers / agents they need to,
>but ideally these should otherwise be clean images, with minimal deviation
>from the stock install.  (Or 'clean' images should be available alongside
>other images)  It would be great if I could

[Openstack] Agreeing a common set of Image Properties

[Justin Santa Barbara]

Is there a (de-facto) standard for image metadata/properties?  I'd like to
be able to able to launch e.g. the Debian Squeeze image provided by the
cloud.  This is particularly important for clouds that don't allow image
upload, but likely this will remain useful because different clouds will
have different tweaks needed (e.g installing the right drivers based on the
hypervisor).

I could try "smart"-parsing the names, but it seems like metadata is the
right way to do this, and I see no reason why any cloud would gain any
advantage from not adopting a common convention.  I know some clouds have
started implementing their own approaches, but I don't believe anyone is
locked into anything.

In the interest of efficiency, I'm going to make a proposal for people to
attack:

3 main pieces of metadata: os:distro, os:version_major, os:version_minor

I'm thinking of the os prefix as standing for OpenStack, not Operating
System.  I'd like to 'reserve' the os prefix for 'agreed' prefixes.  Maybe
this should be openstack.org:distro ?

os:distro would be the primary domain name of the distro, i.e. redhat.com,
ubuntu.com, debian.org, centos.org etc

os:version_major would be the major version of the release:
debian.org: 6.0, 5.0, 4.0, 3.1, 3.0, 2.2, 2.1, 2.0
ubuntu.com: 10.04, 10.10, 11.04, 11.10, 12.04

Numbers seem more machine-friendly than codenames - 6.0, not squeeze -
humans will probably use the image names.

os:version_minor would be the minor version of the release (because it's a
minor revision, hopefully we shouldn't normally have to check this,
although we might want to select the latest always).

So os:distro="debian.org" os:version_major "6.0" os:version_minor "3" would
be an image of debian 6.0.3.

Questions / ideas for other properties:

   - Some clouds will automatically respin images e.g. weekly with the
   latest updates.  This could also be exposed through metadata.
os:updated_through= "20120301" ?
   - Some clouds will offer only bare images, some will provide a variety
   e.g. bare, LAMP, Hadoop, etc.  Should we use the native package names to
   indicate additional packages?  e.g. os:packages="apache2,mysql,php5" ?

As a (programmatic) consumer of these images, my wishlist:

   - A cloud will have to put on whatever drivers / agents they need to,
   but ideally these should otherwise be clean images, with minimal deviation
   from the stock install.  (Or 'clean' images should be available alongside
   other images)  It would be great if I could be launch a 'clean' image on
   any OpenStack cloud and have it behave more or less the same; I shouldn't
   have to second guess any additional tweaks.
   - I would like to be able to launch the clean image and install updates
   myself, in case I don't want a particular update.  Providing a fast apt
   cache is much better than providing respun images, for my use-case.  I
   would be great if old images stuck around, therefore!


Justin

---

Justin Santa Barbara
Founder, FathomDB
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Introducing PlatformLayer: Everything-as-a-service, for OpenStack

[Justin Santa Barbara]

:-)

In a way: it is the knowledge from running an XaaS company distilled
into code, although lots of people worked on the code over the years.
Code libraries = knowledge-as-a-service.

If anyone is thinking of building something as a service, I hope
you'll grab me at the design summit and let me show you the head-start
PlatformLayer gives you; in return I'll get to see more of the areas
where it is lacking!

Justin



On Fri, Apr 6, 2012 at 4:04 PM, Mark Collier  wrote:
> Justin-as-a-Service?
>
>
> Justin Santa Barbara  wrote:
>
>
> FathomDB has just open-sourced PlatformLayer, which makes it easy to
> build anything as a service.  It takes all the know-how and a lot of
> the code from running FathomDB's MySQL-as-a-service offering.
>
> You can now build your:
>
> Memcached-as-a-service (like AWS ElastiCache)
> MySQL-as-a-service (like AWS RDS)
> Nginx-as-a-service (like AWS Elastic Load Balancing)
> DNS-as-a-service (like AWS Route 53)
> etc
>
> This is just a starting point, but I'm hoping to present this more at
> the Design Summit - I'm thinking I'll try live-coding a new service in
> 15 minutes (anyone have any requests?  RabbitMQ-as-a-service?)
>
> Read more: http://blog.justinsb.com/blog/2012/04/06/introducing-platformlayer/
>
> Discussion on hacker news: http://news.ycombinator.com/item?id=3809221
>
> Please contact me if you're interested in using it or helping out!
>
> Thanks,
> Justin
>
> ---
>
> Justin Santa Barbara
> Founder, FathomDB
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Introducing PlatformLayer: Everything-as-a-service, for OpenStack

[Justin Santa Barbara]

FathomDB has just open-sourced PlatformLayer, which makes it easy to
build anything as a service.  It takes all the know-how and a lot of
the code from running FathomDB's MySQL-as-a-service offering.

You can now build your:

Memcached-as-a-service (like AWS ElastiCache)
MySQL-as-a-service (like AWS RDS)
Nginx-as-a-service (like AWS Elastic Load Balancing)
DNS-as-a-service (like AWS Route 53)
etc

This is just a starting point, but I'm hoping to present this more at
the Design Summit - I'm thinking I'll try live-coding a new service in
15 minutes (anyone have any requests?  RabbitMQ-as-a-service?)

Read more: http://blog.justinsb.com/blog/2012/04/06/introducing-platformlayer/

Discussion on hacker news: http://news.ycombinator.com/item?id=3809221

Please contact me if you're interested in using it or helping out!

Thanks,
Justin

---

Justin Santa Barbara
Founder, FathomDB

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Java 7 File Provider for OpenStack Storage

[Justin Santa Barbara]

Java 7 includes a pluggable filesystem abstraction, so I created a basic
filesystem provider for Swift using the Java OpenStack bindings.  It is
available here: https://github.com/platformlayer/openstack-fileprovider

If anyone is interested in using this, please let me know and I'll fix up
any bugs you find!

There's almost no documentation out there on how to do this, so I don't
think any of the legacy clouds support this.  Another small step forward
for OpenStack!

Justin

---

Justin Santa Barbara
Founder, FathomDB
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] OpenStack Plugin for Jenkins

[Justin Santa Barbara]

I've got Compute functionality working with the OpenStack Jenkins plugin,
so it can launch nova instances as on-demand slaves now, run builds on
them, and archive the results into  swift.  I'd like to open GitHub issues
to track your requirements, but I have a few questions.

> We need disposable machines that are only used for one test, which means
spinning up and terminating hundreds of machines per day.

Sounds like we want a function to terminate the machine after the job has
run.
https://github.com/platformlayer/openstack-jenkins/issues/1

> We need to use machines from multiple providers simultaneously so that
we're resilient against errors with one provider.

Label expressions should work here; you would apply a full set of axis
labels to each machine ("rax oneiric python26") but then you would filter
based only on the required axes ("oneric python26").  Are labels sufficient
for this?

> We need to pull nodes from a pool of machines that have been spun up
ahead of time for speed.

This sounds like a custom NodeProvisioner implementation.  The current
implementation is optimized around minimizing CPU hours, by doing load
prediction.  You have a different criteria, based on minimizing launch
latency.  It looks like it should be relatively easy to implement a new
algorithm, although perhaps a bit tricky to figure out how to plug it in.

https://github.com/platformlayer/openstack-jenkins/issues/2


> We need to be able to select from different kinds of images for certain
tests.

Are labels sufficient for this?

> Machines need to be checked for basic functionality before being added to
the pool (we frequently get nodes without a functioning network).

I believe Jenkins does this anyway; a node which doesn't have networking
won't be able to get the agent.  And you can run your own scripts after the
slave boots up ("apt-get install openjdk", for example).  Those scripts can
basically do any checks you want.  Is that enough?

> They need to be started from snapshots with cached data on them to avoid
false negatives from network problems.

Can you explain this a bit more?  This is to protect against the apt
repositories / python sources / github repos being down?  Would an http
proxy be enough?

> We need to keep them around after failures to diagnose problems, and we
 need to delete those after a certain amount of time.

>From the github docs, it sounds like you don't get access anyway because of
the providers' policies.  Would it not therefore be better to take a ZIP or
disk snapshot after a failed test, and then shut down the machine as normal?


Also...

You currently auto-update your images, which is cool
(devstack-update-vm-image).  Do you think this is something a plugin should
do, or do you think this is better done through scripts and a matrix job?
 I'm leaning towards keeping it in scripts.  The one thing I think we
definitely need here is some sort of 'best match' image launching, rather
than hard-coding to a particular ID, so that the cloud plugin will always
pick up the newest image.

https://github.com/platformlayer/openstack-jenkins/issues/3


Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] OpenStack Plugin for Jenkins

[Justin Santa Barbara]

Thanks - there are some good feature requests here.

Have you proposed a design summit session where you are going to talk about
OpenStack features that you would like to see?  Things like supporting
cloud federation, instance pooling, or fixing that networking bug?

The Jenkins features (e.g. associating a test to an image) should be done
as part of Jenkins: JClouds plugin, OpenStack plugin, it doesn't matter -
may the best code win.

However, all the OpenStack features should be done as part of OpenStack
itself, not buried in some third party project that is only available to
people that use Java.

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] OpenStack Plugin for Jenkins

[Justin Santa Barbara]

This is a plugin for OpenStack; it won't support any other clouds.  This
means there will be no baggage from having to support EC2, or any of the
other clouds that JClouds supports.  We can support all the OpenStack
goodness directly.

JClouds is great, but it's an abstraction layer that doesn't give OpenStack
users anything.

I hope the OpenStack CI team will join me in working on something that
advances OpenStack, rather than spending any more time propping up the
dying clouds :-)

Justin



On Wed, Apr 4, 2012 at 2:54 PM, Andrew Hutchings wrote:

> On 04/04/12 20:42, Justin Santa Barbara wrote:
> > I've created a quick OpenStack plugin for Jenkins, using the Java
> > bindings that Luis & I created.
> >
> > It's available on github here:
> >  https://github.com/platformlayer/openstack-jenkins  (no binaries -
> yet!)
>
> Whilst it is cool that there is effort going into this what does it give
> that the Jenkins Jclouds plugin does not? (I think it is at
> https://github.com/jenkinsci/jclouds-plugin)
>
> The OpenStack CI team are also assisting in the development of the
> Jcouds plugin with a view to using it on OpenStack's Jenkins server.
>
> Kind Regards
> --
> Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] OpenStack Plugin for Jenkins

[Justin Santa Barbara]

I've created a quick OpenStack plugin for Jenkins, using the Java bindings
that Luis & I created.

It's available on github here:
https://github.com/platformlayer/openstack-jenkins  (no binaries - yet!)

Right now, it has the same (jenkins) functionality as the S3 plugin; it can
store artifacts into Swift.  I'm going to implement compute next so that we
can use OpenStack VMs to 'burst' building, like the EC2 plugin does.

I'd love to hear from anyone that would be interested in using this, to
better understand what everyone wants.  For example, the S3 plugin flattens
all directories when copying artifacts, that just seems wrong to me, so I'd
like to just fix it.

Justin

---

Justin Santa Barbara
Founder, FathomDB
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Performance diagnosis of metadata query

[Justin Santa Barbara]

I think _we_ should stand behind OpenStack.

That said, I understand why it's a tough call.  Hopefully most problems can
be fixed with a simple-ish SQL script post-release.


On Thu, Mar 29, 2012 at 11:06 AM, David Kranz wrote:

>  Well, there are diablo-stable packages. If Ubuntu, Debian, Red Hat,  etc.
> keep hearing from customers that Essex in an LTS release is not adequate,
> there will be essex-stable packages too. They are the ones who have to
> stand behind the product. It is perfectly understandable that there is
> resistance to putting anything other than fixes for critical bugs in a week
> or so from release.  I am not saying this is great, but if release dates
> are fixed and features, performance the things that are allowed to vary,
> then what else is there to do? Just my opinion.
>
>  -David
>
>
> On 3/29/2012 1:55 PM, Justin Santa Barbara wrote:
>
> I'm not saying it can't be rationalized; I'm saying it is frustrating to
> me.
>
>  My understanding is that Essex is going to be baked into both Ubuntu &
> Debian for the long term - 5 years plus.   That's a long time to have to
> keep explaining why X is broken; I'd rather just fix X.
>
>
>
> On Thu, Mar 29, 2012 at 10:22 AM, David Kranz wrote:
>
>>   On 3/29/2012 12:46 PM, Justin Santa Barbara wrote:
>>
>>  Is there a good way to map back where in the code these calls are
>>> coming from?
>>
>>
>>  There's not a great way currently.  I'm trying to get a patch in for
>> Essex which will let deployments easily turn on SQL debugging (though this
>> is proving contentious); it will have a configurable log level to allow for
>> future improvements, and one of the things I'd like to do is add later is
>> something like a stack trace on 'problematic' SQL (large row count, long
>> query time).  But that'll be in Folsom, or in G if we don't get logging
>> into Essex.
>>
>>  In the meantime, it's probably not too hard to follow the code and
>> infer where the calls are coming from.  In the full log, there's a bit more
>> context, and I've probably snipped some of that out; in this case the
>> relevant code is get_metadata in the compute API service and
>> get_instance_nw_info in the network service.
>>
>>
>>>  Regardless, large table scans should be eliminated, especially if the
>>> table is mostly read, as the hit on an extra index on insert will be
>>> completely offset by the speedups on select.
>>
>>
>>  Agreed - some of these problems are very clear-cut!
>>
>>  It does frustrate me that we've done so much programming work, but then
>> not do the simple stuff at the end to make things work well.  It feels a
>> bit like shipping we're shipping C code which we've compiled with -O0
>> instead of -O3.
>>
>>
>>  Well, in a project with the style of fixed-date release (short-duration
>> train-model) that openstack has, I think we have to accept that there will
>> never be time to do anything except fight critical bugs "at the end". At
>> least not until the project code is much more mature. In projects I have
>> managed we always allocated time at the *beginning* of a release cycle for
>> fixing some backlogged bugs and performance work. There is less pressure
>> and the code is not yet churning. It is also important to have performance
>> benchmark tests to make sure new features do not introduce performance
>> regressions.
>>
>>  -David
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Performance diagnosis of metadata query

[Justin Santa Barbara]

I'm not saying it can't be rationalized; I'm saying it is frustrating to me.

My understanding is that Essex is going to be baked into both Ubuntu &
Debian for the long term - 5 years plus.   That's a long time to have to
keep explaining why X is broken; I'd rather just fix X.



On Thu, Mar 29, 2012 at 10:22 AM, David Kranz wrote:

>  On 3/29/2012 12:46 PM, Justin Santa Barbara wrote:
>
>  Is there a good way to map back where in the code these calls are coming
>> from?
>
>
>  There's not a great way currently.  I'm trying to get a patch in for
> Essex which will let deployments easily turn on SQL debugging (though this
> is proving contentious); it will have a configurable log level to allow for
> future improvements, and one of the things I'd like to do is add later is
> something like a stack trace on 'problematic' SQL (large row count, long
> query time).  But that'll be in Folsom, or in G if we don't get logging
> into Essex.
>
>  In the meantime, it's probably not too hard to follow the code and infer
> where the calls are coming from.  In the full log, there's a bit more
> context, and I've probably snipped some of that out; in this case the
> relevant code is get_metadata in the compute API service and
> get_instance_nw_info in the network service.
>
>
>>  Regardless, large table scans should be eliminated, especially if the
>> table is mostly read, as the hit on an extra index on insert will be
>> completely offset by the speedups on select.
>
>
>  Agreed - some of these problems are very clear-cut!
>
>  It does frustrate me that we've done so much programming work, but then
> not do the simple stuff at the end to make things work well.  It feels a
> bit like shipping we're shipping C code which we've compiled with -O0
> instead of -O3.
>
>
> Well, in a project with the style of fixed-date release (short-duration
> train-model) that openstack has, I think we have to accept that there will
> never be time to do anything except fight critical bugs "at the end". At
> least not until the project code is much more mature. In projects I have
> managed we always allocated time at the *beginning* of a release cycle for
> fixing some backlogged bugs and performance work. There is less pressure
> and the code is not yet churning. It is also important to have performance
> benchmark tests to make sure new features do not introduce performance
> regressions.
>
>  -David
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Performance diagnosis of metadata query

[Justin Santa Barbara]

>
> Is there a good way to map back where in the code these calls are coming
> from?


There's not a great way currently.  I'm trying to get a patch in for Essex
which will let deployments easily turn on SQL debugging (though this is
proving contentious); it will have a configurable log level to allow for
future improvements, and one of the things I'd like to do is add later is
something like a stack trace on 'problematic' SQL (large row count, long
query time).  But that'll be in Folsom, or in G if we don't get logging
into Essex.

In the meantime, it's probably not too hard to follow the code and infer
where the calls are coming from.  In the full log, there's a bit more
context, and I've probably snipped some of that out; in this case the
relevant code is get_metadata in the compute API service and
get_instance_nw_info in the network service.


>  Regardless, large table scans should be eliminated, especially if the
> table is mostly read, as the hit on an extra index on insert will be
> completely offset by the speedups on select.


Agreed - some of these problems are very clear-cut!

It does frustrate me that we've done so much programming work, but then not
do the simple stuff at the end to make things work well.  It feels a bit
like shipping we're shipping C code which we've compiled with -O0 instead
of -O3.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] KVM crash.

[Justin Santa Barbara]

The libvirt hang was a threading deadlock within libvirt, whereas this is a
kernel crash.  I'd be surprised if they are the same issue.

The current theory on the kernel bugzilla is that it might be related to
bridge + netfilter.  That does tally with the observation of it happening
under a network load test.

Guilherme: I've never seen the issue personally, but I haven't run network
stress tests, I run Debian, I prefer FlatManager etc.  Is it fairly easy to
reproduce e.g. by running ab against a guest?  I suspect this will be
outside of anything we can fix in OpenStack itself though; the final
outcome will likely be e.g. "you must use a kernel with patch X."  You can
either go straight to the latest versions, or if you provide a simple
procedure I'm sure a number of the big clouds will want to try it on their
own configurations.

Justin



On Wed, Mar 28, 2012 at 9:45 AM, Vishvananda Ishaya
wrote:

> This was discussed on the mailing list earlier I believe:
>
> http://www.mail-archive.com/openstack@lists.launchpad.net/msg08475.html
>
> The solution appears to be to upgrade to a newer libvirt.
>
> Vish
>
> On Mar 28, 2012, at 6:01 AM, Guilherme Birk wrote:
>
> No one is having this issue?
>
> --
> From: guib...@hotmail.com
> To: openstack@lists.launchpad.net
> Date: Fri, 23 Mar 2012 19:48:04 +
> Subject: [Openstack] KVM crash.
>
> I'm having problems with KVM on a single node installation. My problem is
> like these ones:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=42703
> http://www.spinics.net/lists/kvm/msg67635.html
>
> The KVM normally crashes when I'm doing a load test with a large number of
> connections on the VM's.
>
> This appears to not happen using a dual node installation, with the
> nova-compute running with a dedicated machine.
>
> ___ Mailing list:
> https://launchpad.net/~openstack Post to : 
> openstack@lists.launchpad.netUnsubscribe :
> https://launchpad.net/~openstack More help :
> https://help.launchpad.net/ListHelp
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Performance diagnosis of metadata query

[Justin Santa Barbara]

The performance of the metadata query with cloud-init has been causing some
people problems (it's so slow cloud-init times out!), and has led to the
suggestion that we need lots of caching.  (My hypothesis is that we
don't...)

By turning on SQL debugging in SQL Alchemy (for which I've proposed a patch
for Essex: https://review.openstack.org/#change,5783), I was able to
capture the SQL statements.

I'm focusing on the SQL statements for the metadata call.

The code does this:

1) Checks the cache to see if it has the data
2) Makes a message-bus call to the network service to get the fixed_ip info
from the address
3) Looks up all sort of metadata in the database
4) Formats the reply

#1 means that the first call is slower than the others, so we need to focus
on the first call.
#2 could be problematic, if the message queue is overloaded or if the
network service is slow to response
#3 could be problematic if the DB isn't working properly
#4 is hopefully not the problem.

The relevant SQL log from the API server:
http://paste.openstack.org/show/12109/

And from the network server: http://paste.openstack.org/show/12116/

I've analyzed each of the SQL statements:

API

http://paste.openstack.org/show/12110/ (Need to check that there isn't a
table scan when instance_info_caches is large)
http://paste.openstack.org/show/12111/ Table scan on services table, but
this is presumably smallish
http://paste.openstack.org/show/12112/ No index.  Table scan on s3_images
table.  Also this table is MyISAM.  Also seems to insert rows on the first
call (not shown).  Evil.
http://paste.openstack.org/show/12113/
http://paste.openstack.org/show/12114/ block_device_mapping is MyISAM.


Network

http://paste.openstack.org/show/12117/
http://paste.openstack.org/show/12118/ (Fetch virtual_interface by
instance_id)
http://paste.openstack.org/show/12119/ (Fetch network by id)
http://paste.openstack.org/show/12120/ Missing index => table scan on
networks.  Unnecessary row re-fetch.
http://paste.openstack.org/show/12121/ Missing index => table scan on
virtual_interfaces.  Unnecessary row-refetch.
http://paste.openstack.org/show/12122/ (Fetch fixed_ips on virtual
interface)
http://paste.openstack.org/show/12123/ Missing index => table scan on
networks.  Unnecessary row re-fetch.  (Double re-fetch.  What does it mean?)
http://paste.openstack.org/show/12124/ Missing index => table scan on
virtual_interfaces.  Another re-re-fetch.
http://paste.openstack.org/show/12125/ Missing index => table scan on
fixed_ips (Uh-oh - I hope you didn't allocate a /8!!).  We do have this row
from the virtual interface lookup; perhaps we could remove this query?
http://paste.openstack.org/show/12126/
http://paste.openstack.org/show/12127/


We still have a bunch of MyISAM tables (at least with a devstack install):
http://paste.openstack.org/show/12115/



As I see it, these are the issues (in sort of priority order):

*Critical*

Table scan of fixed_ips on the network service (row per IP address?)
Use of MyISAM tables, particularly for s3_images and block_device_mapping
Table scan of virtual_interfaces (row per instance?)
Verify that MySQL isn't doing a table scan on
http://paste.openstack.org/show/12110/ when # of instances is large

*Naughty*

*(Mostly because the tables are small)*

Table scan of s3_images
Table scan of services
Table scan of networks

*Low importance*

*(Re-fetches aren't a big deal if the queries are fast)*

Row re-fetches & re-re-fetches



My install in nowhere near big enough for any of these to actually cause a
real problem, so I'd love to get timings / a log from someone that is
having a problem.  Even the table scan of fixed_ips should be OK if you
have enough RAM.

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Caching strategies in Nova ...

[Justin Santa Barbara]

This is great: hard numbers are exactly what we need.  I would love to see
a statement-by-statement SQL log with timings from someone that has a
performance issue.  I'm happy to look into any DB problems that
demonstrates.

The nova database is small enough that it should always be in-memory (if
you're running a million VMs, I don't think asking for one gigabyte of RAM
on your DB is unreasonable!)

If it isn't hitting disk, PostgreSQL or MySQL with InnoDB can serve 10k
'indexed' requests per second through SQL on a low-end (<$1000) box.  With
tuning you can get 10x that.  Using one of the SQL bypass engines (e.g.
MySQL HandlerSocket) can supposedly give you 10x again.  Throwing money at
the problem in the form of multi-processor boxes (or disks if you're I/O
bound) can probably get you 10x again.

However, if you put a DB on a remote host, you'll have to wait for a
network round-trip per query.  If your ORM is doing a 1+N query, the total
read time will be slow.  If your DB is doing a sync on every write, writes
will be slow.  If the DB isn't tuned with a sensible amount of cache (at
least as big as the DB size), it will be slow(er).  Each of these has a
very simple fix for OpenStack.

Relational databases have very efficient caching mechanisms built in.  Any
out-of-process cache will have a hard time beating it.  Let's make sure the
bottleneck is the DB, and not (for example) RabbitMQ, before we go off a
huge rearchitecture.

Justin



On Thu, Mar 22, 2012 at 7:53 PM, Mark Washenberger <
mark.washenber...@rackspace.com> wrote:

> Working on this independently, I created a branch with some simple
> performance logging around the nova-api, and individually around
> glance, nova.db, and nova.rpc calls. (Sorry, I only have a local
> copy and its on a different computer right now, and probably needs
> a rebase. I will rebase and publish it on GitHub tomorrow.)
>
> With this logging, I could get some simple profiling that I found
> very useful. Here is a GH project with the analysis code as well
> as some nova-api logs I was using as input.
>
> https://github.com/markwash/nova-perflog
>
> With these tools, you can get a wall-time profile for individual
> requests. For example, looking at one server create request (and
> you can run this directly from the checkout as the logs are saved
> there):
>
> markw@poledra:perflogs$ cat nova-api.vanilla.1.5.10.log | python
> profile-request.py req-3cc0fe84-e736-4441-a8d6-ef605558f37f
> keycountavg
> nova.api.openstack.wsgi.POST   1  0.657
> nova.db.api.instance_update1  0.191
> nova.image.show1  0.179
> nova.db.api.instance_add_security_group1  0.082
> nova.rpc.cast  1  0.059
> nova.db.api.instance_get_all_by_filters1  0.034
> nova.db.api.security_group_get_by_name 2  0.029
> nova.db.api.instance_create1  0.011
> nova.db.api.quota_get_all_by_project   3  0.003
> nova.db.api.instance_data_get_for_project  1  0.003
>
> key  count  total
> nova.api.openstack.wsgi  1  0.657
> nova.db.api 10  0.388
> nova.image   1  0.179
> nova.rpc 1  0.059
>
> All times are in seconds. The nova.rpc time is probably high
> since this was the first call since server restart, so the
> connection handshake is probably included. This is also probably
> 1.5 months stale.
>
> The conclusion I reached from this profiling is that we just plain
> overuse the db (and we might do the same in glance). For example,
> whenever we do updates, we actually re-retrieve the item from the
> database, update its dictionary, and save it. This is double the
> cost it needs to be. We also handle updates for data across tables
> inefficiently, where they could be handled in single database round
> trip.
>
> In particular, in the case of server listings, extensions are just
> rough on performance. Most extensions hit the database again
> at least once. This isn't really so bad, but it clearly is an area
> where we should improve, since these are the most frequent api
> queries.
>
> I just see a ton of specific performance problems that are easier
> to address one by one, rather than diving into a general (albeit
> obvious) solution such as caching.
>
>
> "Sandy Walsh"  said:
>
> > We're doing tests to find out where the bottlenecks are, caching is the
> > most obvious solution, but there may be others. Tools like memcache do a
> > really good job of sharing memory across servers so we don't have to
> > reinvent the wheel or hit the db at all.
> >
> > In addition to looking into caching technologies/approaches we're gluing
> > together some tools for finding those bottlenecks. Our first step will
> > be finding them, then squashing them ... however.
> >
> > -S
> >
> > On 03/22/2012 06:25 PM, Mark Washenberger wrote:
> >> What problems are caching s

Re: [Openstack] Please stop the devstack non-sense!

[Justin Santa Barbara]

Hi Thomas,

I think devstack has done a lot for the developer's use-case, but I believe
we should also have a official / semi-official project that does some sort
of packaging to help the production use-case.  I've proposed a summit
discussion: http://summit.openstack.org/sessions/view/26

The background: I want a semi-production deployment, but as a developer I
still want to be able to edit the code (which makes packages inconvenient).
 devstack is orientated towards e.g. wiping the databases.

I'm hoping that all the various OS packagers can work together, or at least
tell us what sucks.  As a community, we should solve these problems once,
and the OpenStack project shouldn't treat them as externalities.  I've been
doing some initial coding here:
https://github.com/justinsb/openstack-simple-config

The first use case I'm trying to solve is "single node installation of
OpenStack" that is as easy as possible, but also isn't painting the user
into the corner.  Think "apt-get openstack", then the user finds they like
it and grows to a 4 node cluster, all the way up to a 100 node cluster.  So
it uses KVM, FlatManager, config drive injection, Postgresql, etc. - I'm
afraid it is still quite "opinionated"!  I have Keystone, Glance & Nova
installing.  I'm using supervisord to avoid any OS dependencies/flamewars,
but I would imagine that any OS packager could move it to their preferred
init.d flavor easily.  Swift is next on my list - I was facing the problem
that the number of replicas isn't changeable, though I have a patch for
that now.

If you'd like to work together, I'd love to collaborate (and that holds for
anyone doing packaging).  I'm hanging out in #openstack-packaging

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Essex keystone with remote glance endpoint

[Justin Santa Barbara]

>
> # glance  -I adminUser -K ... -S keystone -N
> http://192.168.131.141:5000/v2.0' index
> Failed to show index. Got error:
> Response from Keystone does not contain a Glance endpoint.
>

I think that means that the glance client can't find a suitable glance
endpoint in the response from Keystone.  Relevant code is in get_endpoint
around line 200 of glance/common/auth.py

So, I don't think you're ever hitting the glance server.

Keystone appears to be working, or at least passing basic tests:
> keystone # curl -d '{"auth": {"tenantName": "service",
> "passwordCredentials":{"username": "glance", "password": "glance"}}}' -H
> "Content-type: application/json" http://192.168.131.141:35357/v2.0/tokens| 
> python -mjson.tool
> http://paste.openstack.org/show/11078/


The response (in the paste) looks good to me.  If you're passing in a
region ("OS_REGION_NAME or via the '-R/--region' command line") then maybe
this is wrong?

If not, you could try this curl with the same credentials as you're using
with the glance CLI tool, to verify that the glance endpoint is being
returned correctly here also.

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [NOVA] Snapshotting may require significant disk space (in /tmp). How to properly solve disk space issues?

[Justin Santa Barbara]

We're creating a (huge) temp file, uploading it, and then deleting it.  So
really we should be streaming the snapshot direct to the destination
(glance?)

Checking the code, we are writing it sequentially (particularly if we're
writing in raw):
https://github.com/qemu/QEMU/blob/master/qemu-img.c


But there's more...

> qemu-img --help
qemu-img version 1.0, Copyright (c) 2004-2008 Fabrice Bellard
...
Supported formats: vvfat vpc vmdk vdi *sheepdog* *rbd* raw host_cdrom
host_floppy host_device file qed qcow2 qcow parallels *nbd iscsi* dmg *tftp
ftps ftp https http* cow cloop bochs blkverify blkdebug


So it looks like we really want a "Supported format: glance" there
(particularly as there's already http support in block/curl.c) :-)  I guess
we could then even do crazy things like booting direct from glance?

Or, if we don't want to get back into C, we could at least optimize the
case where glance is backed by Ceph, and stream direct to a Ceph file, and
then hand that file to Glance.

Justin





On Fri, Mar 16, 2012 at 9:11 AM, Jay Pipes  wrote:

> Hi Stackers,
>
> So, in diagnosing a few things on TryStack yesterday, I ran into an
> interesting problem with snapshotting that I'm hoping to get some advice on.
>
> == The Problem ==
>
> The TryStack codebase is Diablo, however the code involved in this
> particular problem I believe is the same in Essex...
>
> The issue that was happening was a user was attempting to snapshot a tiny
> instance (512MB/1-core) through the dashboard. The dashboard returned and
> noted that a snapshot was created and was in Queued status.
>
> The snapshot never goes out of Queued status, and so I logged into the
> compute node that housed the instance in question to see if I could figure
> out what was going on.
>
> Grepping through the compute log, I found the following:
>
> (nova.rpc): TRACE: Traceback (most recent call last):
> (nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-**
> packages/nova/rpc/impl_kombu.**py", line 628, in _process_data
> (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
> (nova.rpc): TRACE:   File 
> "/usr/lib/python2.7/dist-**packages/nova/exception.py",
> line 100, in wrapped
> (nova.rpc): TRACE: return f(*args, **kw)
> (nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-**
> packages/nova/compute/manager.**py", line 687, in snapshot_instance
> (nova.rpc): TRACE: self.driver.snapshot(context, instance_ref,
> image_id)
> (nova.rpc): TRACE:   File 
> "/usr/lib/python2.7/dist-**packages/nova/exception.py",
> line 100, in wrapped
> (nova.rpc): TRACE: return f(*args, **kw)
> (nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-**
> packages/nova/virt/libvirt/**connection.py", line 479, in snapshot
> (nova.rpc): TRACE: utils.execute(*qemu_img_cmd)
> (nova.rpc): TRACE:   File "/usr/lib/python2.7/dist-**packages/nova/utils.py",
> line 190, in execute
> (nova.rpc): TRACE: cmd=' '.join(cmd))
> (nova.rpc): TRACE: ProcessExecutionError: Unexpected error while running
> command.
> (nova.rpc): TRACE: Command: qemu-img convert -f qcow2 -O raw -s
> e7ba4fb5f6f04f99b07d1d222ada02**19 
> /opt/openstack/nova/instances/**instance-0548/disk
> /tmp/tmpIuOQo0/**e7ba4fb5f6f04f99b07d1d222ada02**19
> (nova.rpc): TRACE: Exit code: 1
> (nova.rpc): TRACE: Stdout: ''
> (nova.rpc): TRACE: Stderr: 'qemu-img: error while writing\n'
>
> QEMU was unhelpfully returning a vague error message of "error while
> writing".
>
> It turned out, after speaking with a couple folks on IRC (thx vishy and
> rmk!) that the snapshot process (qemu-img convert ... above) is storing the
> output of the process (the snapshot) in a temporary directory created using
> tempfile.mkdtemp() in the nova/virt/libvirt/connection.**py file.
>
> As it turns out, the base operating system we install on our compute nodes
> in TryStack has a (very) small root partition -- only 2GB in size (we use
> the devstack build_pxe_env.sh script to create the base Ubuntu image that
> is netbooted on the compute nodes.
>
> Looking at the free disk space on the compute node in question, the
> problem was apparent:
>
> root@freecloud102:/var/log/**nova# df -h
> FilesystemSize  Used Avail Use% Mounted on
> /dev/ram0 2.0G  1.4G  535M  73% /
> devtmpfs   48G  240K   48G   1% /dev
> none   48G 0   48G   0% /dev/shm
> none   48G  212K   48G   1% /var/run
> none   48G 0   48G   0% /var/lock
> /dev/md0  5.4T   93G  5.1T   2% /opt/openstack
>
> There simply isn't enough free space on the root partition (which is where
> /tmp is housed) for the snapshot to be created.
>
> == Possible Solutions ==
>
> So, there are a number of solutions that we can work on here, and I'm
> wondering what the preference would be. Here are the solutions I have come
> up with, along with a no-brainer improvement to Nova that would help in
> diagnosing this problem:
>
> The no-brainer: Detect before attempting a snapshot 

[Openstack] Networking guru needed: problem with FlatManager ARP when guest and bridge MACs the same

[Justin Santa Barbara]

We recently changed the MAC address assigned to guests so that they started
with 0xfe, in the hope of avoiding (theoretical?) issues with MAC addresses
changing on the bridge device as machines are shut down (because supposedly
the bridge grabs the lowest MAC address numerically):
https://bugs.launchpad.net/nova/+bug/921838

However, it looks we bumped into some similar behavior done by libvirt: It
also sets the first byte to 0xfe for the host network device, in the hope
of avoiding the same bug.  Thus, with the patch, the host vnetX and the
guest eth0 have the same MAC address.  I think this breaks FlatManager, but
I don't know why, and I really don't know why it wouldn't break other
modes, and I'm hoping a network guru can explain/confirm.

When they have the same MAC address, ARP resolution isn't working: the
guest issues an ARP request for the gateway, on the host I can see the ARP
request and response, but the guest doesn't appear to see/accept the ARP
response and so it just keeps retrying.

This message appears in dmesg:
[ 2199.836114] br100: received packet on vnet1 with own address as source
address

I'm guessing that 'address' means 'MAC address', and this is why ARP is
failing, it sounds like the bridge might be dropping the packet.

Changing to 0x02, or 0xfc does fix it (although my arithmetic was wrong,
and vishy points out we should use 0xfa instead of 0xfc).

Networking guru questions:

   - Does this explanation make sense?
   - Why didn't other networking modes break?
   - Should we simply revert the change and go back to 0x02?
   - Should we switch to 0xfa to try to avoid the bridge interface
   problems?  Or does it simply not matter if libvirt is changing the MAC for
   us?
   - Can anyone explain https://bugs.launchpad.net/nova/+bug/908194 ?  Is
   that bug because there was no 'real' device on the bug reporter's bridge?


Vishy has proposed this patch, which looks good to me:
https://review.openstack.org/#patch,sidebyside,5351,1,nova/utils.py


Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Random libvirt hangs

[Justin Santa Barbara]

I certainly understand your position Thierry.  However, I think it is
important that we target one 'golden' platform, and that we take
responsibility for any issues with that platform.  Otherwise we simply
end up pointing fingers and being blocked on backports, and the end
result is a system that just doesn't work for the people actually
deploying it.  "File a bug upstream" is an appropriate response for
me, but it's not really OK for end-users.

We could then have a policy that 'if Essex fails on TargetPlatform
it's an OpenStack issue, otherwise it's a distro issue'.  We can
either work around the bug or work with TargetPlatform to get a bugfix
integrated.  Other distros can look to the golden platform to
understand what patches are needed and how things are supposed to
work.

It sounds like Precise is a good candidate for Essex: it is an LTS
release, and we have time to ensure that any required bugfixes (that
we don't want to work around) make it into the official release.

If that's agreeable, then e.g. we probably retarget devstack and our
documentation from Oneiric to Precise.  We should probably gate on
Precise as well.

I will be much happier if we just say "we aim to support X"; I don't
really care what X is.  I'm just going to be running OpenStack on the
machine, so I'm not picking my distro e.g. based on how I feel about
Unity.  I'd imagine most users are in a similar camp.

Justin



On Tue, Mar 13, 2012 at 5:00 AM, Thierry Carrez  wrote:
> Justin Santa Barbara wrote:
>> Which operating system(s) are we aiming to support for Essex?  Is the
>> plan to backport the latest libvirt to Oneric, or are we going to wait
>> for Precise?
>
> The question is the other way around: which operating systems aim to
> support Essex ? We try to set the dependencies for OpenStack to a
> reasonable set of versions (generally compatible with the release under
> development of the major Linux distributions), but it's up to the
> distributions themselves to make sure they align if they want to support
> a given version of OpenStack.
>
> Ubuntu will ship Essex in 12.04 LTS. I don't think there are any plans
> to backport it to 11.10. Fedora will support Essex in Fedora 17, etc.
>
> --
> Thierry Carrez (ttx)
> Release Manager, OpenStack
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Random libvirt hangs

[Justin Santa Barbara]

I just today was able to diagnose a libvirt hang.  It appears to be
(similar to) a known bug in libvirt, likely fixed in the latest
Fedora, but it does not appear to be fixed in Ubuntu Oneirc; I think
the fix is in Precise: https://bugs.launchpad.net/nova/+bug/953656

I believe this is the upstream bug:
https://bugzilla.redhat.com/show_bug.cgi?id=757382


Which operating system(s) are we aiming to support for Essex?  Is the
plan to backport the latest libvirt to Oneric, or are we going to wait
for Precise?

Justin

On Mon, Mar 12, 2012 at 1:31 PM, Daniel P. Berrange  wrote:
> On Mon, Mar 12, 2012 at 02:17:49PM -0400, David Kranz wrote:
>> In the spirit of Jay's message, we have a long-running cluster
>> (diablo/kvm) where about once every 3-4 weeks a user will complain
>> that she cannot connect to a vm. Examining the compute node shows
>> that libvirt-bin is hung. Sometimes restarting this process fixes
>> the problem. Sometimes it does not, but rebooting the compute node
>> and then the vm does. I just heard from people in my company
>> operating another cluster (essex/kvm) that they have also seen this.
>> I filed a bug about a month ago
>>
>> https://bugs.launchpad.net/nova/+bug/931540
>>
>> Has any one been running a kvm cluster for a long time with real
>> users and never seen this issue?
>
> There have been various scenarios which can cause libvirtd to hang in
> the past, but that bug report doesn't have enough useful data to
> diagnose the issue. If libvirtd itself is hanging, then you need to
> attach to the daemon with GDB, and run 'thread apply all bt' to collect
> stack traces across all threads. Make sure you have debug symbols
> available when you do this.
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Libvirt Snapshots

[Justin Santa Barbara]

Thanks for the background.  My thoughts:

* Telling a user to build from source isn't a great option for them -
it's painful, they don't get updates automatically etc.  Are we going
to start distributing packages again?

* I can't imagine any open source project removing functionality like
the QEMU pass-through - those sort of tactics are confined to
commercial products that are trying to lock you in, in my experience.
I'm sure libvirt would never try that, and if they did I'm sure it
would rapidly be forked.

* I can imagine that libvirt might switch to the JSON QEMU monitor
protocol (I don't understand why it isn't already, I'm probably wrong
here?).  Thankfully the QEMU JSON protocol also has a pass-through,
but we would ideally switch to the better protocol when/if libvirt
switches over.

* It sounds like this is the exact use-case for which the pass-through
functionality was designed - libvirt doesn't support what we need (at
least in commonly-utilized versions), we have some additional
potential roadmap features that may not be supported, so we _should_
use the pass-through.


Once commonly distributed versions of libvirt have the functionality
we need, then we can remove this functionality from OpenStack, but we
do have to fix this for Essex.  Hopefully this can continue long-term:
OpenStack can develop the functionality that clouds need that libvirt
is missing, in Python instead of in C, and then libvirt can
incorporate this functionality where you want to.

Justin


> The command line and monitor passthrough capability is there as a means
> to perform short term workarounds/hacks on a specific version of libvirt,
> for functionality not already available via a supported API.
>
> We make absolutely no guarentee that usage of passthrough capabilities
> will continue to work correctly if you upgrade to a newer libvirt. As
> such it is not something you really want to use as the basis for a
> production release of OpenStack that expects compatibility with future
> libvirt releases
>
> http://berrange.com/posts/2011/12/19/using-command-line-arg-monitor-command-passthrough-with-libvirt-and-kvm/
>
...
>
> And it possible for people to build any new libvirt release for
> any old Ubuntu release they desire to support. So if someone really
> wants to use new OpenStack on older distros they aren't locked out.
>
> Regards,
> Daniel

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Libvirt Snapshots

[Justin Santa Barbara]

Pedantry: It's QEMU/KVM, not libvirt, that holds the disks open.  The
pedantry does make a difference here I think...

A more sustainable option than being on the bleeding edge of libvirt
may be to try to bypass libvirt and issue those safe QEMU monitor
commands directly.  Libvirt would normally prevent this, but it looks
like there is a QEMU monitor command built into libvirt:
http://blog.vmsplice.net/2011/03/how-to-access-qemu-monitor-through.html

So what does a libvirt snapshot actually do?  Here's the code:
http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_driver.c;h=be678f36527fd7918a1aaa69f54a6c939f258714;hb=HEAD

qemuDomainSnapshotCreateXML is the main function; it looks like we
want to pass in VIR_DOMAIN_SNAPSHOT_CREATE_DISK_ONLY
That calls qemuDomainSnapshotCreateDiskActive, which then pauses the
VM, snapshots each disk in series using the QEMU monitor, and then
resumes the VM.

We should do the same thing, but better:

Suspend the domain using libvirt
Snapshot each disk we want to snapshot _in parallel_, by using the
libvirt QEMU monitor pass-through.  Remote volumes could use the
correct driver so that e.g. a SAN disk could make use of a hardware
snapshot,
Resume the domain using libvirt

For essex, it  sounds like step 2 probably means "snapshot the root
volume only" using qemu, and don't snapshot remote volumes.

Post Essex:
We could try to avoid suspending / resuming the domain, by using a
filesystem thaw.  It looks like libvirt has some (very new) support
for this, but as this relies on a QEMU guest agent, I suspect we'd do
better to roll our own here that could be cross-hypervisor
We can allow selective snapshotting of disks.  On a database, for
example, you really do want to snapshot all the disks together.
We could also support "optimistiic" snapshots, which just do a
snapshot without suspending anything.  The use-case is that the caller
issues a filesystem thaw e.g. over SSH,  then requests a snapshot on
each disk that they care about though Openstack, then resumes normal
operation over SSH.

I actually like that third option the best.  I'd like to be able to
snapshot the root volume like I do any other volume, and I'd like to
be in control of the suspension mechanism.  Suspending the entire VM
is a little extreme, particularly if I'm using a
filesystem/application that offers me a lower-impact alternative!
However, an easy way to "just snapshot everything" with a single call
is also attractive, and I'd imagine people using OpenStack directly
(rather than though code) would definitely use that.

Justin


On Fri, Mar 9, 2012 at 12:18 AM, Daniel P. Berrange  wrote:
>
> On Thu, Mar 08, 2012 at 06:02:54PM -0800, Vishvananda Ishaya wrote:
> > So I could use some specific feedback from kvm/libvirt folks on the 
> > following questions:
> >
> > a) is it safe to use qemu-img to create/delete a snapshot in a disk file 
> > that libvirt is writing to.
> > if not:
> > b) is it safe to use qemu-img to delete a snapshot in a disk file that 
> > libvirt is writing to but not actively using.
> > if not:
> > c) is it safe to use qemu-img to create/delete a snapshot in a disk file 
> > that libvirt has an open file handle to.
>
> Sadly, the answer is no to all those questions. For Qcow2 files, using
> internal snapshots, you cannot make *any* changes to the qcow2 file,
> while QEMU has it open. The reasons are that QEMU may have metadata
> changes pending to the file which have not yet flushed to disk, and
> second, creating/deleteing the snapshot with qemu-img may cause
> metadat changes that QEMU won't be aware of. Either way you will likely
> cause corruption of the qcow2 file.
>
> For these reasons, QEMU provides monitor commands for snapshotting,
> that libvirt uses whenever the guest is running. Libvirt will only
> use qemu-img, if the the guest is offline.
>
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Basic networking/configuration woes

[Justin Santa Barbara]

> The instance firewall should be configured to only allow DHCP
> responses from the IP it believes to be the correct DHCP server.
> Perhaps it has the wrong idea?

Ah yes - it probably does because of my unusual network "subrange" config.

It seems I'm not "missing something", so I've proposed a init script
for contrib that can apply the configuration from a config_drive.  It
solved my problem, and it avoids all of the DHCP/cloud-init complexity
without needing direct injection.  Still very much a version 1 though:
https://review.openstack.org/#change,4498

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Basic networking/configuration woes

[Justin Santa Barbara]

> If you're going to go the cloud-init route... you wouldn't need DHCP, right?  
>  There should be iptables rules to allow you to talk to the metadata service 
> over 169.254.*  (And linux should give you a default link-local address that 
> allows you to talk to the MD service magically)
>
> Do you have a non-nova DHCP server running as well?

Yes, I do have a non-nova DHCP server.  However, even if I didn't, and
even if iptables allowed talking to 169.254 with the magic link-local,
cloud-init still couldn't configure the IP address... :-(

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Basic networking/configuration woes

[Justin Santa Barbara]

Thanks for chipping in.

I have contributed a patch (which has merged) which should allow you
to stop editing the SQL:  https://review.openstack.org/#change,3816
With that, you should be able to pass the full range, with an
additional argument specifying the subset that nova controls:
e.g.-fixed_cidr=10.200.0.0/16

When I boot my VM, I think it gets a real address from my DHCP server
(because the VM can reach the DHCP server), but not the address nova
assigned it!  I believe the nova iptables rules mean that the machine
can't then do TCP/IP, but even if I am wrong/could overcome that, I
don't think cloud-init could then configure the correct address.

Justin


On Thu, Feb 23, 2012 at 3:38 PM, Chris Behrens  wrote:
> I'd assume FlatDHCPManager works much like FlatManager, but maybe I'm wrong.  
> I use FlatManager and I always end up having to modify the fixed_ips table 
> manually after running nova-manage because I think I'm trying to do something 
> similar as you.  I have a /23... and I want to give nova a /25 out of it.   
> Though I'm giving nova a /25, it's still really a /23.   I use nova-manage to 
> add my /23 and then I edit the fixed_ips table and mark a lot of addresses as 
> 'reserved'... or just remove them altogether.  (When I try to specify the /25 
> to nova-manage, it doesn't go so well)
>
> As far as 169.254...  you can reach that without any address assigned.  Your 
> NIC should receive a link local address when there's no other IP assigned 
> which is in the 169.254.* range.
>
> Not sure if that helped much :)
>
> - Chris
>
> On Feb 23, 2012, at 3:12 PM, Justin Santa Barbara wrote:
>
>> I'm trying to use OpenStack in what I think to be the typical
>> non-public-cloud deployment, and my experience is not what it
>> could/should be.  I'm hoping someone can point me to the "right way",
>> or we can figure out what needs to change.
>>
>> My wishlist:
>> * I want my instances to be on "my network" e.g. 10.0.0.0/16
>> * As Nova can't pull IPs from my DHCP server, I'm willing to allocate
>> it a sub-range, e.g. 10.200.0.0/16
>>
>> First decision: Choosing a networking mode:
>> * I don't want / need VLANs
>> * If I use FlatDHCPManager, I can't do the subrange stuff - it seems
>> that this mode assumes it controls the entire address range.
>> * So it's FlatManager.  It works, but now I don't have DHCP, so I just
>> have to inject info into the instance.
>>
>> Next decision: How to inject info (at least the IP address):
>> * Supposedly the 'right way' is to use cloud-init.  It looks like I'd
>> still need DHCP before I can reach 169.254..., and I don't have that.
>> It looks like cloud-init can't do network configuration even if nova
>> passed the information in.  And I'd be locked into cloud-init images -
>> no Windows, no Debian etc.
>> * The next best way is config_drive.  It looks like I'd have to bundle
>> my own image.  Maybe I could use cloud-init, maybe with an OVF
>> formatted config_drive, but even then I couldn't configure networking
>> (?)
>> * So now I'm back to file injection.  That just works.
>>
>> So now I'm using FlatManager and file injection; and yet I feel this
>> is the dodgy back alley of OpenStack, and I should be in the well-lit
>> nice area.  I worry that things like file injection and FlatManager
>> are less favored and may be deprecated in future.  But every time I
>> try to do things "right" I just waste a lot of time and make no
>> progress.
>>
>> Yet I feel I didn't really have a choice here.   How are other people
>> making this work?  What is the "right way"?
>>
>> Justin
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Basic networking/configuration woes

[Justin Santa Barbara]

I'm trying to use OpenStack in what I think to be the typical
non-public-cloud deployment, and my experience is not what it
could/should be.  I'm hoping someone can point me to the "right way",
or we can figure out what needs to change.

My wishlist:
* I want my instances to be on "my network" e.g. 10.0.0.0/16
* As Nova can't pull IPs from my DHCP server, I'm willing to allocate
it a sub-range, e.g. 10.200.0.0/16

First decision: Choosing a networking mode:
* I don't want / need VLANs
* If I use FlatDHCPManager, I can't do the subrange stuff - it seems
that this mode assumes it controls the entire address range.
* So it's FlatManager.  It works, but now I don't have DHCP, so I just
have to inject info into the instance.

Next decision: How to inject info (at least the IP address):
* Supposedly the 'right way' is to use cloud-init.  It looks like I'd
still need DHCP before I can reach 169.254..., and I don't have that.
It looks like cloud-init can't do network configuration even if nova
passed the information in.  And I'd be locked into cloud-init images -
no Windows, no Debian etc.
* The next best way is config_drive.  It looks like I'd have to bundle
my own image.  Maybe I could use cloud-init, maybe with an OVF
formatted config_drive, but even then I couldn't configure networking
(?)
* So now I'm back to file injection.  That just works.

So now I'm using FlatManager and file injection; and yet I feel this
is the dodgy back alley of OpenStack, and I should be in the well-lit
nice area.  I worry that things like file injection and FlatManager
are less favored and may be deprecated in future.  But every time I
try to do things "right" I just waste a lot of time and make no
progress.

Yet I feel I didn't really have a choice here.   How are other people
making this work?  What is the "right way"?

Justin

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] DeltaCloud

[Justin Santa Barbara]

In my experience, the API is the most visible yet smallest problem of
working with different clouds.

For example, EC2 and Rackspace Cloud have completely different
approaches to volumes, such that the way you backup your VMs has to be
completely different (disk snapshot vs application level).  Rackspace
Cloud had limited images compared to EC2, so you often couldn't rely
on image-based deployment or the right OS/kernel modules being
available.  Those are the problems that really bite you when trying to
support multiple clouds.

I believe what OpenStack is building is the true cross-cloud API: a
common API with common concepts and functionality.  If you choose the
OpenStack API, I believe there are a half dozen public clouds today
you can use; there are at least as many companies that will help you
build a private cloud.  All with the same API, all with the same basic
behaviour.  By year-end, I suspect the weight of numbers will be
overwhelming.

If you want to add OpenStack support to deltacloud, by all means do
so, but these cross-cloud abstractions are a long-term dead-end, in my
opinion.

Justin


On Tue, Feb 21, 2012 at 8:04 AM, Frans Thamura  wrote:
>
> anyone know deltacloud
>
> i think this is intersting
>
> esp after OpenStack want to make their own standard vs Amazon API :)
>
> i think the foundation need to be hurry and work closely with Apache ;0
>
> http://www.sdtimes.com/APACHE_ADOPTS_NEW_TOP_LEVEL_CLOUD_PROJECT/By_Alex_Handy/About_APACHE_and_CLOUD_and_DELTACLOUD/36373
>
> F
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] OpenStack Java API

[Justin Santa Barbara]

This is awesome.  I was working on a binding myself, but your use of jersey
makes for much less code.

I've extended your work in a github fork.  I added a CLI so that I could
test it out; the few bits of functionality that I added work great and I'm
going to try using it as my primary interface and fixing/adding things that
aren't working.

One goal I have is to do extensions "right".  So we should allow people to
code extensions without changing the core API code (equivalently, we
shouldn't assume that we know all the extensions when we build the API).  I
have an example of how this can be done where extra XML attributes are
returned (which happens on an out-of-the-box server listing); I'm going to
do more work on more advanced scenarios (extra elements, extra REST
endpoints).  I would eventually like to use the (hand-coded) Java models to
generate valid XSD files.

My fork is here:  https://github.com/justinsb/openstack-java-sdk  I'd like
to work together on this!

Justin

---

Justin Santa Barbara
Founder, FathomDB



On Mon, Feb 13, 2012 at 8:53 AM, Luis Gervaso  wrote:

> The Dasein Arch is great and the code is very clean. Congrats for it.
>
> I can't find a fully implementation of OS API.
>
> are using EC2 API to talk with OS?
>
> Cheers!
>
>
>
>
>
> On Sat, Feb 11, 2012 at 8:15 PM, George Reese 
> wrote:
>
>> There's also Dasein Cloud if you are interested at
>> http://dasein-cloud.sf.net.
>>
>> -George
>>
>> On Feb 11, 2012, at 12:28 AM, Monty Taylor wrote:
>>
>> Hi!
>>
>> Awesome, and thanks for the work!
>>
>> Just in case you didn't know about it:
>>
>> http://www.jclouds.org/
>>
>> Is a Java library with multi-cloud support, including OpenStack, which
>> might be a fun place for you to hack - and I know Adrian loves
>> contributors.
>>
>> On the other hand, any amount of Java story for OpenStack is good news.
>>
>> Thanks!
>> Monty
>>
>> On 02/10/2012 12:08 PM, Luis Gervaso wrote:
>>
>> Till i know Nova 1.0 is deprecated, so it will not be implemented.
>>
>>
>> Nova 1.1 is almost implemented (working now with extensions : volumes /
>>
>> snapshots / storagearrays)
>>
>>
>> Nova 2.0 is a must
>>
>>
>> Glance (working now on it, this is the most easy to implement API)
>>
>>
>> Swift Java API from Rackspace is stable enough, so I will integrate at
>>
>> the end.
>>
>>
>> Hope to hear about this roadmap.
>>
>>
>> Luis
>>
>>
>> On Fri, Feb 10, 2012 at 8:56 PM, Marton Kiss >
>> <mailto:marton.k...@gmail.com>> wrote:
>>
>>
>>Hi,
>>
>>
>>Nice start Luis. Do you have some plans to support different OS API
>>
>>versions? Anybody knows about a similar effort to write a PHP client?
>>
>>
>>Regards,
>>
>> Márton Kiss, CTO
>>
>> Xemeti
>>
>>
>>2012/2/10 Luis Gervaso mailto:l...@woorea.es>>:
>>
>> Hi,
>>
>>
>> My name is Luis Gervaso. I just upload a developer preview of
>>
>>OpenStack Java
>>
>> SDK on
>>
>>
>> https://github.com/woorea/openstack-java-sdk/
>>
>>
>> I want to know if other development efforts have been done in this
>>
>> way in
>>
>> order to contribute.
>>
>>
>> Regards
>>
>>
>> --
>>
>>
>> Luis Alberto Gervaso Martín
>>
>> Java EE Architect & Instructor
>>
>> C/ Cuenca 4A, 2ºB
>>
>> Getafe (Madrid)
>>
>> SPAIN
>>
>> l...@woorea.es <mailto:l...@woorea.es>
>>
>>
>> ___
>>
>> Mailing list: https://launchpad.net/~openstack
>>
>> Post to : openstack@lists.launchpad.net
>>
>><mailto:openstack@lists.launchpad.net>
>>
>> Unsubscribe : https://launchpad.net/~openstack
>>
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>> Luis Alberto Gervaso Martín
>>
>> Java EE Architect & Instructor
>>
>> C/ Cuenca 4A, 2ºB
>>
>> Getafe (Madrid)
>>
>> SPAIN
>>
>> mobile: (+34) 627983344
>>
>> l...@woorea.es <mailto:l...@woorea.es>
>>
>>
>>
>> ___
>>
>> Mailing list: https://launchpad.net/~openstack
>>
>> Post to

Re: [Openstack] Getting pagination right

[Justin Santa Barbara]

I definitely agree that the paging logic is questionable; we definitely
should be specifying the sort order if we're relying on it, as I believe a
database is otherwise free to return results however it sees fit.

I'm not convinced that the proposed logic around the queries and update
timestamps is typo-free.  But even if it was correct, the problem with
timestamps is that I don't think you can get a sensible set of results.  If
you order by increasing timestamp, then I think you see items repeatedly if
they are concurrently updated.  If you order by decreasing timestamp, I
think you can miss an item altogether if it's updated in between page
fetches.

I think the 'normal' way to make this work is to order by primary key, and
to have the marker be the PK of the 'last seen row'.  Because the client
can't know what the pk is, the marker is normally just an opaque value.
 It's also (normally) much easier for the DB to sort by primary key than by
any other value.

The behaviour of an iteration using the PK is fairly nice.  You never see a
row twice, so you don't need to de-dup client side.  You are guaranteed to
see all rows, unless they are concurrently inserted or deleted.  If a row is
inserted you'll see it if you haven't yet fetched the page it would be on.
 If a row is deleted you don't see it unless you've already fetched the page
it was on.

Zones may complicate any scheme, though I think both proposed methods could
be made to work.  I think in general the next N rows have to be fetched from
each child zone and then combined (top N), throwing away the extra records.
  One way around this would be to define an order on the child zones, and
then show the records from zone 1, then those from zone 2 etc etc.  You just
need to map from the marker to a child zone, and if you don't get the full
complement of rows you need to merge in the first results from the next zone
as well.

The SQL query using the PK would look like this (and there's no need for an
OFFSET clause):

SELECT *
FROM 
ORDER BY 
WHERE  > @marker
AND ()
LIMIT 

Without additional filters, that's a _very_ efficient query for a sensible
database.

Justin



On Wed, May 25, 2011 at 9:28 AM, Jay Pipes  wrote:

> When a user issues a GET request for a resource collection, it's clear
> that you don't want to return ALL results in the collection. Returning
> hundreds of thousands of records per request isn't particularly a good
> idea.
>
> Therefore, we use the concept of "pagination" -- returning a subset of
> records matching the search criteria.
>
> In Glance's API, we've added this ability (currently in code review),
> but our approach is a bit at odds with the way that the OpenStack
> Compute API is structured. I'd like to explain why I think our
> approach is better, both in terms of usability and in terms of
> efficiency.
>
> First, here is how Nova does its pagination in the OpenStack API.
>
> 1) Look for a "marker" query parameter in the request
> 2) Look for a "limit" query parameter in the request, otherwise use a
> default max limit.
> 3) If marker parameter is found, the value of the "marker" query
> parameter is assumed to be an *integer* and assumed to be the
> *identifier of the last object on the previous "page" of results*
> 4) Request *all* results
> 5) In the code, find the "page" of results by looking for the object
> in the set of results that matches the marker parameter
> 6) When found, grab the next object in the result set, plus X objects,
> where X is the limit parameter
>
> The relevant code is in /nova/api/openstack/common.py, and looks like so:
>
> def limited_by_marker(items, request, max_limit=FLAGS.osapi_max_limit):
>"""Return a slice of items according to the requested marker and
> limit."""
>
> 
>
>limit = min(max_limit, limit)
>start_index = 0
>if marker:
>start_index = -1
>for i, item in enumerate(items):
>if item['id'] == marker:
>start_index = i + 1
>break
>if start_index < 0:
>raise webob.exc.HTTPBadRequest(_('marker [%s] not found' %
> marker))
>range_end = start_index + limit
>return items[start_index:range_end]
>
> There are three MAJOR problems with the above method of pagination:
>
> 1) Very inefficient
>
> The Python code does the job of determining the page of results to
> find. This means that the database query returns EVERY row in the
> resultset, instead of only returning a small subset of the results
> using the LIMIT X OFFSET Y SQL construct. With hundreds of thousands
> of records in the various databases, this method is simply not
> scalable.
>
> 2) There is no ordering of the results in the queries the Nova
> database API makes
>
> The only queries in the Nova database API that order results is the
> instance_type_get_all() method.
>
> Without an order to the search results, paging is almost meaningless.
> A page of results depends on the order of the list of results. The
> reason the existing 

Re: [Openstack] Proposal for Justin Santa Barbara to join Nova-Core

[Justin Santa Barbara]

Great news - thank you all!

Justin




On Fri, Apr 15, 2011 at 1:24 AM, Soren Hansen  wrote:
> 2011/4/12 Soren Hansen :
>> +1 from me, too.
>>
>> As per the process, if no-one objects within 5 business (my
>> interpretation) days, i.e. before Thursday, I'll get Justin added to
>> the nova-core team.
>
> Noone objected, so this is now done. Congrats, Justin. Welcome to the team :)
>
> --
> Soren Hansen
> Ubuntu Developer    http://www.ubuntu.com/
> OpenStack Developer http://www.openstack.org/
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Summit Talk: Information session on Zones? Any interest?

[Justin Santa Barbara]

I'm not sure what you mean by an 'informational session'?  That sounds
rather didactic - is everything sufficiently agreed for a session of
this type?  I was under the impression that a lot of the details of
zones were not yet agreed, and that a lot of the use cases we would
like to see in future were not yet designed in.

I would very much appreciate a "Current State of Zones" presentation
that would precede a discussion session on what zones should look like
in Diablo and beyond.  We want to be sure that our zones design
supports our ambitions e.g. federation / bursting.

Or is this what you're suggesting?  In which case +1 !

Justin






On Thu, Apr 14, 2011 at 6:07 AM, Sandy Walsh  wrote:
> I've been getting a lot of questions about Zones lately.
> How much interest is there for an informational session on Zones and, I
> guess, Distributed Scheduler and roadmap?
> (pending an available slot at the summit ... things are filling up quickly I
> gather)
> -S
>
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of
> the
> individual or entity to which this message is addressed, and unless
> otherwise
> expressly indicated, is confidential and privileged information of
> Rackspace.
> Any dissemination, distribution or copying of the enclosed material is
> prohibited.
> If you receive this transmission in error, please notify us immediately by
> e-mail
> at ab...@rackspace.com, and delete the original message.
> Your cooperation is appreciated.
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Proposal to defer existing OS API v1.1 to Diablo, for greater stability in Cactus

[Justin Santa Barbara]

I think there are a few distinct issues:

1) XML support.  I was thinking that most XML issues would also be issues in
the JSON, so validating the XML will also serve as validating the JSON.  I'd
appreciate an example here, but I agree in general that focusing on JSON is
acceptable - as long as its not just that we don't see the problems in JSON
because we're not validating it as thoroughly.

2) Missing features.  I don't think of this as an API issue, unless these
are supported features which we simply aren't exposing.  My understanding is
that it's the underlying support that's holding this back, not the API
itself.

3) The v1.1 schema changes.  To be honest, I'm still confused here.  I think
you want to maximize compatibility with Cloud Servers v1.0, so it sounds
like you would also want a minimal v1.1 that was just the bare minimum
additional features to support additional features in nova (e.g.
changePassword action)  I don't really understand where the 'nice to have'
(e.g. atom links) changes came from in that case.  I think everyone would
support a minimal set of changes to the CloudServers v1.0 API; even if
people would rather have their own OpenStack API long-term, supporting the
two existing cloud APIs is an obvious win.


> Given this, what makes the most sense to me is to focus on stability for
version 1.0 API excluding XML support.  The bindings that are out there have
strong support for the JSON data formats in v1.0.  As suggested, I think we
call the current mostly implemented 1.1 API experimental so as to give
access to any features that we need that are only in 1.1.

I think v1.0 is a good API, and has the advantage that a lot of existing
clients work with it.  I would personally like XML support to be in there,
but I understand that we need to make sacrifices, so accept that the focus
should be on JSON in v1.0.

I think calling v1.1 experimental and discussing it at the Design Summit is
the right thing to do.  If you need to extend v1.0 (e.g. to support
restartServer in "v1.0"), then I think you can make the call on
re-numberings.

> One point I would like to ask about is how important is XML to the 1.0
implementation?

I personally would like to see XML support shipped, even if marked
experimental and not 100%.  I had thought that getting the XML right would
mean we got the JSON right also, and it's easier to test the XML.  However,
I totally agree that JSON should be our top priority.  Given the time
situation, I think XML work should only be done to help the JSON (e.g.
adding namespaces to the XML lets us do a schema validation, which
indirectly helps us check the JSON => yes.  Fixing XML output where the JSON
output is already valid => not a priority)


Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Proposal to defer existing OS API v1.1 to Diablo, for greater stability in Cactus

[Justin Santa Barbara]

I have been doing some work on getting XML support in our APIs in better
shape in the OS API.  However we feel about XML, the big benefit is that it
gives us really strict validation for free.  I am only at the point of
adding namespace declarations, but this has already uncovered a lot of
issues where our current output isn't entirely to spec (as defined by the
CloudServers v1.0 API and the OpenStack v1.1 API)  This applies to our JSON
output as much as it applies to our XML output.

The Titan team have been doing great work, but I think we've handed them a
death march if they're to complete all this work for both v1.0 and v1.1 in
time.  I'd rather they were able to focus on one API, or even have time for
the many other bugs we still have :-)

The big problem with the v1.1 spec as I see it, is that it includes a lot of
changes which I don't believe are strictly necessary (hopefully simply
because I don't understand them).  For example:

   - The atom links collection (images, flavors, servers)
   - Random changes (e.g. limits, addresses), where e.g. attributes move to
   their parent elements.  Probably more correct, but is it really worth the
   implementation cost on our side or on the caller's side?
   - Changing from a numeric ID to a URI (e.g. on images), but then not
   really using the URI (or even having two URIs)
   - And these are just the 'known knowns' :-)

To reiterate - this is in no way a criticism of the Titan team.  They've got
many of these already implemented, and with luck and great effort they could
deliver all of them.  If they're willing and able to do that, great.

However, we need to have an API that works 100% in Cactus, and I don't think
we should demand that the Titan team death-march to deliver _two_ APIs.  I
also think if we want to get good client support, we should minimize the
number of changes we require API callers to make, particularly if we don't
have a big carrot to offer them in return.

I would like to propose a fallback scenario for Cactus release management
purposes, which would free up Titan resources and get us a better, more
stable API with greater client support:

   - We defer any non-essential changes in the V1.1 API to post-Cactus.
   - We can then discuss these thoroughly at the design summit (I do not
   believe we had a design summit discussion on these API changes, for timing
   reasons)
   - We make the V1.1 API a minimal extension to V1.0, to support the Cactus
   features we're going to ship.  For example, Brian Waldon pointed out that we
   would need a new element for the changePassword action, and for extensions
   also.
   - These minimal differences would be driven by the people that need them
   (primarily the Titan team)  I am confident they're not going to be
   introducing things that are not strictly necessary at this stage of the game
   :-)
   - We may have to postpone extensions that inject additional content into
   existing elements, and stick to extensions that extend the URI space, so
   that the XML schema for V1.1 is minimal.  Extension of existing elements
   probably warrants a Design Summit discussion anyway.  We do not yet have any
   (non-test) extensions that inject content.
   - We would rename the current V1.1 API to V1.2 or V2.0 - we are just
   deferring it to Diablo, not abandoning it.
   - We could still ship the renamed API as "experimental", even in Cactus
   - The goal is to focus resources on one API that will then work 100%.
   - Yes, it's still sort of going to be two APIs, but if we're really lucky
   we might get away with almost no 'switching' code.  For example, if we _add_
   a URI or an action, a V1.0 client will never discover it.
   - In particular, we want to avoid the output format changing between
   versions wherever we can.
   - I hope by virtue of this approach that Cactus will be 100% compatible
   with existing v1.0 (CloudServers) clients
   - I hope also that V1.0 clients can just switch to use the V1.1 endpoint
   when they're ready, and need make code changes only for things which have
   actually changed.


I believe this is entirely in line with our goals for Cactus.  I am less
confident that the current V1.1 API is in line with our Cactus goals.

Thoughts?  Anyone from the Titan team want to comment on how they feel about
the timetable for delivering the APIs in Cactus?  ttx?


Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Feature Freeze status

[Justin Santa Barbara]

>
> We'd all love to have more people working on OpenStack.
>

Though I'm loathe to continue this debate, I would suggest that blueprints
may not be helping us on this front.

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Feature Freeze status

[Justin Santa Barbara]

>
> Rubbish. Open development means knowing the general directions and
> specifications that people are working on by open discussions, open
> blueprints/specs, and active communication between teams. I can go to
> github and see how many people "forked this" (ugh.). That doesn't give
> me any clue as to what people are attempting to do with the code in
> the long term.
>

I am not the biggest fan of LP, but the 'recent commits' feature on LP is
awesome.  If I want to know what people are really working on, I just look
at the most recent commits:

Cerberus is working on something cool involving notifications, presumably
the first step in addressing the issue that we're almost entirely
request-driven
The titan team is fixing the date/time parsing bug with glance images
The titan team is working on support for changing passwords
ZulCss is working on LXC support
The USC team is working on HPC stuff on hardware that's unfairly cool
Termie's working on libvirt-snapshot support
Termie's clarifying the docstring rules

etc.

That gives me a much better feel for the pulse of the project than anything
else - the blueprints feel like reading a printed newspaper - full of
yesterday's news.  It doesn't answer why, but we should probably make a
bigger push on the blueprints to make sure they always answer the "why"
question.

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Feature Freeze status

[Justin Santa Barbara]

No objection to a discussion during the summit, but I've been able to watch
all of these branches and others evolve here:
https://code.launchpad.net/nova

For example, when I wanted to add VNC support because my system wouldn't
boot, it was easy for me to look at the vnc_console branch and get the
libvirt XML magic from there.  If I was feeling brave, I could have grabbed
the whole branch and merged it into my tree.  That feels _very_ open to me.

 Just another benefit of bazaar being a centralized version
control system, I guess 

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

>
>  So I'm going to implement a partition of 1 billion integers per zone,
> which should allow for approximately 1 billion zones, given a 64 bit integer
> for the PK. This should be workable for now, and after the design summit,
> when we've come to a consensus on changing the API to accept something other
> than integer identifiers, it should not be too difficult to retrofit.
>

The type of a server @id in CloudServers is xsd:int, which is a 32-bit
signed integer:
http://docs.rackspacecloud.com/servers/api/v1.0/xsd/server.xsd

So if you have 1 billion integers per zone, you only get 2 zones.  You can
have 4 if you're willing to go negative, but surely it's too early in the
campaign. 

I think the only way long-term we're going to have CloudServers
v1.0 compatibility is by having a proxy that bridges between legacy APIs
(EC2 and CS) and future APIs (OpenStack).  I'm guessing that proxy will have
to be stateful to implement mappings of server IDs etc.  Yes, this sucks.
 But at some stage you have to say "you know, maybe 640KB wasn't enough, and
we have to make some changes"

How about this as a solution: use ranges as you suggest, but let the
starting points for the zone-ids that child-zones draw from be
customer-configured.  We're pushing the problem onto the end-user, but they
probably know best anyway, and we don't really expect anyone to use
sub-zones in anger anyway until Diablo or later, right?

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

>
> Migrations outside of a zone would require a new
> instance ID, but this should be fine, since other things would also
> change (such as IP, available volumes, ...). A cross-zone migration
> will be more of a copy+delete than a proper move.


+1 on this.  If the IP is changing, there's little point in trying to keep
the ID the same.  Great point.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

Indeed, migrations are a major fly in the ointment for any strategy for
meaningful naming (i.e. anything that doesn't use a central DB).  It's not
clear to me with cross-zone migrations that we (a) can keep the same ID and
(b) want to keep the same ID even if we could...

We could look at tricks like storing a 'redirect' pointer after a migration.
 I think it all boils down to our use-case for migrations:

   - If it's the cloud provider that notices that one machine is overloaded
   / failing / whatever and wants to move a VM to another host in the same
   zone, that should keep the same ID because it should be transparent to the
   user, and I think this shouldn't be a problem, because each zone has a DB.
   - If the cloud provider has a problem affecting an entire zone (e.g. 5
   more minutes of UPS power), can they live migrate those machines to another
   zone?  This could be problematic, and is where redirect pointers might have
   to come in.  So the 'parent zone' would have to know that 'childzone1' is
   down and those machines are potentially now scattered amongst other zones.
   - If it's the user that wants to migrate a machine e.g. because they are
   fed up with the AWS datacenter and want to go to a competitor, then we don't
   necessarily have to guarantee the same ID.  This presumes this is indeed a
   supported scenario; it may be that even if we make this easy, we don't
   support _live_ migration here.

Do we have any scoping on the use cases for migrations?

In my suggestion, I wasn't really talking about an external registry (other
than DNS to locate the top-level zones)  I don't think my proposed solution
addresses the issue of changing IDs in cross-zone migrations.

Looks like we might need an extended session for this at the Design Summit
:-)

Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

> Pure numeric ids will not work in a federated model at scale.
>

Agreed


> Maybe I'm missing something, but I don't see how you could inject a
> collision ID downstream - you can just shoot yourself in your own foot.


I think that you can get away with it only in simple hierarchical
structures.  Suppose cloud users are combining multiple public clouds into
their own 'megaclouds'.  If I'm an evil public cloud operator, I can start
handing out UUIDs that match any UUIDs I can discover on the Rackspace
cloud, and anyone that has constructed a cloud that combines my cloud and
Rackspace would have collisions.  Users wouldn't easily know who to blame
either.

The other option apart from UUID is a globally unique string prefix.  If
> Rackspace had 3 global API endpoints (ord, dfw, lon) each with 5 zones,
> the ID would need to be something like rax:dfw:1:12345 (I would actually
> want to hash the zone id "1" portion with something unique per customer so
> people couldn't coordinate info about zones and target attacks, etc.).
> This is obviously redundant with the Rackspace URI since we are
> representing Rackspace and the region twice, e.g.
> http://dfw.servers.rackspace.com/v1.1/12345/servers/rax:dfw:1:6789.
>

I am in favor of this option, but with a few tweaks:

1) We use DNS, rather than inventing and administering our own scheme
2) I think the server ID looks like
dfw.rackspace.com/servers/a282-a6-cj7aks89.  It's not necessarily a valid
HTTP endpoint, because there's a mapping to a protocol request
3) The client maps it by "filling in" the http/https protocol (or whatever
protocol it is e.g. direct queuing), and it fills in v1.1 because that's the
dialect it speaks.
4) Part of the mapping could be to map from a DNS name to an endpoint,
perhaps using _srv records (I'm sure I'm mangling all the terminology here)
5) This also allows a form of discovery ... if I tell my cloud controller I
want to use rackspace.com, it can then look up the _srv record, find the
endpoint (e.g. openstack.rackspace.com), then do a zone listing request and
find child zones etc.  If I ask my monitoring system to monitor "
rackspace.com/servers/a6cj7aks89", it knows how to map that to an openstack
endpoint.  Auth is another story of course.

Using strings also means people could make ids whatever they want as long
> as they obeyed the prefix/suffix.  So one provider could be
> rax:dfw:1:12345 and another could be osprovider:8F792#@*jsn.  That is
> technically not a big deal, but there is something for consistency and
> simplicity.


True.  We could restrict the character set to A-Z,0-9 and a few other 'safe
characters' if this is a real problem.  We probably should eliminate
difficult-to-encode characters anyway, whether encoding means umlauts or
url-encoding.


> The fundamental problem I see here is URI is intended to be the universal
> resource identifier but since zone federation will create multiple URIs
> for the same resource, the server id now has to be ANOTHER universal
> resource identifier.
>

I think the server ID should be the unique identifier, and is more important
than the REST representation.  I think we should avoid remapping the URI
unless we have to... (more later)

It will be obvious in which deployment the servers live.  This will
> effectively prevent whitelabel federating.  UUID would be more opaque.
>

Whitelabel federation for reselling an underlying provider can easily be
done by rewriting strings: id.replace("rackspace.com", "a.justinsbcloud.com
").replace("citrix.com", "b.justinsbcloud.com").  I suspect the same
approach would work for internal implementation zones also.  The truly
dedicated will discover the underlying structure whatever scheme you put in
place.

Would users ever get returned a "downstream" URI they could hit directly?
>

So now finally I think I can answer this (with my opinion)... Users should
usually get the downstream URI.  Just like DNS, they can either use that URI
directly, or - preferably - use a local openstack endpoint, which acts a bit
like a local DNS resolver.  Your local openstack "proxy" could also do
things like authentication, so - for example - I authenticate to my local
proxy, and it then signs my request before forwarding it.  This could also
deal with the billing issue - the proxy can do charge-back and enforce
internal spending limits and policies, and the public clouds can then bill
the organization in aggregate.

If you need the proxy to sign your requests, then you _can't_ use the
downstream URI directly, which is a great control technique.

Some clouds will want to use zones for internal operational reasons, and
will want to keep the inner zones secret.  So there, we need something like
NAT: the front-end zone translates between "public" and "private" IDs as
they travel in and out.  How that translation works is
deployment-dependent... they could have a mapping database, or could try to
figure out a function which is aware of their internal structure to do thi

Re: [Openstack] Openstack API - Volumes?

[Justin Santa Barbara]

I am happy to do a first pass over the volume API.  I can do it as an
extension or not, whatever is more likely to be merged.

My personal feeling is that we shouldn't use extensions for something this
core, but we can do a 'full lifecycle' if we want: I'll write it as an
extension, we work on it as a community, then we promote it into core
if/when it is ready.

The thing I don't like about the extension promotion process is that it
breaks clients when the volume API is promoted.  But I'm all for trying it
and seeing what happens.

Justin




On Tue, Mar 22, 2011 at 1:20 PM, Jay Pipes  wrote:

> On Tue, Mar 22, 2011 at 3:21 PM, Justin Santa Barbara
>  wrote:
> > So: When can we expect volume support in nova?  If I repackaged my
> volumes
> > API as an extension, can we get it merged into Cactus?
>
> I would personally support this.
>
> Wasn't one of the ideas of the extensions API to provide a bit of a
> playground for features to bake that, at some future time, be made a
> core resource endpoint in the OpenStack API? This would be a perfect
> example of that, no?
>
> -jay
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

Let's take a leadership position here and go with strings; we're not
breaking Amazon's API.  AWS will have to make the same changes when they
reach our scale and ambition :-)

We should also start engaging with client tools, because we're never going
to be 100% EC2 compatible.  At the least, our endpoints will be different.

I think we should discuss this at the Design Summit, and then make an effort
on this front as part of Diablo.



On Tue, Mar 22, 2011 at 12:58 PM, Vishvananda Ishaya
wrote:

> Yes, that is what they say,  Unfortunately all of the ec2 tools expect the
> current format that they are using to various degrees.
>
> Some just need the proper prefix (euca2ools)
> Others need the prefix + hex (elasticfox, irrc)
> Others allow a string but limit it to 11 chars, etc.
>
> So to keep compatibility we are stuck mimicking amazon's string version for
> now.
>
> Vish
>
> On Mar 22, 2011, at 12:51 PM, Justin Santa Barbara wrote:
>
> EC2 uses xsd:string for their instance id.  I can't find any additional
> guarantees.
>
> Here's a (second hand) quote from Amazon:
>
>
> http://serverfault.com/questions/58401/is-the-amazon-ec2-instance-id-unique-forever
> "Instance ids are unique. You'll never receive a duplicate id. However, the
> current format of the instance id is an implementation detail that is
> subject to change. If you use the instance id as a string, you should be
> fine."
>
> So, strings it is then? :-)
>
>
>
> On Tue, Mar 22, 2011 at 12:44 PM, Vishvananda Ishaya <
> vishvana...@gmail.com> wrote:
>
>> The main issue that drove integers is backwards compatibility to the
>> ec2_api and existing ec2 toolsets.  People seemed very opposed to the idea
>> of having two separate ids in the database, one for ec2 and one for the
>> underlying system.  If we want to move to another id scheme that doesn't fit
>> in a 32 bit integer we have to provide a way for ec2 style ids to be
>> assigned to instances, perhaps through a central authority that hands out
>> unique ids.
>>
>> Vish
>>
>> On Mar 22, 2011, at 12:30 PM, Justin Santa Barbara wrote:
>>
>> The API spec doesn't seem to preclude us from doing a fully-synchronous
>> method if we want to (it just reserves the option to do an async
>> implementation).  Obviously we should make scheduling fast, but I think
>> we're fine doing synchronous scheduling.  It's still probably going to be
>> much faster than CloudServers on a bad day anyway :-)
>>
>> Anyone have a link to where we chose to go with integer IDs?  I'd like to
>> understand why, because presumably we had a good reason.
>>
>> However, if we don't have documentation of the decision, then I vote that
>> it never happened, and instance ids are strings.  We've always been at war
>> with Eastasia, and all ids have always been strings.
>>
>> Justin
>>
>>
>>
>>
>> On Tue, Mar 22, 2011 at 12:20 PM, Paul Voccio 
>> wrote:
>>
>>> I agree with the sentiment that integers aren't the way to go long term.
>>> The current spec of the api does introduce some interesting problems to
>>> this discussion. All can be solved. The spec calls for the api to return
>>> an id and a password upon instance creation. This means the api isn't
>>> asynchronous if it has to wait for the zone to create the id. From page
>>> 46
>>> of the API Spec states the following:
>>>
>>> "Note that when creating a server only the server ID and the admin
>>> password are guaranteed to be returned in the request object. Additional
>>> attributes may be retrieved by performing subsequent GETs on the server."
>>>
>>>
>>>
>>> This creates a problem with the bursting if Z1 calls to Z2, which is a
>>> public cloud, which has to wait for Z3-X to find out where it is going be
>>> placed. How would this work?
>>>
>>> pvo
>>>
>>> On 3/22/11 1:39 PM, "Chris Behrens"  wrote:
>>>
>>> >
>>> >I think Dragon got it right.  We need a zone identifier prefix on the
>>> >IDs.  I think we need to get away from numbers.  I don't see any reason
>>> >why they need to be numbers.  But, even if they did, you can pick very
>>> >large numbers and reserve some bits for zone ID.
>>> >
>>> >- Chris
>>> >
>>> >
>>> >On Mar 22, 2011, at 10:48 AM, Justin Santa Barbara wrote:
>>> >
>>> >> I think _if_ we want to stick wit

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

EC2 uses xsd:string for their instance id.  I can't find any additional
guarantees.

Here's a (second hand) quote from Amazon:

http://serverfault.com/questions/58401/is-the-amazon-ec2-instance-id-unique-forever
"Instance ids are unique. You'll never receive a duplicate id. However, the
current format of the instance id is an implementation detail that is
subject to change. If you use the instance id as a string, you should be
fine."

So, strings it is then? :-)



On Tue, Mar 22, 2011 at 12:44 PM, Vishvananda Ishaya
wrote:

> The main issue that drove integers is backwards compatibility to the
> ec2_api and existing ec2 toolsets.  People seemed very opposed to the idea
> of having two separate ids in the database, one for ec2 and one for the
> underlying system.  If we want to move to another id scheme that doesn't fit
> in a 32 bit integer we have to provide a way for ec2 style ids to be
> assigned to instances, perhaps through a central authority that hands out
> unique ids.
>
> Vish
>
> On Mar 22, 2011, at 12:30 PM, Justin Santa Barbara wrote:
>
> The API spec doesn't seem to preclude us from doing a fully-synchronous
> method if we want to (it just reserves the option to do an async
> implementation).  Obviously we should make scheduling fast, but I think
> we're fine doing synchronous scheduling.  It's still probably going to be
> much faster than CloudServers on a bad day anyway :-)
>
> Anyone have a link to where we chose to go with integer IDs?  I'd like to
> understand why, because presumably we had a good reason.
>
> However, if we don't have documentation of the decision, then I vote that
> it never happened, and instance ids are strings.  We've always been at war
> with Eastasia, and all ids have always been strings.
>
> Justin
>
>
>
>
> On Tue, Mar 22, 2011 at 12:20 PM, Paul Voccio 
> wrote:
>
>> I agree with the sentiment that integers aren't the way to go long term.
>> The current spec of the api does introduce some interesting problems to
>> this discussion. All can be solved. The spec calls for the api to return
>> an id and a password upon instance creation. This means the api isn't
>> asynchronous if it has to wait for the zone to create the id. From page 46
>> of the API Spec states the following:
>>
>> "Note that when creating a server only the server ID and the admin
>> password are guaranteed to be returned in the request object. Additional
>> attributes may be retrieved by performing subsequent GETs on the server."
>>
>>
>>
>> This creates a problem with the bursting if Z1 calls to Z2, which is a
>> public cloud, which has to wait for Z3-X to find out where it is going be
>> placed. How would this work?
>>
>> pvo
>>
>> On 3/22/11 1:39 PM, "Chris Behrens"  wrote:
>>
>> >
>> >I think Dragon got it right.  We need a zone identifier prefix on the
>> >IDs.  I think we need to get away from numbers.  I don't see any reason
>> >why they need to be numbers.  But, even if they did, you can pick very
>> >large numbers and reserve some bits for zone ID.
>> >
>> >- Chris
>> >
>> >
>> >On Mar 22, 2011, at 10:48 AM, Justin Santa Barbara wrote:
>> >
>> >> I think _if_ we want to stick with straight numbers, the following are
>> >>the 'traditional' choices:
>> >>
>> >> 1) "Skipping" - so zone1 would allocate numbers 1,3,5, zone2 numbers
>> >>2,4,6.  Requires that you know in advance how many zones there are.
>> >> 2) Prefixing - so zone0 would get 0xxx, zone1 1xx.
>> >> 3) Central allocation - each zone would request an ID from a central
>> >>pool.  This might not be a bad thing, if you do want to have a quick
>> >>lookup table of ID -> zone.  Doesn't work if the zones aren't under the
>> >>same administrative control.
>> >> 4) Block allocation - a refinement of #3, where you get a bunch of IDs.
>> >> Effectively amortizes the cost of the RPC.  Probably not worth the
>> >>effort here.
>> >>
>> >> (If you want central allocation without a shared database, that's also
>> >>possible, but requires some trickier protocols.)
>> >>
>> >> However, I agree with Monsyne: numeric IDs have got to go.  Suppose I'm
>> >>a customer of Rackspace CloudServers once it is running on OpenStack,
>> >>and I also have a private cloud that the new Rackspace Cloud Business
>> >>unit has built for me.  I like bo

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

The API spec doesn't seem to preclude us from doing a fully-synchronous
method if we want to (it just reserves the option to do an async
implementation).  Obviously we should make scheduling fast, but I think
we're fine doing synchronous scheduling.  It's still probably going to be
much faster than CloudServers on a bad day anyway :-)

Anyone have a link to where we chose to go with integer IDs?  I'd like to
understand why, because presumably we had a good reason.

However, if we don't have documentation of the decision, then I vote that it
never happened, and instance ids are strings.  We've always been at war with
Eastasia, and all ids have always been strings.

Justin




On Tue, Mar 22, 2011 at 12:20 PM, Paul Voccio wrote:

> I agree with the sentiment that integers aren't the way to go long term.
> The current spec of the api does introduce some interesting problems to
> this discussion. All can be solved. The spec calls for the api to return
> an id and a password upon instance creation. This means the api isn't
> asynchronous if it has to wait for the zone to create the id. From page 46
> of the API Spec states the following:
>
> "Note that when creating a server only the server ID and the admin
> password are guaranteed to be returned in the request object. Additional
> attributes may be retrieved by performing subsequent GETs on the server."
>
>
>
> This creates a problem with the bursting if Z1 calls to Z2, which is a
> public cloud, which has to wait for Z3-X to find out where it is going be
> placed. How would this work?
>
> pvo
>
> On 3/22/11 1:39 PM, "Chris Behrens"  wrote:
>
> >
> >I think Dragon got it right.  We need a zone identifier prefix on the
> >IDs.  I think we need to get away from numbers.  I don't see any reason
> >why they need to be numbers.  But, even if they did, you can pick very
> >large numbers and reserve some bits for zone ID.
> >
> >- Chris
> >
> >
> >On Mar 22, 2011, at 10:48 AM, Justin Santa Barbara wrote:
> >
> >> I think _if_ we want to stick with straight numbers, the following are
> >>the 'traditional' choices:
> >>
> >> 1) "Skipping" - so zone1 would allocate numbers 1,3,5, zone2 numbers
> >>2,4,6.  Requires that you know in advance how many zones there are.
> >> 2) Prefixing - so zone0 would get 0xxx, zone1 1xx.
> >> 3) Central allocation - each zone would request an ID from a central
> >>pool.  This might not be a bad thing, if you do want to have a quick
> >>lookup table of ID -> zone.  Doesn't work if the zones aren't under the
> >>same administrative control.
> >> 4) Block allocation - a refinement of #3, where you get a bunch of IDs.
> >> Effectively amortizes the cost of the RPC.  Probably not worth the
> >>effort here.
> >>
> >> (If you want central allocation without a shared database, that's also
> >>possible, but requires some trickier protocols.)
> >>
> >> However, I agree with Monsyne: numeric IDs have got to go.  Suppose I'm
> >>a customer of Rackspace CloudServers once it is running on OpenStack,
> >>and I also have a private cloud that the new Rackspace Cloud Business
> >>unit has built for me.  I like both, and then I want to do cloud
> >>bursting in between them, by putting an aggregating zone in front of
> >>them.  I think at that stage, we're screwed unless we figure this out
> >>now.  And this scenario only has one provider (Rackspace) involved!
> >>
> >> We can square the circle however - if we want numbers, let's use UUIDs
> >>- they're 128 bit numbers, and won't in practice collide.  I'd still
> >>prefer strings though...
> >>
> >> Justin
> >>
> >>
> >>
> >> On Tue, Mar 22, 2011 at 9:40 AM, Ed Leafe  wrote:
> >>I want to get some input from all of you on what you think is
> >>the best way to approach this problem: the RS API requires that every
> >>instance have a unique ID, and we are currently creating these IDs by
> >>use of an auto-increment field in the instances table. The introduction
> >>of zones complicates this, as each zone has its own database.
> >>
> >>The two obvious solutions are a) a single, shared database and
> >>b) using a UUID instead of an integer for the ID. Both of these
> >>approaches have been discussed and rejected, so let's not bring them
> >>back up now.
> >>
> >>Given integer IDs and separate databases, the

Re: [Openstack] Openstack API - Volumes?

[Justin Santa Barbara]

There are many existing volume solutions.  Some are in production use (AoE,
iSCSI).  There are enterprise-class SAN providers (Solaris, HP/LeftHand
SANs).  There are existing projects to build 'cloud' storage solutions that
are shaping up very well (SheepDog, RBD).

It's great that you're thinking about how to add another option to the mix.
 I do think you should consider supporting one of the existing efforts
unless you have a radically different approach, but it's open source, so
you're free to scratch your own itch.

However: this should not be allowed to impact the volumes API.  I consider
volumes to be a core feature of the cloud; and OpenStack has great volume
support today.  We have API support for volumes today in the EC2 API, and if
we want people to use the OpenStack API, then API support for volumes needs
to be there as well.  This does not mean that volumes must be enabled on
every OpenStack deployment, so even if CloudServers can't support volumes
initially, we should still provide the API.

So: When can we expect volume support in nova?  If I repackaged my volumes
API as an extension, can we get it merged into Cactus?

Justin


On Tue, Mar 22, 2011 at 11:53 AM, Chuck Thier  wrote:

> Hi Adam,
>
> We have just begun an R&D effort for building a scalable block storage
> service with commodity hardware.  We are still working more on the back end
> specifics for this, so we haven't spent much time looking at the end user
> apis yet.  We are open to hearing feedback on that front though, and plan on
> discussing it more at the next developer summit.
>
> This project will be developed in the open, and when complete will
> be submitted to the Policy Board as a candidate for inclusion in Openstack
> proper.  I don't have a lot of details to share yet, but plan on discussing
> much at the next summit.
>
> --
> Chuck
>
> On Tue, Mar 22, 2011 at 11:20 AM, John Purrier  wrote:
>
>> I know that creiht is looking at this for Rackspace. Chuck, anything to
>> add
>> to this discussion?
>>
>> John
>>
>> -Original Message-
>> From: openstack-bounces+john=openstack@lists.launchpad.net
>> [mailto:openstack-bounces+john=openstack@lists.launchpad.net] On
>> Behalf
>> Of Adam Johnson
>> Sent: Monday, March 21, 2011 10:15 PM
>> To: openstack@lists.launchpad.net
>> Subject: [Openstack] Openstack API - Volumes?
>>
>> Hey everyone,
>>
>> I wanted to bring up the topic of volumes in the OpenStack API.  I
>> know there was some discussion about this before, but it seems to have
>> faded on the ML.   I know Justinsb has done some work on this already,
>> and has a branch here:
>>
>> https://code.launchpad.net/~justin-fathomdb/nova/justinsb-openstack-api-volu
>> mes
>>
>> Im wondering what the consensus is on what the API should look like,
>> and when we could get this merged into Nova?
>>
>> Thanks,
>> Adam Johnson
>> Midokura
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

Totally agree with Eric.

Two questions that I think can help us move forward:


   1. Is the decision to stick with integers still valid?  Can someone that
   was there give us the reason for the decision?  Is it documented anywhere?
   2. If "we must have integers" means that we get 128 bit 'random'
   integers, do we still want integers?



Justin





On Tue, Mar 22, 2011 at 11:25 AM, Eric Day  wrote:

> On Tue, Mar 22, 2011 at 12:40:21PM -0400, Ed Leafe wrote:
> >   The two obvious solutions are a) a single, shared database and b)
> using a UUID instead of an integer for the ID. Both of these approaches have
> been discussed and rejected, so let's not bring them back up now.
>
> We shouldn't dismiss previous ideas just because we've not chosen
> them in the past, but lets not have the same discussion.
>
> >   Given integer IDs and separate databases, the only obvious choice
> is partitioning the numeric space so that each zone starts its
> auto-incrementing at a different point, with enough room between starting
> ranges to ensure that they would never overlap. This would require some
> assumptions be made about the maximum number of instances that would ever be
> created in a single zone in order to determine how much numeric space that
> zone would need. I'm looking to get some feedback on what would seem to be
> reasonable guesses to these partition sizes.
>
> I think we need:
>
> * No central authority such as a globally shared DB. This also
>  means not partitioning some set and handing them out to zones as
>  offset (this is just another form of a shared DB).
>
> * Ability to seamlessly join existing zones without chance of namespace
>  collisions for peering and bursting. This means a globally unique
>  zone naming scheme, and for this I'll reiterate the idea of using
>  DNS names for zones.
>
> If we want to stick with a single DB per zone, as it looks like we
> are, this can simply be the auto-increment value from the instance
> table and the zone as: ..
>
> >   The other concern is more aesthetic than technical: we can make the
> numeric spaces big enough to avoid overlap, but then we'll have very large
> ID values; e.g., 10 or more digits for an instance. Computers won't care,
> but people might, so I thought I'd at least bring up this potential
> objection.
>
> I'm not concerned with aesthetic issues to be honest. We have
> copy/paste, DNS, and various techniques for presentation layers.
>
> -Eric
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Instance IDs and Multiple Zones

[Justin Santa Barbara]

I think _if_ we want to stick with straight numbers, the following are the
'traditional' choices:

1) "Skipping" - so zone1 would allocate numbers 1,3,5, zone2 numbers 2,4,6.
 Requires that you know in advance how many zones there are.
2) Prefixing - so zone0 would get 0xxx, zone1 1xx.
3) Central allocation - each zone would request an ID from a central pool.
 This might not be a bad thing, if you do want to have a quick lookup table
of ID -> zone.  Doesn't work if the zones aren't under the same
administrative control.
4) Block allocation - a refinement of #3, where you get a bunch of IDs.
 Effectively amortizes the cost of the RPC.  Probably not worth the effort
here.

(If you want central allocation without a shared database, that's also
possible, but requires some trickier protocols.)

However, I agree with Monsyne: numeric IDs have got to go.  Suppose I'm a
customer of Rackspace CloudServers once it is running on OpenStack, and I
also have a private cloud that the new Rackspace Cloud Business unit has
built for me.  I like both, and then I want to do cloud bursting in between
them, by putting an aggregating zone in front of them.  I think at that
stage, we're screwed unless we figure this out now.  And this scenario only
has one provider (Rackspace) involved!

We can square the circle however - if we want numbers, let's use UUIDs -
they're 128 bit numbers, and won't in practice collide.  I'd still prefer
strings though...

Justin



On Tue, Mar 22, 2011 at 9:40 AM, Ed Leafe  wrote:

>I want to get some input from all of you on what you think is the
> best way to approach this problem: the RS API requires that every instance
> have a unique ID, and we are currently creating these IDs by use of an
> auto-increment field in the instances table. The introduction of zones
> complicates this, as each zone has its own database.
>
>The two obvious solutions are a) a single, shared database and b)
> using a UUID instead of an integer for the ID. Both of these approaches have
> been discussed and rejected, so let's not bring them back up now.
>
>Given integer IDs and separate databases, the only obvious choice is
> partitioning the numeric space so that each zone starts its
> auto-incrementing at a different point, with enough room between starting
> ranges to ensure that they would never overlap. This would require some
> assumptions be made about the maximum number of instances that would ever be
> created in a single zone in order to determine how much numeric space that
> zone would need. I'm looking to get some feedback on what would seem to be
> reasonable guesses to these partition sizes.
>
>The other concern is more aesthetic than technical: we can make the
> numeric spaces big enough to avoid overlap, but then we'll have very large
> ID values; e.g., 10 or more digits for an instance. Computers won't care,
> but people might, so I thought I'd at least bring up this potential
> objection.
>
>
>
> -- Ed Leafe
>
>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] GLANCE: How to deal with SQLAlchemy-migrate problems

[Justin Santa Barbara]

Do we have patches for approach #3?  I'm thinking that if we have a few
specific issues, that maybe we could maintain our own branch and/or
monkey-patch SA-migrate, and get a better handle on the problem, and the
upstream's project receptiveness to patches (rather than bug reports).

It's a bit more work now, but avoids a major change at this stage of the
release.  One upside is that we'll hopefully have a better understanding
when it comes time to write a replacement if we decide to go down that road.

My experience of SA-migrate was that it seemed powerful, but the
documentation wasn't worth the paper it was written on :-)  It may be that
we can fix a few bugs, figure out the "SA" way to do things, and end up
happy.

Justin





On Mon, Mar 21, 2011 at 8:39 AM, Jay Pipes  wrote:

> Hey all,
>
> For the better part of the last week, I've been struggling to use
> SQLAlchemy-Migrate to produce database migration scripts for Glance.
> In Glance, unlike Nova, we have a unit test that tests that the
> database for Glance's registry can be downgraded and upgraded
> smoothly. In Nova, none of the migrate scripts even contain a
> downgrade method, so this isn't even testable.
>
> We've found a number of problems in using SA-Migrate:
>
> 1. When using the SQLite engine, if a table has a column with
> index=True on it and you drop any column on that table, SA-Migrate
> will die if you use Python migrate scripts
> (http://code.google.com/p/sqlalchemy-migrate/issues/detail?id=117)
> 2. When using the MySQL engine, you cannot use SQL scripts with
> multiple statements in them as upgrade/downgrade scripts
> (http://code.google.com/p/sqlalchemy-migrate/issues/detail?id=99)
> because SA-Migrate pipes all the statements into the
> Connection.execute() method as a single statement, causing the
> underlying MySQLdb driver to vomit an error 2014 "Commands out of
> sync"
> 3. In order to solve #1 above, you need to do your own SQL scripting
> to bypass bugs in SA-Migrate. Writing the SQL script means you run
> into #2 above. This means you can't test both SQLite and MySQL
> migrations using the same scripts. This sucks. A lot.
> 4. It seems the contributors to SA-Migrate are not responsive to bug
> reports. This makes it less than ideal as a dependency for Glance,
> especially considering the critical nature that database migrations
> play in installation and upgrading of our software.
>
> So, what to do? I see these options, none of which I particularly like:
>
> 1) Don't test migration scripts - the Nova approach. This would mean
> we write all our migrate scripts as Python scripts, which work better
> with MySQL but break completely for SQLite. No idea about PostgreSQL.
> 2) Don't use SA-Migrate and write a replacement using python-sqlparse
> and straight SQL scripts. While I can't stand SA-Migrate at this
> point, it remains really the only thing that remotely works.
> 3) Submit patches to SA-Migrate and keep patched versions of
> SA-Migrate in the Glance PPA.
>
> Thoughts? I'm leaning towards #2, then #1, then #3.
>
> -jay
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Seems that the person writing the code (Sandy) wants _not_ to do a single DB
initially.  It sounds like there are back channels where a single DB is
being pushed on Sandy.

To me, it sounds like we have these choices:

   1. We can have a zones implementation in Cactus.  As specified in the
   blueprint, it will use recursive querying, and there will be no caching
   initially, nor will there be a single DB.
   2. We can go 'off blueprint', and simply not have a multi-zones
   implementation in Cactus.


Given that, I don't see why we would deviate from what we've agreed (and I'm
normally all for flexibility); let's get a baseline implementation into
Cactus.  People that want to add caching or a single DB are then free to do
so in their own branches, but at least those enhancements will be starting
from a common base.  I'm not against adding caching / a single DB if it
proves necessary / good later.

Hopefully we'll actually learn of any real-world issues with the simple
approach by running Sandy's code, and we can discuss those facts at the
design conference, rather than talking in hypotheticals.

Sandy: Have I got the wrong end of the stick here?  Are these our choices?

Justin



On Wed, Mar 16, 2011 at 1:13 PM, Ed Leafe  wrote:

> On Mar 16, 2011, at 3:39 PM, Justin Santa Barbara wrote:
>
> > I agree that we could have a better marker, but I'm just going off the
> spec at the moment.
> >
> > I've checked the agreed blueprint, and caching in zones is out of scope
> for Cactus.
> >
> > Please propose a discussion topic for the Design Summit.
>
> Can we get back to the original topic? The only reason caching came
> up was as an alternative to a single DB to hold all instance information.
> That was an implementation solution suggested for multi-cluster/zones, so it
> is definitely in scope for Cactus.
>
>
>
> -- Ed Leafe
>
>
>
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of
> the
> individual or entity to which this message is addressed, and unless
> otherwise
> expressly indicated, is confidential and privileged information of
> Rackspace.
> Any dissemination, distribution or copying of the enclosed material is
> prohibited.
> If you receive this transmission in error, please notify us immediately by
> e-mail
> at ab...@rackspace.com, and delete the original message.
> Your cooperation is appreciated.
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

I agree that we could have a better marker, but I'm just going off the spec
at the moment.

I've checked the agreed blueprint, and caching in zones is out of scope for
Cactus.

Please propose a discussion topic for the Design Summit.

Justin





On Wed, Mar 16, 2011 at 12:21 PM, Eric Day  wrote:

> On Wed, Mar 16, 2011 at 06:47:03PM +, Ed Leafe wrote:
> > > With our approach to pagination, without caching, the answer is always
> correct: each query always returns the next {limit} values whose ID is >=
> {start-id}.
> >
> >   But for this example, you have to traverse *all* the zones in order
> to get the list of instances for a customer. How else would you define
> "instances 101-199 of 823 total instances"? How would you know where #101
> is?
>
> The marker should be the canonical zone:instance, so you just query
> that zone with instance marker and limit, and move on to the next
> zone(s) if your limit is not reached. You never need to build the
> full list.
>
> -Eric
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Inline...

> Can someone explain _why_ we need caching?
>
> We don't *need* caching - it is simply the most direct way to avoid
> multiple expensive calls.


So if we don't need it...?  You cite avoiding expensive calls, but I think
it's entirely unproven that those call are too expensive.

If it makes you happier, think of the child zones as a cache that just
happens to not have any inconsistency issues.  I see Eric's point that it
would be better to integrate it up-front, but let's get ten zones working
first, before we worry out how we can scale to ten thousand zones.

I think this should be Sandy's call.  My interpretation is that he'd like to
implement a correct algorithm first, to understand where the bottlenecks
are.  I think that's the right way.  Equally, if Sandy says he wants to do
caching, then that's fine with me also.



> > With our approach to pagination, without caching, the answer is always
> correct: each query always returns the next {limit} values whose ID is >=
> {start-id}.
>
> But for this example, you have to traverse *all* the zones in order
> to get the list of instances for a customer. How else would you define
> "instances 101-199 of 823 total instances"? How would you know where #101
> is?
>

You do need to traverse all zones, correct.  (Though note that you don't
need to know where #101 is)  I think you may have missed the point about
token-based pagination: the query is not "instances at indexes 101-199 of
823", it is e.g. "the next 100 instances whose id > 'i-198273'.

But how is this a problem?  How many zones are we talking about?  If this is
bigger than a few dozen, why?


>
> > I agree that in practice this means that there's no way to guarantee you
> get all values while they're changing behind the scenes, but this is a
> shortcoming of pagination, not caching.  Caching doesn't solve this, it just
> creates thornier edge cases.  The solution here is a more sensible ordering
> than 'last modified', and I question the value of pagination (other than for
> compatibility)
>
> The question is performance in this particular use case. As pvo
> said, you can accept a certain level of inconsistency as a trade-off for
> scaling. We would then allow the TTL to be controlled so that different
> OpenStack deployments can adjust this to their needs.
>

Except every binding to the CloudServers API has a hacky way to bypass
caching.  We have direct evidence that people don't want to accept these
caching trade-offs / edge cases.

It sounds to me like you're saying: "But caching is web-scale" :-)


Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Can someone explain _why_ we need caching?

With our approach to pagination, without caching, the answer is always
correct: each query always returns the next {limit} values whose ID is >=
{start-id}.

I agree that in practice this means that there's no way to guarantee you get
all values while they're changing behind the scenes, but this is a
shortcoming of pagination, not caching.  Caching doesn't solve this, it just
creates thornier edge cases.  The solution here is a more sensible ordering
than 'last modified', and I question the value of pagination (other than
for compatibility)

Justin

---

Justin Santa Barbara
Founder, FathomDB





On Wed, Mar 16, 2011 at 11:14 AM, Andrew Shafer wrote:

>
> Global temporal consistency is a myth.
>
> If you decide not to cache and support pagination then querying every zone
> for every page is potentially as misleading as caching because what should
> be on each page could change for every request.
>
> +1 for cache with ttl
>
>
> On Wed, Mar 16, 2011 at 11:58 AM, Paul Voccio 
> wrote:
>
>> Ed,
>>
>> I would agree. The caching would go with the design tenet #7: Accept
>> eventual consistency and use it where it is appropriate.
>>
>> If we're ok with accepting that the list may or may not always be up to
>> date and feel its appropriate, we should be good with the caching.
>>
>> pvo
>>
>>
>> On 3/16/11 11:45 AM, "Ed Leafe"  wrote:
>>
>> >On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:
>> >
>> >> Not only is this expensive, but there is no way I can see at the moment
>> >>to do pagination, which is what makes this really expensive. If someone
>> >>asked for an entire list of all their instances and it was > 10,000 then
>> >>I would think they're ok with waiting while that response is gathered
>> >>and returned. However, since the API spec says we should be able to do
>> >>pagination, this is where asking each zone for all its children every
>> >>time gets untenable.
>> >
>> >This gets us into the caching issues that were discussed at the last
>> >summit. We could run the query and then cache the results at the
>> >endpoint, but this would require accepting some level of staleness of the
>> >results. The cache would handle the paging, and some sort of TTL would
>> >have to be established as a balance between performance and staleness.
>> >
>> >
>> >
>> >-- Ed Leafe
>> >
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Good point that pagination makes this harder.  However, thankfully the limit
is implemented using a token (the last ID seen), not an absolute offset, so
I believe we can still do pagination even in loosely coordinated DBs.  Good
job whoever dodged that bullet (Jorge?)

(Aside #1: Sorting by uptime time - now _that's_ expensive, and has to be
done for every page.)
(Aside #2: What is the point of pagination really?  I mean how many API
users that are getting a list actually stop before fetching them all?)

Justin


On Wed, Mar 16, 2011 at 9:23 AM, Paul Voccio wrote:

>  Sandy,
>
>  Not only is this expensive, but there is no way I can see at the moment
> to do pagination, which is what makes this really expensive. If someone
> asked for an entire list of all their instances and it was > 10,000 then I
> would think they're ok with waiting while that response is gathered and
> returned. However, since the API spec says we should be able to do
> pagination, this is where asking each zone for all its children every time
> gets untenable.
>
>  Looking forward to the discussion. More below.
>
>   From: Sandy Walsh 
> Date: Wed, 16 Mar 2011 14:53:37 +
> To: "openstack@lists.launchpad.net" 
> Subject: [Openstack] A single cross-zone database?
>
>   Hi y'all, getting any sleep before Feature Freeze?
>
>  As you know, one of the main design tenants of OpenStack is Share Nothing
> (where possible). http://wiki.openstack.org/BasicDesignTenets
>
>  That's the mantra we've been chanting with Zones. But it does cause a
> problem with a particular Use Case:
>
>  *"Show me all Customer X Instances, across all Zones."*
>
>  This is an expensive request. We have to poll all zones and ask them to
> return a list of matching instances.
>
>  There has been some water cooler chat about some things we could do to
> make this more efficient in the near term. One proposal has been to assume a
> single database, replicated across zones. I'll call it SDB for short. With
> SDB we can have a join table that links Zone to Instance ... keeping a
> record of all instances across zones. Maybe it's a completely separate set
> of tables? Maybe it's a separate replicated db? The intention is to let us
> talk to the appropriate zone directly.
>
>  Sure, there are a ton more optimizations we could make if we go further
> with SDB. We could store all the Zone capabilities in the db to make Zone
> selection faster. We could store all the customers in the db to make
> multi-tenant easier. But that's not what we're talking about here. We're
> talking about the* bare minimum *required to make the get_instances query
> fast.
>
>  Conversely, there are issues with a single DB. The largest being the
> implication it has on Bursting (Hybrid Private/Public clouds) ... a pretty
> funky feature imho.
>
>
>
>
>
>  Personally, I think the same query gains can be obtained by creating a
> separate db using off-the-shelf ETL tools to create cache/read-only db's.
> http://en.wikipedia.org/wiki/Extract,_transform,_load
>
>
>  Isn't the hard part keeping this in sync with what the zones have?
>
>
>  I was considering SDB for Zones (phase 4), but for now, I'm going to
> stick with the original plan of separate databases (1 per zone) and see what
> the performance implications are.
>
>  What are your thoughts on this issue?
>
>  ... let the games begin!
>
>  -S
>
>
>  Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is 
> prohibited.
> If you receive this transmission in error, please notify us immediately by 
> e-mail
> at ab...@rackspace.com, and delete the original message.
> Your cooperation is appreciated.
>
>  ___ Mailing list:
> https://launchpad.net/~openstack Post to : 
> openstack@lists.launchpad.netUnsubscribe :
> https://launchpad.net/~openstack More help :
> https://help.launchpad.net/ListHelp
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Thanks for raising this Sandy: +1 on keeping separate DBs until a problem
arises.

I don't see a performance problem with recursively querying child zones.  I
guess this will partially depend on our zone topology: if the intent is to
have child zones that are geographically distributed where the latency
becomes an issue (and it's not clear to me that even this will be a
problem), then we could use a 'caching' approach rather than the 'total
consistency' approach that a database implies.

It does seem like there's a separate issue, which is finding the appropriate
zone for a given instance.  Again, we could easily use caching here, and
simply retry on cache miss.

What sort of complexity of zones hierarchies are we contemplating?  Even at
our full target scales, I don't believe we're talking about more than a
dozen zones (?), and I believe that even the naive implementation which
simply recursively queries child zones with no caching will be more than
good enough.

Justin



On Wed, Mar 16, 2011 at 7:53 AM, Sandy Walsh wrote:

>  Hi y'all, getting any sleep before Feature Freeze?
>
>  As you know, one of the main design tenants of OpenStack is Share Nothing
> (where possible). http://wiki.openstack.org/BasicDesignTenets
>
>  That's the mantra we've been chanting with Zones. But it does cause a
> problem with a particular Use Case:
>
>  *"Show me all Customer X Instances, across all Zones."*
>
>  This is an expensive request. We have to poll all zones and ask them to
> return a list of matching instances.
>
>  There has been some water cooler chat about some things we could do to
> make this more efficient in the near term. One proposal has been to assume a
> single database, replicated across zones. I'll call it SDB for short. With
> SDB we can have a join table that links Zone to Instance ... keeping a
> record of all instances across zones. Maybe it's a completely separate set
> of tables? Maybe it's a separate replicated db? The intention is to let us
> talk to the appropriate zone directly.
>
>  Sure, there are a ton more optimizations we could make if we go further
> with SDB. We could store all the Zone capabilities in the db to make Zone
> selection faster. We could store all the customers in the db to make
> multi-tenant easier. But that's not what we're talking about here. We're
> talking about the* bare minimum *required to make the get_instances query
> fast.
>
>  Conversely, there are issues with a single DB. The largest being the
> implication it has on Bursting (Hybrid Private/Public clouds) ... a pretty
> funky feature imho.
>
>  Personally, I think the same query gains can be obtained by creating a
> separate db using off-the-shelf ETL tools to create cache/read-only db's.
> http://en.wikipedia.org/wiki/Extract,_transform,_load
>
>  I was considering SDB for Zones (phase 4), but for now, I'm going to
> stick with the original plan of separate databases (1 per zone) and see what
> the performance implications are.
>
>  What are your thoughts on this issue?
>
>  ... let the games begin!
>
>  -S
>
>
>  Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is 
> prohibited.
> If you receive this transmission in error, please notify us immediately by 
> e-mail
> at ab...@rackspace.com, and delete the original message.
> Your cooperation is appreciated.
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] [Nova] Review and bug priorities

[Justin Santa Barbara]

I've been moaning about the fact that Nova is like a happy oblivious little
child for way too long, so I'll take on the two bugs that teach it that
things can go wrong in the real world:

When a node dies, its instances should be marked !running
https://bugs.launchpad.net/nova/+bug/661214

nova-compute doesn't notice when a VM crashes
https://bugs.launchpad.net/nova/+bug/661260

Justin




On Mon, Mar 14, 2011 at 3:09 AM, Thierry Carrez wrote:

> At the last meeting we discussed having a recurrent post to the ML to,
> in Devin's words, "push priorities in front of eyeballs"...
>
> Here is my first attempt at it, trying to single out a few branches and
> bugs that need attention.
>
> PS: IMO this email duplicates already-available information at:
>
> http://wiki.openstack.org/releasestatus/
> http://tinyurl.com/nova-bugs
>
> Let me know if you find it useful or not.
>
> 
> We are 10 days to Feature Freeze, so the focus must be on (in descending
> order of priority):
>
> 1- Writing code and proposing branches on accepted Cactus specs
> 2- Review already-proposed Cactus feature branches
> 3- Review already-proposed bugfix branches
> 4- Fixing reported bugs
> 5- Testing and report new bugs
>
> 
> 4 Cactus feature branch merge proposals needing some extra attention:
>
> Live migration
>  Large patch, needs to land early, already has needed approvals
>  https://code.launchpad.net/~nttdata/nova/live-migration/+merge/49699
>
> Flat Manager IPv6 support
>  Essential to have full IPv6 support in cactus
>  https://code.launchpad.net/~ntt-pf-lab/nova/flatmanager-ipv6/+merge/53064
>
> VMWare/vSphere support
>  Large patch
>
> https://code.launchpad.net/~citrix-openstack/nova/vmware-vsphere-support/+merge/53060
>
> LXC support
>  Needs to land early to get some testing
>  https://code.launchpad.net/~zulcss/nova/nova-lxc/+merge/51469
>
> 
> 4 complex Cactus bugs that need some love:
>
> nova-compute: can't spawn new instances after restarting RabbitMQ
> https://bugs.launchpad.net/nova/+bug/718869
>
> "python setup.py install" actually does not install a lot stuff...
> https://bugs.launchpad.net/nova/+bug/727794
>
> When a node dies, its instances should be marked !running
> https://bugs.launchpad.net/nova/+bug/661214
>
> nova-compute doesn't notice when a VM crashes
> https://bugs.launchpad.net/nova/+bug/661260
>
> --
> Thierry Carrez (ttx)
> Release Manager, OpenStack
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp