Re: [Openstack] OSAPI and Zones
Thanks a lot for your answers. But why do you want to move the Zone code into the extension part ? It's a core part of OpenStack, why it doesn't stay in the core code ? Another question about extensions. I had understand that an extension will be integrated to the core OSAPI when it will be mature, is that true ? The extension mechanism is like an incubator for OSAPI functionalities ? Regards, Édouard. On Mon, Nov 14, 2011 at 7:09 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: Jorge is correct. The zones stuff was added before the API was finalized and before the extensions mechanism was in place. We simply haven't taken the time to convert it yet. -S From: openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net [openstack-bounces+sandy.walsh=rackspace@lists.launchpad.net] on behalf of Anne Gentle [a...@openstack.org] Sent: Monday, November 14, 2011 12:25 PM To: Doude Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] OSAPI and Zones Hi Édouard - I believe zones are documented in the Developer docs in http://nova.openstack.org/devref/zone.html. The howto scenarios are documented using the nova python client only currently (probably due to the refactoring Jorge mentions). When you run into these gaps, please do log a bug against either nova or openstack-manuals (http://bugs.launchpad.net/openstack-manuals). The nova-docs team is small and working through a backlog of such items, and bugs definitely help with prioritization and tracking. Thanks, Anne On Mon, Nov 14, 2011 at 9:49 AM, Doude doudou...@gmail.com wrote: Hi all, I'm trying to understand the multi-zone architecture of OpenStack. I saw zone commands (list, show, select ...) have been added to the OSAPI v1.1 (not as an extension but as a core component of the API) but I cannot find any documentations in the OSAPI book: http://docs.openstack.org/trunk/openstack-compute/developer/openstack-compute-api-1.1/content/ Where I can find this documentation ? In OpenStack wiki ? Where I can open a bug about this lack of documentation ? Regards, Édouard. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Git release tags?
Hi, Would it be possible to use git tags to mark released version? For example, I'm trying to find the latest Keystone release in Git, but there's no tag. It seems a bit messy, or do I miss something? -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Reminder: OpenStack team meeting - 21:00 UTC
Hello everyone, Our general meeting will take place at 21:00 UTC this Tuesday in #openstack-meeting on IRC. PTLs, if you can't make it, please name a substitute on [2]. We'll present essex-2 plans, and discuss if Keystone current stable/diablo branch is ripe for a tag and tarball. Please double-check the meeting time at: [1] http://www.timeanddate.com/worldclock/fixedtime.html?iso=2015T21 See the meeting agenda, edit the wiki to add new topics for discussion: [2] http://wiki.openstack.org/Meetings/TeamMeeting Cheers, -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OSAPI and Zones
Inline: On Nov 15, 2011, at 3:36 AM, Doude wrote: Thanks a lot for your answers. But why do you want to move the Zone code into the extension part ? It's a core part of OpenStack, why it doesn't stay in the core code ? If something is in core then it's guaranteed to be available always. A client should be able to count on the functionality in core for a particular version of an API. Zone's offer admin level functionality that may not be available to all users. I don't think that Rackspace will expose Zones to it's customers right away, for example. By having Zones as an extension a client can detect whether zone support is available or not. Another question about extensions. I had understand that an extension will be integrated to the core OSAPI when it will be mature, is that true ? The extension mechanism is like an incubator for OSAPI functionalities ? Yes, the extension mechanism can be used as an incubator for functionality. But, not all extensions are destine to make it to the core. Some extension for example will cover niche functionality that may be useful for a small set of users. Other extension may cover functionality that would set the bar high for folks deploying OS clouds. Other extension may expose functionality that's applicable to a specific hypervisor etc. That said, at the end of the day, the PTL decides which of these extensions make it to core and which stay as extensions. I've got a draft of a write up to explain extensions in some detail here: https://github.com/RackerWilliams/OpenStack-Extensions/blob/master/apix-intro.pdf Nothing there is set entirely in stone, but it drafts the concept as we're currently thinking of it. I'm in the process of setting up an extension registry and documenting a number of extensions. These certainly inform the doc above -- so expect some changes. As soon as things stabilize I'll publish the doc in our doc site. -jOrGe W. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Git release tags?
Julien Danjou wrote: Would it be possible to use git tags to mark released version? For example, I'm trying to find the latest Keystone release in Git, but there's no tag. It seems a bit messy, or do I miss something? I think you're missing something. keystone (master)$ git tag 2011.3 essex-1 Tags are used for every core project milestone and release (the first release of Keystone as an official openstack core project will happen in April). The git master repo for keystone therefore has a essex-1 tag: https://github.com/openstack/keystone/tree/essex-1 Previous to that, projects are free to use tags as well. Keystone used a 2011.3 tag for their own Diablo release: https://github.com/openstack/keystone/tree/2011.3 -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nexenta Driver proposal
Hi! A few thoughts here, purely from a dev-process and CI perspective. I'm personally thrilled to have driver around that has a team of folks supporting it - but for OpenStack that has a few specific meanings. First of all, I'm not in charge, but I'm pretty certain that there is no way to get this added for diablo. The only things being merged into the stable/diablo branch are backported fixes from trunk... so I'd definitely focus your efforts on essex in the master branch. To get the code into essex, you'll need to submit the code as a patch via gerrit: http://wiki.openstack.org/GerritWorkflow When you are ready, you'll want to rebase and squash this into a single commit on top of master. (it looks like it's a single commit in your stable/diablo branch already, so you may be in good shape there) Once you get it ready for master, there is the question of testing. The CI team has been working with hooking in specialized hardware or environments from vendors with needs into the OpenStack Jenkins. The biggest needs we'll have here in order for this to be a first-class citizen are: - unittests for this code that can run in a normal x86-based ubuntu cloud server - a functional/integration test environment owned and operated by you that is connected to and triggered by our jenkins on which versions of openstack to be tested can be installed. I would expect this to be an environment with solaris/nexenta and an installation of openstack configured to use it. We can totally work with you on figuring out how to integrate your resources into our environment. As a starting place, check out how we're doing this with our first bare-metal integration environment, which is a set of machines provided by rackspace: http://ci.openstack.org/jenkins.html#integration-testing The specific steps don't need to look like that at all, but it might give you a better idea of what we'll need to figure out how to get set up. I don't think the integration environment needs to be a blocker to getting stuff merged - after all, we're still working on getting that for the mainline trunk - but ensuring that we've got unittests that are runnable as part of our normal set of gating tests is essential, from my point of view. Thanks for the work so far! Monty On 11/14/2011 01:48 PM, Yuriy Taraday wrote: All driver code is stamped with Apache Public License 2.0. Is it compatible with OpenStack licensing model? Currently driver supports Diablo branch of Nova, but we're working on changes to make it work on master. This driver was developed and tested in tight cooperation with Nexenta team, and will be supported by Nexenta. Kind regards, Yuriy. PS: Lost CC to mail list in my previous message, sorry. 2011/11/14 Diego Parrilla Santamaría diego.parrilla..santama...@gmail.com mailto:diego.parrilla.santama...@gmail.com Thank you Yuriy, it sounds really interesting. You have our support to move, test and improve this code in order to integrate it with master branch of Essex (and backport it to Diablo if necessary) once you release the code under the Openstack licensing model. It would be great if Nexenta supports this project too. Meanwhile, we will continue using the Solaris derivative version that also runs smoothly in the Nexenta Core Platform ;-) Regards Diego -- Diego Parrilla http://www.stackops.com/*CEO* **www.stackops.com* http://www.stackops.com/ | * diego.parri...@stackops.com mailto:diego.parri...@stackops.com | +34 649 94 43 29 | skype:diegoparrilla* * http://www.stackops.com/ * * ADVERTENCIA LEGAL Le informamos, como destinatario de este mensaje, que el correo electrónico y las comunicaciones por medio de Internet no permiten asegurar ni garantizar la confidencialidad de los mensajes transmitidos, así como tampoco su integridad o su correcta recepción, por lo que STACKOPS TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias. Si no consintiese en la utilización del correo electrónico o de las comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro conocimiento de manera inmediata. Este mensaje va dirigido, de manera exclusiva, a su destinatario y contiene información confidencial y sujeta al secreto profesional, cuya divulgación no está permitida por la ley. En caso de haber recibido este mensaje por error, le rogamos que, de forma inmediata, nos lo comunique mediante correo electrónico remitido a nuestra atención y proceda a su eliminación, así como a la de cualquier documento adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o utilización de este mensaje, o de cualquier documento adjunto al mismo, cualquiera que fuera su finalidad, están prohibidas por la ley. *
Re: [Openstack] Git release tags?
As thierry mentioned, the releases are tagged. It is also worth mentioning that each project has a branch for release + fixes. For the last release the branch name is stable/diablo On Nov 15, 2011 2:43 AM, Julien Danjou julien.dan...@enovance.com wrote: Hi, Would it be possible to use git tags to mark released version? For example, I'm trying to find the latest Keystone release in Git, but there's no tag. It seems a bit messy, or do I miss something? -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Git release tags?
On Tue, Nov 15 2011, Thierry Carrez wrote: I think you're missing something. That is *always* an option. :-) Previous to that, projects are free to use tags as well. Keystone used a 2011.3 tag for their own Diablo release: https://github.com/openstack/keystone/tree/2011.3 Ok, so my next question is: where is the tag 2011.3.1, which should probably reference the commit 6baa62c28fd5594127017be3680eee6578b4b7f6 ? (Cc'ing Dolph Mathews) -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Git release tags?
Tagging of the stable/diablo branch (currently versioned as 2011.3.1) is being discussed in today's #openstack-meeting (21:00 UTC I believe), if you'd like to join! On Tue, Nov 15, 2011 at 9:42 AM, Julien Danjou julien.dan...@enovance.comwrote: On Tue, Nov 15 2011, Thierry Carrez wrote: I think you're missing something. That is *always* an option. :-) Previous to that, projects are free to use tags as well. Keystone used a 2011.3 tag for their own Diablo release: https://github.com/openstack/keystone/tree/2011.3 Ok, so my next question is: where is the tag 2011.3.1, which should probably reference the commit 6baa62c28fd5594127017be3680eee6578b4b7f6 ? (Cc'ing Dolph Mathews) -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Git release tags?
On Tue, Nov 15 2011, Dolph Mathews wrote: Tagging of the stable/diablo branch (currently versioned as 2011.3.1) is being discussed in today's #openstack-meeting (21:00 UTC I believe), if you'd like to join! Ok, thanks. That answers my question. :) I won't be able to join, but anyway I don't really have an opinion on the subject. It's just that I saw the commit and though the release was done but without the proper tag. Sorry for the noise then. :) -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nexenta Driver proposal
Passing standard integration tests that shows an option dealing with a special installation has no adverse effect on a reference environment makes sense. It may be possible to support central test equipment in this case, but that will definitely not be the case for every possible enhancement. We probably need to also talk in terms of test scripts that come from the central team that you expect someone supporting an option to run on their own test equipment. How often would such a remote test be needed? After each nightly build might be a bit too often, and only once per release cycle might not be frequent enough. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OSAPI and Zones
On Nov 15, 2011, at 7:18 AM, Jorge Williams wrote: But why do you want to move the Zone code into the extension part ? It's a core part of OpenStack, why it doesn't stay in the core code ? If something is in core then it's guaranteed to be available always. A client should be able to count on the functionality in core for a particular version of an API. Zone's offer admin level functionality that may not be available to all users. I don't think that Rackspace will expose Zones to it's customers right away, for example. By having Zones as an extension a client can detect whether zone support is available or not. It was my understanding that zones would be transparent to the user; in fact, we went to great pains to ensure that zone information was *not* available to the user. Clients would not need to detect if zones exist. -- Ed Leafe ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Glance API semantics when image sizes aren't known
Hi, What are the expected semantics for the Glance API when uploading an image who's size you do not know? The docs at http://glance.openstack.org/glanceapi.html say x-image-meta-size: This header is optional. ... When not present, Glance will calculate the image's size based on the size of the request body. I read this as If x-image-meta-size is missing, then Glance will accept the upload, and then update the metadata to match whatever what was sent. However (with Glance-over-Swift at least) that's not what's happening. What's actually happening is If x-image-meta-size is missing, then the Content-Length header is read, and if both are missing then the image size is set to 0. I'm not sure whether I should clarify the docs to say that x-image-meta-size is required if you don't set Content-Length, or whether the Swift backend needs to be updated to record / recover the uploaded image size. Thanks, Ewan. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
On Tue, Nov 15, 2011 at 3:03 PM, Ewan Mellor ewan.mel...@eu.citrix.com wrote: Hi, What are the expected semantics for the Glance API when uploading an image who’s size you do not know? The docs at http://glance.openstack.org/glanceapi.html say “x-image-meta-size: This header is optional. … When not present, Glance will calculate the image’s size based on the size of the request body.” I read this as “If x-image-meta-size is missing, then Glance will accept the upload, and then update the metadata to match whatever what was sent”. However (with Glance-over-Swift at least) that’s not what’s happening. What’s actually happening is “If x-image-meta-size is missing, then the Content-Length header is read, and if both are missing then the image size is set to 0.” What version of Glance are you using? Also, how are you uploading the image to Glance? I’m not sure whether I should clarify the docs to say that x-image-meta-size is required if you don’t set Content-Length, or whether the Swift backend needs to be updated to record / recover the uploaded image size. The X-Image-Meta-Size header is only used in the image registration process -- i.e. when you are reserving an image identifier and want to set some metadata about the image you plan to upload. When the actual image is uploaded (either in the POST /images or PUT /images/IMAGE_ID), the block of code that calculates what expected size to pass to the backend store is this: if req.content_length: image_size = int(req.content_length) elif 'x-image-meta-size' in req.headers: image_size = int(req.headers['x-image-meta-size']) else: logger.debug(_(Got request with no content-length and no x-image-meta-size header)) image_size = 0 location, size, checksum = store.add(image_meta['id'], req.body_file, image_size) The else: block is ONLY met when you are not using the Python glance client, the glance CLI tool, and are not setting either the Content-Length or X-Image-Meta-Size header. If you use the Python glance client or CLI tool, the image you are feeding to the client automatically does a seek/tell to determine the size of the image you are uploading. The only reason that the image_size is passed to the backend store is because of Swift's inability to automatically handle files larger than 5GB. The image_size passed to the swift backend store's add() method is used to determine if the image file is larger than a configurable size (defaults to 5GB). If it is larger, the backend store writes the image file in chunks and manually uploads those segments to Swift, writing the manifest file on successful upload to Swift. Unfortunately, if an image_size of 0 is given, we have no choice but to attempt to write the image file to Swift directly and hope the image file is not greater than Swift's maximum (before manifest-ing) object size. The alternative would be to have to write the image file to disk on the Glance API node for every single upload to Swift where image_size was 0, essentially doubling the writes for any image files not greater than 5GB. My guess is that you are either using an older version of the Glance client library (that doesn't seek/tell to set the Content-Length header) or you are using a non-standard way of uploading (Java perhaps? ;) Cheers, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
From: Jay Pipes [mailto:jaypi...@gmail.com] The else: block is ONLY met when you are not using the Python glance client, the glance CLI tool, and are not setting either the Content-Length or X-Image-Meta-Size header. If you use the Python glance client or CLI tool, the image you are feeding to the client automatically does a seek/tell to determine the size of the image you are uploading. Or if you're using something that can't seek/tell, like a stream. In that case, you get neither a Content-Length nor an X-Image-Meta-Size. Hence my question. Ewan. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
On Tue, Nov 15, 2011 at 3:37 PM, Ewan Mellor ewan.mel...@eu.citrix.com wrote: From: Jay Pipes [mailto:jaypi...@gmail.com] The else: block is ONLY met when you are not using the Python glance client, the glance CLI tool, and are not setting either the Content-Length or X-Image-Meta-Size header. If you use the Python glance client or CLI tool, the image you are feeding to the client automatically does a seek/tell to determine the size of the image you are uploading. Or if you're using something that can't seek/tell, like a stream. In that case, you get neither a Content-Length nor an X-Image-Meta-Size. Hence my question. And hence my answer ;) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
-Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Tuesday, November 15, 2011 12:45 PM To: Ewan Mellor Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Glance API semantics when image sizes aren't known On Tue, Nov 15, 2011 at 3:37 PM, Ewan Mellor ewan.mel...@eu.citrix.com wrote: From: Jay Pipes [mailto:jaypi...@gmail.com] The else: block is ONLY met when you are not using the Python glance client, the glance CLI tool, and are not setting either the Content-Length or X-Image-Meta-Size header. If you use the Python glance client or CLI tool, the image you are feeding to the client automatically does a seek/tell to determine the size of the image you are uploading. Or if you're using something that can't seek/tell, like a stream. In that case, you get neither a Content-Length nor an X-Image-Meta-Size. Hence my question. And hence my answer ;) I don't think you answered the question. My question was, what is the intended behavior when given a request that doesn't have either X-Image-Meta-Size or Content-Length set? Because the docs imply that the size will be inferred from the body of the upload request, but that's not what is happening in the Swift backend. Thanks, Ewan. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
On Tue, Nov 15, 2011 at 4:28 PM, Ewan Mellor ewan.mel...@eu.citrix.com wrote: -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Tuesday, November 15, 2011 12:45 PM To: Ewan Mellor Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Glance API semantics when image sizes aren't known On Tue, Nov 15, 2011 at 3:37 PM, Ewan Mellor ewan.mel...@eu.citrix.com wrote: From: Jay Pipes [mailto:jaypi...@gmail.com] The else: block is ONLY met when you are not using the Python glance client, the glance CLI tool, and are not setting either the Content-Length or X-Image-Meta-Size header. If you use the Python glance client or CLI tool, the image you are feeding to the client automatically does a seek/tell to determine the size of the image you are uploading. Or if you're using something that can't seek/tell, like a stream. In that case, you get neither a Content-Length nor an X-Image-Meta-Size. Hence my question. And hence my answer ;) I don't think you answered the question. My question was, what is the intended behavior when given a request that doesn't have either X-Image-Meta-Size or Content-Length set? It depends. If you are just reserving/registering image metadata, Glance will happily store 0, Content-Length or X-Image-Meta-Size. On upload, once an image file is successfully uploaded to the backend store, Glance will set the image's size to the number of bytes that the backend store reported that it wrote. On uploading an image when Swift is the backing store, Glance does its best to upload the image file if no Content-Length or X-Image-Meta-Size is specified, but on the event of a failed Swift upload, the image size does not get set. Because the docs imply that the size will be inferred from the body of the upload request, but that's not what is happening in the Swift backend. Would it be possible to paste the code you are using to do the upload? Cheers, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Tuesday, November 15, 2011 1:36 PM
To: Ewan Mellor
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Glance API semantics when image sizes aren't
known
On Tue, Nov 15, 2011 at 4:28 PM, Ewan Mellor
ewan.mel...@eu.citrix.com wrote:
-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Tuesday, November 15, 2011 12:45 PM
To: Ewan Mellor
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Glance API semantics when image sizes
aren't
known
On Tue, Nov 15, 2011 at 3:37 PM, Ewan Mellor
ewan.mel...@eu.citrix.com wrote:
From: Jay Pipes [mailto:jaypi...@gmail.com]
The else: block is ONLY met when you are not using the Python
glance
client, the glance CLI tool, and are not setting either the
Content-Length or X-Image-Meta-Size header. If you use the Python
glance client or CLI tool, the image you are feeding to the
client
automatically does a seek/tell to determine the size of the image
you
are uploading.
Or if you're using something that can't seek/tell, like a stream.
In
that case, you get neither a Content-Length nor an X-Image-Meta-
Size.
Hence my question.
And hence my answer ;)
I don't think you answered the question. My question was, what is
the intended behavior when given a request that doesn't have either X-
Image-Meta-Size or Content-Length set?
It depends. If you are just reserving/registering image metadata,
Glance will happily store 0, Content-Length or X-Image-Meta-Size.
On upload, once an image file is successfully uploaded to the backend
store, Glance will set the image's size to the number of bytes that
the backend store reported that it wrote.
On uploading an image when Swift is the backing store, Glance does its
best to upload the image file if no Content-Length or
X-Image-Meta-Size is specified, but on the event of a failed Swift
upload, the image size does not get set.
It also does not set the image size if the upload is successful.
image_size is returned unchanged from glance.store.swift.Store.add, so
0 is passed in, and passed out again.
So the question is whether this is a bug. Are you expecting the
backend to return a correct image size if the upload is successful?
Because the docs imply that the size will be inferred from the body
of the upload request, but that's not what is happening in the Swift
backend.
Would it be possible to paste the code you are using to do the upload?
~ # cat test_glance.py
import sys
import glance.client
client = glance.client.Client('localhost', 9292, auth_tok=999888777666)
print client.add_image({}, sys.stdin)
~ # echo a | python26 ./test_glance.py
{u'status': u'active', u'name': None, u'deleted': False, u'container_format':
None, u'created_at': u'2011-11-15T21:44:21', u'disk_format': None,
u'updated_at': u'2011-11-15T21:44:22', u'id': 6, u'owner': u'Administrator',
u'location': u'swift+http://root:password@localhost:5000/v1.0/glance/6',
u'min_ram': 0, u'checksum': u'60b725f10c9c85c70d97880dfe8191b3', u'min_disk':
0, u'is_public': False, u'deleted_at': None, u'properties': {}, u'size': 0}
Note that size is returned as 0, not 1.
Cheers,
Ewan.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
On Tue, Nov 15, 2011 at 4:51 PM, Ewan Mellor ewan.mel...@eu.citrix.com wrote:
-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Tuesday, November 15, 2011 1:36 PM
To: Ewan Mellor
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Glance API semantics when image sizes aren't
known
On Tue, Nov 15, 2011 at 4:28 PM, Ewan Mellor
ewan.mel...@eu.citrix.com wrote:
-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Tuesday, November 15, 2011 12:45 PM
To: Ewan Mellor
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Glance API semantics when image sizes
aren't
known
On Tue, Nov 15, 2011 at 3:37 PM, Ewan Mellor
ewan.mel...@eu.citrix.com wrote:
From: Jay Pipes [mailto:jaypi...@gmail.com]
The else: block is ONLY met when you are not using the Python
glance
client, the glance CLI tool, and are not setting either the
Content-Length or X-Image-Meta-Size header. If you use the Python
glance client or CLI tool, the image you are feeding to the
client
automatically does a seek/tell to determine the size of the image
you
are uploading.
Or if you're using something that can't seek/tell, like a stream.
In
that case, you get neither a Content-Length nor an X-Image-Meta-
Size.
Hence my question.
And hence my answer ;)
I don't think you answered the question. My question was, what is
the intended behavior when given a request that doesn't have either X-
Image-Meta-Size or Content-Length set?
It depends. If you are just reserving/registering image metadata,
Glance will happily store 0, Content-Length or X-Image-Meta-Size.
On upload, once an image file is successfully uploaded to the backend
store, Glance will set the image's size to the number of bytes that
the backend store reported that it wrote.
On uploading an image when Swift is the backing store, Glance does its
best to upload the image file if no Content-Length or
X-Image-Meta-Size is specified, but on the event of a failed Swift
upload, the image size does not get set.
It also does not set the image size if the upload is successful.
image_size is returned unchanged from glance.store.swift.Store.add, so
0 is passed in, and passed out again.
So the question is whether this is a bug. Are you expecting the
backend to return a correct image size if the upload is successful?
Absolutely. If it's not, that's a bug.
Because the docs imply that the size will be inferred from the body
of the upload request, but that's not what is happening in the Swift
backend.
Would it be possible to paste the code you are using to do the upload?
~ # cat test_glance.py
import sys
import glance.client
client = glance.client.Client('localhost', 9292, auth_tok=999888777666)
print client.add_image({}, sys.stdin)
~ # echo a | python26 ./test_glance.py
{u'status': u'active', u'name': None, u'deleted': False, u'container_format':
None, u'created_at': u'2011-11-15T21:44:21', u'disk_format': None,
u'updated_at': u'2011-11-15T21:44:22', u'id': 6, u'owner': u'Administrator',
u'location': u'swift+http://root:password@localhost:5000/v1.0/glance/6',
u'min_ram': 0, u'checksum': u'60b725f10c9c85c70d97880dfe8191b3', u'min_disk':
0, u'is_public': False, u'deleted_at': None, u'properties': {}, u'size': 0}
Note that size is returned as 0, not 1.
Yes, indeed. That is because the client is not designed to be used
that way, hence this in the glance.client code:
if hasattr(image_data, 'seek') and hasattr(image_data, 'tell'):
try:
image_data.seek(0, os.SEEK_END)
image_size = image_data.tell()
image_data.seek(0)
return image_size
except IOError, e:
if e.errno == errno.ESPIPE:
# Illegal seek. This means the user is trying
# to pipe image data to the client, e.g.
# echo testdata | bin/glance add blah..., or
# that stdin is empty
return None
else:
raise
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
On upload, once an image file is successfully uploaded to the
backend
store, Glance will set the image's size to the number of bytes that
the backend store reported that it wrote.
On uploading an image when Swift is the backing store, Glance does
its
best to upload the image file if no Content-Length or
X-Image-Meta-Size is specified, but on the event of a failed Swift
upload, the image size does not get set.
It also does not set the image size if the upload is successful.
image_size is returned unchanged from glance.store.swift.Store.add,
so
0 is passed in, and passed out again.
So the question is whether this is a bug. Are you expecting the
backend to return a correct image size if the upload is successful?
Absolutely. If it's not, that's a bug.
OK, I'll file a bug.
Because the docs imply that the size will be inferred from the
body
of the upload request, but that's not what is happening in the Swift
backend.
Would it be possible to paste the code you are using to do the
upload?
~ # cat test_glance.py
import sys
import glance.client
client = glance.client.Client('localhost', 9292,
auth_tok=999888777666)
print client.add_image({}, sys.stdin)
~ # echo a | python26 ./test_glance.py
{u'status': u'active', u'name': None, u'deleted': False,
u'container_format': None, u'created_at': u'2011-11-15T21:44:21',
u'disk_format': None, u'updated_at': u'2011-11-15T21:44:22', u'id': 6,
u'owner': u'Administrator', u'location':
u'swift+http://root:password@localhost:5000/v1.0/glance/6', u'min_ram':
0, u'checksum': u'60b725f10c9c85c70d97880dfe8191b3', u'min_disk': 0,
u'is_public': False, u'deleted_at': None, u'properties': {}, u'size':
0}
Note that size is returned as 0, not 1.
Yes, indeed. That is because the client is not designed to be used
that way [snip]
But it could perfectly well be used this way, modulo the bug above, and
this is a very useful thing to be able to do (stdin here could be a
decompression or decryption pipeline, or a read from a remote socket, or
whatever.
With the bug in the Swift backend fixed, I think it will work just fine
to stream through a glance client in this way.
Cheers,
Ewan.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
On Nov 15, 2011, at 2:17 PM, Ewan Mellor wrote: With the bug in the Swift backend fixed, I think it will work just fine to stream through a glance client in this way. Only if the image is 5 GB. Otherwise swift will blow up trying to store it. Vish Cheers, Ewan. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
From: Ewan Mellor
From: Jay Pipes [mailto:jaypi...@gmail.com]
From: Ewan Mellor
~ # cat test_glance.py
import sys
import glance.client
client = glance.client.Client('localhost', 9292,
auth_tok=999888777666)
print client.add_image({}, sys.stdin)
~ # echo a | python26 ./test_glance.py
{u'status': u'active', u'name': None, u'deleted': False,
u'container_format': None, u'created_at': u'2011-11-15T21:44:21',
u'disk_format': None, u'updated_at': u'2011-11-15T21:44:22', u'id':
6,
u'owner': u'Administrator', u'location':
u'swift+http://root:password@localhost:5000/v1.0/glance/6',
u'min_ram':
0, u'checksum': u'60b725f10c9c85c70d97880dfe8191b3', u'min_disk': 0,
u'is_public': False, u'deleted_at': None, u'properties': {}, u'size':
0}
Note that size is returned as 0, not 1.
Yes, indeed. That is because the client is not designed to be used
that way [snip]
But it could perfectly well be used this way, modulo the bug above, and
this is a very useful thing to be able to do (stdin here could be a
decompression or decryption pipeline, or a read from a remote socket,
or
whatever.
With the bug in the Swift backend fixed, I think it will work just fine
to stream through a glance client in this way.
Erm, well it will until we hit the 5GB Swift chunking limit, anyway.
Do you know what this means (glance/store/swift.py +347)?
Why can't we stream from body_file?
else:
# Write the image into Swift in chunks. We cannot
# stream chunks of the webob.Request.body_file, unfortunately,
# so we must write chunks of the body_file into a temporary
# disk buffer, and then pass this disk buffer to Swift.
Thanks,
Ewan.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack] OpenStack Asia Movement - Barrier
hi all based on my experience visiting Japan (Osaka/Tokyo) dan see there are a collective protective model of culture.. esp in Japan :) sorry to OS Leader, cannot meet u, because of this culture.. but I think that will be good idea if we can united and share more extensively about OS with all of us.. the chapter leader 1. culture barier to share outside country, in several case, outside internal country's community 2. language barier, esp we are not english speaking country, and we know. English is suck language.. word and pronouciation unmatch. a lot of asian country using double byte language, and that hard to understand. 3. goverment model to pull the program, and translation become trend, so the local language journal become a barrier for the better movement, esp in opensource world. 4. cost to meet in several country like indonesia become main issue, but less of asian people wanna to share in global mailing list, they love local mailing list. that is not happen in korea or japan, but here in indonesia.. i believe the 2 country malaysia and singapore better, they speak english.. but i dont see good community program there.. :) esp in singapore, sharing is not their culture, but sales-ing.. benefit unification 1. goverment have funding, and need greater program than local program.. i believe openstack can get benefit from this. 2. better content to bring new value to openstack in focus ... i think this is good for input for OpenStack Foundation model and local chapter model -- Frans Thamura (曽志胜) Chief of Advisory Meruvian. Integrated Hypermedia Java Solution Provider. Mobile: +628557888699 Blog: http://blogs.mervpolis.com/roller/flatburger (id) FB: http://www.facebook.com/meruvian TW: http://www.twitter.com/meruvian / @meruvian Website: http://www.meruvian.org We grow because we share the same belief. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance API semantics when image sizes aren't known
On Tue, Nov 15, 2011 at 5:26 PM, Ewan Mellor ewan.mel...@eu.citrix.com wrote:
From: Ewan Mellor
From: Jay Pipes [mailto:jaypi...@gmail.com]
From: Ewan Mellor
~ # cat test_glance.py
import sys
import glance.client
client = glance.client.Client('localhost', 9292,
auth_tok=999888777666)
print client.add_image({}, sys.stdin)
~ # echo a | python26 ./test_glance.py
{u'status': u'active', u'name': None, u'deleted': False,
u'container_format': None, u'created_at': u'2011-11-15T21:44:21',
u'disk_format': None, u'updated_at': u'2011-11-15T21:44:22', u'id':
6,
u'owner': u'Administrator', u'location':
u'swift+http://root:password@localhost:5000/v1.0/glance/6',
u'min_ram':
0, u'checksum': u'60b725f10c9c85c70d97880dfe8191b3', u'min_disk': 0,
u'is_public': False, u'deleted_at': None, u'properties': {}, u'size':
0}
Note that size is returned as 0, not 1.
Yes, indeed. That is because the client is not designed to be used
that way [snip]
But it could perfectly well be used this way, modulo the bug above, and
this is a very useful thing to be able to do (stdin here could be a
decompression or decryption pipeline, or a read from a remote socket,
or
whatever.
With the bug in the Swift backend fixed, I think it will work just fine
to stream through a glance client in this way.
Erm, well it will until we hit the 5GB Swift chunking limit, anyway.
Do you know what this means (glance/store/swift.py +347)?
Why can't we stream from body_file?
else:
# Write the image into Swift in chunks. We cannot
# stream chunks of the webob.Request.body_file, unfortunately,
# so we must write chunks of the body_file into a temporary
# disk buffer, and then pass this disk buffer to Swift.
We can (and we do in the case when image_size=0 when calling the Swift
backend store's add() method). Unfortunately, since you don't know the
file size, you begin streaming to Swift (essentially by passing
body_file on to the swift.client.Connection.put_object() method).
Unfortunately, if the file is 5GB, what will happen is that Swift
will vomit after attempting to upload 5G of data, resulting in a
situation where the Glance Swift storage driver would then have to try
to restart the upload using a segmented approach. But in order to do
that, the driver would have to seek(0) back on the body_file.
However, by default, webob.Request.body_file is NOT seekable. To make
it seekable, you need to either do this:
webob.Request.make_body_seekable()
or this:
webob.Request.is_body_seekable = True
Unfortunately, doing either of the above results in the entire request
body being buffered to disk on the server side in order for webob to
convert the (unseekable) StringIO it uses for webob.Request.body into
a (seekable) regular file object that it then assigns to its
webob.Request.body_file attribute.
So, let's say we tried to handle re-trying a segmented upload after we
first try to upload a 6GB image file *without* setting the image_size
to the actual size. What would happen is this:
Write 5GB of data to the swift socket
Get error, make body_file seekable in order to seek(0) back to the
request body's starting point, which results in copying the entire
body_file (6GB) into a temporary file on the server
seek(SEEK_END) on that body_file to get the real image size, and then
stream 6GB of data in chunks to Swift using its segmented upload
method.
Total data written to disk and network for 6GB image file: 5GB + 6GB +
6GB = 17GB
Or you can set the image_size properly by determining the file size
ahead of time. :)
You can read more about this issue here:
https://bugs.launchpad.net/glance/+bug/794718
https://bugs.launchpad.net/glance/+bug/818292
upstream:
https://bitbucket.org/ianb/webob/issue/12/fix-for-issue-6-broke-chunked-transfer
Cheers,
-jay
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack-poc] PPB Meeting
Reminder that we have a PPB meeting scheduled at 2000UTC/2:00 PM CST. Does anyone have anything they'd like to cover? One thing that might be worth discussing is a concern several people raised to me last week at Cloud Expo around the Essex release and having some sort of quality gate. It seems like a lot of people definitely want to avoid another release that isn't fully baked at release time. We may want to discuss if there's anything else we should start putting in place now towards that goal. Thoughts? Jonathan. ___ Mailing list: https://launchpad.net/~openstack-poc Post to : openstack-poc@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-poc More help : https://help.launchpad.net/ListHelp
Re: [Openstack-poc] PPB Meeting
I'd love to chat about the status of increased quality gating. We've been working on this pretty solidly since the design summit and have the infrastructure in place to be able to do the testing. We're currently working with folks on getting an install and a test suitably solid enough that we can put it in place as a gate. Happy to update everyone on the progress and status there though. On 11/15/2011 12:25 PM, Jonathan Bryce wrote: Reminder that we have a PPB meeting scheduled at 2000UTC/2:00 PM CST. Does anyone have anything they'd like to cover? One thing that might be worth discussing is a concern several people raised to me last week at Cloud Expo around the Essex release and having some sort of quality gate. It seems like a lot of people definitely want to avoid another release that isn't fully baked at release time. We may want to discuss if there's anything else we should start putting in place now towards that goal. Thoughts? Jonathan. ___ Mailing list: https://launchpad.net/~openstack-poc Post to : openstack-poc@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-poc More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack-poc Post to : openstack-poc@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-poc More help : https://help.launchpad.net/ListHelp
