Re: [Openstack] Memory meters from ceilometer
Hi Jobin, As explained earlier, the existing memory and disk.* meters are not measures of utilization in the classic sense (i.e. the % of an existing limited resource currently used and available). If you simply want to know the quantum of memory allocated to the instance by the hypervisor, then the existing memory meter is fine. Similarly if all you need to know is ephermal disk size or the rate of read/write IOPs, then the existing disk.* meters address this. If on the other hand you want to know how much free memory or unused diskspace remains available to processes running on the instance, then this would require new meters as I explained earlier on this thread. Cheers, Eoghan - Original Message - Thanks Eoghan for your detailed response regarding the current scenario and progress related to this project. Had been investigating about this for while since they had been documented here in the official ceilometer measurements pagehttp://docs.openstack.org/developer/ceilometer/measurements.html. Thanks for your enlightenment! On Thu, Jul 18, 2013 at 8:43 PM, Eoghan Glynn egl...@redhat.com wrote: Hi Jobin, The memory utilization metering will require a new release of libvirt which will not be available for another few weeks. After that, it will depend on there being developer bandwidth available to put the ceilometer support in place for a new meter type. We have no current plans that I'm aware of to implement metering of disk utilization (in the sense of available diskspace, equivalent to that reported by say running the df command on the instance). But I could see it being a useful thing. The information is certainly available via the libvirt domain blockInfo, though there may be some wrinkles around multiple block devices versus volumes attached. In any case, please submit[1] a bug with a request for enhancement. Cheers, Eoghan [1] https://bugs.launchpad.net/ceilometer/+filebug - Original Message - Thanks a lot Eoghan for your detailed response. I have enabled instance usage auditing in my nova.conf http://pastebin.ubuntu.com/5887592/. Is there anyway I could get these meters(memory and disk utilization) for VM's provisioned using OpenStack? Thanks for your efforts. On Thu, Jul 18, 2013 at 4:57 PM, Eoghan Glynn egl...@redhat.com wrote: Hey Jobin, Thanks for your perceptive question. The reason is that the conduits for gathering CPU metering and memory metering are quite different in ceilometer currently: * cpu/cpu_util are derived by polling the libvirt daemon * memory is derived from the compute.instance.exists notification sent by nova (This is the pollster versus notification-handler dichotomy you'll see throughout ceilometer). The reason you're not seeing the memory meter being collected is probably because you don't have instance usage auditing enabled in your nova config (see the Configure nova section in [1]). However, be warned that this meter is probably not the memory utilization statistic that you're expecting. Rather it's likely to be a static value reflecting the quantum of memory allocated by the hypervisor to the instance (as opposed to the sort of number you'd see when running free -m on the instance). I'm looking into adding support for more useful memory utilization metering (that could for example drive autoscaling logic), but this will require usage of an upcoming release of libvirt. Cheers, Eoghan [1] http://docs.openstack.org/developer/ceilometer/install/manual.html Hey! I am running openstack on an Ubuntu 12.04 desktop 64-bit virtual machine. I am trying to use ceilometer to get the CPU and memory utilization from my compute node on a KVM host which has a few VM's running on it. However, this is the list of meters I get from ceilometer: cpu, cpu_util, disk.read.bytes, disk.write.bytes, disk.read.requests, disk.write.requests, image, image.size, image.download, instance, instance:m1.small, network.incoming.bytes, network.outgoing.bytes, network.incoming.packets, network.outgoing.packets Why am I not getting memory usage meters? I don't see the logs to have any interpretation of this. Here is my ceilometer config file . -- Thanks and regards, Jobin Raju George Third Year, Information Technology College of Engineering Pune Alternate e-mail: georgejr10...@coep.ac.in ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net
Re: [Openstack] Memory meters from ceilometer
Hey Jobin, Thanks for your perceptive question. The reason is that the conduits for gathering CPU metering and memory metering are quite different in ceilometer currently: * cpu/cpu_util are derived by polling the libvirt daemon * memory is derived from the compute.instance.exists notification sent by nova (This is the pollster versus notification-handler dichotomy you'll see throughout ceilometer). The reason you're not seeing the memory meter being collected is probably because you don't have instance usage auditing enabled in your nova config (see the Configure nova section in [1]). However, be warned that this meter is probably not the memory utilization statistic that you're expecting. Rather it's likely to be a static value reflecting the quantum of memory allocated by the hypervisor to the instance (as opposed to the sort of number you'd see when running free -m on the instance). I'm looking into adding support for more useful memory utilization metering (that could for example drive autoscaling logic), but this will require usage of an upcoming release of libvirt. Cheers, Eoghan [1] http://docs.openstack.org/developer/ceilometer/install/manual.html Hey! I am running openstack on an Ubuntu 12.04 desktop 64-bit virtual machine. I am trying to use ceilometer to get the CPU and memory utilization from my compute node on a KVM host which has a few VM's running on it. However, this is the list of meters I get from ceilometer: cpu, cpu_util, disk.read.bytes, disk.write.bytes, disk.read.requests, disk.write.requests, image, image.size, image.download, instance, instance:m1.small, network.incoming.bytes, network.outgoing.bytes, network.incoming.packets, network.outgoing.packets Why am I not getting memory usage meters? I don't see the logs to have any interpretation of this. Here is my ceilometer config file . -- Thanks and regards, Jobin Raju George Third Year, Information Technology College of Engineering Pune Alternate e-mail: georgejr10...@coep.ac.in ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Memory meters from ceilometer
Hi Jobin, The memory utilization metering will require a new release of libvirt which will not be available for another few weeks. After that, it will depend on there being developer bandwidth available to put the ceilometer support in place for a new meter type. We have no current plans that I'm aware of to implement metering of disk utilization (in the sense of available diskspace, equivalent to that reported by say running the df command on the instance). But I could see it being a useful thing. The information is certainly available via the libvirt domain blockInfo, though there may be some wrinkles around multiple block devices versus volumes attached. In any case, please submit[1] a bug with a request for enhancement. Cheers, Eoghan [1] https://bugs.launchpad.net/ceilometer/+filebug - Original Message - Thanks a lot Eoghan for your detailed response. I have enabled instance usage auditing in my nova.conf http://pastebin.ubuntu.com/5887592/. Is there anyway I could get these meters(memory and disk utilization) for VM's provisioned using OpenStack? Thanks for your efforts. On Thu, Jul 18, 2013 at 4:57 PM, Eoghan Glynn egl...@redhat.com wrote: Hey Jobin, Thanks for your perceptive question. The reason is that the conduits for gathering CPU metering and memory metering are quite different in ceilometer currently: * cpu/cpu_util are derived by polling the libvirt daemon * memory is derived from the compute.instance.exists notification sent by nova (This is the pollster versus notification-handler dichotomy you'll see throughout ceilometer). The reason you're not seeing the memory meter being collected is probably because you don't have instance usage auditing enabled in your nova config (see the Configure nova section in [1]). However, be warned that this meter is probably not the memory utilization statistic that you're expecting. Rather it's likely to be a static value reflecting the quantum of memory allocated by the hypervisor to the instance (as opposed to the sort of number you'd see when running free -m on the instance). I'm looking into adding support for more useful memory utilization metering (that could for example drive autoscaling logic), but this will require usage of an upcoming release of libvirt. Cheers, Eoghan [1] http://docs.openstack.org/developer/ceilometer/install/manual.html Hey! I am running openstack on an Ubuntu 12.04 desktop 64-bit virtual machine. I am trying to use ceilometer to get the CPU and memory utilization from my compute node on a KVM host which has a few VM's running on it. However, this is the list of meters I get from ceilometer: cpu, cpu_util, disk.read.bytes, disk.write.bytes, disk.read.requests, disk.write.requests, image, image.size, image.download, instance, instance:m1.small, network.incoming.bytes, network.outgoing.bytes, network.incoming.packets, network.outgoing.packets Why am I not getting memory usage meters? I don't see the logs to have any interpretation of this. Here is my ceilometer config file . -- Thanks and regards, Jobin Raju George Third Year, Information Technology College of Engineering Pune Alternate e-mail: georgejr10...@coep.ac.in ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Thanks and regards, Jobin Raju George Third Year, Information Technology College of Engineering Pune Alternate e-mail: georgejr10...@coep.ac.in ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Summit Runners
+1 Thanks, Eoghan - Original Message - G'day, Would anyone be interested in morning runs (5K) during the Summit in PDX next week? If you are, let's meet in the lobby of the Portland Hilton on Sixth Avenue at 0600 on Monday and 0700 from Tuesday to Friday. Some of the Monday morning runners have breakfast meetings, hence the early start. It's a great way to see a bit of Portland! Luke ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [openstack-dev] [metering][ceilometer] Unified Instrumentation, Metering, Monitoring ...
Here's a first pass at a proposal for unifying StackTach/Ceilometer and other instrumentation/metering/monitoring efforts. It's v1, so bend, spindle, mutilate as needed ... but send feedback! http://wiki.openstack.org/UnifiedInstrumentationMetering Thanks for putting this together Sandy, We were debating on IRC (#heat) earlier the merits of moving the ceilometer emission logic into the services, e.g. directly into the nova-compute node. At first sight, this seemed to be what you were getting at with the suggestion: Remove the Compute service that Ceilometer uses and integrate the existing fanout compute notifications into the data collected by the workers. There's no need for yet-another-worker. While this could be feasible for measurements driven directly by notifications, I'm struggling with the idea of moving say the libvirt polling out of the ceilometer compute agent, as this seems to leak too many monitoring-related concerns directly into nova (cadence of polling, semantics of libvirt stats reported etc.). So I just wanted to clarify whether the type of low level unification you're proposing includes both push pull (i.e. notification polling) or whether you mainly had just former in mind when it comes to ceilometer. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ceilometer] meter data volume
if you have: Time | Value 0 | 10 1 | 30 2 | 50 3 | 80 4 | 100 If your delta-pollster is down at 1 and 2, you restart at 3, therefore at 4 you'll send 20 as usage (100 minus 80). So you miss the delta between 10 (time 0) and 80 (time 3) (therefore 70 for free!). If you send right away 80 at time 3 when restarting, the API will be able to guess that between 0 and 3 the value went from 10 to 80. With delta approach, the API cannot guess that. Sure it can, you just need to move where the caching is done. Using a local cache to maintain the previous time a value was published you would know at time 3 that the last published value was 10, and so send 70. So the total will be correct. Good point, previously IIUC there was an implicit assumption that any prev time caching would be done in-memory, hence lost across process restarts. But as you point out, these data could be persisted locally by the compute agent. What would be the best way to achieve this? A small sqlite DB per-agent, or even simpler just a pickled dict? The latter would avoid the complexity of DB versioning and migration. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ceilometer] meter data volume
Hi Yawei Wu, The root of the confusion is the fact the cpu meter is reporting the cumlative cpu_time stat from libvirt. This libvirt counter is reset when the associated qemu process is restarted (an artifact of how cpuacct works). So when you stop/start or suspend/resume, a fresh qemu process is sparked up, then the cumulative time is reset. Thanks for bringing this up, as it has implications as to how we meter CPU time and utilization[1]. We may need to start metering the delta between CPU times on subsequent polling cycles, instead of using a cumulative meter (dealing with the edge case where the instance has been restarted within a polling period). Cheers, Eoghan [1] https://review.openstack.org/14921 I am still testing ceilometer now. I am confused about the meter volume in the mongodb. Let's talk about cpu usage. After I create and boot a vm named vm_1, meter data record about cpu usage will be inserted into db in cycle(default 10 minutes). For example,the 'counter_volume' of the first record is '5206000',and the second one is '12389000'. 1) '12389000' nanoseconds means '123.89' seconds or two minutes,it seem like to be 1238.9 seconds actually, is there something wrong ? 2) If I never reboot or suspend vm_1, will the 'counter_volume' of cpu usage record increase all the time ? Just like '8 minutes' - '18 minutes' - '28 minutes' ? 3) If I reboot or suspend vm_1, I find that the 'counter_volume' of cpu usage record will count from zero. Just like '8 minutes' - '18 minutes' - '28 minutes' [- '0 minutes'] -'5 minutes' - '15 minutes'. Does it mean that 'counter_volume' just represents how long has vm_1 been booted up ? 4) This one is about Web API. I find that GET /v1/resources/(resource)/meters/(meter)/volume/sum just return the sum value of all the cpu 'counter_volume', like '8 minutes' + '18 minutes'. Is it reduplicate ? 5) If I want to know how long has vm_1's cpu been used yesterday, how can I do ? It seems like that I have too many questions.. Thank you very much ! --- Yawei Wu Dalian Hi-Think Computer Technology,Corp. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ceilometer] meter data volume
Not at all. It means the CPU time consumed is reset to 0, but that's not an issue in itself, the API should be capable to deal with that if you ask for the total usage. Would that total usage be much more apparent if we started metering the delta between CPU times on subsequent polling periods as a gauge measure? (As opposed to treating it as a cumulative measure) /v1/resources/(resource)/meters/(meter)/volume/sum just return the sum value of all the cpu 'counter_volume', like '8 minutes' + '18 minutes'. Is it reduplicate ? Don't understand what you mean, but the CPU counter is a cumulative one, and asking for its sum is a non-sense. You want to ask for (max - min) to get the used value, something which is not in the API yet. I don't think (max - min) would suffice to give an accurate measure of the actual CPU time used, as the counter may have reset multiple times in the course of the requested duration. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ceilometer] meter data volume
I don't think (max - min) would suffice to give an accurate measure of the actual CPU time used, as the counter may have reset multiple times in the course of the requested duration. It is, because /max in the API should be aware of the fact a reset can occur and computes accordingly. We started to discuss this a bit in: https://bugs.launchpad.net/ceilometer/+bug/1061817 A-ha, OK, so not so much (max - min) as: (\Sigma local maxima) - first Sounds computationally expensive to produce on the fly, but maybe the local maxima can be efficiently recorded as the data is being ingested. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ceilometer] meter data volume
If your pollster is not running to compute delta and you have no state stored, you'll miss a part of what has been used. Would we have also have some 'misses' with the cumulative approach when the ceilometer agent was down? If I understood the (\Sigma local maxima)-first idea correctly, the usage up to the first polling cycle would always be discounted from any duration. Similarly, calculating the time delta as a gauge measure would discount only the usage up to the first libvirt poll after each ceilo agent restart. As long as the ceilo compute agent was restarted only rarely, I'm not sure the under-reporting would be a huge issue in either case. A more pernicious problem would occur if the instance was being regularly restarted with a higher frequency than the polling period. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [ceilometer] meter data volume
Would we have also have some 'misses' with the cumulative approach when the ceilometer agent was down? No, unless the counter resets several times while your agent is down. But delta has the same issue. If I understood the (\Sigma local maxima)-first idea correctly, the usage up to the first polling cycle would always be discounted from any duration. No, because if you have: Time | Value 0| 10 1| 30 2| 50 3| 80 4| 100 If your delta-pollster is down at 1 and 2, you restart at 3, therefore at 4 you'll send 20 as usage (100 minus 80). So you miss the delta between 10 (time 0) and 80 (time 3) (therefore 70 for free!). If you send right away 80 at time 3 when restarting, the API will be able to guess that between 0 and 3 the value went from 10 to 80. With delta approach, the API cannot guess that. Yep the sum of local maxima is not lossy as long as the requested duration completely encapsulates the compute agent outage (and the instance doesn't restart during the outage). However I was more thinking of the scenario where the duration requested via the API is say t1..t4 in your example above. In any case, do we need a new measurement type, in addition to the existing CUMULATIVE type, that captures the non-monotonic nature of the measure and alerts the API that special handling is required to compute say max-min? Something like TRANSIENT_CUMULATIVE, if that's not too much of a mouthful. A more pernicious problem would occur if the instance was being regularly restarted with a higher frequency than the polling period. Yes, but in that case, whatever counting method you use, you're screwed. :) True that. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Discussion / proposal: Ability to reset tenant's quotas to default
HI All,
I would like to open a discussion on a topic user should have a
option to reset the tenant’s quotas( to the default).
Hi Vijaya,
I don't think a new nova command is needed for this use-case,
just add a simple custom script:
nova quota-update `nova quota-defaults $1 | tail -n +4 | tr '_' '-' | awk
'/|/ {printf( --%s %s, $2,$4)}'` $1
then call with the tenant ID as command line arg.
Cheers,
Eoghan
At present nova client has following commands for the quota
operation.
$nova --help | grep quota
quota-defaults List the default quotas for a tenant.
quota-show List the quotas for a tenant.
quota-update Update the quotas for a tenant.
It will be very helpful to have a command to reset quota values to
defaults .
For ex: User who wants to do huge tests on the system and rollback
once the test is done.
So my proposal is to add a new command quota-reset to the nova client
which reverts the quota value supplied for the tenant ,to the
default.
Some thing similar to nova quota-reset( tenant-id key)
Let me know your suggestion/thoughts on the same.
Thanks,
Vijaya
DISCLAIMER == This e-mail may contain privileged and
confidential information which is the property of Persistent Systems
Ltd. It is intended only for the use of the individual or entity to
which it is addressed. If you are not the intended recipient, you
are not authorized to read, retain, copy, print, distribute or use
this message. If you have received this communication in error,
please notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus
infected mails.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Discussion / proposal: Ability to reset tenant's quotas to default
Isn't that just changing one custom limit with another ?
A true reset to the defaults would see the user stay in step with any
changes to the default values.
Do you mean configured changes to the defaults?
AFAIK 'nova quota-defaults' returns the current set of defaults,
which seems to be the logical point to reset against.
HI All,
I would like to open a discussion on a topic user should have a
option
to reset the tenant’s quotas( to the default).
Hi Vijaya,
I don't think a new nova command is needed for this use-case, just
add a simple custom script:
nova quota-update `nova quota-defaults $1 | tail -n +4 | tr '_' '-'
| awk '/|/ {printf( --%s %s, $2,$4)}'` $1
then call with the tenant ID as command line arg.
Cheers,
Eoghan
At present nova client has following commands for the quota
operation.
$nova --help | grep quota
quota-defaults List the default quotas for a tenant.
quota-show List the quotas for a tenant.
quota-update Update the quotas for a tenant.
It will be very helpful to have a command to reset quota values to
defaults .
For ex: User who wants to do huge tests on the system and rollback
once the test is done.
So my proposal is to add a new command quota-reset to the nova
client
which reverts the quota value supplied for the tenant ,to the
default.
Some thing similar to nova quota-reset( tenant-id key)
Let me know your suggestion/thoughts on the same.
Thanks,
Vijaya
DISCLAIMER == This e-mail may contain privileged and
confidential information which is the property of Persistent
Systems
Ltd. It is intended only for the use of the individual or entity to
which it is addressed. If you are not the intended recipient, you
are
not authorized to read, retain, copy, print, distribute or use this
message. If you have received this communication in error, please
notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus
infected mails.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Discussion / proposal: Ability to reset tenant's quotas to default
My point was that if a user is currently configured to have a quota of 50 VMs, and the default is currently configured to be 20 VMs then there is a difference between configuring the user to have a quota of 20 and configuring a user to have the default quota.The first is just a subsequent update to give a user a different but still specific quota value, whereas the second undoes any quota value that has been specifically assigned. And the intent here is the second case. A-ha, got it now. Perhaps quota delete would be a more appropriate description (and the right way to implement this in the API) ? Yep, that would probably be clearer. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] One potential issue on the normalize_time() in timeutils
Thanks Yunhong for pointing this issue out and submitting a patch in quick order. Your reasoning for switching from if offset to if offset is None, in order to avoid including the offset==0 case, makes perfect sense. You'll just have to propose the change first to openstack-common, from where it will be copied to the nova, glance etc. codebases. Cheers, Eoghan - Original Message - I create a patch for it https://review.openstack.org/#/c/12705/ and please help to review. Thanks --jyh -Original Message- From: openstack-bounces+yunhong.jiang=intel@lists.launchpad.net [mailto:openstack-bounces+yunhong.jiang=intel@lists.launchpad.net] On Behalf Of Jiang, Yunhong Sent: Monday, September 10, 2012 8:17 AM To: Robert Collins Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] One potential issue on the normalize_time() in timeutils Rob, thanks for comments. I totally agree with you that aware datetime object is better than naive one. The key thing is, utcnow() will return naive object, that means in some time, we have to use naive object to compare with utcnow(), and we need a conversion function to convert from aware to naive. The normalize_time() is the best candidate for this purpose, but it will fail to convert to naive datetime object in some situation. That's why I send the mail. I just want to change the normalize_time() to make sure it will always return naive object. Thanks --jyh -Original Message- From: Robert Collins [mailto:robe...@robertcollins.net] Sent: Monday, September 10, 2012 3:25 AM To: Jiang, Yunhong Cc: egl...@redhat.com; openstack@lists.launchpad.net Subject: Re: [Openstack] One potential issue on the normalize_time() in timeutils On Mon, Sep 10, 2012 at 3:09 AM, Jiang, Yunhong yunhong.ji...@intel.com wrote: Hi, Eoghan and all, When I implement an enhancement to trusted_filter, I need utilize timeutils() to parse ISO time. However, I suspect there is one potential issue in normalize_time() and want to get some input from your side. In normalize_time(), if the parameter timestamp is an aware object (http://docs.python.org/library/datetime.html) , it's output will vary depends on the input. If the timestamp is UTC time, it will be return as is without convention, i.e still an aware object. However, if it's not an UTC time, it will be converted to be a naive object. This mean that the function will return different type depends on input, that's not so good IMHO. The worse is, any compare/substract between naïve object and aware object will fail. Because the timeutils.utcnow() and datetime.datetime.now() returns naive object, so compare/substract between the uctnow() and normalize_time() may fail, or not, depends on input from the API user. I'm a bit surprised that changes-since works on such situation :) I suspect this is caused by the if offset. When timestamp is naïve object, the offset is None. Thus check if offset will avoid operation type exception. However, if the timestamp is UTC time, the offset will be date.timeslot(0), which will return false also for if offset. Are there any special reason that we need keep aware object if input is at UTC time already? Can I changes the function to always return naive object? If yes, I can create a patch for it. You are probably better off creating an aware datetime object, and using them pervasively across the codebase, than using naive objects. As you note, they are not interoperable, and I've seen countless bugs where folk try to mix and match. If we want to show local date time to users *anywhere*, we'll need TZ aware objects, which is why that variation is the one to standardise on. Otherwise, you end up with multiple conversion points, and you can guarantee that at least one will get it wrong. -Rob ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] A plea from an OpenStack user
I think its great that we're having this discussion.
+1, excellent discussion in terms of both tone content.
In the hope that its informative, I'd like to give some info on
issues
we're looking at when moving our Glance deployment to Folsom. A lot
of
this is in common with Ryan, but there are a couple of twists due to
our
goals of maximization of uptime (ie we are hoping to do a rolling
rather
than lock-step upgrade) and decoupling upgrades. Also, I mention the
case where you may have existing Glance clients which you don't
control.
In our case the upgrade of the various components (swift/glance/nova
etc) will be staggered rather than performed simultaneously. (For
large
organisations I imagine this may often be the case.) If we are to
avoid
downtime for Glance we must simultaneously support ids and uuids.
These
must be presented appropriately to Nova nodes which we must assume
are
moving targets -- ie will initially be on older code, but will
upgrade
to Folsom.
We have some ideas on how this may be possible but haven't worked
through
all the details yet (in particular the Nova database entries)... but
there could be some coding for Nova/Glance and some deploys of the
interoperable code before eventually switching to standard Folsom.
Its unfortunate that the glance.images.id-uuid migration didn't follow
the nova.instances.{id|uuid} co-existence pattern, where the old IDs
are maintained in the DB and also continue to be supported for
identification purposes in the API.
But in any case, I presume your migration scenario is pre-Essex
to Folsom? (as the glance UUIDs were already in place for Essex)
I wonder though for the more tractable migration of Essex to Folsom,
should we consider building in tolerance for mixed old/new glance
deployments during a rolling upgrade of horizontally scaled glance-api
services?
Given that (a) the api service is now DB-aware, whereas previously this
was limited to the registry service, and (b) the amount of churn in the
glance DB schema was relatively limited between Essex and Folsom (a
new image_tags table, an additional images.protected column and some
munging of any swift images.location URIs).
For an Essex glance-api service to continue to run against the Folsom DB
schema, it might only require that any *new* swift location URIs are quoted
after the fact, and we bake in tolerance for unquoted URIs in the Folsom
code that interacts with the swift backend. (I would need to confirm that ...)
(Jay -- I don't think scripts are sufficient here?)
If Glance were publically available its not clear how the id/uuid
change
could be worked through gracefully where we didn't have control over
the
glance client. Ie the upgrade would break existing customers' clients
which expected an id rather than a uuid.
Other than maintaining *both* the existing numeric id and the new varchar
UUID as described above, while allowing the image to identified by either,
I don't see how support for old clients could work.
Cheers,
Eoghan
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] boot command fails
nova.api.openstack self._iterator.next() 2012-09-04 13:09:16 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py, line 572, in iterconsume 2012-09-04 13:09:16 TRACE nova.api.openstack yield self.ensure(_error_callback, _consume) 2012-09-04 13:09:16 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py, line 503, in ensure 2012-09-04 13:09:16 TRACE nova.api.openstack error_callback(e) 2012-09-04 13:09:16 TRACE nova.api.openstack File /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py, line 553, in _error_callback 2012-09-04 13:09:16 TRACE nova.api.openstack raise rpc_common.Timeout() 2012-09-04 13:09:16 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response. 2012-09-04 13:09:16 TRACE nova.api.openstack 2012-09-04 13:09:16 INFO nova.api.openstack [req-ebaafb8a-770f-467e-b0c4-d89527bc87b4 cf8971efd8934844b559d26e238506cc 141d12e2431f47a5bf77f90da4800960] http://ip:8774/v2/141d12e2431f47a5bf77f90da4800960/servers/2314f007-d446-477a-bcf0-6a5f77d4d25b/action returned with HTTP 500 2012/9/3 Eoghan Glynn egl...@redhat.com While trying to create a VM instance on openstack, the boot command (nova boot) returns the following error: --- ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) --- everything seems to be working (nova services are starting). I am using an Ubuntu 12.04 server (amd64) with Xen as a virtualization technology + the Essex version of openstack The used image was manually created (it is an ubuntu also). I can start it via xm commands. Any idea how to solve this problem? The first step would be to surface more detailed information on the failure that has occurred. In Essex, most internal nova exceptions are mapped directly to 500 Server Error, which effectively hides the underlying error condition from the client. Folsom is more permissive in this regard, so that internal exceptions declared safe for exposure are returned to the user. So in your case, you'll need to scour the nova-api, nova-scheduler nova-compute logs to get visibility on the underlying error condition. A quick short-cut would be to note the request ID returned (use nova --debug boot ...) and then grep for this in the logs mentioned above. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] boot command fails
While trying to create a VM instance on openstack, the boot command (nova boot) returns the following error: --- ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) --- everything seems to be working (nova services are starting). I am using an Ubuntu 12.04 server (amd64) with Xen as a virtualization technology + the Essex version of openstack The used image was manually created (it is an ubuntu also). I can start it via xm commands. Any idea how to solve this problem? The first step would be to surface more detailed information on the failure that has occurred. In Essex, most internal nova exceptions are mapped directly to 500 Server Error, which effectively hides the underlying error condition from the client. Folsom is more permissive in this regard, so that internal exceptions declared safe for exposure are returned to the user. So in your case, you'll need to scour the nova-api, nova-scheduler nova-compute logs to get visibility on the underlying error condition. A quick short-cut would be to note the request ID returned (use nova --debug boot ...) and then grep for this in the logs mentioned above. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Unable to start NOVA VOLUME
I have Installed Nova Volume in the Openstack Essex Controller. But When I restart the nova-volume service, I get the following error [...] 2012-09-03 12:26:22 TRACE nova OperationalError: (OperationalError) (1054, Unknown column 'volumes.instance_id' in 'field list') Hi Trinath, One possibility is that you've mismatched versions of nova core (specifically the DB schema) and nova volume. The volumes.instance_id column exists in the Essex schema, but was later dropped and replaced with an instance_uuid column (to reflect the foreign key type changing from int to UUID-as-varchar). Have you run a DB sync to up the schema version beyond Essex (i.e. past migration index 82)? Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Unable to start NOVA VOLUME
I have Installed Nova Volume in the Openstack Essex Controller. But When I restart the nova-volume service, I get the following error [...] 2012-09-03 12:26:22 TRACE nova OperationalError: (OperationalError) (1054, Unknown column 'volumes.instance_id' in 'field list') Hi Trinath, One possibility is that you've mismatched versions of nova core (specifically the DB schema) and nova volume. The volumes.instance_id column exists in the Essex schema, but was later dropped and replaced with an instance_uuid column (to reflect the foreign key type changing from int to UUID-as-varchar). Have you run a DB sync to up the schema version beyond Essex (i.e. past migration index 82)? I should have mentioned that a quick way to check if a migration has been applied, and if so report the current version, just run the following SQL statement against your nova DB: SELECT MAX(id) FROM migrations Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] AWS CreateImage compat
Hi Jorge, What version are you testing against? I recently got a series of patches onto master that addressed a bunch of issues in the EC2 CreateImage support, so that it now works smoothly with volume-backed nova instances: https://review.openstack.org/9732 https://review.openstack.org/9833 https://review.openstack.org/9915 https://review.openstack.org/9925 https://review.openstack.org/9715 This functionality is tested with a new devstack exercise, awaiting approval: https://review.openstack.org/9853 Also, I have backported the patches to stable/essex where they await review: https://review.openstack.org/10224 https://review.openstack.org/10225 https://review.openstack.org/10226 https://review.openstack.org/10228 https://review.openstack.org/10230 https://review.openstack.org/10238 Cheers, Eoghan - Original Message - Hello guys, Can you point me if there is any effort (patch, proposal) to provide a CreateImage API endpoint fully compatible with the Amazon EC2 API?. Boto create_image method is broken with the current API. -- Jorge Niedbalski R. -- ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance image add error
Can you provide relevant glance-api and -registry log excerpts?
Also probably best to track this as a glance question[1] or bug[2].
Cheers,
Eoghan
[1] https://answers.launchpad.net/glance
[2] https://bugs.launchpad.net/glance
- Original Message -
Hello
I'm getting this error from glance when i try to add an image, plz
help
Failed to add image. Got error:
The response body:
{badMethod: {message: The server could not comply with the
request since it is either malformed or otherwise incorrect.,
code: 405}}
Note: Your image metadata may still be in the registry, but the
image's status will likely be 'killed'.
I try to add the image with a command like this one:
glance add name=ubuntu-oneiric is_public=true container_format=ovf
disk_format=qcow2 ubuntu-11.10-server-cloudimg-amd64-disk1.img
I'm using ubuntu server 12.04 as my OS and I'm using the ubuntu
packages to install OpenStack (by hand not using scripts)
Thanks in advance!
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] quota question
We're running a system with a really wide variety of node types. This variety (nodes with 24GB, 48GB, GPU nodes, and 1TB mem nodes) causes some real trouble with quotas. Basically, for any tenant that is going to use the large memory nodes (even in smaller slices), we need to set quotas that are high enough that they are useless when it comes to all other instance types. Also, instances aren't comparable across our range of hardware either. Is there a clever solution to this problem that I'm missing? Is anyone else running into this sort of properly operationally? I'm a little bit worried that a slightly too simple notion of quotas has been baked into openstack at a fairly deep level. thanks for any insight.. Hi Narayan, I had the idea previously of applying a weighting function to the resource usage being allocated from the quota, as opposed to simply counting raw instances. The notion I had in mind was more related to image usage in glance, where the image footprint can vary very widely. However I think it could be useful for some nova resources also. Now for some resource types, for example say volumes, usage can be controlled along multiple axes (i.e. number of volumes and total size), so that gives more flexibility. But if I'm hearing you correctly, you'd want to apply a lower weighting to instances that are scheduled onto one of the higher-memory compute nodes, and vice versa a higher weighting to instances that happen to be run on lower-memory nodes. Does that sum it up, or have I misunderstood? BTW what kind of nova-scheduler config are you using? Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] quota question
Sounds like one solution alright.. But - what about making quotas pluggable, like the scheduler? Yeah, that could certainly be a direction to head in the longer term. The way things stand though, the decision as to which quota is being checked against in made at the enforcement point, and the question posed of the quotas engine is really just a dict mapping resource names to the numbers requested (i.e. there isn't any further context provided). So in order to allow the quotas engine ask more detailed questions about the kind of resource required (i.e. is the instance requested SSD-backed or whatever), we'd have provide a lot more context than we currently do. Cheers, Eoghan This would allow for even more complex quotas, like limiting the number of SSD backed instances across the entire cloud per tenant, while still keeping the core implementation lean. Kiall On Jul 20, 2012 3:48 PM, Eoghan Glynn egl...@redhat.com wrote: The harder part is that we need to be able to specify independent/orthogonal quota constraints on different flavors. It would be really useful to be able to say basically, you can have 2TB of memory from this flavor, and 4TB of memory from that flavor. This would allow saying something like you can have up to 3 1TB instances, and independently have up to 3TB of small instances as well. OK, so its the as well aspect that's problematic here. (If it were an either-or situation as opposed to a both, then obviously a combination of the instances and RAM quotas would get you part of the way at least). So just thinking aloud, we could potentially add new per-flavor quota resources so that for each existing instance-type, there was the potential to add a new quota limiting *only* that instance type (and maybe keep the existing instances quotas as an over-arching limit). For example, if the following quotas where set: instances: 100 instances-m1.xlarge: 10 instances-m1.large: 20 instances-m1.small: 50 instances-m1.tiny: 100 and a user requested an additional xlarge instance, we'd first check if we had headroom on the instances-m1.xlarge quota and then if we also had headroom on the over-arching instances quota (before going on to check the ram cores if necessary). Whereas, if a medium instance was requested, we would only check the overarching limit, as there is no instances-medium quota defined. This would require some change to the quotas logic, to allow the set of resources that may be limited by quotas to be more dynamic (currently we have a fairly fixed set, whereas new instance types may be defined at any time). Would that address your requirement? Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] quota question
Would that address your requirement? I think so. If these acted as a hard limit in conjunction with existing quota constraints, I think it would do the trick. I've raised this a nova blueprint, so let's see if it gets any traction: https://blueprints.launchpad.net/nova/+spec/flavor-specific-instance-quotas Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova and asynchronous instance launching
Note that I do distinguish between a 'real' async op (where you really return little more than a 202) and one that returns a skeleton of the resource being created - like instance.create() does now. So the latter approach at least provides a way to poll on the resource status, so as to figure out if and when it becomes usable. In the happy-path, eventually the instance status transitions to ACTIVE and away we go. However, considering the unhappy-path for a second, is there a place for surfacing some more context as to why the new instance unexpectedly went into the ERROR state? For example even just an indication that failure occurred in the scheduler (e.g. resource starvation) or on the target compute node. Is the thought that such information may be operationally sensitive, or just TMI for a typical cloud user? Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [Continuous-Integration] What else is running on the Jenkins slaves?
Folks, A question for the CI side-of-the-house ... What else is running on the Jenkins slaves, concurrently with the gating CI tests? The background is the intermittent glance service launch failure - the recently added strace-on-failure logic reveals the issue to be an EADDRINUSE when the registry service listen socket is bound to a supposedly unused port. Two possible explanations for this: 1. A race whereby some other process jumps in grabs this port before the registry service is launched (the window of opportunity is not too narrow, as the API service is being launched in the meantime). 2. We identify the unused port by quickly opening a closing a socket on port zero - there could I guess be some lag in recycling the port, but this seems unlikely as no connections were established, hence no need for TIME_WAIT. Option #1 seems the more likely, so I wanted to confirm there is indeed other port-grabbing stuff running on the Jenkins slaves. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Continuous-Integration] What else is running on the Jenkins slaves?
Thanks for the quick response ... Very basic things, not much other than the Jenkins Slave service and SSH. Nothing that should cause conflicts that you are seeing. We also intentionally only run one test run per slave at a time. Interesting, seems the alternate explanation of a lag-on-closure is the more likely in that case. Are you closing ports with SO_REUSEADDR? If the registry service or something else isn't then I guess that could cause it. We do set SO_REUSEADDR on the registry server socket, but not on the dummy socket used to identify an unused port. But I think setting SO_REUSEADDR on the latter would defeat the purpose of the dummy socket, by breaking the constraint that the port should be previously unused. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [nova][ec2] EC2 CreateImage API and nova boot-from-volume
Hi Folks, I've been looking into the (currently broken) EC2 CreateImage API support and just wanted to get a sanity check on the following line of reasoning: - EC2 CreateImage should *only* apply to booted-from-volume nova servers, for fidelity with the EC2 limitation to EBS-based instances (i.e. should raise InvalidParameterValue when the API is applied to an instance-store style server). So my previous crack at this https://review.openstack.org/8532 was going completely in the wrong direction. - Normally, a snapshot of a bootable-volume is booted via the native API tooling with something like: nova boot --image IMAGE_ID --block-device-mapping vd[a-z]=SNAP_ID:snap::0 ... where AFAICS the IMAGE_ID is *only* used to determine the kernel and ramdisk IDs and is otherwise irrelevant. - The EC2 CreateImage on the other hand requires that a usable image ID be returned, not set of a volume snapshot IDs. - The resulting image should be bootable via EC2 RunInstances, but doesn't necessarily need to be portable, as it depends on local snapshot ID(s). Here a few different potential approaches to the creation of this image: 1. Create a place-holder image in glance with the image data being effectively empty, and the following properties set: * the imaged instance's kernel and ramdisk IDs * block device mapping containing the appropriate snapshot ID(s) so that we can boot from this image without providing additional context (such as the via nova boot --block-device-mapping option) 2. Extend the s3_image mapping logic such that an ami-* style ID can be mapped directly to a set of properties, snapshot IDs etc (so there would no image registration with glance). 3. Construct an AMI manifest containing the appropriate blockDeviceMapping and then leverage EC2 RegisterImage logic to create an AMI image. Some questions: - Does any of the above approaches to the image creation make sense? Option #3 in particular seems uneccessarily complex to me, and would I think require some reverse-engineering of the manifest format. I'm not sure there's much benefit in aping the EC2 image bundling/ registration approach given that the resulting image wouldn't be portable outside the current openstack deployment. However I may be missing something there ... - How should the lifecycle of the image and the corresponding snapshot(s) be intertwined? (e.g. should a deletion of the snapshot cause the corresponding image to be also deleted or marked unusable?) - Would a corresponding feature for the native API make sense? i.e. an equivalent of the nova createImage action that does not depend on the qemu-img command, but instead works for booted-from-vol servers. This would avoid the counter-intuitive use of an imageRef that's only used to grab the kernel and ramdisk IDs. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [Continuous-Integration] using strace from a test suite running on Ubuntu
Hi Folks, I wanted to use strace(1) to get to the bottom of the glance service launch failures that have been plaguing Smokestack and Jenkins in the past few weeks: https://review.openstack.org/8722 However I just realized that Ubuntu from Maverick onward no longer allows ptrace to attach to a user's own processes that are not direct children :( So I'm wondering whether the CI side-of-the-house would be prepared to enable this temporarily on the Jenkins slaves, by running: echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope prior to the glance tests being kicked off, then reverting straight after. Or, for a more permanent setting: sed -e 's/kernel\.yama.\ptrace_scope *= *1/kernel.yama.ptrace_scope = 0/' /etc/sysctl.d/10-ptrace.conf and then re-cast the image used for the Jenkins slaves. Thanks, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Glance] Replication implementations
BTW that patch is up for review as: https://review.openstack.org/7302 Cheers, eoghan I'm working on a patch to at least make the glance UUID - S3 image ID not totally depend on an on-demand insertion order as it does now. Agreed, collisions are inevitable given the relative domain and range sizes (122 unique bit UUID versus 32-bit hex string) - in testing, the first colliding UUID tends to occur after ~75k-80k images IDs have been generated. So at least it would be useful for smaller deployments to have a semi-predictable ID mapping (modulo collisions). For larger deployments, the mapping data to be replicated could be much reduced by limiting it to the colliding IDs. Cheers, Eoghan Alternatively, we could just consider the ec2 mapping layer to be global data that must be replicated somehow across the system. I don't think we can really ensure no collisions mapping from uuid - ec2_id deterministically, and I don't see a clear path forward when we do get a collision. Vish On May 8, 2012, at 12:24 AM, Michael Still wrote: On 04/05/12 20:31, Eoghan Glynn wrote: Sorry for the slow reply, I've been trapped in meetings. [snip] So the way things currently stand, the EC2 image ID isn't really capable of migration. I was thinking however that we should change the EC2 image generation logic, so that there is a reproducible glance UUID - EC2 mapping (with a small chance of collision). This change would allow the same EC2 ID to be generated in multiple regions for a given glance UUID (modulo collisions). Would that be helpful in your migration use-case? I do think this is a good idea. Or even if the column wasn't auto-increment, but just picked a random number or something (because that would be marginally less likely to clash). Without somehow making these ec2 ids more global, replication between regions is going to suffer from ec2 api users having to somehow perform a lookup out of band. Now, my use case is a bit special, because I can enforce that images are only ever uploaded to one master region, and then copied to all others. I think that's probably not true for other users though. Mikal ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Glance] Replication implementations
I'm working on a patch to at least make the glance UUID - S3 image ID not totally depend on an on-demand insertion order as it does now. Agreed, collisions are inevitable given the relative domain and range sizes (122 unique bit UUID versus 32-bit hex string) - in testing, the first colliding UUID tends to occur after ~75k-80k images IDs have been generated. So at least it would be useful for smaller deployments to have a semi-predictable ID mapping (modulo collisions). For larger deployments, the mapping data to be replicated could be much reduced by limiting it to the colliding IDs. Cheers, Eoghan Alternatively, we could just consider the ec2 mapping layer to be global data that must be replicated somehow across the system. I don't think we can really ensure no collisions mapping from uuid - ec2_id deterministically, and I don't see a clear path forward when we do get a collision. Vish On May 8, 2012, at 12:24 AM, Michael Still wrote: On 04/05/12 20:31, Eoghan Glynn wrote: Sorry for the slow reply, I've been trapped in meetings. [snip] So the way things currently stand, the EC2 image ID isn't really capable of migration. I was thinking however that we should change the EC2 image generation logic, so that there is a reproducible glance UUID - EC2 mapping (with a small chance of collision). This change would allow the same EC2 ID to be generated in multiple regions for a given glance UUID (modulo collisions). Would that be helpful in your migration use-case? I do think this is a good idea. Or even if the column wasn't auto-increment, but just picked a random number or something (because that would be marginally less likely to clash). Without somehow making these ec2 ids more global, replication between regions is going to suffer from ec2 api users having to somehow perform a lookup out of band. Now, my use case is a bit special, because I can enforce that images are only ever uploaded to one master region, and then copied to all others. I think that's probably not true for other users though. Mikal ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Glance] Replication implementations
Current warts: ... - maintaining amazon ec2 ids across regions requires twiddling the nova database where this mapping is stored Hi Mikal, We discussed that nova s3_images table earlier in the week on IRC. Now at the time, I wasn't fully clear on the mechanics of the glance UUID - EC2-style image ID mapping. I've had a dig into the code since then and it turns out that the EC2 image ID is effectively determined by an auto_increment column in the s3_images table. So there is nothing globally significant about that ID, in fact it's completely determined by the insertion order into the nova s3_images table, which occurs *on demand* the first time the image is being referenced in an EC2-related context (for example the first call to DescribeImages after the image has been uploaded to glance). For example in a fresh openstack deployment, if three images are uploaded with types aki, ari aki respectively, the first EC2-related reference to the images will result in the following EC2 IDs always being generated: aki-0001 ari-0002 ami-0003 Now, if the underlying glance images are migrated to another long-standing openstack deployment, it's likely that there are already multiple images referenced in the local s3_images table, so that the identifiers a[krm]i-000[1-3] are already taken. So the way things currently stand, the EC2 image ID isn't really capable of migration. I was thinking however that we should change the EC2 image generation logic, so that there is a reproducible glance UUID - EC2 mapping (with a small chance of collision). This change would allow the same EC2 ID to be generated in multiple regions for a given glance UUID (modulo collisions). Would that be helpful in your migration use-case? Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Glance] Replication implementations
Current warts: ... - maintaining amazon ec2 ids across regions requires twiddling the nova database where this mapping is stored Hi Mikal, We discussed that nova s3_images table earlier in the week on IRC. Now at the time, I wasn't fully clear on the mechanics of the glance UUID - EC2-style image ID mapping. I've had a dig into the code since then and it turns out that the EC2 image ID is effectively determined by an auto_increment column in the s3_images table. So there is nothing globally significant about that ID, in fact it's completely determined by the insertion order into the nova s3_images table, which occurs *on demand* the first time the image is being referenced in an EC2-related context (for example the first call to DescribeImages after the image has been uploaded to glance). For example in a fresh openstack deployment, if three images are uploaded with types aki, ari aki respectively, the first EC2-related reference to the images will result in the following EC2 IDs always being generated: aki-0001 ari-0002 ami-0003 Now, if the underlying glance images are migrated to another long-standing openstack deployment, it's likely that there are already multiple images referenced in the local s3_images table, so that the identifiers a[krm]i-000[1-3] are already taken. So the way things currently stand, the EC2 image ID isn't really capable of migration. I was thinking however that we should change the EC2 image generation logic, so that there is a reproducible glance UUID - EC2 mapping (with a small chance of collision). This change would allow the same EC2 ID to be generated in multiple regions for a given glance UUID (modulo collisions). Would that be helpful in your migration use-case? Of course, the other way of looking at this would be to just follow the lead of EC2 itself and not provide an image ID portability across regions. Though I think we can do better than that by compressing the UUID in a predictable way. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance problem
Hi Andrei,
The underlying issue is starvation of the storage space used to store
image content(as opposed to the image metadata, which takes up very
little space).
The reason the killed image isn't showing up in the output of glance index
is that non-viable images are sanitized from the list.
If you want to be rid of this image, you can easily find the offending image
ID from the glance registry log, e.g.:
$ grep killed /var/log/glance/registry.log
...
012-04-30 11:26:48 9198 INFO [sqlalchemy.engine.base.Engine] ('2012-04-30
10:26:48.806445', u'killed', u'e75a8d52-0cc5-481d-b765-3a056d80ad28')
Then you can just delete the metadata directly from the glance registry DB.
For example if using the default sqlite DB, use:
$ sudo sqlite3 glance.sqlite DELETE from images where id =
'e75a8d52-0cc5-481d-b765-3a056d80ad28'
If on the other hand you're using mysql or postgres, just issue the DELETE query
via the appropriate DB client.
Cheers,
Eoghan
Hi , please help me with this. I try to add a image to glance and i
receive the following message:
glance add -A 651abbc762f84e8e9b25baeec35c54bd name=$name
is_public=true container_format=bare disk_format=raw
$imageUploading image 'server'
[ 99%] 19.7M/s, ETA
0h 0m 0sFailed to add image. Got error:
The request returned a 413 Request Entity Too Large. This generally
means that rate limiting or a quota threshold was breached.
The response body:
413 Request Entity Too Large
The body of your request was too large for this server.
Image storage media is full: There is not enough disk space on the
image storage media.
Note: Your image metadata may still be in the registry, but the
image's status will likely be 'killed'.
=[100%] 19.7M/s, ETA
0h 0m 0s
The problem is that i don't know how to delete this metadata (which i
believe is ocupying my glance). When i try to see my images i see
only this ones.
ctrl@ubuntu:~/devstack$ glance index -A
651abbc762f84e8e9b25baeec35c54bd
ID Name Disk Format Container Format Size
--
--
ffd57095-0bb5-4678-b6ea-2460140d3084 cirros-0.3.0-x86_64-uec-ramdis
ari ari 2254249
48806fc7-2f95-4c83-b7d6-091d59731187 cirros-0.3.0-x86_64-uec-kernel
aki aki 4731440
465d5101-9cdd-4a2c-b55d-534c33949491 cirros-0.3.0-x86_64-uec ami ami
25165824
ctrl@ubuntu:~/devstack$ glance details -A
651abbc762f84e8e9b25baeec35c54bd
Can anybody help?
Andrei-Cosmin Ion
telefon: 0727 768 281
email: andrei_t...@yahoo.com
Munceste ca si cum n-ai muri niciodata, dar ingrijeste-te de sufletul
tau ca si cum ai muri maine!
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack][Nova] Minimum required code coverage per file
- Original Message - Kevin, should we start copying openstack-common tests to client projects? Or just make sure to not count openstack-common code in the code coverage numbers for client projects? That's a tough one. If we copy in the tests, they end up being somewhat redundant, but slow down the project unit tests, but on the other hand, we'd be able to easily demonstrate that that code works properly. I think I'd prefer if we just try to not count openstack-common code for code coverage numbers… Yep, one approach would be for the copy script to decorate the common code with whatever signals the coverage exclusion (e.g. #pragma: no cover) so that the code is counted in the openstack-common coverage metrics, but not counted after being copied into nova/glance/wherever. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] database migration cleanup
https://review.openstack.org/#/c/6847/ Nice! * Migrations added during Folsom release cycle could be compacted during E release cycle. TBD if/when we do the next compaction. An alternative idea would be to do the compaction *prior* to the Folsom relase instead of after, so that the cleanest possible migration path is presented to non-trunk-chasing users. It could for example be a task that's part of spinning up the first Folsom-RC. Its unlikely that after new migrations are added after the release candidate goes out the door (as these are generally associated with non-trivial new features, which would have missed the boat at that late stage). But if there are any, these would have to be adde to the squashed aggregate migration from the get-go. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Pending reviews
There's something like 7 pages of open reviews on gerrit. The project has a good kind of problem with so many people trying to contribute. The question now is how to scale the development processes to handle that growth. It was nice to see a number of discussions at the summit in this area. The biggest backlog is nova, and there are discussions of both splitting parts out to make nova smaller, as well as adopting feature branches and merge windows. The feature branches could have more reviewers that are experts in that area, but not necessarily nova-core. Hopefully these things will help in the Folsom cycle. Thanks to all of the core reviewers who regularly invest time into reviewing submissions! :-) Some simple processes that I've seen improve matters on seemingly unmanagable backlogs: 1. An initial short concerted queue draining exercise (e.g. a review-busting day where all core team members agree to dedicate a significant portion of their openstack time to reviews). The intended outcome is a much leaner queue as a starting point (at the cost of potential instability with many more patches landing on master than would normally be the case, so it makes sense to do this early in the release cycle). 2. Prominent visibility to a number of simple stats that capture the trend on responsiveness: - age of the oldest unreviewed patch - average turnaround time from submission to merge or -2 - number of open unreviewed patches - number of reviewed patches needing approval There would an implicit goal not to leave the stats in worse shape than yesterday at the end of each core-team members' rostered review day. 3. A loose SLA indicating the level of responsiveness that patch submitters can expect, e.g. we strive to respond within X working days, average turnaround time is currently Y days. 4. If things get out of hand again GOTO #1. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] New Gerrit version (and server)
We've just upgraded Gerrit to version 2.3. There are a lot of changes behind the scenes that we've been looking forward to (like being able to store data in innodb rather than myisam tables for extra data longevity). And there are a few visible changes that may be of interest to OpenStack developers. Am I imagining things, or has a little bug crept into the way the Reviewer Verified Code-Review Approved matrix is rendered? i.e. if a reviewer +2's the patch but doesn't also approve it, this is now indicated with a green tick in the Code-Review column and also a red x in Approved column. Previously this would have been a green tick and a blank. The new style seems a tad misleading, i.e. it suggests a Do not submit on a first glance. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] control user quota
I try to assign quota to individual users, to control how many instances each user can run concurrently. But I don't see a doc describing how to do that. I use diablo release. Any help or doc pointer will be greatly appreciated. Quotas apply at the nova project/tenant granularity, as opposed to an individual user. Project-specific quotas may be set via the nova CLI, e.g. $ nova quota-update tenant_ID --instances=50 otherwise the configured default quota is inherited. Since you're still on diablo, the new quota classes mechanism would not be relevant. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Swift block-level deduplication
Folks, From previous posts on the ML, it seems there are a couple of efforts in train to add distributed content deduping to Swift. My question is whether either or both these approaches involve active client participation in enabling duplicate chunk detection? One could see a spectrum ranging between: 1. Client actively breaks the object into chunks, selects the hashing algorithm, calculates fingerprint and then only uploads if Swift reports that fingerprint is unknown. 2. Client determines which objects are worth deduping, maybe has some influence on chunk size and/or hashing, but fingerprint calculation is all handled internally by Swift. 3. Client is entirely uninvolved, deduplication is handled transparently in the object storage layer and enabled either globally or per-container. If anyone involved has insight into the above, I'd be interested in hearing your thoughts (the context is leveraging dedupe in glance). Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Swift block-level deduplication
Thanks for the response Caitlin, The versioning/dedup ring we are working on at Nexenta will support both 1 and 3. I'll be presenting at the Summit on this. Great, I'll look forward to your presentation. The ultimate goal of distributed dedup is scenario #1. Only the client software can determine the optimum chunk boundaries, Agreed, this is where the maximum savings can be acheived, at the cost of imposing some complexity on the client layer. I'd be interested in using glance as a proof-point for this approach, its one of the topics I intended to discuss at the summit as part of this session: http://summit.openstack.org/sessions/view/78 Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Image API v2 Draft 4
APPENDIX B: Outstanding issues
...
2) How do we fit the existing 'copy_from' functionality in?
Is the v2 API retaining some equivalent of the existing
x-image-meta-location header, to allow an externally-stored
image be registered with glance?
e.g. via an image field specified on create or update:
POST /images HTTP/1.1
{external-location: s3://access:sec...@s3.amzonaws.com/image, ...}
or:
PUT /images/IMAGE_ID HTTP/1.1
{external-location: s3://access:sec...@s3.amzonaws.com/image, ...}
If so, the most straight-forward approach for copy-from would be to
follow a similar pattern with an image field such as:
POST /images HTTP/1.1
{copy-from: s3://access:sec...@s3.amzonaws.com/image, ...}
... etc.
Cheers,
Eoghan
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Quota classes
Eoghan Glynn wrote: - how is the mapping between project and quota-class established? I was expecting a project_quota_class_association table or some-such in the nova DB. Is this association maintained by keystone instead? - is the quota_class attribute currently being set on the request context anywhere in the dispatch path? Is the idea that the auth middleware takes care of this? Kevin Mitchell wrote: The basic answer is that there isn't anything in nova right now that does this, partly because it's a slightly difficult question to answer correctly for everyone. In my testing environment, for instance, I use a Turnstile preprocessor to set the quota_class attribute on the request context to be the same as the selected rate limit class. I envisioned that, ultimately, the quota_class would be set by the authentication processing middleware(s), but I'm not against adding an association to nova to manage that. One more follow-up question on this. So IIUC the mechanism as it currently stands implies a 1:M mapping between quota classes and projects/tenants. Only a single quota class may be selected for each request context, so *all* the individual hard limits for the different resource types encompassed by that class are inherited as default quotas for the request. But might there be a usecase for mutiple quota classes applying to an individual project (for different resource types)? e.g. compute-hungry/storage-light customers might select the premium package for instances/cores/RAM combined with the basic package for volumes/gigabytes (or vice versa for storage-heavy but narrowly-scaled apps). If we were to go down that road, we'd require an N:M association between quota class names projects, right? Or equivalently, a 1:M mapping between quota class IDs and projects. Also the single quota class name in the request context would have to be replaced with a list of either quota class IDs or (quota class name, resource type) pairs. I guess the other alternative would be to simply create new quota classes for each permutation of resource limits required, e.g. premium-compute-basic-storage, basic-compute-premium-storage etc. We'd get asome proliferation on the number of quota classes, but in realistic scenarios it wouldn't go combinatorial. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OpenStack Community Newsletter –March 30, 2012
COMMUNITY STATISTICS • Activity on the main branch of OpenStack repositories, lines of code added and removed per developer during week 7 of 2012 (from Mon Mar 19 00:00:00 UTC 2012 to Mon March 26 00:00:00 UTC 2012) Hi Stefano, Assuming you're using git-log to generate these stats, you might want to throw the --find-copies-harder option into the mix, to avoid skewing the stats on minor re-orgs of the codebase directory structure. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Quota classes
I wanted to let everyone know about a quota classes blueprint I've submitted; you can find the details here: * https://blueprints.launchpad.net/nova/+spec/quota-classes * http://wiki.openstack.org/QuotaClass I've already implemented this blueprint and pushed to Gerrit, but have it -2'd for right now since we haven't opened trunk yet for Folsom. Hi Kevin, A couple of quick questions on how this quota class mechanism is intended to work ... - how is the mapping between project and quota-class established? I was expecting a project_quota_class_association table or some-such in the nova DB. Is this association maintained by keystone instead? - is the quota_class attribute currently being set on the request context anywhere in the dispatch path? Is the idea that the auth middleware takes care of this? Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Quota classes
Eoghan Glynn wrote: A couple of quick questions on how this quota class mechanism is intended to work ... - how is the mapping between project and quota-class established? I was expecting a project_quota_class_association table or some-such in the nova DB. Is this association maintained by keystone instead? - is the quota_class attribute currently being set on the request context anywhere in the dispatch path? Is the idea that the auth middleware takes care of this? Kevin Mitchell wrote: The basic answer is that there isn't anything in nova right now that does this, partly because it's a slightly difficult question to answer correctly for everyone. In my testing environment, for instance, I use a Turnstile preprocessor to set the quota_class attribute on the request context to be the same as the selected rate limit class. I envisioned that, ultimately, the quota_class would be set by the authentication processing middleware(s), but I'm not against adding an association to nova to manage that. Presumably we'd also need some additional logic in the quota-classes API extension to allow tenant-to-quota-class mappings be established and torn down? e.g. POST /v2/class-id/os-quota-class-sets/tenant-id and: DELETE /v2/class-id/os-quota-class-sets/tenant-id to establish and tear down respectively. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Quota classes
Presumably we'd also need some additional logic in the quota-classes API extension to allow tenant-to-quota-class mappings be established and torn down? Well, yeah :) Cool, captured in https://bugs.launchpad.net/nova/+bug/969537 I'll propose a patch early next week. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Specify an owner when adding an image in Glance
Hi Juerg, That's because 'owner' is not supported as an explicit parameter to 'glance add'. So as a result the CLI treats it a generic image property, and passes this to the API service via the header: x-image-meta-property-owner: 2 The 'x-image-meta-property-' prefix is used to distinguish additional image properties, separate to the first class image attributes. Please raise a bug against the glance CLI so that 'owner' is supported as an explicit field for the add command (as it currently is for the update command). Cheers, Eoghan Is there a particular reason why an owner can't be specified when adding an image? I.e., the following: $ glance add name=testing owner=99 testing results in: URI: http://jabba:9292/v1/images/22 Id: 22 Public: No Name: testing Status: active Size: 36614 Disk format: None Container format: None Minimum Ram Required (MB): 0 Minimum Disk Required (GB): 0 Owner: 2 Property 'owner': 99 whereas I expect it to be: URI: http://jabba:9292/v1/images/22 Id: 22 Public: No Name: testing Status: active Size: 36614 Disk format: None Container format: None Minimum Ram Required (MB): 0 Minimum Disk Required (GB): 0 Owner: 99 Regards ...Juerg ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Specify an owner when adding an image in Glance
Done. https://bugs.launchpad.net/glance/+bug/962998 Thanks, fixed here: https://review.openstack.org/5727 Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] nova-manage quota --headroom
Folks, One thing that's been on my wishlist since hitting a bunch of quota exceeded issues when first running Tempest and also on the Fedora17 openstack test day. It's ability to easily see the remaining headroom for each per-project quota, e.g. $ nova-manage quota --headroom --project=admin ... instances:10 (of which 1 remaining) floating_ips: 10 (of which 8 remaining) ... This would give an immediate indication of an impending resource starvation issue - shoot, I'll have to clean out some of those old instances before spinning up two more, else increase the quota. It would only really be useful for quotas that represent a threshold on overall resource usage that may grow or shrink over time, as opposed to some fixed limit (think, max instances versus max injected files per instance). So the question is whether there's already a means to acheive this in one fell swoop? And if not, would it be best tracked with a small-scale nova blueprint, or as an enhancement request expressed in a launchpad bug? Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-manage quota --headroom
Thanks Jay for the feedback and background info, comments inline ... Eoghan Glynn wrote: So the question is whether there's already a means to acheive this in one fell swoop? Jay Pipes wrote: Well, Horizon, in the launch instance modal dialog does show similar information. See attached screenshot. That dialog uses a call to the contrib/simple_usage extension of the compute API Lovely, that's exactly the type of representation I had in mind (but wearing my old-school CLI-oriented hat). Jay Pipes wrote: Basically, you could take any of these approaches: a) Modify the simple_usage extension to return information about the quotas along with the usage information b) Make a new extension to the API that marries the usage with the quota c) Make a new novaclient call that exposes the married information A new extension that combines the quota with a report on current usage levels (i.e. options b+c) seems neatest to me. Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-manage quota --headroom
Kevin Mitchell wrote: I recently got the quota classes stuff merged into master (after the RC branch for Essex was cut, of course). After I had completed that work, I started thinking about quotas in general, and I think there's a better way to organize how we do quotas in the first place. One potential side effect I see of that possible work would be to address this particular feature. My idea here is not really very well fleshed out or architected yet, and I have other things I'm focusing on right now (caching), but perhaps I should mention them on the list, in order to get a discussion going…and perhaps a summit discussion of the topic would be in order? Yep, that's a good idea. Let me know if you'd be interested in collaborating on a summit session. I'll have a detailed look at your recent work on quota classes before responding to your quota infrastructure thoughts below ... but on a first look, definitely sounds sane. Cheers, Eoghan The basic thought I had here was to create a quota manager or quota plugin infrastructure, instead of a bunch of fixed functions in quota.py that we use to test certain specific quotas. Having it as a plugin that you specify in configuration means you can easily substitute out the default implementation, if necessary—maybe you want quotas to be enforced across several cells instead of just one, for instance. Then, checking a quota would be a matter of simply calling a method on the quota plugin class. The side effect is that, in order to properly impose quotas in a flexible architecture, you would need to keep the quota plugin up-to-date on the current numbers of whatever resource is being limited anyway—in the example above, of having one quota applied across a number of cells, you don't want to have to ask each cell how many instances the user currently has. That makes implementation of your headroom concept pretty trivial, and I could see it implemented by attaching a little more data to the /limits endpoint's returned data. (The current quota code has to count how many instances there are each time it checks the quota; I think other parts that use quotas may do the counting themselves and just ask quota.py what the upper limit is.) -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] When how are glance-cache* (.conf, -paste.ini) files loaded / parsed ?
Florian, The key point in the split between glance-api.conf, glance-registry.conf, glance-cache.conf etc. is the glance application intended to consume that config. This follows directly from the naming: bin/glance-api by default consumes glance-api.conf bin/glance-registry by default consumes glance-registry.conf bin/glance-cache-* by default consumes glance-cache.conf bin/glance-scrubber by default consumes glance-scrubber.conf This is merely a convention, which can be overridden for example by naming the glance API service config as foobar.conf: bin/glance-api --config-file /path/to/foobar.conf However the naming convention is convenient as it may allow the pathname of the config file to be inferred if not explicitly specified, for example for the glance-api application, the follow search order is used: ~/.glance/glance-api.conf ~/glance-api.conf /etc/glance/glance-api.conf /etc/glance-api.conf or, in general, replace glance-api above with the program name, i.e. basename(sys.argv[0]) The intended consumer then determines what options should be specified in each config file, for example there would be no point in defining the the backend s3/swift store config in glance-cache.conf, similarly no need to define the max cache size in glance-api.conf, nor the scrubber wakeup time in glance-registry.conf. Then each .conf file has a corresponding -paste.ini, which splits along a different axis. Here the idea is to separate the core and paste deploy config for a particular glance application. So for example the default paste config for glance-api is glance-api-paste.ini, to be found in the same directory as glance-api.conf. Again this is merely a convenient convention that may be overridden. Does that all make sense? Cheers, Eoghan Can someone help me understand what options need to be in glance-api.conf and what options can be left to glance-cache.conf , resp glance-cache-paste.ini ? Case in point: If I wanted to use the xattr driver, I need to specify that in glance-api.conf -- specifying that in glance-cache.conf is ignored. The trivial solution is to copy the options from glance-cache.conf and paste them in glance-api.conf but I hardly think this was the intention of having the two split. So I'd like to understand when how those two files are loaded / parsed. The questions above is for the image cache, but I guess the same type of questions can be asked for the scrubber (i.e. glance-scrubber.conf resp glance-scrubber-paste.ini) Least I forget: This is for the E4 code release. TIA for the help, Florian ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] When how are glance-cache* (.conf, -paste.ini) files loaded / parsed ?
Yes, it does make perfect sense. Kind thanks for the explanation.
However, what is still unclear is what config iteems that pertain to
other apps must still be present (ie. duplicated in) glance-api.conf
(e.g. image_cache_driver , etc )
This is probably something we should document more carefully so it's clear
to users from the outset which config options are required and when.
However I don't have a definitive mapping of config items to apps, so I'd
normally consider them on a case-by-case basis and just check where the config
option is used. For example it may be always required by a particular app, or
only be required if a particular middleware is enabled.
In this particular case, a little grep'ing in the codebase soon reveals when
the image_cache_driver config item is required.
$ find glance -name *.py | grep -v test | xargs grep -n -B 5
image_cache_driver
glance/image_cache/__init__.py-33-class ImageCache(object):
glance/image_cache/__init__.py-34-
glance/image_cache/__init__.py-35-Provides an LRU cache for image
data.
glance/image_cache/__init__.py-36-
glance/image_cache/__init__.py-37-opts = [
glance/image_cache/__init__.py:38:cfg.StrOpt('image_cache_driver',
default='sqlite'),
--
glance/image_cache/__init__.py-48-
glance/image_cache/__init__.py-49-def init_driver(self):
glance/image_cache/__init__.py-50-
glance/image_cache/__init__.py-51-Create the driver for the cache
glance/image_cache/__init__.py-52-
glance/image_cache/__init__.py:53:driver_name =
self.conf.image_cache_driver
So we see that this config option is pulled in by the
glance.image_cache.ImageCache
class, which is then used as follows:
$ find glance -name *.py | grep -v test | xargs grep ImageCache | cut -f1
-d: | sort | uniq
glance/api/cached_images.py
glance/api/middleware/cache.py
glance/image_cache/cleaner.py
glance/image_cache/__init__.py
glance/image_cache/prefetcher.py
glance/image_cache/pruner.py
glance/image_cache/queue_image.py
Looking the first two source files, we see that the image_cache_driver config
option
would be required in the glance-api application iff a caching or
cachemanagement-based
pipeline is selected.
Cheers,
Eoghan
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] can not start glance-api in glance E4
Does /etc/glance/policy.json exist?
Is is readable?
- Original Message -
From: .。o 0 O泡泡 501640...@qq.com
To: openstack openstack@lists.launchpad.net
Sent: Wednesday, 7 March, 2012 2:06:50 PM
Subject: [Openstack] can not start glance-api in glance E4
hi all:
In glance E4 ,when I enter follows:
sudo glance-api glance-api.conf --debug
It returns :
2012-03-07 21:53:50 26589 ERROR [glance.store.filesystem] Could not
find filesystem_store_datadir in configuration options.
2012-03-07 21:53:50 26589 ERROR [glance.store.base] Failed to
configure store correctly. Disabling add method.
2012-03-07 21:53:50 26589 ERROR [glance.store.s3] Could not find
s3_store_host in configuration options.
2012-03-07 21:53:50 26589 ERROR [glance.store.base] Failed to
configure store correctly. Disabling add method.
2012-03-07 21:53:50 26589 ERROR [glance.store.swift] Could not find
swift_store_auth_address in configuration options.
2012-03-07 21:53:50 26589 ERROR [glance.store.base] Failed to
configure store correctly. Disabling add method.
Traceback (most recent call last):
File /usr/local/bin/glance-api, line 5, in module
pkg_resources.run_script('glance==2012.1', 'glance-api')
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 467,
in run_script
self.require(requires)[0].run_script(script_name, ns)
File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 1200,
in run_script
execfile(script_filename, namespace, namespace)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/EGG-INFO/scripts/glance-api,
line 48, in module
app = config.load_paste_app(conf)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/glance/common/config.py,
line 174, in load_paste_app
app = wsgi.paste_deploy_app(conf_file, app_name, conf)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/glance/common/wsgi.py,
line 647, in paste_deploy_app
return deploy.loadapp(config:%s % paste_config_file, name=app_name)
File /usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py,
line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File /usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py,
line 272, in loadobj
return context.create()
File /usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py,
line 710, in create
return self.object_type.invoke(self)
File /usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py,
line 203, in invoke
app = context.app_context.create()
File /usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py,
line 710, in create
return self.object_type.invoke(self)
File /usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py,
line 146, in invoke
return fix_call(context.object, context.global_conf,
**context.local_conf)
File /usr/lib/python2.7/dist-packages/paste/deploy/util.py, line
56, in fix_call
val = callable(*args, **kw)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/glance/common/wsgi.py,
line 573, in __call__
return factory(self.conf, **local_conf)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/glance/api/v1/router.py,
line 37, in __init__
images_resource = images.create_resource(conf)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/glance/api/v1/images.py,
line 965, in create_resource
return wsgi.Resource(Controller(conf), deserializer, serializer)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/glance/api/v1/images.py,
line 100, in __init__
self.policy = policy.Enforcer(conf)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/glance/api/policy.py,
line 41, in __init__
self.policy_path = self._find_policy_file(conf)
File
/usr/local/lib/python2.7/dist-packages/glance-2012.1-py2.7.egg/glance/api/policy.py,
line 66, in _find_policy_file
raise cfg.ConfigFilesNotFoundError(('policy.json',))
glance.common.cfg.ConfigFilesNotFoundError: Failed to read some
config files: policy.json
because I don't want to use swift and s3,I add # in the head of
every line about swift and s3..and configure
filesystem_store_datadir = /var/lib/glance/images/..
but It still the same return when I enter the cmd again..
and now ,I have no idea of how to get it start ..
Looking for anyone's help :)
thanks a lot..
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] running Tempest continuously in the openstack project?
1. Add catalog_name=compute to tempest.conf 2. Change name to type in rest_client.py Yep, easiest to just apply this patch: git fetch https://review.openstack.org/p/openstack/tempest refs/changes/59/4259/1 git format-patch -1 --stdout FETCH_HEAD Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] DevStack and Fedora
This is great news Dean, thank you! I'll try using your patch to get tempest running on F16, and I'll get back to you with any issues I encounter. Cheers, Eoghan - Original Message - From: Dean Troyer dtro...@gmail.com To: openstack@lists.launchpad.net Sent: Wednesday, 22 February, 2012 5:04:03 PM Subject: [Openstack] DevStack and Fedora I have proposed a DevStack branch that supports Fedora 16 at https://review.openstack.org/4364. Not everything is working yet, as outlined below. I am proposing now anyway to get feedback on the direction and some of the decisions I made along the way. Even though this release is targeted specifically to Fedora 16, the direction needs to include consideration for Debian and RHEL/CentOS/etc. Status = Ubuntu: runs the same as before; i.e. anything broken without this patch is still broken. Fedora: stack.sh runs to completion WITHOUT n-vol, quantum, melange enabled. Swift installs but doesn't respond. Open Questions Internally I am using the following convention to refer to the distros: Ubuntu uses the codename adjective as expected; other Debian-based distros use the codename directly (lenny, squeeze). In the Red Hat/Fedora world the codenames are not as commonly used, rather vendor/release names such as fedora16 (or f16), rhel5.5 or centos60. The default for anything not otherwise recognized will be vendor-release.update. Many of the differences are due to the packaging (deb vs rpm) and that is what should be tested rather than release names. At this point my plan is to only maintain package lists for Ubuntu and Fedora. Contributions are always welcome. Remaining FIXMEs == lxc and cgroups: cgroup filesystems are mounted at /sys/fs/cgroup: lxc is untested tgtd is misbehaving on fedora16 for me: volumes don't work packaging for OpenVSwitch isn't in fedora's default repos (yet), need to be located: quantum and melange are untested dt -- Dean Troyer dtro...@gmail.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] DeltaCloud
Deltacloud already has support for OpenStack: http://deltacloud.apache.org/drivers.html Yep, though the existing support is a thin extension over the original deltacloud Rackspace driver, so is limited to the 1.0 version of the openstack compute API. However work is under way on a new deltacloud driver for the v1.1/2.0 compute API, see: https://issues.apache.org/jira/browse/DTACLOUD-130 Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Glance]: which part of the source codes handle the receiving of the upload image data
I'm not good in WSGI. I have a foolish question to ask. Which part of the source codes handle the receiving of the uploading data. As far as I know, the uploading data is in body_file from webob. I traced the webob code but it made my head blowed. --- send chunked data - | (webob) this mechanism is unclear to me| --- body_file Would somebody kindly give a guide on this issue ? Hi Reynolds, Are you asking for a full description of the mechanics of dispatching the incoming HTTP request entity-body to the webob.Request.body_file? Or are you just interested in the interface between WSGI and webob, i.e. the mapping between environ['wsgi.input'] and webob.Request.body_file? Generally I've found that the WSGI/webob innards only become relevant in Glance when chasing an apparent bug in the dispatch path (e.g. the recent issue with premature disconnection under webob 1.1.1-1 on ubuntu precice). But even if it's not usually crucial to understanding Glance, it would still be good to add to the tribal knowledge on the subject. It might make sense to direct your detailed questions at the webob community[1] and report back here. Cheers, Eoghan [1] http://www.webob.org ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] FFE request - glance image upload copied from external store
Folks, I like to request an Essex feature freeze exception for this blueprint: https://blueprints.launchpad.net/glance/+spec/retrieve-image-from as implemented by the following patch: https://review.openstack.org/#change,4096 The blueprint was raised in response to a late-breaking feature request from the Horizon team, as discussed on the mailing list: https://lists.launchpad.net/openstack/msg07417.html Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Adding a Glance image via API without POSTing the image data?
Yep, that's pretty much exactly the implementation we were hoping might exist. If it can be built that would be phenomenal. Any thoughts on whether that might be possible before E4 closes, or will it have to wait until Folsom? I'll propose a blueprint and see if I can get it approved for E4. Hi Gabriel, FYI that blueprint is: https://blueprints.launchpad.net/glance/+spec/retrieve-image-from implemented by this patch under review: https://review.openstack.org/#change,4096 Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Heads-up: new glance dependency on pysendfile
Folks, Just a quick heads-up that this review[1] if accepted will result in glance taking a soft dependency on pysendfile. The import is conditional, so where pysendfile is unavailable on a particular distro, the 'glance add' command will simply fallback to the pre-existing chunk-at-a-time logic. Russell Bryant is working on packaging pysendfile for Fedora[2], so that base will be covered. I also spoke to the maintainer of pysendfile and there are ogoing efforts in his community to update the python-sendfile packages for Debian/Ubuntu - the existing packages[3] were based on the now-obsolete 1.4.x version of pysendfile. Cheers, Eoghan [1] https://review.openstack.org/#change,3863 [2] http://www.spinics.net/linux/fedora/fedora-cloud/msg01224.html [3] http://packages.ubuntu.com/search?keywords=python-sendfile ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Heads-up: new glance dependency on pysendfile
BTW, does anybody knows who is taking care of it for Debian? Apparently Janoš Guljaš ja...@resenje.org was looking at packaging it for Debian. But apparently the original mainntainer of the python-sendfile package is uncontactable so a team upload (Debian Python Modules Team) would be needed Cheers, Eoghan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Adding a Glance image via API without POSTing the image data?
Yep, that's pretty much exactly the implementation we were hoping might exist. If it can be built that would be phenomenal. Any thoughts on whether that might be possible before E4 closes, or will it have to wait until Folsom? I'll propose a blueprint and see if I can get it approved for E4. My feeling is that it should be do-able in that timeframe. Cheers, Eoghan From: Eoghan Glynn [mailto:egl...@redhat.com] A-ha, I see what you mean. AFAIK that mode of upload separate to the image POST is not currently supported, but it would be quite straightforward to add say support for a header that identifies an image location for glance to retrieve from (as opposed to just recording the external location). Something like: X-Image-Meta-Retrieve-From: URI where URI is a HTTP, S3, or Swift location that's accessible to the glance API service. Would that address your use-case, Gabriel? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Easy blueprint for a new contributor to pick up
Hey Jay, I'll take this one (assuming no-one else was thinking of grabbing it?). Cheers, Eoghan - Original Message - From: Jay Pipes jaypi...@gmail.com To: openstack@lists.launchpad.net Sent: Tuesday, 7 February, 2012 2:37:17 AM Subject: [Openstack] [GLANCE] Easy blueprint for a new contributor to pick up Hey Stackers, I don't have the bandwidth to get this blueprint done, but if someone is interested in completing it, I would be inclined to support a freeze exception to get it into Essex since it would not be changing any interfaces or implementation -- just adding new event notifications to the outbound event pipeline... https://blueprints.launchpad.net/glance/+spec/glance-usage-notifications Assign yourself if you're interested! Cheers, -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] glance performance gains via sendfile()
Hi Reynolds, I've been looking into your interesting idea around sendfile()[1] usage, here are a few initial thoughts: - There's potentially even more speed-up to be harnessed in serving out images from the filesystem store via sendfile(), than from using it client-side on the initial upload (based on the assumption that images would typically uploaded once but downloaded many times, also that the download time is more crucial for perceived responsiveness as an instance being spun up by nova may be held up until the image is retreived from glance, unless already cached). - I'd suspect that some of the potential gain on the client side is currently thrown away by the syntax of the glance add CLI, specifically the use of shell redirection to pass the image content: glance add name=MyImage /path/to/image/file I'm open to correction, but this seems to needlessly copy the image content via user-space, even if the glance client avoids a second copy internally via the sendfile() usage. So I'd propose to also add a new cmd line arg to allow the file be directly specified, e.g. glance add name=MyImage path=/path/to/image/file This would have different semantics to the location field, e.g location=file:///path/to/image/file (which would imply that the content is not uploaded to the remote store). - The structure of typical pysendfile usage gets in the way of glance's image iterator pattern. On the client side this is more an an incovenience, requiring some restructuring of the code. However on the service-side, it seems we're a bit a hamstrung by the WSGI/webob APIs. For example the webob.response.body_file is filelike but doesn't expose a fileno attribute as there's no real underlying FD available intially, so the following kind of approach isn't workable: sendfile(response.body_file.fileno(), filestore_path.fileno(), ...) Seems a better approach would be to allow glance to be optionally deployed on a WSGI container that directly supports the [2] wsgi.file_wrapper extension (e.g. mod_wsgi on Apache, or uWSGI) or even allow one of the non-standard headers like X-Sendfile or X-Accel-Redirect to be set where supported. In case, I'll crack on with the basic client-side usage to begin with, so that we can quantify the performance gain. Cheers, Eoghan [1] http://code.google.com/p/pysendfile/ [2] http://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] WADL for compute API v1.1
Hi Folks, The describedby links in nova/api/openstack/compute/versions.py contain broken hrefs to a v1.1 WADL document[1] and PDF[1]. Looks like a copy'n'paste from the corresponding 1.0 versions of the WADL[3] and PDF[4], both of which are present and correct. So I was wondering whether there was an intention to publish a v1.1 (AKA v2.0) WADL or whether these links are purely a throw-back to 1.0 and should be removed? Cheers, Eoghan [1] http://docs.rackspacecloud.com/servers/api/v1.1/application.wadl [2] http://docs.rackspacecloud.com/servers/api/v1.1/cs-devguide-20110125.pdf [3] http://docs.rackspacecloud.com/servers/api/v1.0/application.wadl [4] http://docs.rackspacecloud.com/servers/api/v1.0/cs-devguide-20110415.pdf ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] WADL for compute API v1.1
So I was wondering whether there was an intention to publish a v1.1 WADL ... Follow up question: would it be nasty to serve out that WADL directly from github? e.g https://github.com/openstack/compute-api/blob/essex-final-tag/openstack-compute-api-1.1/src/os-compute-1.1.wadl ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] WADL for compute API v1.1
Thanks Anne! The use-case is simply to ensure that we serve out accurate links in the version details reponse from the compute API. (I just happened to notice the broken links when poking around the v1.0-v1.1 API changes) BTW I've proposed a change with the WADL being served out of github: https://review.openstack.org/3421 but if you publish the WADL at a well-known path under docs.openstack.org, that would be much better. Cheers, Eoghan - Original Message - From: Anne Gentle a...@openstack.org To: Eoghan Glynn egl...@redhat.com Cc: openstack@lists.launchpad.net Sent: Wednesday, 25 January, 2012 5:33:02 PM Subject: Re: [Openstack] WADL for compute API v1.1 Hi Eoghan - We will be using that WADL for the API reference site so we've got a vested interest in its accuracy. Not sure if Github has any policies about serving files directly. We could copy it to docs.openstack.org with a Jenkins job if that's helpful. We publish schemas at http://docs.openstack.org/api/openstack-compute/1.1/xsd/ already. Tell us more about your use case and let's see what we can do. Thanks, Anne On Wed, Jan 25, 2012 at 10:30 AM, Eoghan Glynn egl...@redhat.com wrote: So I was wondering whether there was an intention to publish a v1.1 WADL ... Follow up question: would it be nasty to serve out that WADL directly from github? e.g https://github.com/openstack/compute-api/blob/essex-final-tag/openstack-compute-api-1.1/src/os-compute-1.1.wadl ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
