Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image
Actually, this is unlikely because there was locking for image fetching already. This patch only adds it for case when several compute processes share same $instances_dir. - disk full really might be the case. Simple check is to extend available space and try to reproduce it again. 2012/9/25 Wangpan hzwang...@corp.netease.com ** Thanks for your clue. yes, the occured condition of my issue is the same as BUG *1014227*, but the difference is that my instance directory is NOT shared between compute nodes, the instances' images are saved in local disk on compute nodes. I believe the bug Boris found may occur in NOT shared storage circumstance as mine, and my issue may be resloved by this patch which adds a lock to cache image method. Thanks again. 2012-09-25 -- Wangpan -- *发件人：*Boris Filippov *发送时间：*2012-09-24 19:59 *主题：*Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image *收件人：*Razique Mahrouarazique.mahr...@gmail.com *抄送：*Wangpanhzwang...@corp.netease.com,openstack openstack@lists.launchpad.net Your instance directory is shared between compute nodes? Similar issue was fixed for folsom-rc1 https://review.openstack.org/#/c/12024/ https://bugs.launchpad.net/nova/+bug/1014227 2012/9/24 Razique Mahroua razique.mahr...@gmail.com Here is what I found http://comments.gmane.org/gmane.comp.emulators.qemu/153254 http://osdir.com/ml/qemu-devel/2012-05/msg04234.html What do you think ? *Nuage Co - Razique Mahroua** * razique.mahr...@gmail.com Le 24 sept. 2012 à 10:43, Wangpan hzwang...@corp.netease.com a écrit : qemu-img check -f qcow2 test.qcow2 No errors were found on the image. 2012-09-24 -- Wangpan -- *发件人：*Razique Mahroua *发送时间：*2012-09-24 16:30 *主题：*Re: Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image *收件人：*Wangpanhzwang...@corp.netease.com *抄送：*openstackopenstack@lists.launchpad.net hi, what about a qemu-img check -f qcow2 test.qcow2 *Nuage Co - Razique Mahroua** * razique.mahr...@gmail.com NUAGECO-LOGO-Fblan_petit(1).jpg Le 24 sept. 2012 à 10:19, Wangpanhzwang...@corp.netease.com a écrit : Thanks for your attention! - disk full # I believe there is enough space, because I have tried to boot serveral instances after this error with the same image, they are all booted successfully. - permission issues# Not this reason, too. explain as above. I still believe this is a race problem, if we convert to image from qcow2 to raw at a short interval, one of the convertion may fail, I have verified this by running two same commands continuously such as: qemu-img convert -f qcow2 -O raw test.qcow2 test.part and one of them give me the same error: qemu-img: error while writing sector 1052672: Input/output error ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image
Your instance directory is shared between compute nodes? Similar issue was fixed for folsom-rc1 https://review.openstack.org/#/c/12024/ https://bugs.launchpad.net/nova/+bug/1014227 2012/9/24 Razique Mahroua razique.mahr...@gmail.com Here is what I found http://comments.gmane.org/gmane.comp.emulators.qemu/153254 http://osdir.com/ml/qemu-devel/2012-05/msg04234.html What do you think ? *Nuage Co - Razique Mahroua** * razique.mahr...@gmail.com Le 24 sept. 2012 à 10:43, Wangpan hzwang...@corp.netease.com a écrit : qemu-img check -f qcow2 test.qcow2 No errors were found on the image. 2012-09-24 -- Wangpan -- *发件人：*Razique Mahroua *发送时间：*2012-09-24 16:30 *主题：*Re: Re: [Openstack] [Nova] Got 'qemu-img: error while writing sector 634880: Input/output error' in _create_image *收件人：*Wangpanhzwang...@corp.netease.com *抄送：*openstackopenstack@lists.launchpad.net hi, what about a qemu-img check -f qcow2 test.qcow2 *Nuage Co - Razique Mahroua** * razique.mahr...@gmail.com NUAGECO-LOGO-Fblan_petit(1).jpg Le 24 sept. 2012 à 10:19, Wangpanhzwang...@corp.netease.com a écrit : Thanks for your attention! - disk full # I believe there is enough space, because I have tried to boot serveral instances after this error with the same image, they are all booted successfully. - permission issues# Not this reason, too. explain as above. I still believe this is a race problem, if we convert to image from qcow2 to raw at a short interval, one of the convertion may fail, I have verified this by running two same commands continuously such as: qemu-img convert -f qcow2 -O raw test.qcow2 test.part and one of them give me the same error: qemu-img: error while writing sector 1052672: Input/output error ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] snapshot-temp-directory
This actually was fixed in upstream ( https://github.com/openstack/nova/commit/70129ed19db187cc90f74abd5c93c86098d29c27 ). Which OpenStack version you use? Maybe this patch can be backported there. There is workaround for this problem, but it's (not a bit) ugly: http://www.mail-archive.com/openstack@lists.launchpad.net/msg13730.html 2012/8/13 Wolfgang Hennerbichler wolfgang.hennerbich...@risc-software.at: hello fellow openstack users, I just tried to create a snapshot of a virtual machine, which was larger than average. Bad news: The snapshot couldn't be saved, but what I see here is some kind of design problem: 2012-08-13 13:44:41 DEBUG nova.utils [req-a678e738-9c18-4ee6-8e2e-f1f176a4674b 9b4ad74ecb69492db1bd3818efe6a47f 7edc3284f53f4e02b92d498db41b842d] Running cmd (subprocess): qemu-img convert -f qcow2 -O raw -s 34dd12038e7b400b9f1d772de4e01 70a /var/lib/nova/instances/instance-005c/disk /tmp/tmp1gRMY9/34dd12038e7b400b9f1d772de4e0170a from (pid=25365) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219 that's bad, I don't have so much disk space on /, but I have plenty in /var/lib/nova/instances. df -h | grep nova /dev/dm-8 1.4T 150G 1.3T 11% /var/lib/nova/instances is there a generic way to change this path? Wolfgang -- DI (FH) Wolfgang Hennerbichler Software Development Unit Advanced Computing Technologies RISC Software GmbH A company of the Johannes Kepler University Linz IT-Center Softwarepark 35 4232 Hagenberg Austria Phone: +43 7236 3343 245 Fax: +43 7236 3343 250 wolfgang.hennerbich...@risc-software.at http://www.risc-software.at ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Support for snapshot of LVM backend image
Current snapshot implementation for libvirt driver doesn't support snapshotting for everything except qcow2 (as i remember, because virDomainManagedSave will fail on everything except qcow2). So, maybe we can discuss it here. This for sure can be implemented for lvm using lvm snapshots but it will be unsafe doing snapshot on alive instance. LVM is much like raw right now and raw doesn't support snapshots with current logic too. 2012/7/16 Li Wang fox...@gmail.com hi, all As LVM backend image support was introduced in the trunk version of nova, is there any plan to support snapshot of the LVM backend image? I am using libvirt and Xen, with lvm as the instance image backend. There are two issues when creating snapshot. 1. when calling managedSave, libvirt complained there's no supported function by the driver. libvirtError: this function is not supported by the connection driver: virDomainManagedSave 2. when actually creating snapshot, libvirt calls qemu-img, which does not support lvm. As a result, snapshot status is always 'queue', and actually it's failed. For issue 1, it seems that libvirt does not support save or similar functions for xen, some modifications to the libvirt Xen driver maybe help? For issue 2, I think the snapshot action should call lvm related command but not qemu-img when using lvm. For example:lvcreate --snapshot .. Further discussions and replies are welcome. Thanks all Regards Wang Li ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Support for snapshot of LVM backend image
Well, according to http://libvirt.org/html/libvirt-libvirt.html#virDomainManagedSave domain will be suspended. After snapshotting there is new _create_domain call: if state == power_state.RUNNING: virt_dom.managedSave(0) ... snapshotting... finally: libvirt_utils.delete_snapshot(disk_path, snapshot_name) if state == power_state.RUNNING: self._create_domain(domain=virt_dom) So, Is this actually live snapshot right now? But qemu can also write the vm state outside of the backend image, which should be usable with all image backends. This can be exploited for saving VM states. There is blueprint for that: https://blueprints.launchpad.net/nova/+spec/live-snapshots ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Support for snapshot of LVM backend image
But qemu can also write the vm state outside of the backend image, which should be usable with all image backends. Use that instead managedSave? This should suffice. Save VM state, suspend domain - do snapshot - restore VM with previous state? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] How do I stop image-create from using /tmp?
2012/7/1 Lars Kellogg-Stedman l...@seas.harvard.edu: So, maybe setting any of this environment variables for nova-compute to desired value sholuld help. Yeah, I was expecting that. Given that this could easily take out a compute host I'd like to see it get an explicit configuration value (or default to instance_dir, I guess). If I can sort out the corporate contributor agreement stuff I may try to submit a patch... -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | https://review.openstack.org/#/c/9239/ Wrote patch for it. We had same issue with our OpenStack installation before and this behavior is quite surprise for people who believe that nobody will store Big Files in /tmp. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] How do I stop image-create from using /tmp?
2012/7/2 Daniel P. Berrange berra...@redhat.com: On Mon, Jul 02, 2012 at 10:24:02AM -0700, Matt Joyce wrote: I like the idea of making this a flagfile option. In the particular case of the qemu-img command described in earlier in this thread, I'm not convinced we need a new option. Instead of using /tmp when extracting a snapshot from an existing disk image, it could just use the path where the source image already resides. ie the existing FLAGS.instances_path directory, which can be assumed to be a suitably large persistent data store. Other uses of temporary files should be analysed ona case by case basis to figure out a suitable storage location. This might perhaps identify a need for a generic temp file location for nova, such as /var/run/nova/ or /var/cache/nova or both (depending on use case). Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp Sure, we can place snapshots inside instances_path. At least, just separate them from instances and place them in %instances_path%/snapshots. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] How do I stop image-create from using /tmp?
As I remember, destination directory for snapshot is created by mkdtemp. What module documantation says about default temp directory: tempfile.tempdir When set to a value other than None, this variable defines the default value for the dir argument to all the functions defined in this module. If tempdir is unset or None at any call to any of the above functions, Python searches a standard list of directories and sets tempdir to the first one which the calling user can create files in. The list is: The directory named by the TMPDIR environment variable. The directory named by the TEMP environment variable. The directory named by the TMP environment variable. A platform-specific location:On RiscOS, the directory named by the Wimp$ScrapDir environment variable. On Windows, the directories C:\TEMP, C:\TMP, \TEMP, and \TMP, in that order. On all other platforms, the directories /tmp, /var/tmp, and /usr/tmp, in that order. As a last resort, the current working directory. (http://docs.python.org/library/tempfile.html) So, maybe setting any of this environment variables for nova-compute to desired value sholuld help. 2012/6/30 Lars Kellogg-Stedman l...@seas.harvard.edu I'm running nova image-create for the first time, and on the compute node I see: qemu-img convert -f qcow2 -O qcow2 -s 98a8efe9ec114b489eb163c64661441a /virt/pools/openstack_2/instance-0011/disk /tmp/tmpH9KkUZ/98a8efe9ec114b489eb163c64661441a I am concerned to see this operation dropping a disk image into /tmp. What if the disk image is larger than the root filesystem? Is there a way to have OpenStack put these temporary images somewhere else? I would prefer instance_dir by default, instead of /tmp, because instance_dir has already been provisioned with enough space to meet the needs of disk images, but an explicit parameter would probably be a better option. -- Lars Kellogg-Stedman l...@seas.harvard.edu | Senior Technologist| http://ac.seas.harvard.edu/ Academic Computing | http://code.seas.harvard.edu/ Harvard School of Engineering and Applied Sciences | ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
