Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Dear Kevin, thank you very much! For the moment, I did stabilize my systems with these commands. For a next step, will explore three further options, which should all solve the issue better: - move to ovirt altogether - probably as foolproof as needed at my skill level - install ovirt's qemu-kvm-rhev to use the more extensive options - move to QMP Thanks again also to Paolo and Eric for providing substantial insights! Regards, Michael -Ursprüngliche Nachricht- Von: Kevin Wolf [mailto:kw...@redhat.com] Gesendet: Dienstag, 25. November 2014 12:34 An: Prof. Dr. Michael Schefczyk Cc: Eric Blake; Paolo Bonzini; qemu-devel Betreff: Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request) Am 22.11.2014 um 18:02 hat Prof. Dr. Michael Schefczyk geschrieben: Dear All, after some trying, my impression is that the following steps do work with plain Centos 7: virsh snapshot-create-as VM backsnap qemu-img convert -f qcow2 -s backsnap -O qcow2 VM.img backup.img virsh snapshot-delete VM backsnap Am I on the right track with these commands? I won't tell you that you're bound to break your image with this sequence, because chances are that you won't break it. (In practice, this happens to work in most cases, because the essential snapshot metadata generally isn't written to without explicit user actions on that snapshot.) But you're violating the rule of one writer or many readers, so as far as qemu is concerned, we won't be careful to avoid breaking this setup. Don't blame us if it fails one day. With the QMP solutions (either drive-backup or snapshot/commit) you're using official APIs, so I'd encourage you to use these. Kevin smime.p7s Description: S/MIME Cryptographic Signature
Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Dear All, after some trying, my impression is that the following steps do work with plain Centos 7: virsh snapshot-create-as VM backsnap qemu-img convert -f qcow2 -s backsnap -O qcow2 VM.img backup.img virsh snapshot-delete VM backsnap Am I on the right track with these commands? The further features seem to be reserved to RHEL (and potentially other distributions) but not included in Centos: virsh snapshot-create-as issues Live Disk Snapshot not supported in this version of QEMU (retranslated from German). virsh blockcommit issues Online Transfer not supported ... Thus, with Centos 7 it should be possible to back up VMs without shutting them down. They are being paused, however, as long as the snapshot is created. The qemu-guest-agent does not help in this context, as the features which depend on it are not available in Centos. Regards, Michael -Ursprüngliche Nachricht- Von: Eric Blake [mailto:ebl...@redhat.com] Gesendet: Mittwoch, 19. November 2014 19:13 An: Prof. Dr. Michael Schefczyk; Paolo Bonzini; qemu-devel Betreff: Re: AW: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request) On 11/19/2014 10:32 AM, Prof. Dr. Michael Schefczyk wrote: Dear Eric, dear all, Again, thank you very much. I now gather that I took the wrong path towards nightly backups of running VM. I remain surprised that I did work for a relatively long time. [can you convince your mailer to wrap long lines? also, we tend to frown on top-posting on technical lists] Yeah, you were just getting lucky that two different processes weren't both trying to allocate a cluster for different purposes at the same time. When the collision finally did happen, it had catastrophic results on your image. A major book on KVM in German language by Kofler Spenneberg recommends the following approach for online backups (with and without --disk-only): virsh snapshot-create-as vm XXX --disk-only qemu-img convert -f qcow2 -s XXX -O qcow2 XXX.img /backup/YYY.img virsh snapshot-delete vm XXX Yes, virsh is using QMP commands under the hood, so this method is safe. One slight issue is that this sequence is incomplete (it does not shrink the backing file chain after the copy is complete), so if you keep repeating it, you will eventually cause reduced performance when you have a really long chain of multiple qcow2 overlays, or even cause qemu to run into fd exhaustion. Also, this command does not show that unless you clean things up, then the second time you run this you do not want to copy XXX.img into backup, but instead the qcow2 wrapper file that was created by the first snapshot (and which itself wrapped by the second snapshot). With new enough libvirt and qemu, you can shrink the chain back down with an active commit, as in: virsh blockcommit vm XXX --verbose --active --pivot Also, the use of --disk-only means that your disks have a snapshot taken as if at a point in time when you yank the power cord; reverting to such a backup may require fsck, and may suffer from other problems from anything that was depending on I/O that had not yet been flushed to disk. If you add the --quiesce option (which implies that you set up your guest to use qemu-guest-agent, and told libvirt to manage the agent), then you can guarantee that your guest has flushed and frozen all filesystems prior to the point in time where the snapshot is created; and you can even install hooks in the guest to extend that stability to things like databases. Then your backups are much easier to use. If you omit --disk-only, and take a full snapshot, then you have the guest memory state that is necessary to restore all pending I/O, and don't need to worry about freezing the guest file systems; but then you have to remember to back up the memory state in addition to your disk state. Would this be any better than my script, because it uses virsh snapshot-create-as instead of qemu-img snapshot? The second command is still qemu-img convert which may be problematical. No, remember what I said. qemu-img may not be used on any image that is opened read-write by qemu, but is perfectly safe to do read-only operations on any image that is opened read-only by qemu. That sequence of commands goes from the initial: disk.img [read-write] then the snapshot-create command causes libvirt to issue a QMP command to switch qemu to: disk.img [read-only] - overlay.qcow2 [read-write] at which point you can do anything read-only to disk.img (whether 'qemu-img convert' or 'cp' or any other command that doesn't alter the contents of the file) finally, the 'virsh blockcommit' command would take you back to: disk.img [read-write] The problem I am facing is that the documentation is not easy to understand for the average user/administrator who is not among the kvm developers and experts. I have of course tried to read section 14.2.3 of RHEL 7 Virtualization
Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Yes! My level of knowledge is that one uses the qcow2 format in order to be able to create live snapshots/backups. Otherwise one would tend to use the more efficient raw format. Is this not correct and did I apply the backup mechanism in the wrong way? -Ursprüngliche Nachricht- Von: Paolo Bonzini [mailto:pbonz...@redhat.com] Gesendet: Mittwoch, 19. November 2014 12:54 An: Prof. Dr. Michael Schefczyk; qemu-devel Betreff: Re: AW: File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request) On 19/11/2014 12:48, Prof. Dr. Michael Schefczyk wrote: Thank you very much. To execute the backup, I run a script. For the machine in question, it looks as follows: #!/bin/bash dt=`date +%y%m%d` qemu-img snapshot -c gatewayb72 /kvm02/gatewayb72.img qemu-img convert -f qcow2 -O qcow2 /kvm02/gatewayb72.img /backup/gatewayb72$dt.img qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img /bin/find /backup/* -mtime +100 -exec rm {} \; Is it done while the VM is running? Paolo smime.p7s Description: S/MIME Cryptographic Signature
Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Dear Paolo, Thank you very much. To execute the backup, I run a script. For the machine in question, it looks as follows: #!/bin/bash dt=`date +%y%m%d` qemu-img snapshot -c gatewayb72 /kvm02/gatewayb72.img qemu-img convert -f qcow2 -O qcow2 /kvm02/gatewayb72.img /backup/gatewayb72$dt.img qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img /bin/find /backup/* -mtime +100 -exec rm {} \; Regards, Michael -Ursprüngliche Nachricht- Von: Paolo Bonzini [mailto:pbonz...@redhat.com] Gesendet: Mittwoch, 19. November 2014 12:42 An: Prof. Dr. Michael Schefczyk; qemu-devel Betreff: File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request) On 19/11/2014 12:39, Prof. Dr. Michael Schefczyk wrote: Dear Paolo, thank you very much for your prompt reply. Hi, please keep the replies on the mailing list. I've added the qemu-devel mailing list, where more people should be able to see the message and hopefully reply with something helpful. Also, how do you do the backups? Paolo For example, I have a guest named gatewayb72.img where the backup failed. If I thereafter try to create or delete a snapshot, the following reply occurs on the command line: [root@linuxhost2 kvm02]# qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img qemu-img: Could not open '/kvm02/gatewayb72.img': Could not read snapshots: File too large If I want to reboot that machine, I get the following error: Fehler beim Starten der Domain: Interner Fehler: Prozess wurde während der Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2, cache=none: could not open disk image /kvm02/gatewayb72.img: Could not read snapshots: File too large Traceback (most recent call last): File /usr/share/virt-manager/virtManager/asyncjob.py, line 100, in cb_wrapper callback(asyncjob, *args, **kwargs) File /usr/share/virt-manager/virtManager/asyncjob.py, line 122, in tmpcb callback(*args, **kwargs) File /usr/share/virt-manager/virtManager/domain.py, line 1220, in startup self._backend.create() File /usr/lib64/python2.7/site-packages/libvirt.py, line 698, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: Interner Fehler: Prozess wurde während der Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2, cache=none: could not open disk image /kvm02/gatewayb72.img: Could not read snapshots: File too large Based on the facts I can see, the file is not too large. When reading the first error, the file size was 13.8 GB, while the limit is 14.5 GB. The same does also happen with files which are only, for example 6 GB big, while their limit is also 14.5 GB. Therefore, I think that the file too large really stands for something else. Can I provide any additional information? Regards, Michael ___ Technische Universität Dresden Fakultät Wirtschaftswissenschaften Lehrstuhl für Entrepreneurship und Innovation Prof. Dr. Michael Schefczyk D-01062 Dresden Fon: +49-3 51-4 63-3 68 81 Fax: +49-3 51-4 63-3 68 83 www.gruenderlehrstuhl.de Since some months ago, I am facing the problem, that backups fail unpredictably. A failed backup does not generate a backup file and thereafter, snapshots can no longer be created or deleted and the guest cannot be started anymore. The resulting error message is File too large. Who is reporting the File too large error? Can you please include the error in full detail? Paolo smime.p7s Description: S/MIME Cryptographic Signature
Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Dear Eric, dear all, Again, thank you very much. I now gather that I took the wrong path towards nightly backups of running VM. I remain surprised that I did work for a relatively long time. A major book on KVM in German language by Kofler Spenneberg recommends the following approach for online backups (with and without --disk-only): virsh snapshot-create-as vm XXX --disk-only qemu-img convert -f qcow2 -s XXX -O qcow2 XXX.img /backup/YYY.img virsh snapshot-delete vm XXX Would this be any better than my script, because it uses virsh snapshot-create-as instead of qemu-img snapshot? The second command is still qemu-img convert which may be problematical. The problem I am facing is that the documentation is not easy to understand for the average user/administrator who is not among the kvm developers and experts. I have of course tried to read section 14.2.3 of RHEL 7 Virtualization Deployment and Administration Guide on backups, but I found that too cryptic for someone like myself to draw practical consequences from it. Regards, Michael ___ Technische Universität Dresden Fakultät Wirtschaftswissenschaften Lehrstuhl für Entrepreneurship und Innovation Prof. Dr. Michael Schefczyk D-01062 Dresden Fon: +49-3 51-4 63-3 68 81 Fax: +49-3 51-4 63-3 68 83 www.gruenderlehrstuhl.de -Ursprüngliche Nachricht- Von: Eric Blake [mailto:ebl...@redhat.com] Gesendet: Mittwoch, 19. November 2014 16:44 An: Paolo Bonzini; Prof. Dr. Michael Schefczyk; qemu-devel Betreff: Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request) On 11/19/2014 07:54 AM, Paolo Bonzini wrote: On 19/11/2014 13:07, Prof. Dr. Michael Schefczyk wrote: Yes! My level of knowledge is that one uses the qcow2 format in order to be able to create live snapshots/backups. Otherwise one would tend to use the more efficient raw format. Is this not correct and did I apply the backup mechanism in the wrong way? That's correct, but you still have to create live snapshots from within QEMU. You absolutely CANNOT modify a qcow2 file via qemu-img or any other external means while qemu is using the file in read-write mode, or you risk file corruption (which appears to be what happened to you). Monitor commands to the qemu process are the only supported means for taking a live snapshot/backup of an in-use disk. This is done with a QMP (QEMU Management Protocol) command like { execute: blockdev-snapshot-internal-sync, arguments: { device: ide-hd0, name: snapshot0 } } QMP is accessed through normal sockets, or via libvirt. However, I'm not sure if running qemu-img convert on the resulting snapshot is possible though, and there is no equivalent of qemu-img snapshot -d. qemu-img snapshot -d can be achieved via the QMP monitor command 'human-monitor-command' forwarding to the HMP 'delvm' command. But you are correct that I don't know of any way to reproduce qemu-img convert on a read-write image. You can instead use QEMU's support for backup, which will do what you wanted directly while the VM is running. For example: { execute: drive-backup, arguments: { device: ide-hd0, sync: full, format: qcow2, target: backup.img } } This does not even require qcow2 for the image. The downside is that you must not turn off the VM until the job has completed. Another option is to create a live snapshot (so that your file under interest is now read-only with a temporary qcow2 overlay as the read-write file), then you can do whatever you want with the file (such as qemu-img convert), and finally do an active commit to get rid of the temporary qcow2 overlay and back to just the main live image. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org smime.p7s Description: S/MIME Cryptographic Signature