Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Dear Kevin, thank you very much! For the moment, I did stabilize my systems with these commands. For a next step, will explore three further options, which should all solve the issue better: - move to ovirt altogether - probably as foolproof as needed at my skill level - install ovirt's qemu-kvm-rhev to use the more extensive options - move to QMP Thanks again also to Paolo and Eric for providing substantial insights! Regards, Michael -Ursprüngliche Nachricht- Von: Kevin Wolf [mailto:kw...@redhat.com] Gesendet: Dienstag, 25. November 2014 12:34 An: Prof. Dr. Michael Schefczyk Cc: Eric Blake; Paolo Bonzini; qemu-devel Betreff: Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request) Am 22.11.2014 um 18:02 hat Prof. Dr. Michael Schefczyk geschrieben: Dear All, after some trying, my impression is that the following steps do work with plain Centos 7: virsh snapshot-create-as VM backsnap qemu-img convert -f qcow2 -s backsnap -O qcow2 VM.img backup.img virsh snapshot-delete VM backsnap Am I on the right track with these commands? I won't tell you that you're bound to break your image with this sequence, because chances are that you won't break it. (In practice, this happens to work in most cases, because the essential snapshot metadata generally isn't written to without explicit user actions on that snapshot.) But you're violating the rule of one writer or many readers, so as far as qemu is concerned, we won't be careful to avoid breaking this setup. Don't blame us if it fails one day. With the QMP solutions (either drive-backup or snapshot/commit) you're using official APIs, so I'd encourage you to use these. Kevin smime.p7s Description: S/MIME Cryptographic Signature
Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Dear All, after some trying, my impression is that the following steps do work with plain Centos 7: virsh snapshot-create-as VM backsnap qemu-img convert -f qcow2 -s backsnap -O qcow2 VM.img backup.img virsh snapshot-delete VM backsnap Am I on the right track with these commands? The further features seem to be reserved to RHEL (and potentially other distributions) but not included in Centos: virsh snapshot-create-as issues Live Disk Snapshot not supported in this version of QEMU (retranslated from German). virsh blockcommit issues Online Transfer not supported ... Thus, with Centos 7 it should be possible to back up VMs without shutting them down. They are being paused, however, as long as the snapshot is created. The qemu-guest-agent does not help in this context, as the features which depend on it are not available in Centos. Regards, Michael -Ursprüngliche Nachricht- Von: Eric Blake [mailto:ebl...@redhat.com] Gesendet: Mittwoch, 19. November 2014 19:13 An: Prof. Dr. Michael Schefczyk; Paolo Bonzini; qemu-devel Betreff: Re: AW: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request) On 11/19/2014 10:32 AM, Prof. Dr. Michael Schefczyk wrote: Dear Eric, dear all, Again, thank you very much. I now gather that I took the wrong path towards nightly backups of running VM. I remain surprised that I did work for a relatively long time. [can you convince your mailer to wrap long lines? also, we tend to frown on top-posting on technical lists] Yeah, you were just getting lucky that two different processes weren't both trying to allocate a cluster for different purposes at the same time. When the collision finally did happen, it had catastrophic results on your image. A major book on KVM in German language by Kofler Spenneberg recommends the following approach for online backups (with and without --disk-only): virsh snapshot-create-as vm XXX --disk-only qemu-img convert -f qcow2 -s XXX -O qcow2 XXX.img /backup/YYY.img virsh snapshot-delete vm XXX Yes, virsh is using QMP commands under the hood, so this method is safe. One slight issue is that this sequence is incomplete (it does not shrink the backing file chain after the copy is complete), so if you keep repeating it, you will eventually cause reduced performance when you have a really long chain of multiple qcow2 overlays, or even cause qemu to run into fd exhaustion. Also, this command does not show that unless you clean things up, then the second time you run this you do not want to copy XXX.img into backup, but instead the qcow2 wrapper file that was created by the first snapshot (and which itself wrapped by the second snapshot). With new enough libvirt and qemu, you can shrink the chain back down with an active commit, as in: virsh blockcommit vm XXX --verbose --active --pivot Also, the use of --disk-only means that your disks have a snapshot taken as if at a point in time when you yank the power cord; reverting to such a backup may require fsck, and may suffer from other problems from anything that was depending on I/O that had not yet been flushed to disk. If you add the --quiesce option (which implies that you set up your guest to use qemu-guest-agent, and told libvirt to manage the agent), then you can guarantee that your guest has flushed and frozen all filesystems prior to the point in time where the snapshot is created; and you can even install hooks in the guest to extend that stability to things like databases. Then your backups are much easier to use. If you omit --disk-only, and take a full snapshot, then you have the guest memory state that is necessary to restore all pending I/O, and don't need to worry about freezing the guest file systems; but then you have to remember to back up the memory state in addition to your disk state. Would this be any better than my script, because it uses virsh snapshot-create-as instead of qemu-img snapshot? The second command is still qemu-img convert which may be problematical. No, remember what I said. qemu-img may not be used on any image that is opened read-write by qemu, but is perfectly safe to do read-only operations on any image that is opened read-only by qemu. That sequence of commands goes from the initial: disk.img [read-write] then the snapshot-create command causes libvirt to issue a QMP command to switch qemu to: disk.img [read-only] - overlay.qcow2 [read-write] at which point you can do anything read-only to disk.img (whether 'qemu-img convert' or 'cp' or any other command that doesn't alter the contents of the file) finally, the 'virsh blockcommit' command would take you back to: disk.img [read-write] The problem I am facing is that the documentation is not easy to understand for the average user/administrator who is not among the kvm developers and experts. I have of course tried to read section 14.2.3 of RHEL 7 Virtualization
Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Yes! My level of knowledge is that one uses the qcow2 format in order to be
able to create live snapshots/backups. Otherwise one would tend to use the more
efficient raw format. Is this not correct and did I apply the backup mechanism
in the wrong way?
-Ursprüngliche Nachricht-
Von: Paolo Bonzini [mailto:pbonz...@redhat.com]
Gesendet: Mittwoch, 19. November 2014 12:54
An: Prof. Dr. Michael Schefczyk; qemu-devel
Betreff: Re: AW: File too large error from qemu-img snapshot (was Re: AW:
Bug Repoting Directions Request)
On 19/11/2014 12:48, Prof. Dr. Michael Schefczyk wrote:
Thank you very much. To execute the backup, I run a script. For the machine
in question, it looks as follows:
#!/bin/bash
dt=`date +%y%m%d`
qemu-img snapshot -c gatewayb72 /kvm02/gatewayb72.img qemu-img convert
-f qcow2 -O qcow2 /kvm02/gatewayb72.img /backup/gatewayb72$dt.img
qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img /bin/find
/backup/* -mtime +100 -exec rm {} \;
Is it done while the VM is running?
Paolo
smime.p7s
Description: S/MIME Cryptographic Signature
Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Dear Paolo,
Thank you very much. To execute the backup, I run a script. For the machine in
question, it looks as follows:
#!/bin/bash
dt=`date +%y%m%d`
qemu-img snapshot -c gatewayb72 /kvm02/gatewayb72.img
qemu-img convert -f qcow2 -O qcow2 /kvm02/gatewayb72.img
/backup/gatewayb72$dt.img
qemu-img snapshot -d gatewayb72 /kvm02/gatewayb72.img
/bin/find /backup/* -mtime +100 -exec rm {} \;
Regards,
Michael
-Ursprüngliche Nachricht-
Von: Paolo Bonzini [mailto:pbonz...@redhat.com]
Gesendet: Mittwoch, 19. November 2014 12:42
An: Prof. Dr. Michael Schefczyk; qemu-devel
Betreff: File too large error from qemu-img snapshot (was Re: AW: Bug
Repoting Directions Request)
On 19/11/2014 12:39, Prof. Dr. Michael Schefczyk wrote:
Dear Paolo,
thank you very much for your prompt reply.
Hi,
please keep the replies on the mailing list. I've added the qemu-devel mailing
list, where more people should be able to see the message and hopefully reply
with something helpful.
Also, how do you do the backups?
Paolo
For example, I have a guest named gatewayb72.img where the backup failed.
If I thereafter try to create or delete a snapshot, the following reply
occurs on the command line:
[root@linuxhost2 kvm02]# qemu-img snapshot -d gatewayb72
/kvm02/gatewayb72.img
qemu-img: Could not open '/kvm02/gatewayb72.img': Could not read
snapshots: File too large
If I want to reboot that machine, I get the following error:
Fehler beim Starten der Domain: Interner Fehler: Prozess wurde während
der Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive
file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2,
cache=none: could not open disk image /kvm02/gatewayb72.img: Could not
read snapshots: File too large
Traceback (most recent call last):
File /usr/share/virt-manager/virtManager/asyncjob.py, line 100, in
cb_wrapper
callback(asyncjob, *args, **kwargs)
File /usr/share/virt-manager/virtManager/asyncjob.py, line 122, in tmpcb
callback(*args, **kwargs)
File /usr/share/virt-manager/virtManager/domain.py, line 1220, in startup
self._backend.create()
File /usr/lib64/python2.7/site-packages/libvirt.py, line 698, in create
if ret == -1: raise libvirtError ('virDomainCreate() failed',
dom=self)
libvirtError: Interner Fehler: Prozess wurde während der
Verbindungsaufnahme zum Monitor beendet : qemu-kvm: -drive
file=/kvm02/gatewayb72.img,if=none,id=drive-virtio-disk0,format=qcow2,
cache=none: could not open disk image /kvm02/gatewayb72.img: Could not
read snapshots: File too large
Based on the facts I can see, the file is not too large. When reading the
first error, the file size was 13.8 GB, while the limit is 14.5 GB. The same
does also happen with files which are only, for example 6 GB big, while their
limit is also 14.5 GB. Therefore, I think that the file too large really
stands for something else.
Can I provide any additional information?
Regards,
Michael
___
Technische Universität Dresden
Fakultät Wirtschaftswissenschaften
Lehrstuhl für Entrepreneurship und Innovation Prof. Dr. Michael
Schefczyk
D-01062 Dresden
Fon: +49-3 51-4 63-3 68 81
Fax: +49-3 51-4 63-3 68 83
www.gruenderlehrstuhl.de
Since some months ago, I am facing the problem, that backups fail
unpredictably. A failed backup does not generate a backup file and
thereafter, snapshots can no longer be created or deleted and the
guest cannot be started anymore. The resulting error message is File
too large.
Who is reporting the File too large error? Can you please include the
error in full detail?
Paolo
smime.p7s
Description: S/MIME Cryptographic Signature
Re: [Qemu-devel] File too large error from qemu-img snapshot (was Re: AW: Bug Repoting Directions Request)
Dear Eric, dear all,
Again, thank you very much. I now gather that I took the wrong path towards
nightly backups of running VM. I remain surprised that I did work for a
relatively long time.
A major book on KVM in German language by Kofler Spenneberg recommends the
following approach for online backups (with and without --disk-only):
virsh snapshot-create-as vm XXX --disk-only
qemu-img convert -f qcow2 -s XXX -O qcow2 XXX.img /backup/YYY.img
virsh snapshot-delete vm XXX
Would this be any better than my script, because it uses virsh
snapshot-create-as instead of qemu-img snapshot? The second command is still
qemu-img convert which may be problematical.
The problem I am facing is that the documentation is not easy to understand for
the average user/administrator who is not among the kvm developers and experts.
I have of course tried to read section 14.2.3 of RHEL 7 Virtualization
Deployment and Administration Guide on backups, but I found that too cryptic
for someone like myself to draw practical consequences from it.
Regards,
Michael
___
Technische Universität Dresden
Fakultät Wirtschaftswissenschaften
Lehrstuhl für Entrepreneurship und Innovation
Prof. Dr. Michael Schefczyk
D-01062 Dresden
Fon: +49-3 51-4 63-3 68 81
Fax: +49-3 51-4 63-3 68 83
www.gruenderlehrstuhl.de
-Ursprüngliche Nachricht-
Von: Eric Blake [mailto:ebl...@redhat.com]
Gesendet: Mittwoch, 19. November 2014 16:44
An: Paolo Bonzini; Prof. Dr. Michael Schefczyk; qemu-devel
Betreff: Re: [Qemu-devel] File too large error from qemu-img snapshot (was
Re: AW: Bug Repoting Directions Request)
On 11/19/2014 07:54 AM, Paolo Bonzini wrote:
On 19/11/2014 13:07, Prof. Dr. Michael Schefczyk wrote:
Yes! My level of knowledge is that one uses the qcow2 format in order
to be able to create live snapshots/backups. Otherwise one would tend
to use the more efficient raw format. Is this not correct and did I
apply the backup mechanism in the wrong way?
That's correct, but you still have to create live snapshots from
within QEMU.
You absolutely CANNOT modify a qcow2 file via qemu-img or any other external
means while qemu is using the file in read-write mode, or you risk file
corruption (which appears to be what happened to you).
Monitor commands to the qemu process are the only supported means for taking a
live snapshot/backup of an in-use disk.
This is done with a QMP (QEMU Management Protocol) command like
{ execute: blockdev-snapshot-internal-sync,
arguments: { device: ide-hd0,
name: snapshot0 }
}
QMP is accessed through normal sockets, or via libvirt.
However, I'm not sure if running qemu-img convert on the resulting
snapshot is possible though, and there is no equivalent of qemu-img
snapshot -d.
qemu-img snapshot -d can be achieved via the QMP monitor command
'human-monitor-command' forwarding to the HMP 'delvm' command. But you are
correct that I don't know of any way to reproduce qemu-img convert on a
read-write image.
You can instead use QEMU's support for backup, which will do what you
wanted directly while the VM is running. For example:
{ execute: drive-backup, arguments: { device: ide-hd0,
sync: full, format: qcow2,
target: backup.img } }
This does not even require qcow2 for the image. The downside is that
you must not turn off the VM until the job has completed.
Another option is to create a live snapshot (so that your file under interest
is now read-only with a temporary qcow2 overlay as the read-write file), then
you can do whatever you want with the file (such as qemu-img convert), and
finally do an active commit to get rid of the temporary qcow2 overlay and back
to just the main live image.
--
Eric Blake eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org
smime.p7s
Description: S/MIME Cryptographic Signature
