On Mon, Jun 26, 2017 at 12:12:05PM +0100, Daniel P. Berrange wrote: > On Mon, Jun 26, 2017 at 10:28:38AM +0200, Kevin Wolf wrote: > > [ Cc: qemu-devel; don't post to qemu-block only! ] > > > > Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben: > > > Hi, > > > > > > I am currently working on optimizing speed for compressed QCOW2 > > > images. We use them for templates and would also like to use them for > > > backups, but the latter is almost infeasible because using gzip for > > > compression is horribly slow. I tried to experiment with different > > > options to deflate, but in the end I think its better to use a > > > different compression algorithm for cases where speed matters. As we > > > already have probing for it in configure and as it is widely used I > > > would like to use LZO for that purpose. I think it would be best to > > > have a flag to indicate that compressed blocks use LZO compression, > > > but I would need a little explaination which of the feature fields I > > > have to use to prevent an older (incompatible) Qemu opening LZO > > > compressed QCOW2 images. > > > > > > I also have already some numbers. I converted a fresh Debian 9 Install > > > which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a > > > compressed QCOW2. With GZIP compression the result is 356MB whereas > > > the LZO version is 452MB. However, the current GZIP variant uses 35 > > > seconds for this operation where LZO only needs 4 seconds. I think is > > > is a good trade in especially when its optional so the user can > > > choose. > > > > > > What are your thoughts? > > > > We had a related RFC patch by Den earlier this year, which never > > received many comment and never got out of RFC: > > > > https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html > > > > So he chose a different algorithm (zstd). When I asked, he posted a > > comparison of algorithms (however a generic one and not measured in the > > context of qemu) that suggests that LZO would be slightly faster, but > > have a considerable worse compression ratio with the settings that were > > benchmarked. > > > > I think it's clear that if there is any serious interest in compression, > > we'll want to support at least one more algorithm. What we still need to > > evaluate is which one(s) to take, and whether a simple incompatible flag > > in the header like in Den's patch is enough or whether we should add a > > whole new header field for the compression algorithm (like we already > > have for encryption). > > We might also want to consider whether doing compression on individual > qcow2 clusters is the best approach for desired usage scenarios, as > compared to adding a general purpose compression block driver in between > the qcow2 driver and the file driver. eg akin to just running 'gzip' over > the entire qcow2 file. The relatively small size of qcow2 clusters limits > the effectiveness of all compression algorithms we might choose between. > By adding a separate compression driver below qcow2, we can choose a much > larger compression block size, independant of qcow2 cluster size. > > Taking a random disk image I have with 1 GB of data. If I tell 'xz' to > compress using 65 KB block sizes (to simulate compression attained if > compressing individual qcow2 clusters in isolation), then I get a file > 124 MB in size, which is barely better than that attained with qcow2's > built-in gzip compression. If I tell 'xz' to compress with 16 MB block > sizes though, the output is 114 MB in size, which is a massive win over > gzip.
I think we need both. qcow2 block compression allows for writable qcow2 files, whereas xz compression over a whole file is read-only. Both have real use cases. liblzma (xz) is a very flexible library which allows for (read-only) decompression of individual blocks. If you prepare your xz file with the --blocksize parameter, keeping it to something small like 16 MB, then you can make random-access xz-compressed disk images quite easily. In fact I wrote code to do exactly that: https://github.com/libguestfs/nbdkit/tree/master/plugins/xz Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top