On 5/17/19 4:05 AM, Denis Plotnikov wrote:
>
>
> On 17.05.2019 2:25, John Snow wrote:
>>
>>
>> On 5/16/19 9:48 AM, Denis Plotnikov wrote:
>>> The patch adds some preparation parts for incompatible compression type
>>> feature into QCOW2 header that indicates that *all* compressed clusters
>>> must be (de)compressed using a certain compression type.
>>>
>>> It is implied that the compression type is set on the image creation and
>>> can be changed only later by image conversion, thus the only compression
>>> algorithm is used for the image.
>>>
>>> The plan is to add support for ZSTD and then may be something more effective
>>> in the future.
>>>
>>> ZSTD compression algorithm consumes 3-5 times less CPU power with a
>>> comparable compression ratio with zlib. It would be wise to use it for
>>> data compression e.g. for backups.
>>>
>>> The default compression is ZLIB.
>>>
>>
>> (Merely a curiosity:)
>>
>> Since this is coming from Virtuozzo, I trust that you've had good luck
>> with ZSTD already in R&D. What do the compression ratios look like in
>> practice? It's touted as "comparable to zlib" which certainly does sound
>> quite nice for streaming compression of backups.
>>
>> I suppose in the worst case it ought to be faster than bandwidth speeds,
>> so no harm in utilizing it.
> Yes, we did some research on it. Actually, there is a patch in the
> mailing list (please, take a look: ) which applies ZSTD compression in
> the migration.
> Here are the numbers from there:
>
> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
> migration to the same host
> VM: 2xVCPU, 8G RAM total
> 5G RAM used, memory populated with postgreqsl data
> produced by pgbench performance benchmark
>
> Threads: 1 compress – 1 decompress
>
> zstd provides slightly less compression ratio with almost the same
> CPU usage but copes with RAM compression roughly 2 times faster
>
> compression type zlib | zstd
> ---------------------------------------------------------
> compression level 1 5 | 1 5
> compression ratio 6.92 7.05 | 6.69 6.89
> cpu idle, % 82 83 | 86 80
> time, sec 49 71 | 26 31
> time diff to zlib, sec | -25 -41
> time diff to zlib, % | -47% -56%
>
> I general ZSTD provides better compression ratio on big dependent chunks
> of data. Than bigger the data size then better ZSTD it compresses.
>
> Since, in our cases (migration: 4K RAM block, qcow2: cluster) we
> have to compress independent chunks the ability of ZSTD to find better
> compression solution is restricted.
>
> Although, the compression ratio is pretty much the same in both cases
> the experiments shown that ZSTD does the compression much faster (x2).
>
> Which is obviously good for us.
>
> Here is other comparison of ZLIB vs ZSTD without any application in qemu
> which shows that ZSTD works faster
>
> zlib = max compression level
> zstd = compression level 5 (max 22)
>
> cycles consumed for compression:
>
> 4k txt 4K ram 4K bin 64K txt 64K ram 64K bin
> zlib 400K 344K 1.3M 13M 5M 92.3M
> zstd 350K 235K 312K 3.3M 1.3M 2.4M
> Diff,% -12 -32 -77 -75 -73 -97
>
Wow, the 4k bin one is drastic. The text is even more prominent. wow!
> size after compression in bytes:
>
> 4k txt 4K ram 4K bin 64K txt 64K ram 64K bin
> zlib 1542 3599 1403 17386 64735 20609
> zstd 1568 3540 1250 17656 65546 20023
> Diff,% -2 2 11 -2 -1 3
>
Yeah, that's pretty close. Seems like absolutely a great tradeoff for
the speed gain. If the little bit of difference matters to you, you can
always do some more heavy-duty compression of your choice in another
layer of the storage stack.
> Data sources for test files [we took 4K and 64K chunks from there]:
>
> txt = linux/Documentation/memory-barriers.txt
> ram = /boot/initramfs-4.20.0-rc6+.img
> bin = x86_64-softmmu/qemu-system-x86_64
>
> Increasing of ZSTD compression ratio didn't give any significant
> improvements of the out size but slowed down the pace of ZSTD
>
> Denis
>
>>
>>> Signed-off-by: Denis Plotnikov <dplotni...@virtuozzo.com>
>
Very useful data, thanks for sharing! Seems like this would indeed be a
great thing to have for qcow2.
