Re: zlib-ng as a compat replacement for zlib

[Neal Gompa]

On Tue, Oct 17, 2023 at 1:59 PM Mattia Verga via devel
 wrote:
>
> Is there any chance we can have this drafted as a change request for F40?
>

It is being drafted, don't worry:
https://fedoraproject.org/wiki/Changes/ZlibNGTransition

We'll hopefully submit it soon.




--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Mattia Verga via devel]

Is there any chance we can have this drafted as a change request for F40?

Inviato da Proton Mail mobile

 Messaggio originale 
Il 19 Set 2023, 18:26, Daniel Alley ha scritto:

> Will this be posted soon? ___ 
> devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an 
> email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List 
> Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List 
> Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org 
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel Alley]

Will this be posted soon?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Leslie Satenstein via devel]

Thank you. 


Leslie Satenstein
 

On Saturday, September 2, 2023 at 09:29:36 a.m. EDT, Daniel Alley 
 wrote:  
 
 Yeah, the Zstd default of 3 is tuned to be roughly on par with alternatives 
but much faster.  To get better compression than gzip you generally need to go 
to higher levels.  
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue
  ___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel Alley]

Yeah, the Zstd default of 3 is tuned to be roughly on par with alternatives but 
much faster.  To get better compression than gzip you generally need to go to 
higher levels.  
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 06:45:22PM +0200, Florian Weimer wrote:
> * Richard W. M. Jones:
> 
> > I tested the speed of decompression using:
> >
> >   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw 
> > test.out'
> >   (qemu 8.0.0-4.fc39.x86_64)
> >
> >   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run 
> > '\''nbdcopy --request-size "$uri" test.out'\'' '
> >   (nbdkit-1.35.11-2.fc40.x86_64)
> 
> How realistic is that?  Larger cluster sizes will make random access
> perform noticeably worse is some cases.  Think about reading a few bytes
> towards the end of the cluster.  It makes a difference whether you have
> to decompress 64 KiB bytes for that, or 2 MiB.  As far as I understand
> it, the above commands use all data decompressed, so they don't suffer
> from this issue (particularly with read-ahead to deal with unfortunate
> cluster boundaries).
> 
> Time to first HTTP request served after boot or something like that
> might be a better comparison.

Yes, this is a good point.

Current Fedora images use 64k cluster size which I think is also the
default:

$ qemu-img info 
https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-Base-38-1.6.x86_64.qcow2
 
image: 
https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-Base-38-1.6.x86_64.qcow2
file format: qcow2
virtual size: 5 GiB (5368709120 bytes)
disk size: unavailable
cluster_size: 65536<
Format specific information:
compat: 0.10
compression type: zlib
refcount bits: 16

And the tests showed there's barely any performance gain above the
default cluster size anyway.  Only very small clusters should be
avoided, but there are good reasons to avoid those already such as
metadata overhead and small maximum image size.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Florian Weimer]

* Richard W. M. Jones:

> I tested the speed of decompression using:
>
>   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw 
> test.out'
>   (qemu 8.0.0-4.fc39.x86_64)
>
>   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run 
> '\''nbdcopy --request-size "$uri" test.out'\'' '
>   (nbdkit-1.35.11-2.fc40.x86_64)

How realistic is that?  Larger cluster sizes will make random access
perform noticeably worse is some cases.  Think about reading a few bytes
towards the end of the cluster.  It makes a difference whether you have
to decompress 64 KiB bytes for that, or 2 MiB.  As far as I understand
it, the above commands use all data decompressed, so they don't suffer
from this issue (particularly with read-ahead to deal with unfortunate
cluster boundaries).

Time to first HTTP request served after boot or something like that
might be a better comparison.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard Shaw]

On Fri, Sep 1, 2023 at 9:02 AM Richard W.M. Jones  wrote:

> On Fri, Sep 01, 2023 at 07:10:59AM -0500, Richard Shaw wrote:
> > Drive by comment because I found this thread interesting...
> >
> > What compression level was used for zstd? My understanding is that higher
> > levels take longer to compress but decompression time remains virtually
> the
> > same.
>
> Seems to be the default level?  Or at least it doesn't seem like it is
> being set.  Can you tell from this code:
>
>
> https://gitlab.com/qemu-project/qemu/-/blob/17780edd81d27fcfdb7a802efc870a99788bd2fc/block/qcow2-threads.c#L176


Yeah didn't see anything referencing the `compression_level` property, and
the default is 3. If compression performance doesn't matter as much I would
try increasing it to maybe 9? The levels are 1 to 22 but I'm sure there's a
diminishing return.

Thanks,
Richard
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 07:10:59AM -0500, Richard Shaw wrote:
> Drive by comment because I found this thread interesting...
> 
> What compression level was used for zstd? My understanding is that higher
> levels take longer to compress but decompression time remains virtually the
> same.

Seems to be the default level?  Or at least it doesn't seem like it is
being set.  Can you tell from this code:

https://gitlab.com/qemu-project/qemu/-/blob/17780edd81d27fcfdb7a802efc870a99788bd2fc/block/qcow2-threads.c#L176

Rich.

> Thanks,
> Richard

> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue


-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard Shaw]

Drive by comment because I found this thread interesting...

What compression level was used for zstd? My understanding is that higher
levels take longer to compress but decompression time remains virtually the
same.

Thanks,
Richard
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 01:10:16PM +0200, Kevin Wolf wrote:
> And nbdkit seems to get worse instead of better with larger cluster
> size, no matter whether zlib or zstd is used.

It's caused by nbdcopy's default request size being 256k.  Increasing
it to 2M cures the scaling problem - see updated results below.  (Note
nbdkit & nbdcopy are being used together so we're copying between two
programs.  The request size is the size of NBD requests between the two.)

> If you think using more threads is the key for the remaining difference
> at 64k, would increasing QCOW2_MAX_THREADS (currently only 4) help on
> the qemu-img side?

Results:

  qemu800 = qemu-img-8.0.0-4.fc39.x86_64 [previous results]
  qemugit = qemu @ 17780edd81d
  qemuthr = qemu @ 17780edd81d with QCOW2_MAX_THREADS changed from 4 to 16
  nbdkit = nbdkit-1.35.11-2.fc40.x86_64 [previous results]
  nbdkit2M = nbdkit with nbdcopy --request-size=$((2*1024*1024))


Cluster  Compression  Compressed size  Prog  Decompression speed

4k   zlib 3228811264   qemu800   5.921 s ±  0.074 s
4k   zstd 3258097664   qemu800   5.189 s ±  0.158 s

4k   zlib 3228811264   qemugit   7.021 s ±  0.234 s
4k   zstd 3258097664   qemugit   6.594 s ±  0.170 s

4k   zlib 3228811264   qemuthr   6.744 s ±  0.111 s
4k   zstd 3258097664   qemuthr   6.428 s ±  0.206 s

4k   zlib 3228811264   nbdkit1.390 s ±  0.094 s
4k   zstd 3258097664   nbdkit1.328 s ±  0.055 s


64k  zlib 3164667904   qemu800   3.579 s ±  0.094 s
64k  zstd 3132686336   qemu800   1.770 s ±  0.060 s

64k  zlib 3164667904   qemugit   3.644 s ±  0.018 s
64k  zstd 3132686336   qemugit   1.814 s ±  0.098 s

64k  zlib 3164667904   qemuthr   1.356 s ±  0.058 s
64k  zstd 3132686336   qemuthr   1.266 s ±  0.064 s

64k  zlib 3164667904   nbdkit1.254 s ±  0.065 s
64k  zstd 3132686336   nbdkit1.315 s ±  0.037 s


512k zlib 3158744576   qemu800   4.008 s ±  0.058 s
512k zstd 3032697344   qemu800   1.503 s ±  0.072 s

512k zlib 3158744576   qemugit   4.015 s ±  0.040 s
512k zstd 3032697344   qemugit   1.557 s ±  0.025 s

512k zlib 3158744576   qemuthr   1.233 s ±  0.050 s
512k zstd 3032697344   qemuthr   1.149 s ±  0.032 s

512k zlib 3158744576   nbdkit1.702 s ±  0.026 s
512k zstd 3032697344   nbdkit1.593 s ±  0.039 s


2048kzlib 3197569024   qemu800   4.327 s ±  0.051 s
2048kzstd 2995143168   qemu800   1.465 s ±  0.085 s

2048kzlib 3197569024   qemugit   4.323 s ±  0.031 s
2048kzstd 2995143168   qemugit   1.484 s ±  0.067 s

2048kzlib 3197569024   qemuthr   1.299 s ±  0.055 s
2048kzstd 2995143168   qemuthr   1.229 s ±  0.046 s

2048kzlib 3197569024   nbdkit2M  1.636 s ±  0.071 s
2048kzstd 2995143168   nbdkit2M  1.644 s ±  0.040 s


Increasing the number of threads makes a big difference, so I think
changing the default (or making it run-time adjustable somehow) is a
good idea, also an easy win.

Increased qcow2 threads + zlib-ng would be _very_ interesting.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Kevin Wolf]

Am 01.09.2023 um 12:03 hat Richard W.M. Jones geschrieben:
> On Fri, Sep 01, 2023 at 10:55:50AM +0100, Daniel P. Berrangé wrote:
> > On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
> > > On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
> > > > I understand the context and didn't really think about file size at all.
> > > > 
> > > > My question was essentially if decompressing many small blocks (as we do
> > > > today) performs significantly different from decompressing fewer larger
> > > > blocks (as we would do with a larger cluster size).
> > > 
> > > I did a quick test just by adjusting the cluster size of a qcow2 file:
> > > 
> > >   $ virt-builder fedora-36
> > >   $ ls -lsh fedora-36.img 
> > >   1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
> > >   $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > 
> > > test.raw
> > >   $ ls -lsh test.raw 
> > >   4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
> > >   $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o 
> > > compression_type=zlib,cluster_size=4096
> > > 
> > > (for cluster sizes 4k, 64k, 512k, 2048k, and
> > > compression types zlib & zstd)
> > > 
> > > I tested the speed of decompression using:
> > > 
> > >   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw 
> > > test.out'
> > >   (qemu 8.0.0-4.fc39.x86_64)
> > > 
> > >   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run 
> > > '\''nbdcopy --request-size "$uri" test.out'\'' '
> > >   (nbdkit-1.35.11-2.fc40.x86_64)
> > > 
> > > Results:
> > > 
> > >   Cluster  Compression  Compressed size  Prog   Decompression speed
> > > 
> > >   4k   zlib 3228811264   qemu   5.921 s ±  0.074 s
> > >   4k   zstd 3258097664   qemu   5.189 s ±  0.158 s
> > > 
> > >   4k   zlib/zstd nbdkit failed, bug!!
> > > 
> > >   64k  zlib 3164667904   qemu   3.579 s ±  0.094 s
> > >   64k  zstd 3132686336   qemu   1.770 s ±  0.060 s
> > > 
> > >   64k  zlib 3164667904   nbdkit 1.254 s ±  0.065 s
> > >   64k  zstd 3132686336   nbdkit 1.315 s ±  0.037 s
> > > 
> > >   512k zlib 3158744576   qemu   4.008 s ±  0.058 s
> > >   512k zstd 3032697344   qemu   1.503 s ±  0.072 s
> > > 
> > >   512k zlib 3158744576   nbdkit 1.702 s ±  0.026 s
> > >   512k zstd 3032697344   nbdkit 1.593 s ±  0.039 s
> > > 
> > >   2048kzlib 3197569024   qemu   4.327 s ±  0.051 s
> > >   2048kzstd 2995143168   qemu   1.465 s ±  0.085 s
> > > 
> > >   2048kzlib 3197569024   nbdkit 3.660 s ±  0.011 s
> > >   2048kzstd 2995143168   nbdkit 3.368 s ±  0.057 s
> > > 
> > > No great surprises - very small cluster size is inefficient, but
> > > scaling up the cluster size gain performance, and zstd performs better
> > > than zlib once the cluster size is sufficiently large.

It's interesting that for zstd, qemu-img keeps getting better with
increasing cluster size, while zlib numbers get worse again above 64k.
And nbdkit seems to get worse instead of better with larger cluster
size, no matter whether zlib or zstd is used.

> > The default qcow2 cluster size is 64k, which means we've already
> > got the vast majority of the perfornmance and file size win. Going
> > beyond 64k defaults doesn't seem massively compelling.
> > 
> > zstd does have a small space win over zlib as expected, but again
> > nothing so drastic that it seems compelling to change - that win
> > will be line noise in the overall bigger picture of image storage
> > and download times.
> 
> Yeah, I was a bit surprised by this.  I expected zstd files to be
> significantly smaller than zlib even though that's not what zstd is
> optimized for.  Not that they'd be about the same.
> 
> > The major difference here is that zstd is much faster than zlib
> > at decompress. I'd be curious if zlib-ng closes that gap ?
> 
> It's quite hard to use zlib-ng in Fedora (currently) since it requires
> changes to the source code.  That is what the pull request being
> discussed would change, as you could simply install zlib-ng-compat
> which would replace libz.so.  But anyway I can't easily get results
> for qemu + zlib-ng, but we expect it would be ~ 40% faster at
> decompression, and decompression is what is taking most of the time in
> the qemu numbers above.
> 
> I forgot to say that nbdkit is using zlib-ng, since I made the source
> level changes a few weeks back (but most of the nbdkit performance
> improvement comes from being able to use lots of threads).

Ah, that's actually a very important detail. I was wondering why zlib
was performing so much better in nbdkit when zstd is more or less
comparable in both.

If you think using more threads is the key for the remaining difference
at 64k, would increasing QCOW2_MAX_THREADS (currentl

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 11:06:20AM +0100, Daniel P. Berrangé wrote:
> On Fri, Sep 01, 2023 at 11:03:54AM +0100, Richard W.M. Jones wrote:
> > I forgot to say that nbdkit is using zlib-ng, since I made the source
> > level changes a few weeks back (but most of the nbdkit performance
> > improvement comes from being able to use lots of threads).
> 
> Ah that last point is interesting. If we look at nbdkit results we can
> see that while zstd is clearly faster, the margin of the win is massively
> lower. So I presume we can infer similar margins if qemu-img were
> switched too.

FWIW here's the nbdkit commit which added zlib-ng support (assuming
you don't want to wait for a compat package).  It's not a massive
change so something similar might be done for qemu:

https://gitlab.com/nbdkit/nbdkit/-/commit/1b67e323e998a5d719f1afe43d5be84e45c6739b

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel P . Berrangé]

On Fri, Sep 01, 2023 at 11:03:54AM +0100, Richard W.M. Jones wrote:
> On Fri, Sep 01, 2023 at 10:55:50AM +0100, Daniel P. Berrangé wrote:
> > On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
> > > On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
> > > > Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
> > > > > On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
> > > > > > [ Cc: qemu-block ]
> > > > > > 
> > > > > > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
> > > > > > > On Tue, Aug 29, 2023 at 05:49:24PM -, Daniel Alley wrote:
> > > > > > > > > The background to this is I've spent far too long trying to 
> > > > > > > > > optimize
> > > > > > > > > the conversion of qcow2 files to raw files.  Most existing 
> > > > > > > > > qcow2 files
> > > > > > > > > that you can find online are zlib compressed, including the 
> > > > > > > > > qcow2
> > > > > > > > > images provided by Fedora.  Each cluster in the file is 
> > > > > > > > > separately
> > > > > > > > > compressed as a zlib stream, and qemu uses zlib library 
> > > > > > > > > functions to
> > > > > > > > > decompress them.  When downloading and decompressing these 
> > > > > > > > > files, I
> > > > > > > > > measured 40%+ of the total CPU time is doing zlib 
> > > > > > > > > decompression.
> > > > > > > > > 
> > > > > > > > > [You don't need to tell me how great Zstd is, qcow2 supports 
> > > > > > > > > this for
> > > > > > > > > compression also, but it is not widely used by existing 
> > > > > > > > > content.]
> > > > > > 
> > > > > > You make it sound like compressing each cluster individually has a 
> > > > > > big
> > > > > > impact. If so, does increasing the cluster size make a difference, 
> > > > > > too?
> > > > > > That could be an change with less compatibility concerns.
> > > > > 
> > > > > The issue we're discussing in the original thread is speed of
> > > > > decompression.  We noted that using zlib-ng (a not-quite drop-in
> > > > > replacement for zlib) improves decompression speed by 40% or more.
> > > > > 
> > > > > Original thread:
> > > > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CDNPJ4SOTRQMYVCDI3ZSY4SP4FYESCWD/
> > > > > zlib-ng proposed change:
> > > > > https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
> > > > > 
> > > > > Size of the compressed file is also a concern, but wasn't discussed.
> > > > 
> > > > I understand the context and didn't really think about file size at all.
> > > > 
> > > > My question was essentially if decompressing many small blocks (as we do
> > > > today) performs significantly different from decompressing fewer larger
> > > > blocks (as we would do with a larger cluster size).
> > > 
> > > I did a quick test just by adjusting the cluster size of a qcow2 file:
> > > 
> > >   $ virt-builder fedora-36
> > >   $ ls -lsh fedora-36.img 
> > >   1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
> > >   $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > 
> > > test.raw
> > >   $ ls -lsh test.raw 
> > >   4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
> > >   $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o 
> > > compression_type=zlib,cluster_size=4096
> > > 
> > > (for cluster sizes 4k, 64k, 512k, 2048k, and
> > > compression types zlib & zstd)
> > > 
> > > I tested the speed of decompression using:
> > > 
> > >   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw 
> > > test.out'
> > >   (qemu 8.0.0-4.fc39.x86_64)
> > > 
> > >   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run 
> > > '\''nbdcopy --request-size "$uri" test.out'\'' '
> > >   (nbdkit-1.35.11-2.fc40.x86_64)
> > > 
> > > Results:
> > > 
> > >   Cluster  Compression  Compressed size  Prog   Decompression speed
> > > 
> > >   4k   zlib 3228811264   qemu   5.921 s ±  0.074 s
> > >   4k   zstd 3258097664   qemu   5.189 s ±  0.158 s
> > > 
> > >   4k   zlib/zstd nbdkit failed, bug!!
> > > 
> > >   64k  zlib 3164667904   qemu   3.579 s ±  0.094 s
> > >   64k  zstd 3132686336   qemu   1.770 s ±  0.060 s
> > > 
> > >   64k  zlib 3164667904   nbdkit 1.254 s ±  0.065 s
> > >   64k  zstd 3132686336   nbdkit 1.315 s ±  0.037 s
> > > 
> > >   512k zlib 3158744576   qemu   4.008 s ±  0.058 s
> > >   512k zstd 3032697344   qemu   1.503 s ±  0.072 s
> > > 
> > >   512k zlib 3158744576   nbdkit 1.702 s ±  0.026 s
> > >   512k zstd 3032697344   nbdkit 1.593 s ±  0.039 s
> > > 
> > >   2048kzlib 3197569024   qemu   4.327 s ±  0.051 s
> > >   2048kzstd 2995143168   qemu   1.465 s ±  0.085 s
> > > 
> > >   2048kzlib 3197569024   nbdkit 3.660 s ±  0.011 s
> > >   2048kzstd 2995143168   nbdkit 3.368 s 

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 10:55:50AM +0100, Daniel P. Berrangé wrote:
> On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
> > On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
> > > Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
> > > > On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
> > > > > [ Cc: qemu-block ]
> > > > > 
> > > > > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
> > > > > > On Tue, Aug 29, 2023 at 05:49:24PM -, Daniel Alley wrote:
> > > > > > > > The background to this is I've spent far too long trying to 
> > > > > > > > optimize
> > > > > > > > the conversion of qcow2 files to raw files.  Most existing 
> > > > > > > > qcow2 files
> > > > > > > > that you can find online are zlib compressed, including the 
> > > > > > > > qcow2
> > > > > > > > images provided by Fedora.  Each cluster in the file is 
> > > > > > > > separately
> > > > > > > > compressed as a zlib stream, and qemu uses zlib library 
> > > > > > > > functions to
> > > > > > > > decompress them.  When downloading and decompressing these 
> > > > > > > > files, I
> > > > > > > > measured 40%+ of the total CPU time is doing zlib decompression.
> > > > > > > > 
> > > > > > > > [You don't need to tell me how great Zstd is, qcow2 supports 
> > > > > > > > this for
> > > > > > > > compression also, but it is not widely used by existing 
> > > > > > > > content.]
> > > > > 
> > > > > You make it sound like compressing each cluster individually has a big
> > > > > impact. If so, does increasing the cluster size make a difference, 
> > > > > too?
> > > > > That could be an change with less compatibility concerns.
> > > > 
> > > > The issue we're discussing in the original thread is speed of
> > > > decompression.  We noted that using zlib-ng (a not-quite drop-in
> > > > replacement for zlib) improves decompression speed by 40% or more.
> > > > 
> > > > Original thread:
> > > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CDNPJ4SOTRQMYVCDI3ZSY4SP4FYESCWD/
> > > > zlib-ng proposed change:
> > > > https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
> > > > 
> > > > Size of the compressed file is also a concern, but wasn't discussed.
> > > 
> > > I understand the context and didn't really think about file size at all.
> > > 
> > > My question was essentially if decompressing many small blocks (as we do
> > > today) performs significantly different from decompressing fewer larger
> > > blocks (as we would do with a larger cluster size).
> > 
> > I did a quick test just by adjusting the cluster size of a qcow2 file:
> > 
> >   $ virt-builder fedora-36
> >   $ ls -lsh fedora-36.img 
> >   1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
> >   $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > test.raw
> >   $ ls -lsh test.raw 
> >   4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
> >   $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o 
> > compression_type=zlib,cluster_size=4096
> > 
> > (for cluster sizes 4k, 64k, 512k, 2048k, and
> > compression types zlib & zstd)
> > 
> > I tested the speed of decompression using:
> > 
> >   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw 
> > test.out'
> >   (qemu 8.0.0-4.fc39.x86_64)
> > 
> >   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run 
> > '\''nbdcopy --request-size "$uri" test.out'\'' '
> >   (nbdkit-1.35.11-2.fc40.x86_64)
> > 
> > Results:
> > 
> >   Cluster  Compression  Compressed size  Prog   Decompression speed
> > 
> >   4k   zlib 3228811264   qemu   5.921 s ±  0.074 s
> >   4k   zstd 3258097664   qemu   5.189 s ±  0.158 s
> > 
> >   4k   zlib/zstd nbdkit failed, bug!!
> > 
> >   64k  zlib 3164667904   qemu   3.579 s ±  0.094 s
> >   64k  zstd 3132686336   qemu   1.770 s ±  0.060 s
> > 
> >   64k  zlib 3164667904   nbdkit 1.254 s ±  0.065 s
> >   64k  zstd 3132686336   nbdkit 1.315 s ±  0.037 s
> > 
> >   512k zlib 3158744576   qemu   4.008 s ±  0.058 s
> >   512k zstd 3032697344   qemu   1.503 s ±  0.072 s
> > 
> >   512k zlib 3158744576   nbdkit 1.702 s ±  0.026 s
> >   512k zstd 3032697344   nbdkit 1.593 s ±  0.039 s
> > 
> >   2048kzlib 3197569024   qemu   4.327 s ±  0.051 s
> >   2048kzstd 2995143168   qemu   1.465 s ±  0.085 s
> > 
> >   2048kzlib 3197569024   nbdkit 3.660 s ±  0.011 s
> >   2048kzstd 2995143168   nbdkit 3.368 s ±  0.057 s
> > 
> > No great surprises - very small cluster size is inefficient, but
> > scaling up the cluster size gain performance, and zstd performs better
> > than zlib once the cluster size is sufficiently large.
> 
> The default qcow2 cluster size is 64k, which means we've already
> got the vast maj

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
> Results:
> 
>   Cluster  Compression  Compressed size  Prog   Decompression speed
> 
>   4k   zlib 3228811264   qemu   5.921 s ±  0.074 s
>   4k   zstd 3258097664   qemu   5.189 s ±  0.158 s
> 
>   4k   zlib/zstd nbdkit failed, bug!!

Stupid bounds checking bug, fixed now [1]:

4k   zlib 3228811264   nbdkit 1.390 s ±  0.094 s
4k   zstd 3258097664   nbdkit 1.328 s ±  0.055 s

>   64k  zlib 3164667904   qemu   3.579 s ±  0.094 s
>   64k  zstd 3132686336   qemu   1.770 s ±  0.060 s
> 
>   64k  zlib 3164667904   nbdkit 1.254 s ±  0.065 s
>   64k  zstd 3132686336   nbdkit 1.315 s ±  0.037 s
> 
>   512k zlib 3158744576   qemu   4.008 s ±  0.058 s
>   512k zstd 3032697344   qemu   1.503 s ±  0.072 s
> 
>   512k zlib 3158744576   nbdkit 1.702 s ±  0.026 s
>   512k zstd 3032697344   nbdkit 1.593 s ±  0.039 s
> 
>   2048kzlib 3197569024   qemu   4.327 s ±  0.051 s
>   2048kzstd 2995143168   qemu   1.465 s ±  0.085 s
> 
>   2048kzlib 3197569024   nbdkit 3.660 s ±  0.011 s
>   2048kzstd 2995143168   nbdkit 3.368 s ±  0.057 s

[1] 
https://gitlab.com/nbdkit/nbdkit/-/commit/73b58dc13dd7a3bbc7e7b3b718a535ee362caa92

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel P . Berrangé]

On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
> On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
> > Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
> > > On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
> > > > [ Cc: qemu-block ]
> > > > 
> > > > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
> > > > > On Tue, Aug 29, 2023 at 05:49:24PM -, Daniel Alley wrote:
> > > > > > > The background to this is I've spent far too long trying to 
> > > > > > > optimize
> > > > > > > the conversion of qcow2 files to raw files.  Most existing qcow2 
> > > > > > > files
> > > > > > > that you can find online are zlib compressed, including the qcow2
> > > > > > > images provided by Fedora.  Each cluster in the file is separately
> > > > > > > compressed as a zlib stream, and qemu uses zlib library functions 
> > > > > > > to
> > > > > > > decompress them.  When downloading and decompressing these files, 
> > > > > > > I
> > > > > > > measured 40%+ of the total CPU time is doing zlib decompression.
> > > > > > > 
> > > > > > > [You don't need to tell me how great Zstd is, qcow2 supports this 
> > > > > > > for
> > > > > > > compression also, but it is not widely used by existing content.]
> > > > 
> > > > You make it sound like compressing each cluster individually has a big
> > > > impact. If so, does increasing the cluster size make a difference, too?
> > > > That could be an change with less compatibility concerns.
> > > 
> > > The issue we're discussing in the original thread is speed of
> > > decompression.  We noted that using zlib-ng (a not-quite drop-in
> > > replacement for zlib) improves decompression speed by 40% or more.
> > > 
> > > Original thread:
> > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CDNPJ4SOTRQMYVCDI3ZSY4SP4FYESCWD/
> > > zlib-ng proposed change:
> > > https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
> > > 
> > > Size of the compressed file is also a concern, but wasn't discussed.
> > 
> > I understand the context and didn't really think about file size at all.
> > 
> > My question was essentially if decompressing many small blocks (as we do
> > today) performs significantly different from decompressing fewer larger
> > blocks (as we would do with a larger cluster size).
> 
> I did a quick test just by adjusting the cluster size of a qcow2 file:
> 
>   $ virt-builder fedora-36
>   $ ls -lsh fedora-36.img 
>   1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
>   $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > test.raw
>   $ ls -lsh test.raw 
>   4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
>   $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o 
> compression_type=zlib,cluster_size=4096
> 
> (for cluster sizes 4k, 64k, 512k, 2048k, and
> compression types zlib & zstd)
> 
> I tested the speed of decompression using:
> 
>   $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw 
> test.out'
>   (qemu 8.0.0-4.fc39.x86_64)
> 
>   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run 
> '\''nbdcopy --request-size "$uri" test.out'\'' '
>   (nbdkit-1.35.11-2.fc40.x86_64)
> 
> Results:
> 
>   Cluster  Compression  Compressed size  Prog   Decompression speed
> 
>   4k   zlib 3228811264   qemu   5.921 s ±  0.074 s
>   4k   zstd 3258097664   qemu   5.189 s ±  0.158 s
> 
>   4k   zlib/zstd nbdkit failed, bug!!
> 
>   64k  zlib 3164667904   qemu   3.579 s ±  0.094 s
>   64k  zstd 3132686336   qemu   1.770 s ±  0.060 s
> 
>   64k  zlib 3164667904   nbdkit 1.254 s ±  0.065 s
>   64k  zstd 3132686336   nbdkit 1.315 s ±  0.037 s
> 
>   512k zlib 3158744576   qemu   4.008 s ±  0.058 s
>   512k zstd 3032697344   qemu   1.503 s ±  0.072 s
> 
>   512k zlib 3158744576   nbdkit 1.702 s ±  0.026 s
>   512k zstd 3032697344   nbdkit 1.593 s ±  0.039 s
> 
>   2048kzlib 3197569024   qemu   4.327 s ±  0.051 s
>   2048kzstd 2995143168   qemu   1.465 s ±  0.085 s
> 
>   2048kzlib 3197569024   nbdkit 3.660 s ±  0.011 s
>   2048kzstd 2995143168   nbdkit 3.368 s ±  0.057 s
> 
> No great surprises - very small cluster size is inefficient, but
> scaling up the cluster size gain performance, and zstd performs better
> than zlib once the cluster size is sufficiently large.

The default qcow2 cluster size is 64k, which means we've already
got the vast majority of the perfornmance and file size win. Going
beyond 64k defaults doesn't seem massively compelling.

zstd does have a small space win over zlib as expected, but again
nothing so drastic that it seems compelling to change - that win
will be line noise in the overall bigger picture of image storage
and download times.

The major

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 10:42:16AM +0100, Richard W.M. Jones wrote:
>   $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run 
> '\''nbdcopy --request-size "$uri" test.out'\'' '

Sorry, copy and paste error, the command is:

hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run '\''nbdcopy 
"$uri" test.out'\'' '

unless you want to adjust the --request-size parameter to fix
the scaling problem at the largest cluster size.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Fri, Sep 01, 2023 at 10:48:14AM +0200, Kevin Wolf wrote:
> Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
> > On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
> > > [ Cc: qemu-block ]
> > > 
> > > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
> > > > On Tue, Aug 29, 2023 at 05:49:24PM -, Daniel Alley wrote:
> > > > > > The background to this is I've spent far too long trying to optimize
> > > > > > the conversion of qcow2 files to raw files.  Most existing qcow2 
> > > > > > files
> > > > > > that you can find online are zlib compressed, including the qcow2
> > > > > > images provided by Fedora.  Each cluster in the file is separately
> > > > > > compressed as a zlib stream, and qemu uses zlib library functions to
> > > > > > decompress them.  When downloading and decompressing these files, I
> > > > > > measured 40%+ of the total CPU time is doing zlib decompression.
> > > > > > 
> > > > > > [You don't need to tell me how great Zstd is, qcow2 supports this 
> > > > > > for
> > > > > > compression also, but it is not widely used by existing content.]
> > > 
> > > You make it sound like compressing each cluster individually has a big
> > > impact. If so, does increasing the cluster size make a difference, too?
> > > That could be an change with less compatibility concerns.
> > 
> > The issue we're discussing in the original thread is speed of
> > decompression.  We noted that using zlib-ng (a not-quite drop-in
> > replacement for zlib) improves decompression speed by 40% or more.
> > 
> > Original thread:
> > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CDNPJ4SOTRQMYVCDI3ZSY4SP4FYESCWD/
> > zlib-ng proposed change:
> > https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
> > 
> > Size of the compressed file is also a concern, but wasn't discussed.
> 
> I understand the context and didn't really think about file size at all.
> 
> My question was essentially if decompressing many small blocks (as we do
> today) performs significantly different from decompressing fewer larger
> blocks (as we would do with a larger cluster size).

I did a quick test just by adjusting the cluster size of a qcow2 file:

  $ virt-builder fedora-36
  $ ls -lsh fedora-36.img 
  1.2G -rw-r--r--. 1 rjones rjones 6.0G Sep  1 09:53 fedora-36.img
  $ cat fedora-36.img fedora-36.img fedora-36.img fedora-36.img  > test.raw
  $ ls -lsh test.raw 
  4.7G -rw-r--r--. 1 rjones rjones 24G Sep  1 09:53 test.raw
  $ qemu-img convert -f raw test.raw -O qcow2 test.qcow2.zlib.4k -c -o 
compression_type=zlib,cluster_size=4096

(for cluster sizes 4k, 64k, 512k, 2048k, and
compression types zlib & zstd)

I tested the speed of decompression using:

  $ hyperfine 'qemu-img convert -W -m 16 -f qcow2 test.qcow2.XXX -O raw 
test.out'
  (qemu 8.0.0-4.fc39.x86_64)

  $ hyperfine 'nbdkit -U - --filter=qcow2dec file test.qcow2.XXX --run 
'\''nbdcopy --request-size "$uri" test.out'\'' '
  (nbdkit-1.35.11-2.fc40.x86_64)

Results:

  Cluster  Compression  Compressed size  Prog   Decompression speed

  4k   zlib 3228811264   qemu   5.921 s ±  0.074 s
  4k   zstd 3258097664   qemu   5.189 s ±  0.158 s

  4k   zlib/zstd nbdkit failed, bug!!

  64k  zlib 3164667904   qemu   3.579 s ±  0.094 s
  64k  zstd 3132686336   qemu   1.770 s ±  0.060 s

  64k  zlib 3164667904   nbdkit 1.254 s ±  0.065 s
  64k  zstd 3132686336   nbdkit 1.315 s ±  0.037 s

  512k zlib 3158744576   qemu   4.008 s ±  0.058 s
  512k zstd 3032697344   qemu   1.503 s ±  0.072 s

  512k zlib 3158744576   nbdkit 1.702 s ±  0.026 s
  512k zstd 3032697344   nbdkit 1.593 s ±  0.039 s

  2048kzlib 3197569024   qemu   4.327 s ±  0.051 s
  2048kzstd 2995143168   qemu   1.465 s ±  0.085 s

  2048kzlib 3197569024   nbdkit 3.660 s ±  0.011 s
  2048kzstd 2995143168   nbdkit 3.368 s ±  0.057 s

No great surprises - very small cluster size is inefficient, but
scaling up the cluster size gain performance, and zstd performs better
than zlib once the cluster size is sufficiently large.

Something strange happens with 2048k clusters + zlib - the file gets
larger and decompression gets slower again.

nbdkit has several issues, fails completely at 4k, and scaling issues
at the largest cluster size.  The scaling issue turns out to be caused
by the default request size used by nbdcopy (256k, increasing it cures
the problem).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
devel mail

Re: zlib-ng as a compat replacement for zlib

[Kevin Wolf]

Am 31.08.2023 um 11:20 hat Richard W.M. Jones geschrieben:
> On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
> > [ Cc: qemu-block ]
> > 
> > Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
> > > On Tue, Aug 29, 2023 at 05:49:24PM -, Daniel Alley wrote:
> > > > > The background to this is I've spent far too long trying to optimize
> > > > > the conversion of qcow2 files to raw files.  Most existing qcow2 files
> > > > > that you can find online are zlib compressed, including the qcow2
> > > > > images provided by Fedora.  Each cluster in the file is separately
> > > > > compressed as a zlib stream, and qemu uses zlib library functions to
> > > > > decompress them.  When downloading and decompressing these files, I
> > > > > measured 40%+ of the total CPU time is doing zlib decompression.
> > > > > 
> > > > > [You don't need to tell me how great Zstd is, qcow2 supports this for
> > > > > compression also, but it is not widely used by existing content.]
> > 
> > You make it sound like compressing each cluster individually has a big
> > impact. If so, does increasing the cluster size make a difference, too?
> > That could be an change with less compatibility concerns.
> 
> The issue we're discussing in the original thread is speed of
> decompression.  We noted that using zlib-ng (a not-quite drop-in
> replacement for zlib) improves decompression speed by 40% or more.
> 
> Original thread:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CDNPJ4SOTRQMYVCDI3ZSY4SP4FYESCWD/
> zlib-ng proposed change:
> https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
> 
> Size of the compressed file is also a concern, but wasn't discussed.

I understand the context and didn't really think about file size at all.

My question was essentially if decompressing many small blocks (as we do
today) performs significantly different from decompressing fewer larger
blocks (as we would do with a larger cluster size).

Kevin
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Florian Weimer]

* Richard W. M. Jones:

> On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
>> Unfortunately, we seem to build the RHEL packages with --disable-zstd
>> (I suppose just because we tend to disable everything nobody explicitly
>> asked for). Maybe we should check other distros. If zstd is commonly
>> enabled, we could still make it the default upstream (because honestly,
>> it probably does makes sense from an upstream perspective). Of course,
>> for RHEL this would mean that images out there are likely to use zstd
>> soon, so it might need to enable zstd in newer versions, too.
>
> Oh that is bad actually.  We really should enable zstd support in RHEL :-/

The zstd package was added to RHEL 8 only after its release, so it's not
that you had much choice initially.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Thu, Aug 31, 2023 at 11:05:55AM +0200, Kevin Wolf wrote:
> [ Cc: qemu-block ]
> 
> Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
> > On Tue, Aug 29, 2023 at 05:49:24PM -, Daniel Alley wrote:
> > > > The background to this is I've spent far too long trying to optimize
> > > > the conversion of qcow2 files to raw files.  Most existing qcow2 files
> > > > that you can find online are zlib compressed, including the qcow2
> > > > images provided by Fedora.  Each cluster in the file is separately
> > > > compressed as a zlib stream, and qemu uses zlib library functions to
> > > > decompress them.  When downloading and decompressing these files, I
> > > > measured 40%+ of the total CPU time is doing zlib decompression.
> > > > 
> > > > [You don't need to tell me how great Zstd is, qcow2 supports this for
> > > > compression also, but it is not widely used by existing content.]
> 
> You make it sound like compressing each cluster individually has a big
> impact. If so, does increasing the cluster size make a difference, too?
> That could be an change with less compatibility concerns.

The issue we're discussing in the original thread is speed of
decompression.  We noted that using zlib-ng (a not-quite drop-in
replacement for zlib) improves decompression speed by 40% or more.

Original thread:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CDNPJ4SOTRQMYVCDI3ZSY4SP4FYESCWD/
zlib-ng proposed change:
https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3

Size of the compressed file is also a concern, but wasn't discussed.

> > > Independent from the decision to use zlib-ng as a distro-wide zlib
> > > replacement, which is a good idea regardless.
> > >
> > > Are there reasons why Fedora's qcow2 images cannot switch to Zstd
> > > compression?  Zstd support appears to have been added by QEMU 5.1 in
> > > August 2020, and both EL8 and EL9 appear to have newer versions QEMU
> > > available (therefore, they ought to be able to support those
> > > images).
> > 
> > TBH I think the most probable reason is that people don't know about
> > it and it is not obvious that you have to enable it.  To generate a
> > zlib-compressed qcow2 file, you simply add the -c option, easy.  To
> > use zstd compression you have to use this mouthful:
> > 
> >   qemu-img convert -f raw disk.img -O qcow2 disk.qcow2 -c -o 
> > compression_type=zstd
> > 
> > The qemu-img man page doesn't even mention it.
> 
> Good point, that needs to be fixed.
> 
> (Though I don't think that '-o compression_type=zstd' in an unreasonable
> mouthful for enabling a non-standard compression algorithm.)
> 
> > I think all recent qcow2-based tools should be fine with zstd, but I
> > didn't check them all (RHEL 7 is still quite popular so that platform
> > would no longer be compatible).
> 
> So my first thought was that maybe we can just change the default now
> that the option has been there for quite a while. But then it occurred
> to me that it's not a hard dependency. So at least, the default would
> still have to be zlib if zstd isn't even compiled in, which makes it a
> bit less nice in theory. In practice, it depends on what build options
> distros actually use.
> 
> Unfortunately, we seem to build the RHEL packages with --disable-zstd
> (I suppose just because we tend to disable everything nobody explicitly
> asked for). Maybe we should check other distros. If zstd is commonly
> enabled, we could still make it the default upstream (because honestly,
> it probably does makes sense from an upstream perspective). Of course,
> for RHEL this would mean that images out there are likely to use zstd
> soon, so it might need to enable zstd in newer versions, too.

Oh that is bad actually.  We really should enable zstd support in RHEL :-/

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Tue, Aug 29, 2023 at 05:49:24PM -, Daniel Alley wrote:
> > The background to this is I've spent far too long trying to optimize
> > the conversion of qcow2 files to raw files.  Most existing qcow2 files
> > that you can find online are zlib compressed, including the qcow2
> > images provided by Fedora.  Each cluster in the file is separately
> > compressed as a zlib stream, and qemu uses zlib library functions to
> > decompress them.  When downloading and decompressing these files, I
> > measured 40%+ of the total CPU time is doing zlib decompression.
> > 
> > [You don't need to tell me how great Zstd is, qcow2 supports this for
> > compression also, but it is not widely used by existing content.]
>
> Independent from the decision to use zlib-ng as a distro-wide zlib
> replacement, which is a good idea regardless.
>
> Are there reasons why Fedora's qcow2 images cannot switch to Zstd
> compression?  Zstd support appears to have been added by QEMU 5.1 in
> August 2020, and both EL8 and EL9 appear to have newer versions QEMU
> available (therefore, they ought to be able to support those
> images).

TBH I think the most probable reason is that people don't know about
it and it is not obvious that you have to enable it.  To generate a
zlib-compressed qcow2 file, you simply add the -c option, easy.  To
use zstd compression you have to use this mouthful:

  qemu-img convert -f raw disk.img -O qcow2 disk.qcow2 -c -o 
compression_type=zstd

The qemu-img man page doesn't even mention it.

I think all recent qcow2-based tools should be fine with zstd, but I
didn't check them all (RHEL 7 is still quite popular so that platform
would no longer be compatible).

> Sure, it doesn't help for existing content, but it would help for
> future content.  And I'm pretty sure zstd remains faster than
> zlib-ng despite the speed improvements in the latter.

Yup.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel Alley]

> The background to this is I've spent far too long trying to optimize
> the conversion of qcow2 files to raw files.  Most existing qcow2 files
> that you can find online are zlib compressed, including the qcow2
> images provided by Fedora.  Each cluster in the file is separately
> compressed as a zlib stream, and qemu uses zlib library functions to
> decompress them.  When downloading and decompressing these files, I
> measured 40%+ of the total CPU time is doing zlib decompression.
> 
> [You don't need to tell me how great Zstd is, qcow2 supports this for
> compression also, but it is not widely used by existing content.]

Independent from the decision to use zlib-ng as a distro-wide zlib replacement, 
which is a good idea regardless.

Are there reasons why Fedora's qcow2 images cannot switch to Zstd compression?  
Zstd support appears to have been added by QEMU 5.1 in August 2020, and both 
EL8 and EL9 appear to have newer versions QEMU available (therefore, they ought 
to be able to support those images). 

Sure, it doesn't help for existing content, but it would help for future 
content.  And I'm pretty sure zstd remains faster than zlib-ng despite the 
speed improvements in the latter.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

RFC: Pull request for zlib-ng compat Was: zlib-ng as a compat replacement for zlib

[Tulio Magno Quites Machado Filho]

I went ahead and started to modify Jacek's initial proposal [1] in order
to integrate the feedback from this thread as well as some changes I had
in mind.

I have created a RFC pull request [2]. Hopefully this will help answer
some of the questions we've seen here. Feedback is appreciated.

I have also created packages in a Copr project [3], but I haven't tested
this yet.

Future work:
 - Run tests using these packages.
 - Write the F40 change request based on the RFC pull request.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2145239
[2] https://src.fedoraproject.org/rpms/zlib-ng/pull-request/3
[3] https://copr.fedorainfracloud.org/coprs/tuliom/zlib-ng/

-- 
Tulio Magno
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Owen Taylor]

One practical example - the process of creating an OCI container image of
the Fedora Flatpak runtime does a *lot* of zlib compression and
decompression. (https://pagure.io/flatpak-module-tools/issue/22). A lot of
that is temporary to save disk space and IO bandwidth and could be switched
to zstd or a lower zlib compression level, but without any changes,
dropping in zlib-ng for the system zlib on my system reduced the wall time
for building the runtime from 9m13s to 6m49s. I suspect a lot of developer
tasks would get noticeably faster with a faster zlib, if not by that margin.

Regards,
Owen


On Sat, Aug 5, 2023 at 11:12 AM Richard W.M. Jones 
wrote:

> The background to this is I've spent far too long trying to optimize
> the conversion of qcow2 files to raw files.  Most existing qcow2 files
> that you can find online are zlib compressed, including the qcow2
> images provided by Fedora.  Each cluster in the file is separately
> compressed as a zlib stream, and qemu uses zlib library functions to
> decompress them.  When downloading and decompressing these files, I
> measured 40%+ of the total CPU time is doing zlib decompression.
>
> [You don't need to tell me how great Zstd is, qcow2 supports this for
> compression also, but it is not widely used by existing content.]
>
> I was looking around for alternative zlib implementations and there
> are several.  This AWS blog is a decent summary:
>
>
> https://aws.amazon.com/blogs/opensource/improving-zlib-cloudflare-and-comparing-performance-with-other-zlib-forks/
>
> We already package zlib-ng in Fedora:
>
>   https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec
>   https://github.com/zlib-ng/zlib-ng
>
> In my test zlib-ng is about 40% faster.
>
> Sadly zlib-ng is not compiled with the ZLIB_COMPAT option.  What this
> means in practice is that the zlib functions have different names
> (eg. zng_inflateInit instead of inflateInit).  It is not a drop-in
> replacement for zlib and software would need to be adjusted to use it.
>
> However there is this bug / RFE to package the compat library.
>
>   https://bugzilla.redhat.com/show_bug.cgi?id=2145239
>
> It literally replaces /usr/lib64/libz.so.1, so pretty high stakes.  It
> has the appropriate Provides/Conflicts.
>
> Anyway, 40% (or whatever, but significant) performance improvement
> sounds good for a very widely used operation.
>
> So I'd like to ask Fedora ...
>
> What do we think about the opt in approach of adding a patch similar
> to the one proposed in bug 2145239, where I think you could "simply"
> install zlib-ng to get better performance with existing software?
> ("simply" because it seems high risk of going wrong)
>
> What about replacing zlib with zlib-ng?
>
> Other ideas ...?  Does anyone know what other distros are doing?
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical machines to virtual machines.  Boot with a
> live CD or over the network (PXE) and turn machines into KVM guests.
> http://libguestfs.org/virt-v2v
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel Alley]

As far as I can tell it's mostly legacy removal and adoption of (now) 
standardized types, such as "z_size_t" being replaced by "size_t"

There is a document here that tries to describe the different considerations: 
https://github.com/zlib-ng/zlib-ng/blob/develop/PORTING.md

"The zlib-ng native has implemented some modernization and simplifications in 
its API, intended to make life easier for application developers."

"In certain places zlib-ng native uses more appropriate data types, removing 
the need for some workarounds in the API compared to zlib."

In another discussion the maintainer said that compat mode still includes all 
of the performance improvements.  

There is one compatibility hiccup mentioned on the porting document which also 
affects compat mode, which is that if you provide your own buffer, zlib-ng 
needs a larger buffer than zlib regardless of which way it is compiled.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Lukas Javorsky]

@Tulio Magno Quites Machado Filho  there is one
question I have due to the zlib-ng...

What is the difference between zlib-ng compiled with and without the
`ZLIB_COMPAT=ON` option? Are there any differences in the performance, or
is it only the names of the functions?
I'm interested in this because I want to know why would someone use the
"non-compat" zlib-ng? Are there any advantages in using it against the
"compat" zlib-ng?

Thanks for the information.

On Wed, Aug 23, 2023 at 11:00 AM Lukas Javorsky  wrote:

> In the end most of the patches were dropped the uplift wasn't worth the
>> effort of maintaining them downstream, when the effort can be better
>> spent getting a 10X uplift using a more modern compression
>> implementation (ex zstd/lz4/lzo/etc) that isn't written with so many
>> byte oriented assumptions.
>>
>
> Yes, generally having downstream patches is not the goal we want to chase
> in Fedora/RHEL.
> It takes too much effort and knowledge which could be used much more
> efficiently.
>
> And also one of the key values of Red Hat is "upstream first", so we
> always propose the patches to upstream making it a win-win for both sides.
> Unfortunately, zlib's upstream isn't too welcoming for complex patches
> such as these, and most of them are stuck in open PR.
>
> That is a big plus for zlib-ng as its upstream is open for such PRs.
> On the other hand, changes like these could lead to broken compatibility
> which is a big concern for our products in the case of widely used
> packages such as zlib.
>
>
>
> On Tue, Aug 22, 2023 at 10:26 PM Jeremy Linton 
> wrote:
>
>> Hi,
>>
>> On 8/6/23 08:33, John Reiser wrote:
>> > On 8/6/23 02:00, Peter Robinson wrote:
>> >> We tried to pull some of the optimisations in some time ago to the
>> >> Fedora package and they caused some issues with compatibility.
>> >
>> > Please provide *any* documentation!  Such as: the dates the work was
>> > performed,
>> > the participants, the nature of the issues, the "other side" of the
>> problem
>> > cases (the other packages, the use cases, etc.)
>>
>> Waves, some of this was my fault.
>>
>> example bug: https://bugzilla.redhat.com/show_bug.cgi?id=1582555
>> look at the zlib rpm history you will see things like:
>>
>>
>> https://src.fedoraproject.org/rpms/zlib/c/71a74f9c8684dd99c0e0657f53affb90a3ca3219?branch=rawhide
>>
>> there were a few others that were ignored, or we reverted part of the
>> original set of 5 or 6 patches. The original patches were aarch64 + NEON
>> optimizations, but there were a number of issues around unittests in
>> various packages that zipped something then validated the results
>> against a known crc/hash/etc which then failed because the hash changed,
>> the size changed, padding issues, the optimized code touched valid parts
>> of the buffer and tripped buffer poisoning logic, etc.
>>
>> Turns out zlib is old school byte oriented and any slight behavioral
>> change can result in compatibility issues. The first obvious
>> optimization is to increase the fetched word/matching sizes, which
>> maintain binary compatibility with the zlib format/decompressor but
>> results in buffer len/compressed size deltas.
>>
>> Of course some of these were potentially the fault of the patches, but
>> you have to decide between perf or compatibility when writing these, and
>> if the goal is faster, then the compatibility gets sacrificed.
>>
>> Bugzilla is taking its time retrieving some of the BZs that were closed
>> without fixes. So you will have to search for them yourself.
>>
>> In the end most of the patches were dropped the uplift wasn't worth the
>> effort of maintaining them downstream, when the effort can be better
>> spent getting a 10X uplift using a more modern compression
>> implementation (ex zstd/lz4/lzo/etc) that isn't written with so many
>> byte oriented assumptions.
>>
>>
>>
>>
>> > ___
>> > devel mailing list -- devel@lists.fedoraproject.org
>> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org
>> > Fedora Code of Conduct:
>> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> > List Archives:
>> >
>> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>> > Do not reply to spam, report it:
>> > https://pagure.io/fedora-infrastructure/new_issue
>> ___
>> devel mailing list -- devel@lists.fedoraproject.org
>> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
>> Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>> Do not reply to spam, report it:
>> https://pagure.io/fedora-infrastructure/new_issue
>>
>
>
> --
> S pozdravom/ Best regards
>
> Lukáš Javo

Re: zlib-ng as a compat replacement for zlib

[Lukas Javorsky]

>
> In the end most of the patches were dropped the uplift wasn't worth the
> effort of maintaining them downstream, when the effort can be better
> spent getting a 10X uplift using a more modern compression
> implementation (ex zstd/lz4/lzo/etc) that isn't written with so many
> byte oriented assumptions.
>

Yes, generally having downstream patches is not the goal we want to chase
in Fedora/RHEL.
It takes too much effort and knowledge which could be used much more
efficiently.

And also one of the key values of Red Hat is "upstream first", so we always
propose the patches to upstream making it a win-win for both sides.
Unfortunately, zlib's upstream isn't too welcoming for complex patches such
as these, and most of them are stuck in open PR.

That is a big plus for zlib-ng as its upstream is open for such PRs.
On the other hand, changes like these could lead to broken compatibility
which is a big concern for our products in the case of widely used
packages such as zlib.



On Tue, Aug 22, 2023 at 10:26 PM Jeremy Linton 
wrote:

> Hi,
>
> On 8/6/23 08:33, John Reiser wrote:
> > On 8/6/23 02:00, Peter Robinson wrote:
> >> We tried to pull some of the optimisations in some time ago to the
> >> Fedora package and they caused some issues with compatibility.
> >
> > Please provide *any* documentation!  Such as: the dates the work was
> > performed,
> > the participants, the nature of the issues, the "other side" of the
> problem
> > cases (the other packages, the use cases, etc.)
>
> Waves, some of this was my fault.
>
> example bug: https://bugzilla.redhat.com/show_bug.cgi?id=1582555
> look at the zlib rpm history you will see things like:
>
>
> https://src.fedoraproject.org/rpms/zlib/c/71a74f9c8684dd99c0e0657f53affb90a3ca3219?branch=rawhide
>
> there were a few others that were ignored, or we reverted part of the
> original set of 5 or 6 patches. The original patches were aarch64 + NEON
> optimizations, but there were a number of issues around unittests in
> various packages that zipped something then validated the results
> against a known crc/hash/etc which then failed because the hash changed,
> the size changed, padding issues, the optimized code touched valid parts
> of the buffer and tripped buffer poisoning logic, etc.
>
> Turns out zlib is old school byte oriented and any slight behavioral
> change can result in compatibility issues. The first obvious
> optimization is to increase the fetched word/matching sizes, which
> maintain binary compatibility with the zlib format/decompressor but
> results in buffer len/compressed size deltas.
>
> Of course some of these were potentially the fault of the patches, but
> you have to decide between perf or compatibility when writing these, and
> if the goal is faster, then the compatibility gets sacrificed.
>
> Bugzilla is taking its time retrieving some of the BZs that were closed
> without fixes. So you will have to search for them yourself.
>
> In the end most of the patches were dropped the uplift wasn't worth the
> effort of maintaining them downstream, when the effort can be better
> spent getting a 10X uplift using a more modern compression
> implementation (ex zstd/lz4/lzo/etc) that isn't written with so many
> byte oriented assumptions.
>
>
>
>
> > ___
> > devel mailing list -- devel@lists.fedoraproject.org
> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> >
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>


-- 
S pozdravom/ Best regards

Lukáš Javorský

Software Engineer, Core service - Databases

Red Hat 

Purkyňova 115 (TPB-C)

612 00 Brno - Královo Pole

ljavo...@redhat.com

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
ht

Re: zlib-ng as a compat replacement for zlib

[Jeremy Linton]

Hi,

On 8/6/23 08:33, John Reiser wrote:

On 8/6/23 02:00, Peter Robinson wrote:

We tried to pull some of the optimisations in some time ago to the
Fedora package and they caused some issues with compatibility.


Please provide *any* documentation!  Such as: the dates the work was 
performed,

the participants, the nature of the issues, the "other side" of the problem
cases (the other packages, the use cases, etc.)


Waves, some of this was my fault.

example bug: https://bugzilla.redhat.com/show_bug.cgi?id=1582555
look at the zlib rpm history you will see things like:

https://src.fedoraproject.org/rpms/zlib/c/71a74f9c8684dd99c0e0657f53affb90a3ca3219?branch=rawhide

there were a few others that were ignored, or we reverted part of the 
original set of 5 or 6 patches. The original patches were aarch64 + NEON 
optimizations, but there were a number of issues around unittests in 
various packages that zipped something then validated the results 
against a known crc/hash/etc which then failed because the hash changed, 
the size changed, padding issues, the optimized code touched valid parts 
of the buffer and tripped buffer poisoning logic, etc.


Turns out zlib is old school byte oriented and any slight behavioral 
change can result in compatibility issues. The first obvious 
optimization is to increase the fetched word/matching sizes, which 
maintain binary compatibility with the zlib format/decompressor but 
results in buffer len/compressed size deltas.


Of course some of these were potentially the fault of the patches, but 
you have to decide between perf or compatibility when writing these, and 
if the goal is faster, then the compatibility gets sacrificed.


Bugzilla is taking its time retrieving some of the BZs that were closed 
without fixes. So you will have to search for them yourself.


In the end most of the patches were dropped the uplift wasn't worth the 
effort of maintaining them downstream, when the effort can be better 
spent getting a 10X uplift using a more modern compression 
implementation (ex zstd/lz4/lzo/etc) that isn't written with so many 
byte oriented assumptions.






___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/

List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Tulio Magno Quites Machado Filho]

Sorry, resending because the original message was rejected by the
mailing list.

Hi Lukas,

Lukas Javorsky  writes:
> Hi,
>
> I'm currently maintaining the zlib package across Fedora and Red Hat products.
>
> I like the proposal for the zlib-ng package, there are just a few questions 
> for @Tulio Magno Quites Machado Filho  :
> 1) Just to clarify, do you want to have two separate packages (zlib-ng and 
> e.g. zlib-ng-compat) in Fedora? One with the `-DZLIB_COMPAT=ON` option 
> enabled and one without it?

Yes. While I do not have a personal preference, I believe it's important to 
provide the zlib-ng API for projects willing to use it instead of the zlib API.
I'm open to other suggestions too, including building zlib-ng twice and 
distributing them in different sub-packages as suggested by Michel.
Would you have any preferences?

> 2) What is your point of view on maintaining these packages? You will be the 
> main contact and I could be the secondary one?

LGTM.
Ali (in Cc.) has also demonstrated interest in this package too. I'd be happy 
to share this with both of you.

> 3) Same as 2) but for CentOS Stream and RHEL products?

Sorry, I'm afraid the decision on supporting RHEL products is beyond my pay 
grade.

> Next, I have a few scary scenarios in my head, which I'm not sure how would 
> be handled:

Please share all of them!
My experience maintaining long term libraries downstream is limited.

> 1) When we decide to migrate from zlib to zlib-ng and zlib-ng-compat, the 
> packages would still need to rewrite their code so they can use the pure (no 
> compat) zlib-ng functions and libraries. How many of the packages will be 
> able (and most importantly willing) to do that?

I disagree that packages "need to rewrite their code".
IMHO, most packages will probably keep using the zlib API and should magically 
link against the zlib-ng-compat package.

> 2) There are 271 RPMs dependent on zlib in ELN repo (there will be more in 
> the Fedora repo). It would mean that we would have to side-tag rebuild all of 
> them when switching to the zlib-ng-compat package. It may be challenging.

I'm planning to use the mass-prebuild tool on Copr first [1].

> If I understood something incorrectly please let me know, I'm trying to 
> understand it completely, what is the plan here. It will be needed to be 
> thoroughly documented in the Fedora Change.

Agreed.

> Overall, I think performance-wise this is a great idea. We just need to be 
> cautious about the compatibility.

Ack.

[1] https://gitlab.com/fedora/packager-tools/mass-prebuild

-- 
Tulio Magno
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Michel Alexandre Salim]

On Mon, Aug 07, 2023 at 09:29:08AM -0300, Tulio Magno Quites Machado Filho 
wrote:
> "Richard W.M. Jones"  writes:
> 
> > The background to this is I've spent far too long trying to optimize
> > the conversion of qcow2 files to raw files.  Most existing qcow2 files
> > that you can find online are zlib compressed, including the qcow2
> > images provided by Fedora.  Each cluster in the file is separately
> > compressed as a zlib stream, and qemu uses zlib library functions to
> > decompress them.  When downloading and decompressing these files, I
> > measured 40%+ of the total CPU time is doing zlib decompression.
> 
> This number may go even higher on s390x [1] because zlib-ng supports
> hardware acceleration.
> 
> qatzip [2] and libnxz [3] should provide performance on the same order of
> magnitude for Intel and Power9 processors, with the negative side of not
> using a single library.
> 
> > We already package zlib-ng in Fedora:
> >
> >   https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec
> >   https://github.com/zlib-ng/zlib-ng
> >
> > In my test zlib-ng is about 40% faster.
> >
> > Sadly zlib-ng is not compiled with the ZLIB_COMPAT option.  What this
> > means in practice is that the zlib functions have different names
> > (eg. zng_inflateInit instead of inflateInit).  It is not a drop-in
> > replacement for zlib and software would need to be adjusted to use it.
> >
> > However there is this bug / RFE to package the compat library.
> >
> >   https://bugzilla.redhat.com/show_bug.cgi?id=2145239
> >
> > It literally replaces /usr/lib64/libz.so.1, so pretty high stakes.  It
> > has the appropriate Provides/Conflicts.
> >
> > Anyway, 40% (or whatever, but significant) performance improvement
> > sounds good for a very widely used operation.
> >
> > So I'd like to ask Fedora ...
> >
> > What do we think about the opt in approach of adding a patch similar
> > to the one proposed in bug 2145239, where I think you could "simply"
> > install zlib-ng to get better performance with existing software?
> > ("simply" because it seems high risk of going wrong)
> 
> Wearing my zlib-ng fedora maintainer hat:
> I like the idea of this patch, although I think it needs a few changes.
> I'd like to merge it and add another package with the same source code,
> but different value for compat, i.e. package zlib-ng would still
> distribute the zlib-ng API while the new package would distribute the
> zlib-compatible API.

Would it help with maintenance, security issues etc. if we keep a single
package, and simply perform the build twice?

e.g. ncurses does this:
https://src.fedoraproject.org/rpms/ncurses/blob/rawhide/f/ncurses.spec#_145-165

Best regards,

-- 
Michel Alexandre Salim
identities: https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Lukas Javorsky]

Hi,

I'm currently maintaining the zlib package across Fedora and Red Hat
products.

I like the proposal for the zlib-ng package, there are just a few questions
for @Tulio Magno Quites Machado Filho  :
1) Just to clarify, do you want to have two separate packages (zlib-ng and
e.g. zlib-ng-compat) in Fedora? One with the `-DZLIB_COMPAT=ON` option
enabled and one without it?
2) What is your point of view on maintaining these packages? You will be
the main contact and I could be the secondary one? Or do you have someone
else in your team who could take the responsibility and our team could
leave those packages to you?
3) Same as 2) but for CentOS Stream and RHEL products?

Next, I have a few scary scenarios in my head, which I'm not sure how would
be handled:
1) When we decide to migrate from zlib to zlib-ng and zlib-ng-compat, the
packages would still need to rewrite their code so they can use the pure
(no compat) zlib-ng functions and libraries. How many of the packages will
be able (and most importantly willing) to do that?
2) There are 271 RPMs dependent on zlib in ELN repo (there will be more in
the Fedora repo). It would mean that we would have to side-tag rebuild all
of them when switching to the zlib-ng-compat package. It may be challenging.

If I understood something incorrectly please let me know, I'm trying to
understand it completely, what is the plan here. It will be needed to be
thoroughly documented in the Fedora Change.

Overall, I think performance-wise this is a great idea. We just need to be
cautious about the compatibility.

On Thu, Aug 17, 2023 at 6:49 AM Daniel Alley  wrote:

> The zlib-ng 2.1 beta, apparently, has some significant further
> enhancements coming down the pipe.  So the potential is there for users to
> see improvements much greater than 40% eventually.
>
> https://www.phoronix.com/news/Zlib-ng-2.1-Beta
>
> "With zlib-ng 2.1 beta there is upwards of 56% faster decompression
> performance when using an AVX2-capable x86_64 CPU. In general the
> decompression performance should be a "lot faster" and headlines this new
> beta release."
>
> "Zlib-ng 2.1 has also been working on compression improvements from levels
> 3 to 9 while the speed-ups are more focused on the decompression side. The
> zlib-ng 2.1 beta update has also been enhancing its CMake build system,
> improved support for the Apple M1, enhanced the EmScripten support for
> compiling to JavaScript, and many other changes."
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>


-- 
S pozdravom/ Best regards

Lukáš Javorský

Software Engineer, Core service - Databases

Red Hat 

Purkyňova 115 (TPB-C)

612 00 Brno - Královo Pole

ljavo...@redhat.com

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel Alley]

The zlib-ng 2.1 beta, apparently, has some significant further enhancements 
coming down the pipe.  So the potential is there for users to see improvements 
much greater than 40% eventually.

https://www.phoronix.com/news/Zlib-ng-2.1-Beta

"With zlib-ng 2.1 beta there is upwards of 56% faster decompression performance 
when using an AVX2-capable x86_64 CPU. In general the decompression performance 
should be a "lot faster" and headlines this new beta release."

"Zlib-ng 2.1 has also been working on compression improvements from levels 3 to 
9 while the speed-ups are more focused on the decompression side. The zlib-ng 
2.1 beta update has also been enhancing its CMake build system, improved 
support for the Apple M1, enhanced the EmScripten support for compiling to 
JavaScript, and many other changes."
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Ali Erdinc Koroglu]

On 07/08/2023 15:29, Tulio Magno Quites Machado Filho wrote:

"Richard W.M. Jones"  writes:


The background to this is I've spent far too long trying to optimize
the conversion of qcow2 files to raw files.  Most existing qcow2 files
that you can find online are zlib compressed, including the qcow2
images provided by Fedora.  Each cluster in the file is separately
compressed as a zlib stream, and qemu uses zlib library functions to
decompress them.  When downloading and decompressing these files, I
measured 40%+ of the total CPU time is doing zlib decompression.


This number may go even higher on s390x [1] because zlib-ng supports
hardware acceleration.

qatzip [2] and libnxz [3] should provide performance on the same order of
magnitude for Intel and Power9 processors, with the negative side of not
using a single library.


We already package zlib-ng in Fedora:

   https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec
   https://github.com/zlib-ng/zlib-ng

In my test zlib-ng is about 40% faster.

Sadly zlib-ng is not compiled with the ZLIB_COMPAT option.  What this
means in practice is that the zlib functions have different names
(eg. zng_inflateInit instead of inflateInit).  It is not a drop-in
replacement for zlib and software would need to be adjusted to use it.

However there is this bug / RFE to package the compat library.

   https://bugzilla.redhat.com/show_bug.cgi?id=2145239

It literally replaces /usr/lib64/libz.so.1, so pretty high stakes.  It
has the appropriate Provides/Conflicts.

Anyway, 40% (or whatever, but significant) performance improvement
sounds good for a very widely used operation.

So I'd like to ask Fedora ...

What do we think about the opt in approach of adding a patch similar
to the one proposed in bug 2145239, where I think you could "simply"
install zlib-ng to get better performance with existing software?
("simply" because it seems high risk of going wrong)


Wearing my zlib-ng fedora maintainer hat:
I like the idea of this patch, although I think it needs a few changes.
I'd like to merge it and add another package with the same source code,
but different value for compat, i.e. package zlib-ng would still
distribute the zlib-ng API while the new package would distribute the
zlib-compatible API.

That way, if there are any projects using the zlib-ng API, they could
still be supported.

[1] https://github.com/rust-lang/libz-sys/pull/72#issuecomment-904020105
[2] https://src.fedoraproject.org/rpms/qatzip
[3] https://src.fedoraproject.org/rpms/libnxz


We were discussing the use of zlib-ng in Hyperscale and ISA SIG with Florian 
Weimer, Honza Horak, Matej Muzila and Neal Gompa for a while.
I've started to prepare a change proposal for Fedora 40, so any additional 
ideas are more than welcome.

--
Ali Erdinc Koroglu
Intel, SSE | Linux OS Systems Engineering
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Tulio Magno Quites Machado Filho]

"Richard W.M. Jones"  writes:

> The background to this is I've spent far too long trying to optimize
> the conversion of qcow2 files to raw files.  Most existing qcow2 files
> that you can find online are zlib compressed, including the qcow2
> images provided by Fedora.  Each cluster in the file is separately
> compressed as a zlib stream, and qemu uses zlib library functions to
> decompress them.  When downloading and decompressing these files, I
> measured 40%+ of the total CPU time is doing zlib decompression.

This number may go even higher on s390x [1] because zlib-ng supports
hardware acceleration.

qatzip [2] and libnxz [3] should provide performance on the same order of
magnitude for Intel and Power9 processors, with the negative side of not
using a single library.

> We already package zlib-ng in Fedora:
>
>   https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec
>   https://github.com/zlib-ng/zlib-ng
>
> In my test zlib-ng is about 40% faster.
>
> Sadly zlib-ng is not compiled with the ZLIB_COMPAT option.  What this
> means in practice is that the zlib functions have different names
> (eg. zng_inflateInit instead of inflateInit).  It is not a drop-in
> replacement for zlib and software would need to be adjusted to use it.
>
> However there is this bug / RFE to package the compat library.
>
>   https://bugzilla.redhat.com/show_bug.cgi?id=2145239
>
> It literally replaces /usr/lib64/libz.so.1, so pretty high stakes.  It
> has the appropriate Provides/Conflicts.
>
> Anyway, 40% (or whatever, but significant) performance improvement
> sounds good for a very widely used operation.
>
> So I'd like to ask Fedora ...
>
> What do we think about the opt in approach of adding a patch similar
> to the one proposed in bug 2145239, where I think you could "simply"
> install zlib-ng to get better performance with existing software?
> ("simply" because it seems high risk of going wrong)

Wearing my zlib-ng fedora maintainer hat:
I like the idea of this patch, although I think it needs a few changes.
I'd like to merge it and add another package with the same source code,
but different value for compat, i.e. package zlib-ng would still
distribute the zlib-ng API while the new package would distribute the
zlib-compatible API.

That way, if there are any projects using the zlib-ng API, they could
still be supported.

[1] https://github.com/rust-lang/libz-sys/pull/72#issuecomment-904020105
[2] https://src.fedoraproject.org/rpms/qatzip
[3] https://src.fedoraproject.org/rpms/libnxz

-- 
Tulio Magno
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Zbigniew Jędrzejewski-Szmek]

On Sat, Aug 05, 2023 at 04:45:36PM +0100, Neal Gompa wrote:
> I think it'd be worth us moving to zlib-ng as the sole zlib provider.

+1. zlib-ng is already widely used, so if it has issues, then we'd have a
major problem. So we're not really saving much by having two implementations.
One library is better than two libraries, and 40% is a significant difference.

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel Alley]

Also to that point, the compatibility issues aren't always compatibility 
issues, but rather poorly written tests.  For example, some tests might 
hardcode an expected checksum [0] for the compressed output, which could be 
broken by any number of changes even if the compressed output is entirely valid 
and decompresses correctly.

[0] https://github.com/rpm-software-management/createrepo_c/pull/336
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[John Reiser]

On 8/6/23 02:00, Peter Robinson wrote:

We tried to pull some of the optimisations in some time ago to the
Fedora package and they caused some issues with compatibility.


Please provide *any* documentation!  Such as: the dates the work was performed,
the participants, the nature of the issues, the "other side" of the problem
cases (the other packages, the use cases, etc.)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

On Sun, Aug 06, 2023 at 04:34:46AM -, Daniel Alley wrote:
> >In my test zlib-ng is about 40% faster.
>
> I did some testing with zlib-ng and createrepo-c a few months ago
> [0], and I also found that the compression portion of the workload
> was about 40% faster.  So this matches my experience, too.
>
> (BTW, if that github comment [0] sounds negative, it isn't meant to,
> it's just that zlib ended up not being the primary culprit of the
> performance issue I was investigating at the time, so I was not as
> immediately interested in replacing it.)
>
> I support this proposal.  A tricky detail is that one of the big
> upsides of zlib-ng is that it has a lot of optimized codepaths for
> ARM, POWER, Z, RISC-V, AVX-256, AVX-512 and so forth which
> madler/zlib does not have.  And that's fantastic, but I expect it
> could make the testing process a bit more painful.

There are indeed quite a lot.  For example this is just x86:

https://github.com/zlib-ng/zlib-ng/tree/develop/arch/x86

> Since each of the arch-specific acceleration codepaths is behind a
> separate build flag [1] though, (I assume) we could easily do a more
> conservative rollout with most arch-specific optimizations disabled
> at first, and then enabled gradually over time.

& the other thing it is important to note is that it does seem to do
CPUID detection at runtime, so we are able to ship a "runs anywhere"
binary.  See:

https://github.com/zlib-ng/zlib-ng/blob/develop/arch/x86/x86_features.c

Rich.

> [0] 
> https://github.com/rpm-software-management/createrepo_c/pull/335#issuecomment-1381362220
> [1] https://github.com/zlib-ng/zlib-ng#advanced-build-options
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Peter Robinson]

On Sun, Aug 6, 2023 at 5:35 AM Daniel Alley  wrote:
>
> >In my test zlib-ng is about 40% faster.
>
> I did some testing with zlib-ng and createrepo-c a few months ago [0], and I 
> also found that the compression portion of the workload was about 40% faster. 
>  So this matches my experience, too.
>
> (BTW, if that github comment [0] sounds negative, it isn't meant to, it's 
> just that zlib ended up not being the primary culprit of the performance 
> issue I was investigating at the time, so I was not as immediately interested 
> in replacing it.)
>
> I support this proposal.  A tricky detail is that one of the big upsides of 
> zlib-ng is that it has a lot of optimized codepaths for ARM, POWER, Z, 
> RISC-V, AVX-256, AVX-512 and so forth which madler/zlib does not have.  And 
> that's fantastic, but I expect it could make the testing process a bit more 
> painful.

We tried to pull some of the optimisations in some time ago to the
Fedora package and they caused some issues with compatibility.

> Since each of the arch-specific acceleration codepaths is behind a separate 
> build flag [1] though, (I assume) we could easily do a more conservative 
> rollout with most arch-specific optimizations disabled at first, and then 
> enabled gradually over time.
>
> [0] 
> https://github.com/rpm-software-management/createrepo_c/pull/335#issuecomment-1381362220
> [1] https://github.com/zlib-ng/zlib-ng#advanced-build-options
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Daniel Alley]

>In my test zlib-ng is about 40% faster.

I did some testing with zlib-ng and createrepo-c a few months ago [0], and I 
also found that the compression portion of the workload was about 40% faster.  
So this matches my experience, too.

(BTW, if that github comment [0] sounds negative, it isn't meant to, it's just 
that zlib ended up not being the primary culprit of the performance issue I was 
investigating at the time, so I was not as immediately interested in replacing 
it.)

I support this proposal.  A tricky detail is that one of the big upsides of 
zlib-ng is that it has a lot of optimized codepaths for ARM, POWER, Z, RISC-V, 
AVX-256, AVX-512 and so forth which madler/zlib does not have.  And that's 
fantastic, but I expect it could make the testing process a bit more painful.

Since each of the arch-specific acceleration codepaths is behind a separate 
build flag [1] though, (I assume) we could easily do a more conservative 
rollout with most arch-specific optimizations disabled at first, and then 
enabled gradually over time.

[0] 
https://github.com/rpm-software-management/createrepo_c/pull/335#issuecomment-1381362220
[1] https://github.com/zlib-ng/zlib-ng#advanced-build-options
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Neal Gompa]

On Sat, Aug 5, 2023 at 4:12 PM Richard W.M. Jones  wrote:
>
> The background to this is I've spent far too long trying to optimize
> the conversion of qcow2 files to raw files.  Most existing qcow2 files
> that you can find online are zlib compressed, including the qcow2
> images provided by Fedora.  Each cluster in the file is separately
> compressed as a zlib stream, and qemu uses zlib library functions to
> decompress them.  When downloading and decompressing these files, I
> measured 40%+ of the total CPU time is doing zlib decompression.
>
> [You don't need to tell me how great Zstd is, qcow2 supports this for
> compression also, but it is not widely used by existing content.]
>
> I was looking around for alternative zlib implementations and there
> are several.  This AWS blog is a decent summary:
>
>   
> https://aws.amazon.com/blogs/opensource/improving-zlib-cloudflare-and-comparing-performance-with-other-zlib-forks/
>
> We already package zlib-ng in Fedora:
>
>   https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec
>   https://github.com/zlib-ng/zlib-ng
>
> In my test zlib-ng is about 40% faster.
>
> Sadly zlib-ng is not compiled with the ZLIB_COMPAT option.  What this
> means in practice is that the zlib functions have different names
> (eg. zng_inflateInit instead of inflateInit).  It is not a drop-in
> replacement for zlib and software would need to be adjusted to use it.
>
> However there is this bug / RFE to package the compat library.
>
>   https://bugzilla.redhat.com/show_bug.cgi?id=2145239
>
> It literally replaces /usr/lib64/libz.so.1, so pretty high stakes.  It
> has the appropriate Provides/Conflicts.
>
> Anyway, 40% (or whatever, but significant) performance improvement
> sounds good for a very widely used operation.
>
> So I'd like to ask Fedora ...
>
> What do we think about the opt in approach of adding a patch similar
> to the one proposed in bug 2145239, where I think you could "simply"
> install zlib-ng to get better performance with existing software?
> ("simply" because it seems high risk of going wrong)
>
> What about replacing zlib with zlib-ng?
>
> Other ideas ...?  Does anyone know what other distros are doing?
>

It's certainly an idea. In CentOS Hyperscale, we've discussed
proposing this for Fedora because of the better maintainability and
performance.

openSUSE ships zlib-ng-compat as a provider of zlib, though it is not
yet the default. They are working on build failures caused by
switching to zlib-ng as the provider of zlib.

I think it'd be worth us moving to zlib-ng as the sole zlib provider.


-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

zlib-ng as a compat replacement for zlib

[Richard W.M. Jones]

The background to this is I've spent far too long trying to optimize
the conversion of qcow2 files to raw files.  Most existing qcow2 files
that you can find online are zlib compressed, including the qcow2
images provided by Fedora.  Each cluster in the file is separately
compressed as a zlib stream, and qemu uses zlib library functions to
decompress them.  When downloading and decompressing these files, I
measured 40%+ of the total CPU time is doing zlib decompression.

[You don't need to tell me how great Zstd is, qcow2 supports this for
compression also, but it is not widely used by existing content.]

I was looking around for alternative zlib implementations and there
are several.  This AWS blog is a decent summary:

  
https://aws.amazon.com/blogs/opensource/improving-zlib-cloudflare-and-comparing-performance-with-other-zlib-forks/

We already package zlib-ng in Fedora:

  https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec
  https://github.com/zlib-ng/zlib-ng

In my test zlib-ng is about 40% faster.

Sadly zlib-ng is not compiled with the ZLIB_COMPAT option.  What this
means in practice is that the zlib functions have different names
(eg. zng_inflateInit instead of inflateInit).  It is not a drop-in
replacement for zlib and software would need to be adjusted to use it.

However there is this bug / RFE to package the compat library.

  https://bugzilla.redhat.com/show_bug.cgi?id=2145239

It literally replaces /usr/lib64/libz.so.1, so pretty high stakes.  It
has the appropriate Provides/Conflicts.

Anyway, 40% (or whatever, but significant) performance improvement
sounds good for a very widely used operation.

So I'd like to ask Fedora ...

What do we think about the opt in approach of adding a patch similar
to the one proposed in bug 2145239, where I think you could "simply"
install zlib-ng to get better performance with existing software?
("simply" because it seems high risk of going wrong)

What about replacing zlib with zlib-ng?

Other ideas ...?  Does anyone know what other distros are doing?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue