One practical example - the process of creating an OCI container image of the Fedora Flatpak runtime does a *lot* of zlib compression and decompression. (https://pagure.io/flatpak-module-tools/issue/22). A lot of that is temporary to save disk space and IO bandwidth and could be switched to zstd or a lower zlib compression level, but without any changes, dropping in zlib-ng for the system zlib on my system reduced the wall time for building the runtime from 9m13s to 6m49s. I suspect a lot of developer tasks would get noticeably faster with a faster zlib, if not by that margin.
Regards, Owen On Sat, Aug 5, 2023 at 11:12 AM Richard W.M. Jones <rjo...@redhat.com> wrote: > The background to this is I've spent far too long trying to optimize > the conversion of qcow2 files to raw files. Most existing qcow2 files > that you can find online are zlib compressed, including the qcow2 > images provided by Fedora. Each cluster in the file is separately > compressed as a zlib stream, and qemu uses zlib library functions to > decompress them. When downloading and decompressing these files, I > measured 40%+ of the total CPU time is doing zlib decompression. > > [You don't need to tell me how great Zstd is, qcow2 supports this for > compression also, but it is not widely used by existing content.] > > I was looking around for alternative zlib implementations and there > are several. This AWS blog is a decent summary: > > > https://aws.amazon.com/blogs/opensource/improving-zlib-cloudflare-and-comparing-performance-with-other-zlib-forks/ > > We already package zlib-ng in Fedora: > > https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec > https://github.com/zlib-ng/zlib-ng > > In my test zlib-ng is about 40% faster. > > Sadly zlib-ng is not compiled with the ZLIB_COMPAT option. What this > means in practice is that the zlib functions have different names > (eg. zng_inflateInit instead of inflateInit). It is not a drop-in > replacement for zlib and software would need to be adjusted to use it. > > However there is this bug / RFE to package the compat library. > > https://bugzilla.redhat.com/show_bug.cgi?id=2145239 > > It literally replaces /usr/lib64/libz.so.1, so pretty high stakes. It > has the appropriate Provides/Conflicts. > > Anyway, 40% (or whatever, but significant) performance improvement > sounds good for a very widely used operation. > > So I'd like to ask Fedora ... > > What do we think about the opt in approach of adding a patch similar > to the one proposed in bug 2145239, where I think you could "simply" > install zlib-ng to get better performance with existing software? > ("simply" because it seems high risk of going wrong) > > What about replacing zlib with zlib-ng? > > Other ideas ...? Does anyone know what other distros are doing? > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat > http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-p2v converts physical machines to virtual machines. Boot with a > live CD or over the network (PXE) and turn machines into KVM guests. > http://libguestfs.org/virt-v2v > _______________________________________________ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue >
_______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
