Bug#1002056: ITP: zlib-ng -- optimized zlib compression library

[David Heidelberg]

Hello,

I think it already makes sense to push zlib-ng and let it co-exist with 
zlib since you can port your software directly to the zlib-ng, which I'm 
currently doing for Mesa3D.


I dropped the zlib-ng sources into https://salsa.debian.org/dh/zlib-ng 
feel free to force push there any Debian relevant changes.


After introducing the zlib-ng, we could continue to the second phase 
migrating software still relying on zlib to zlib-ng compat layer.


What do you think?

David

On Mon, 6 Nov 2023 21:44:05 +0100 Sebastian Andrzej Siewior 
 wrote:


> On 2023-10-25 23:17:06 [+0200], Guillem Jover wrote:
> > Hi!
> Hi,
>
> > Ah, thanks! I had in my mind getting back to this ITP, given that the
> > zlib-ng project has continued to gain traction and seems to have
> > consolidated most of the other forks around it.
> >
> > So I'll draft another mail to Mark and probably to debian-devel to
> > discuss this.
>
> Do you want me to join your efforts? This looks interrestig. I may have
> time ;)
>
> > Thanks,
> > Guillem
>
> Sebastian
>
>

--
David Heidelberg

Bug#1002056: ITP: zlib-ng -- optimized zlib compression library

[Guillem Jover]

Hi!

On Sat, 2023-03-25 at 21:24:44 +0100, David Heidelberg wrote:
> I see you recently pushed some code into git, do you plan to push the code
> also into Debian itself?

Given that it conflicts with the zlib package, and I'm not sure it
makes sense to upload just the zlib-ng specific library without the
compatibility shims, I'm currently not planning on doing so, no. But
I should probably discuss this further with Mark Brown.

Thanks,
Guillem

Bug#1002056: ITP: zlib-ng -- optimized zlib compression library

[David Heidelberg]

Hello,

I see you recently pushed some code into git, do you plan to push the 
code also into Debian itself?


Thank you

David

--
David Heidelberg
Consultant Software Engineer

Bug#1002056: ITP: zlib-ng -- optimized zlib compression library

[Mark Brown]

On Tue, Dec 21, 2021 at 04:45:21AM +0100, Guillem Jover wrote:

> instead of with its native API. But if that happens, I think it would
> make sense to upload, as it's currently being embedded in several
> upstream projects and even if dpkg would not switch to it, it would
> still help with removing embedded code copies, and speeding up other
> packages. Or make other RPFs such as #901490 (an alternative fork)
> unnecessary.

The alternative fork is the big issue here - there's several different
zlib projects active which don't seem to be working towards each other
and there's none of them that is clearly going to be the one that will
become the default in future if it's not the original zlib.  We should
just pick a new zlib if we want a new zlib rather than making users go
and figure out which they want.


signature.asc
Description: PGP signature

Bug#1002056: ITP: zlib-ng -- optimized zlib compression library

[Guillem Jover]

On Mon, 2021-12-20 at 23:15:07 -0500, nick black wrote:
> not that i expect you to have run extensive benchmarks or
> anything, but how do you feel this compares to libdeflate?

I don't think it matters, because libdeflate does not support a
streaming API anyway, so it's of no use in many situations where
large files or streaming content needs to be dealt with. It also
does not provide a compatible zlib API, which means it cannot be
easily migrated to.

> the few comparisons i've seen suggest that they are (or at least
> were) pretty much a wash, performance-wise.

I think when I first noticed libdeflate being uploaded to Debian, I
took a look to also play with it for dpkg, but the above problems
meant that never got anywhere. From the benchmarks I've seen
libdeflate might be a bit faster, but then it would require tons more
memory to handle equivalent inputs and outputs.

The reasons I found the zlib-ng alternative interesting were because:

 - the Intel fork is not going to be adding non-Intel optimizations,
 - the Cloudflare fork does not look very lively,
 - the zlib upstream does not look very active, and apparently outright
   refuses contributions to the main code, and only accepts them into
   its contrib/ directory,

OTOH the zlib-ng looks very lively, accepts contributions, has had its
code modernized and cleaned up, and seems to be performing rather well
in comparison on multiple architectures.

Thanks,
Guillem

Bug#1002056: ITP: zlib-ng -- optimized zlib compression library

[nick black]

not that i expect you to have run extensive benchmarks or
anything, but how do you feel this compares to libdeflate? the
few comparisons i've seen suggest that they are (or at least
were) pretty much a wash, performance-wise.

Bug#1002056: ITP: zlib-ng -- optimized zlib compression library

[Guillem Jover]

Package: wnpp
Severity: wishlist
Owner: Guillem Jover 
X-Debbugs-Cc: debian-de...@lists.debian.org, Mark Brown 

* Package name: zlib-ng
  Version : 2.0.5
  Upstream Author : zlib-ng Team
* URL : http://github.com/zlib-ng/zlib-ng
* License : Zlib, Zlib-RFC, CC-BY-3.0, CC-BY-4.0, Public-Domain
  Programming Lang: C
  Description : optimized zlib compression library

 zlib-ng is a fork of the zlib library implementing the deflate compression
 method found in gzip and PKZIP.
 .
 It includes and consolidates many optimizations found in alternative forks,
 that have not been merged in the official zlib library.


I just discovered this project, and started packaging [P] it to be able
to play with a dpkg branch switching its zlib support to zlib-ng. The
speed up is quite significant, for example when packing the 0ad-data,
on my system it takes 5m50s~ with zlib and 4m30s~ with zlib-ng.

  [P] https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/

The project has a compat mode which generates API "compatible" zlib
replacement libraries, but unfortunately it is stated not to guarantee
to be ABI compatible, so no compat packages or similar could be
produced right now as that could potentially break other packages. I've
filed a report [S] upstream to request a more usable shim instead.

  [S] 


I'm still pondering whether to upload this, although the packing is
already done, but w/o the above mentioned shim its utility seems
restricted as most upstream projects use it via its compat mode,
instead of with its native API. But if that happens, I think it would
make sense to upload, as it's currently being embedded in several
upstream projects and even if dpkg would not switch to it, it would
still help with removing embedded code copies, and speeding up other
packages. Or make other RPFs such as #901490 (an alternative fork)
unnecessary.

Thanks,
Guillem