On Fri, Apr 27, 2018 at 01:45:07PM +0200, Adam Borowski wrote: > (ZSTD) > > On Fri, Apr 27, 2018 at 07:02:12AM +0200, Guillem Jover wrote: > > Recently Julian mentioned it again on IRC, and we each started > > implementing support in dpkg and apt respectively, to allow easier > > evaluation. I stopped when I realized the code was getting too messy, > > but after few weeks Julian and Bálint Réczey ended up implementing > > the support for apt and dpkg [D], so that they could merge it in > > Ubuntu for their upcoming release, which they did. > > > > Their main driving reason (AFAICT) has been (de)compression speed. > > With default level: > > comp decomp size > xz 8.038s 0.356s 4320292 > bz2 2.265s 0.730s 5234516 > zst 0.274s 0.102s 5657626 > gz 0.880s 0.152s 6515505 > Z 0.499s 0.133s 8932459 > lzo 0.100s 0.095s 9198874 > > _Compression_ speed is insane; decompression is merely nice. Obviously, > tuneable level turns this single point into an envelope, but even xz at its > lowest levels can be multiple times faster than gzip while delivering better > compression.
We only care about zstd at level 19 for dpkg and apt. > Don't. For .debs, that is. > > A .deb that is getting processed goes through dpkg, whose status files > writes and all those fsyncs are so slow that it's pointless to optimize > every last bit of decompression. If someone already goes as far as > recompressing packages, we got two fine choices: "xz -0" and "none"; the > former being faster than gzip while compressing better, and the latter being > as fast as cat. You don't get any further than cat on the size-vs-speed > slider... Our major use case is cloud initial setup, image building, CI, buildds, all of which do not require any syncs, and can safely use eatmydata, for example; hence the enormous speed up. There's still a speed up with syncs, but it's less than 10% IIRC. This will get faster over time as SSDs increase their IOPS. > > On the other hand, adding .tar.zst [2] would be a much welcome addition for > files referenced from .dsc. A machine that installs dpkg-dev has enough > storage that a half-MB library is beneath any notice, and zstd is a fine > replacement for gzip and the likes. this was backed out at guillem's request. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en