Daniel Alley wrote:
>>ry xz -9, it should be better than zstd. It will take longer to compress,
>>but should actually be FASTER (!) to decompress, which is what really
>>matters.
>
> Please provide data - any data - to support this claim, because it flies
> completely in the face of every
Daniel Alley wrote:
>>ry xz -9, it should be better than zstd. It will take longer to compress,
>>but should actually be FASTER (!) to decompress, which is what really
>>matters.
>
> Please provide data - any data - to support this claim, because it flies
> completely in the face of every
>ry xz -9, it should be better than zstd. It will take longer to compress, but
>should actually be FASTER (!) to decompress, which is what really matters.
Please provide data - any data - to support this claim, because it flies
completely in the face of every benchmark the internet has to
Sirius via devel writes:
> echo % of original
> echo Time to decompress the file, output to /dev/null
> time gzip -d -c ${INPUTFILE}.gz > /dev/null
Keep in mind that gzip has its own zlib implementation, while
createrepo_c uses the system-provided zlib.
That means, when creating a
In days of yore (Tue, 26 Mar 2024), fedora-devel thus quoth:
> Also note that adding '-T0' to use all available cores of the CPU will
> greatly speed up the results with zstd.
>
> However, all this talking about the optimal compression level, but in
> the end there's no way to set that to
Il 26/03/24 10:41, Sirius via devel ha scritto:
> In days of yore (Tue, 26 Mar 2024), fedora-devel thus quoth:
>> In days of yore (Thu, 21 Mar 2024), Stephen Smoogen thus quoth:
>>> On Wed, 20 Mar 2024 at 22:01, Kevin Kofler via devel <
>>> devel@lists.fedoraproject.org> wrote:
>>>
Aoife
In days of yore (Tue, 26 Mar 2024), fedora-devel thus quoth:
> In days of yore (Thu, 21 Mar 2024), Stephen Smoogen thus quoth:
> > On Wed, 20 Mar 2024 at 22:01, Kevin Kofler via devel <
> > devel@lists.fedoraproject.org> wrote:
> >
> > > Aoife Moloney wrote:
> > > > The zstd compression type
In days of yore (Thu, 21 Mar 2024), Stephen Smoogen thus quoth:
> On Wed, 20 Mar 2024 at 22:01, Kevin Kofler via devel <
> devel@lists.fedoraproject.org> wrote:
>
> > Aoife Moloney wrote:
> > > The zstd compression type was chosen to match createrepo_c settings.
> > > As an alternative, we might
On Mon, Mar 25, 2024 at 10:59:15PM +0100, Kevin Kofler via devel wrote:
> Zbigniew Jędrzejewski-Szmek wrote:
>
> > On Mon, Mar 25, 2024 at 04:50:28PM -0400, Neal Gompa wrote:
> >> Keep in mind we also want to make the compose process faster too, I
> >> don't know if it's worth it to spend 20x
Zbigniew Jędrzejewski-Szmek wrote:
> On Mon, Mar 25, 2024 at 04:50:28PM -0400, Neal Gompa wrote:
>> Keep in mind we also want to make the compose process faster too, I
>> don't know if it's worth it to spend 20x more time compressing
>> repodata when we keep trying to get back hours and minutes
On Mon, Mar 25, 2024 at 04:50:28PM -0400, Neal Gompa wrote:
> On Mon, Mar 25, 2024 at 4:40 PM Zbigniew Jędrzejewski-Szmek
> wrote:
> >
> > On Mon, Mar 25, 2024 at 07:29:09PM +0100, Kevin Kofler via devel wrote:
> > > Daniel Alley wrote:
> > > > One more point: createrepo_c uses zstd compression
On Mon, Mar 25, 2024 at 4:40 PM Zbigniew Jędrzejewski-Szmek
wrote:
>
> On Mon, Mar 25, 2024 at 07:29:09PM +0100, Kevin Kofler via devel wrote:
> > Daniel Alley wrote:
> > > One more point: createrepo_c uses zstd compression level 10, but the range
> > > goes all the way up to level 22. I would
On Mon, Mar 25, 2024 at 07:29:09PM +0100, Kevin Kofler via devel wrote:
> Daniel Alley wrote:
> > One more point: createrepo_c uses zstd compression level 10, but the range
> > goes all the way up to level 22. I would oppose making the default much
> > computationally heavier than it is
Daniel Alley wrote:
> One more point: createrepo_c uses zstd compression level 10, but the range
> goes all the way up to level 22. I would oppose making the default much
> computationally heavier than it is currently, but if spending 20x longer
> to compress the repo 10% more is desirable to the
One more point: createrepo_c uses zstd compression level 10, but the range goes
all the way up to level 22. I would oppose making the default much
computationally heavier than it is currently, but if spending 20x longer to
compress the repo 10% more is desirable to the fedora project, then
Also, to use that squash benchmark you will probably want to run it yourself
with modern libraries and modern hardware, as the data on their website
(assuming it's the same as the data in their github repo) is 8+ years old.
zstd has improved a fair bit during that timeframe.
--
But we're not compressing text, we're compressing XML.
Anyway, I ran an experiment on a local copy of the Fedora 38 release repo and
the differences (while they do exist) aren't very significant. Less than 10%
createrepo_c --update --skip-stat --recycle-pkglist --general-compress-type gz .
Neal Gompa wrote:
> Regular zstd compression is less optimized due to the lack of
> dictionaries, but it's also effectively the fallback path, though much
> faster to decompress while providing pretty good compression (which is
> why we have been gradually switching *everything* to zstd).
"pretty
Il 21/03/24 03:00, Kevin Kofler via devel ha scritto:
> Since xz consistently compresses better than zstd, I would strongly suggest
> using xz everywhere to minimize download sizes. However:
>
>> especially after zlib-ng has been made the default in Fedora and brought
>> performance improvements.
Stephen Smoogen wrote:
> There are two parts to this which users will see as 'slowness'. Part one
> is downloading the data from a mirror. Part two is uncompressing the data.
> In work I have been a part of, we have found that while xz gave us much
> smaller files, the time to uncompress was so
On Thu, Mar 21, 2024 at 8:20 AM Stephen Smoogen wrote:
>
>
>
> On Wed, 20 Mar 2024 at 22:01, Kevin Kofler via devel
> wrote:
>>
>> Aoife Moloney wrote:
>> > The zstd compression type was chosen to match createrepo_c settings.
>> > As an alternative, we might want to choose xz,
>>
>> Since xz
On Wed, 20 Mar 2024 at 22:01, Kevin Kofler via devel <
devel@lists.fedoraproject.org> wrote:
> Aoife Moloney wrote:
> > The zstd compression type was chosen to match createrepo_c settings.
> > As an alternative, we might want to choose xz,
>
> Since xz consistently compresses better than zstd, I
Aoife Moloney wrote:
> The zstd compression type was chosen to match createrepo_c settings.
> As an alternative, we might want to choose xz,
Since xz consistently compresses better than zstd, I would strongly suggest
using xz everywhere to minimize download sizes. However:
> especially after
23 matches
Mail list logo
