Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

An entire *.rpm file needs to be read only because rpm-metadata chose the file digest to be included in metadata. There is no way for a plaintext file to contain its own digest, true for zchunk as well as current rpm format. ___ devel mailing list -- d

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-12-03 at 22:16 +0100, Jan Pokorný wrote: > On 16/11/18 22:07 +, Jonathan Dieter wrote: > > The core idea behind zchunk is that a file is split into independently > > compressed chunks and the checksum of each compressed chunk is stored > > in the zchunk header. When downloading a

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 16/11/18 22:07 +, Jonathan Dieter wrote: > The core idea behind zchunk is that a file is split into independently > compressed chunks and the checksum of each compressed chunk is stored > in the zchunk header. When downloading a new version of the file, you > download the zchunk header firs

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Um, rpm headers are not compressed, so there is no decompression overhead retrieving a tag value, and there is no reason why an entire rpm needs to be decompressed in order to retrieve a value. Presumably you are obliquely referring to deltarpm creation, not rpm tag value access. After watchin

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

> On Nov 23, 2018, at 6:16 AM, Florian Weimer wrote: > > * Jonathan Dieter: > >>> On Tue, 2018-11-20 at 12:45 +, Michael Schroeder wrote: On Mon, Nov 19, 2018 at 08:30:14PM +, Jonathan Dieter wrote: Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't require e

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 11/23/18 3:16 PM, Florian Weimer wrote: * Jonathan Dieter: On Tue, 2018-11-20 at 12:45 +, Michael Schroeder wrote: On Mon, Nov 19, 2018 at 08:30:14PM +, Jonathan Dieter wrote: Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't require extra CPU usage on the client s

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

* Jonathan Dieter: > On Tue, 2018-11-20 at 12:45 +, Michael Schroeder wrote: >> On Mon, Nov 19, 2018 at 08:30:14PM +, Jonathan Dieter wrote: >> > Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't >> > require extra CPU usage on the client side as they don't go through the

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Thu, 2018-11-22 at 11:31 -0500, Josh Boyer wrote: > I'm concerned that this will effectively render EL RPM unable to > handle any Fedora RPMs at all. That's both a practical concern, as > many people develop Fedora using EL and vice versa, and also a broader > ecosystem concern. I would very m

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Wed, Nov 21, 2018 at 3:37 PM Jonathan Dieter wrote: > > On Wed, 2018-11-21 at 11:31 -0500, Colin Walters wrote: > > After having introduced a new format (OSTree) into the ecosystem here, > > as well as working a lot on the Docker/OCI ecosystem, one thing I want > > to emphasize is: > > > > A lo

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 2018-11-16, Jonathan Dieter wrote: For reference, this is in reply to Paul [Frield]'s email about lifecycle objectives, specifically focusing on problem statement #1[1]. Have rpm use zchunk as its compression format, removing the need for deltarpms, and thus reducing compose time. This will

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 11/22/2018 05:55 AM, Kevin Kofler wrote: > > The thing is, with a fast network and a slow CPU, deltarpms actually slow > you down. No wonder users in such a situation hate them. > Actually, with a fast network and even a FAST CPU, deltarpms actually slow you down. There is no CPU where delta

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Matthew Miller wrote: > Leigh, it may be "lame" to you, but it's important to many people with > bandwidth limitations or slower connections. There's always room for > improvement but let's please not talk about other people's work in this > way. Thank you. The thing is, with a fast network and a

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Wed, 2018-11-21 at 14:36 +0100, Kamil Paral wrote: > On Fri, Nov 16, 2018 at 11:13 PM Jonathan Dieter wrote: > > For reference, this is in reply to Paul's email about lifecycle > > objectives, specifically focusing on problem statement #1[1]. > > > > > > Have rpm use zchunk as its compression

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Wed, 2018-11-21 at 11:31 -0500, Colin Walters wrote: > After having introduced a new format (OSTree) into the ecosystem here, > as well as working a lot on the Docker/OCI ecosystem, one thing I want > to emphasize is: > > A lot of Red Hat's customers don't connect their systems to the Internet,

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

+1 to the reducing compose time, although there are many other things that we have to do to make it even faster. There are many good points that people are bringing up here, but I hope it wont have any bottlenecks as that of xz compression. Also since its a very young technology we should spend s

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sat, Nov 17, 2018, at 12:24 PM, Jonathan Dieter wrote: > Agreed, that this would be a massive format change and should therefore > be a major version bump for RPM. New versions of RPM should still be > able to read and install old-format rpms, but, as you point out, old > versions of RPM won

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Dne 21. 11. 18 v 14:36 Kamil Paral napsal(a): On Fri, Nov 16, 2018 at 11:13 PM Jonathan Dieter > wrote: For reference, this is in reply to Paul's email about lifecycle objectives, specifically focusing on problem statement #1[1]. Have rpm use zchunk as

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Fri, Nov 16, 2018 at 11:13 PM Jonathan Dieter wrote: > For reference, this is in reply to Paul's email about lifecycle > objectives, specifically focusing on problem statement #1[1]. > > > Have rpm use zchunk as its compression format, removing the need for > deltarpms, and thus reducing comp

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Tue, 2018-11-20 at 12:45 +, Michael Schroeder wrote: > On Mon, Nov 19, 2018 at 08:30:14PM +, Jonathan Dieter wrote: > > Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't > > require extra CPU usage on the client side as they don't go through the > > decompress-recompress

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Hey, thanks for the detailed analysis. Comments inline. On Tue, 2018-11-20 at 00:47 +0100, s...@gmx.com wrote: > Based on work I've done in this area, I'm somewhat sceptical that this > will work out well. I spent a decent amount of time comparing various > approaches to data saving for both com

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, Nov 19, 2018 at 08:30:14PM +, Jonathan Dieter wrote: > Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't > require extra CPU usage on the client side as they don't go through the > decompress-recompress cycle that deltarpms do. Re-assembling a zchunk > file requires no

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

> Sent: Friday, November 16, 2018 at 10:07 PM > From: "Jonathan Dieter" > To: "Development discussions related to Fedora" > > Subject: Proposal: Faster composes by eliminating deltarpms and using > zchunked rpms instead > > For reference, this

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 16:29 -0500, Simo Sorce wrote: > On Mon, 2018-11-19 at 21:02 +, Jonathan Dieter wrote: > > On Mon, 2018-11-19 at 15:18 -0500, Simo Sorce wrote: > > > I do not think you can just trust random metadata somewhere, one of the > > > points of a rpm reinstall is to fix damaged

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 22:18 +0100, Nicolas Mailhot wrote: > Le lundi 19 novembre 2018 à 20:30 +, Jonathan Dieter a écrit : > > > > The zchunk advantage over deltarpms is much less CPU usage, while its > > disadvantages are slightly larger network usage and increased disk > > usage. > > Unfort

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 22:18 +0100, Nicolas Mailhot wrote: > Le lundi 19 novembre 2018 à 20:30 +, Jonathan Dieter a écrit : > > The zchunk advantage over deltarpms is much less CPU usage, while its > > disadvantages are slightly larger network usage and increased disk > > usage. > > Unfortunate

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 21:14 +, Tom Hughes wrote: > On 19/11/2018 20:30, Jonathan Dieter wrote: > > > On the client: > > The zchunk advantage over regular rpm is decreased network usage, while > > its disadvantage is increased disk usage (since the local chunks will > > be uncompressed). > > >

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 21:02 +, Jonathan Dieter wrote: > On Mon, 2018-11-19 at 15:18 -0500, Simo Sorce wrote: > > On Mon, 2018-11-19 at 19:58 +, Jonathan Dieter wrote: > > > > > That's an interesting thought. I was picturing using the zchunk > > > library in the dnf download stage to buil

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Le lundi 19 novembre 2018 à 20:30 +, Jonathan Dieter a écrit : > > The zchunk advantage over deltarpms is much less CPU usage, while its > disadvantages are slightly larger network usage and increased disk > usage. Unfortunately, that's a bad compromise for most limited clients. A limited cli

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 19/11/2018 20:30, Jonathan Dieter wrote: On the client: The zchunk advantage over regular rpm is decreased network usage, while its disadvantage is increased disk usage (since the local chunks will be uncompressed). The zchunk advantage over deltarpms is much less CPU usage, while its disadv

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 15:18 -0500, Simo Sorce wrote: > On Mon, 2018-11-19 at 19:58 +, Jonathan Dieter wrote: > > That's an interesting thought. I was picturing using the zchunk > > library in the dnf download stage to build a local rpm from the > > verified locally installed files and the dow

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sun, 2018-11-18 at 13:19 -0500, Stephen John Smoogen wrote: > On Sun, 18 Nov 2018 at 12:49, Neal Gompa wrote: > > On Sun, Nov 18, 2018 at 11:54 AM Jonathan Dieter > > wrote: > > > On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: > > > > Jonathan Dieter wrote: > > > > > My proposal would

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 19:58 +, Jonathan Dieter wrote: > On Mon, 2018-11-19 at 20:16 +0100, Jan Pokorný wrote: > > On 19/11/18 13:13 +0100, Nicolas Mailhot wrote: > > > Le 2018-11-19 12:28, Martin Kolman a écrit : > > > > > > > Many people might think RAM would not be an issue in 2018, but in >

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 20:16 +0100, Jan Pokorný wrote: > On 19/11/18 13:13 +0100, Nicolas Mailhot wrote: > > Le 2018-11-19 12:28, Martin Kolman a écrit : > > > > > Many people might think RAM would not be an issue in 2018, but in > > > practice there are > > > and likely always will be memory const

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sun, 2018-11-18 at 07:02 +, Leigh Scott wrote: > +1 to anything to rid me of deltarpms, I currently have to disable > this lame default. The irony is that getting deltarpms into Fedora was largely my fault. ;) Sorry, Leigh. Jonathan ___ devel m

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 14:57 +0100, Miroslav Suchý wrote: > Dne 16. 11. 18 v 23:07 Jonathan Dieter napsal(a): > >1. Downloading a new release of a zchunked rpm would be larger than > > downloading the equivalent deltarpm. This is offset by the fact > > that the client is able to wor

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 19/11/18 13:13 +0100, Nicolas Mailhot wrote: > Le 2018-11-19 12:28, Martin Kolman a écrit : > >> Many people might think RAM would not be an issue in 2018, but in >> practice there are >> and likely always will be memory constrained installation targets, >> such as massive deployments >> of "sm

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 16/11/2018 22:07, Jonathan Dieter wrote: The uncompressed local chunks would be combined with the downloaded compressed chunks to create a local rpm that will pass signature verification without needing to recompress the uncompressed local chunks, making this computationally much faster than

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Dne 16. 11. 18 v 23:07 Jonathan Dieter napsal(a): >1. Downloading a new release of a zchunked rpm would be larger than > downloading the equivalent deltarpm. This is offset by the fact > that the client is able to work out which chunks it needs no matter > what the original r

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sun, Nov 18, 2018 at 07:02:24AM -, Leigh Scott wrote: > +1 to anything to rid me of deltarpms, I currently have to disable this lame > default. Leigh, it may be "lame" to you, but it's important to many people with bandwidth limitations or slower connections. There's always room for improv

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Le 2018-11-19 12:28, Martin Kolman a écrit : Many people might think RAM would not be an issue in 2018, but in practice there are and likely always will be memory constrained installation targets, such as massive deployments of "small" VMs or the IoT use cases mentioned above. Sure, that’s the

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Mon, 2018-11-19 at 11:41 +0100, Sheogorath wrote: > On 11/18/18 7:19 PM, Stephen John Smoogen wrote: > > On Sun, 18 Nov 2018 at 12:49, Neal Gompa wrote: > > > On Sun, Nov 18, 2018 at 11:54 AM Jonathan Dieter > > > wrote: > > > > On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: > > > > >

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 11/18/18 7:19 PM, Stephen John Smoogen wrote: > On Sun, 18 Nov 2018 at 12:49, Neal Gompa wrote: >> >> On Sun, Nov 18, 2018 at 11:54 AM Jonathan Dieter wrote: >>> >>> On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: Jonathan Dieter wrote: > My proposal would be to make zchunk the

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Le 2018-11-19 10:31, Gerd Hoffmann a écrit : caches: doesn't look so good. squid can't store chunks of a file. For range request caching you probably want apache traffic server (ATS), not squid, as cache layer. The video guys implemented lots if range request magic in the cache tech they

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sat, Nov 17, 2018 at 02:43:23PM -0600, Chris Adams wrote: > Once upon a time, Jonathan Dieter said: > > The benefit of zchunked rpms is that, when downloading an updated rpm, > > you would only need to download the chunks that have changed from > > what's on your system. > > How well do web se

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sun, 18 Nov 2018 at 12:49, Neal Gompa wrote: > > On Sun, Nov 18, 2018 at 11:54 AM Jonathan Dieter wrote: > > > > On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: > > > Jonathan Dieter wrote: > > > > My proposal would be to make zchunk the rpm compression format for > > > > Fedora. > > >

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sun, Nov 18, 2018 at 11:54 AM Jonathan Dieter wrote: > > On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: > > Jonathan Dieter wrote: > > > My proposal would be to make zchunk the rpm compression format for > > > Fedora. > > > > Given that: > > 1. zchunk is based on zstd, which is typicall

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: > Jonathan Dieter wrote: > > My proposal would be to make zchunk the rpm compression format for > > Fedora. > > Given that: > 1. zchunk is based on zstd, which is typically less efficient in terms of >compression ratio than xz, depending o

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

+1 to anything to rid me of deltarpms, I currently have to disable this lame default. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.or

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sat, 2018-11-17 at 14:43 -0600, Chris Adams wrote: > Once upon a time, Jonathan Dieter said: > > The benefit of zchunked rpms is that, when downloading an updated rpm, > > you would only need to download the chunks that have changed from > > what's on your system. > > How well do web servers a

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sat, 2018-11-17 at 14:36 -0500, Neal Gompa wrote: > On Sat, Nov 17, 2018 at 1:15 PM Jonathan Dieter wrote: > > Neal, thanks so much for your thoughts on this. Responses inline: > > > > On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote: > > > > > If we're really considering changing the RPM

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Jonathan Dieter wrote: > My proposal would be to make zchunk the rpm compression format for > Fedora. Given that: 1. zchunk is based on zstd, which is typically less efficient in terms of compression ratio than xz, depending on settings (see, e.g., https://github.com/inikep/lzbench), and 2.

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Once upon a time, Jonathan Dieter said: > The benefit of zchunked rpms is that, when downloading an updated rpm, > you would only need to download the chunks that have changed from > what's on your system. How well do web servers and caches handle range requests? I haven't really paid attention

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Sat, Nov 17, 2018 at 1:15 PM Jonathan Dieter wrote: > > Neal, thanks so much for your thoughts on this. Responses inline: > > On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote: > > > If we're really considering changing the RPM file format, then we need > > a proper discussion on rpm-maint@

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

Neal, thanks so much for your thoughts on this. Responses inline: On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote: > If we're really considering changing the RPM file format, then we need > a proper discussion on rpm-maint@ and rpm-ecosystem@ mailing lists on > rpm.org. Can you please start

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Fri, Nov 16, 2018 at 6:03 PM Jonathan Dieter wrote: > > > *Changes* > The zchunk format would need to be extended to allow for a zchunked rpm > to contain both the uncompressed chunks that were already on the local > system and the newly downloaded compressed chunks while still passing > signat

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On 11/16/18 3:00 PM, Adam Williamson wrote: > On Fri, 2018-11-16 at 22:07 +, Jonathan Dieter wrote: >> For reference, this is in reply to Paul's email about lifecycle >> objectives, specifically focusing on problem statement #1[1]. >> >> >> Have rpm use zchunk as its compression format, remo

Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

On Fri, 2018-11-16 at 22:07 +, Jonathan Dieter wrote: > For reference, this is in reply to Paul's email about lifecycle > objectives, specifically focusing on problem statement #1[1]. > > > Have rpm use zchunk as its compression format, removing the need for > deltarpms, and thus reducing com

Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead

For reference, this is in reply to Paul's email about lifecycle objectives, specifically focusing on problem statement #1[1]. Have rpm use zchunk as its compression format, removing the need for deltarpms, and thus reducing compose time. This will require changes to both the rpm format and new f