Re: Delta RPMs in Fedora 34
On Tue, 2021-01-05 at 08:49 -0500, Neal Gompa wrote: > To be blunt, I would have never done Zchunk metadata if it was going > to be used as a tool to kill DeltaRPMs. I firmly believe we need both > to have a comprehensive offering that accommodates the needs of > Fedora users across the world. Hey Neal, I'm not sure where you're going with that first sentence, but I think it's pretty obvious that zchunk and deltarpms solve different problems and, following this thread, I don't think anyone has suggested that we should kill deltarpms *because* we have zchunk metadata. When we first brought deltarpms into Fedora, a savings of 60-90% when doing updates was normal. Now that we're losing the deltarpms after each push (as we have been for the last three years), the savings is significantly lower (I normally see less than 10%) and that makes it hard to be motivated to fix the bugs that inevitably arise. It sounds like there might be a plan to keep deltarpms beyond a single push, and, if that happens, I will be more than happy to keep on dealing with deltarpm bugs. :) Thanks, Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Tue, Jan 05, 2021 at 08:01:04PM +0100, Miroslav Suchý wrote: > Dne 05. 01. 21 v 19:44 Kevin Fenzi napsal(a): > > On the next f33-updates push the entire process runs again. It never > > _updates_ existing repos, it always creates them. > > Ahh. So this all worked when we run the the process once per week. It worked differently back when we used mash to create updates, I think because we also did drpms from the GA/base repo. Which we could do now, but It would only help users on their initial 'dnf update'. > But because we run it every day now, the deltas are > minimal. It just took us several months to notice. It's been this way since we moved to pungi. 3+ years. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
Dne 05. 01. 21 v 19:44 Kevin Fenzi napsal(a): > On the next f33-updates push the entire process runs again. It never > _updates_ existing repos, it always creates them. Ahh. So this all worked when we run the the process once per week. But because we run it every day now, the deltas are minimal. It just took us several months to notice. -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
Dne 05. 01. 21 v 15:31 Zbigniew Jędrzejewski-Szmek napsal(a): > Yet another reason why popcon would be useful. https://github.com/xsuchy/popcon-for-fedora-old Feel free to take it :) -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Tue, Jan 05, 2021 at 09:49:59AM +0100, Miroslav Suchý wrote: > Dne 05. 01. 21 v 0:29 Matthew Miller napsal(a): > > So, the first thing we need to do to fix this is move deltarpm creation out > > of the updates process. > > Right. > > > Kevin Fenzi tells me this would mean we'd need a > > separate delta RPMs repo, > > Why? You can do that in the same repo. You just need once per X days/hours run > createrepo_c --deltas --num-deltas X On Tue, Jan 05, 2021 at 06:30:37PM +0100, Daniel Mach wrote: > > To be honest, I don't understand why drpms are related to Pungi at all. > > Deltas are optional, if they're not available, a normal RPM is used. > They can be processed asynchronously (as mentioned earlier in this thread) > and injected in repos once they're ready. > > Please note that we're talking about 74 drpms in F33.x86_64 updates repo: > http://ftp.fi.muni.cz/pub/linux/fedora/linux/updates/33/Everything/x86_64/drpms/ > > Sometimes I'm wondering if it's worth it and if Fedora shouldn't move away > from drpms. so, ok then. I guess people are still confused about this. Here's my attempt to explain in detail how it currently works: When bodhi does an updates push (say for f33-updates), it does a lot of things. It checks the updates that are pending f33 stable updates, it locks them so no one can mess with them in the ui until it's done, it makes sure they are signed, it tells koji to move the tags the packages are tagged into into f33-updates, then it calls pungi to actually do the heavy lifting. This pungi process then talks to koji and says "hey, give me the latest tagged packages for the 'f33-updates' tag signed with key xyz. It puts them in directories for arch and type and such and runs createrepo_c to make the repodata. This is the point where it makes drpms. In order to make a drpm createpo_c needs to know what you want to make drpms for. It also has to have both the OLD and NEW versions available to make the drpm. createrepo_c also makes the normal repodata In our above case pungi has the current repos it's making and the f33-updates repo. Thus all the drpms it can make are ones where a new package version is being added to the repos and there is an older version available in f33-updates. It doesn't have access to all those versions before or after the ones it has. It only has those two. Once pungi is done, bodhi then does more things (emailing people, updating notes, etc) and... importantly, updates the repodata with the security information (so you can know what are security updates, etc). Then, that entire tree is synced to the master mirrors. On the next f33-updates push the entire process runs again. It never _updates_ existing repos, it always creates them. This means that if you have foo-1.0-1 in f33-updates, foo-1.1-1 comes along, it will make a drpm between them, that will exist _only_ on the day it added foo-1.1-1. The next day it will be gone. This is why there are so few drpms. It's only generating them for the things that it could at the time of the last push. So if you happen to update during a day where there were things updated that you have installed you would see the drpms. If you happen to update the next day, you would not. So, my last thought was to teach pungi about all the old updates trees (which are in the same directory as it makes the new one) and have it gather all the old drpms from those and expire them at some configurable time. This would not use more cycles to make them, and would make the chances of a user being able to use them much higher. But I am not sure this is possible/if pungi maintainers are willing to implement this. It would mean that createrepo_c would need to know about those old drpms to add them to metadata. Failing that we could move the drpm creation to another process/repo, but... drpms have to be in the same repodata as the repo they are for right? Or can they be in another one? Hope that clarifies more than it confuses... kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
Dne 05. 01. 21 v 0:50 Kevin Fenzi napsal(a): On Mon, Jan 04, 2021 at 06:29:13PM -0500, Matthew Miller wrote: On Mon, Jan 04, 2021 at 10:21:15PM +, Matthew Almond via devel wrote: There's been a lot of interesting talk about the state and future of drpm. I'd like to propose we continue the conversation about that with a different subject line :) Okay, fair. I have a proposal. Right now, the problem is that making delta rpms is expensive, and therefore we aren't making very many, which makes them even less useful. Plus, we're only making them between updates and for packages where those updates are frequent, that means you need to keep on top of things, which may be best practice but is most difficult for low-bandwidth users who might most benefit in the first place. So, the first thing we need to do to fix this is move deltarpm creation out of the updates process. Kevin Fenzi tells me this would mean we'd need a separate delta RPMs repo, which doesn't sound like a bad thing to me, but we're not sure offhand if DNF can handle that without modification. Yeah, I don't recall how dnf looks for drpms. Right now they are in the same repo, using the same repodata. If we moved them to a new repo would they get found correctly? This would let us make the delta RPMs asynchronously and not block updates. And, it would also give us the ability to roughly see how important they are to users, because we could see how popular that repository is compared to the updates repo. I also remember when this was a killer feature for Fedora, and without any real way of judging use and demand, I'm hesitant to kill it off. But that's definitely plan B. We can point people who are in low-bandwidth situations at Silverblue, CoreOS, and Kinoite as the preferred approach. Yeah, I came up with one more possible way we could get more drpms with our current setup, but need to talk to pungi maintainers and see if it's doable. :) After that, it's either split things out or drop drpms I think. To be honest, I don't understand why drpms are related to Pungi at all. Deltas are optional, if they're not available, a normal RPM is used. They can be processed asynchronously (as mentioned earlier in this thread) and injected in repos once they're ready. Please note that we're talking about 74 drpms in F33.x86_64 updates repo: http://ftp.fi.muni.cz/pub/linux/fedora/linux/updates/33/Everything/x86_64/drpms/ Sometimes I'm wondering if it's worth it and if Fedora shouldn't move away from drpms. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Tue, Jan 5, 2021 at 3:46 PM Florian Weimer wrote: > The metadata would also be much larger, and so would be the battery > usage to recompress the payload. 8-( And while the bandwidth reduction has value, cpu and wallclock time to rebuild the rpm is substantially increased for low end devices such as ARM SoCs with slow (compared to recent gen x86) cpus, and slow (compared to recent nvme) storage devices such as sd cards compared to just downloading the entire rpm. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
* Matthew Miller: > On Tue, Jan 05, 2021 at 11:30:10AM +0100, Florian Weimer wrote: >> > I also remember when this was a killer feature for Fedora, and without any >> > real way of judging use and demand, I'm hesitant to kill it off. >> >> Is it really saving bandwidth, though? The reported savings are >> generally very small for me. Downloading the metadata costs something >> as well. > > If we made a lot more of them, they could save significant bandwidth. The metadata would also be much larger, and so would be the battery usage to recompress the payload. 8-( Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Tue, Jan 05, 2021 at 11:30:10AM +0100, Florian Weimer wrote: > > I also remember when this was a killer feature for Fedora, and without any > > real way of judging use and demand, I'm hesitant to kill it off. > > Is it really saving bandwidth, though? The reported savings are > generally very small for me. Downloading the metadata costs something > as well. If we made a lot more of them, they could save significant bandwidth. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Tue, 5 Jan 2021 at 03:50, Miroslav Suchý wrote:
> Dne 05. 01. 21 v 0:29 Matthew Miller napsal(a):
> > So, the first thing we need to do to fix this is move deltarpm creation
> out
> > of the updates process.
>
> Right.
>
> > Kevin Fenzi tells me this would mean we'd need a
> > separate delta RPMs repo,
>
> Why? You can do that in the same repo. You just need once per X days/hours
> run
> createrepo_c --deltas --num-deltas X
>
>
Get pungi and all the other tools in the build system which touch the repos
and expect things to be done in a certain way to not break, corrupt or make
releng's life a daily nightmare and you are golden.
Remember the Fedora build system is a Rube Goldberg machine[1] where every
group who has a new idea about how to make Linux easier to
compose/consume/etc have stuck something in. None of them have been
designed really to work with each other and various parts are completely
running on luck and mad patching (hi PDC). Everytime someone says 'just do
this one thing' it turns into a cascade of broken bits where everyone
spends a month or 2 blaming (1) releng for adding in one more thing and (2)
every other group for messing with their perfect tool. Since we usually
have 1-2 months to get something in place before the next release.. that
means we have whatever time left over from the poop-throwing festival to
monkey-patch it for another release and then live with the system breaking
daily for a couple of months. [Then watch Kevin and Mohan grow more
Stockholm syndrome to say that the system is fine.. just like the previous
release engineers who have departed from Fedora.]
Patches, ideas and fixes are indeed helpful, but what would be more helpful
is getting everyone with a say on the build system and their one thing that
they want from Release Engineering in a room to work out what the entire
developer and build system experience should be with an idea of how to make
it more manageable and more able to slot in things in and out versus
'ah {'mass rebuild','beta release','final release','2 week holiday'}
is in 2 days get that tool working'
[1] https://en.wikipedia.org/wiki/Rube_Goldberg_machine
--
> Miroslav Suchy, RHCA
> Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
--
Stephen J Smoogen.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Mon, Jan 04, 2021 at 06:29:13PM -0500, Matthew Miller wrote: > I also remember when this was a killer feature for Fedora, and without any > real way of judging use and demand, I'm hesitant to kill it off. But that's > definitely plan B. We can point people who are in low-bandwidth situations > at Silverblue, CoreOS, and Kinoite as the preferred approach. It is also very difficult to measure - for my part I'm happy for every MB saved when downloading updates at home, because it directly translates into waiting time. At work I don't have a bandwidth problem, so it's way less of an issue there. But I don't see anywhere how much the potential is - iirc correctly it sometimes saves hundreds of MBs, which translates into something like 10-20 Minutes saved time. Even though it's an after-the-fact information I'm still happy it doing its job. (Interestingly on the last update almost half of the md5 sums mismatched for the drpms, which increased download size from 12.3MB to 14.0MB - but this seems to be a rare problem) Personally I'd like to stay on regular Fedora Workstation / Fedora Server - I'd probably decide to take increased download times instead of switching distros/editions if drpm gets removed. All the best, Astra signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
Hi, > we aren't making very many, which makes them even less useful. Plus, we're > only making them between updates and for packages where those updates are > frequent, that means you need to keep on top of things, which may be best > practice but is most difficult for low-bandwidth users who might most > benefit in the first place. I'm a low bandwidth user. And my setup basically is: (1) route all updates though a squid caching proxy. (2) configure all fedora machines to use the same fixed mirror. (3) disable drpms. (4) disable zchunk. There are always cases where you need the full rpm anyway (for example fresh installs with update repo enabled), so just loading (+caching!) the full rpms and don't bother with drpms works better overall. The problem with zchunk is that it isn't cache-friendly. squid can't cache range requests. And even in case of a full download (fresh install) I've seen zchunk metadata being re-downloaded when requested again ... take care, Gerd ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Tue, Jan 05, 2021 at 08:49:20AM -0500, Neal Gompa wrote: > On Tue, Jan 5, 2021 at 8:34 AM Colin Walters wrote: > > Now speaking of deltas - really delta implementations are going to benefit > > from a stronger "cadence" to releases, exactly much like what we do for > > CoreOS (but not Silverblue/IoT) today. The relationship of such a system > > and Bodhi is...messy. > > ostree deltas are also *much* better than deltarpm in various ways (most > > notably the CPU intensive part is bsdiff which we only use selectively > > instead of the whole thing). > > > > Cadence only matters if the infrastructure around delta fetching > requires it to care. In the case of RPM-OSTree and Flatpak, this is > not a problem as long as you're using native OSTree remotes. If you're > using OCI image remotes instead, then you *do* have to care about > cadence because you have to maintain images and generate deltas based > on possible options. The latter option is how we deliver Flatpaks, and > so we have the same problem we have with DeltaRPMs. > > > On the other hand, we really want deltas too for containers; that's > > https://github.com/containers/image/pull/902 > > > > A very tricky case is the intersection of all of these; for my "dev > > container"/toolbox on my Silverblue workstation I use a custom container > > built on a server with all of my tools, but I do often `yum update` inside > > it since that works incrementally and online. (But I do periodically flush > > and re-pull) If we implemented container deltas I'd be a lot more likely > > to use `podman` to update it instead. > > Addressing the underlying issue here: container deltas and OSTree > deltas are considerably worse for constrained bandwidth than RPM > deltas. Outside of the USA (in the general case) and within the USA > (in several parts of the country), it is extremely common to have > extremely limited bandwidth availability and even more common to have > low throughput. As that will basically never change, we have to work > with that framework. > > Regular Fedora variants offer users the ability to pick and choose > updates based on their situation. If DeltaRPMs were implemented in our > infrastructure correctly, those users would be very well-served by > being able to publish a sliding window of the last 30 days of delta > RPM content. What's particularly galling here is that we have all the > necessary inputs to do it, since Koji keeps everything and Bodhi knows > everything that's ever been pushed. It's pretty much a Pungi > restriction that we've never been able to do this properly. Another thought: we could use popcon-like information to generate delta rpms only for N% most popular packages (10%?). This would significantly cut down on the cost of generation, without really affecting average user savings. Yet another reason why popcon would be useful. > To be blunt, I would have never done Zchunk metadata if it was going > to be used as a tool to kill DeltaRPMs. I firmly believe we need both > to have a comprehensive offering that accommodates the needs of Fedora > users across the world. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Tue, Jan 5, 2021 at 8:34 AM Colin Walters wrote: > > > > On Mon, Jan 4, 2021, at 6:29 PM, Matthew Miller wrote: > > > > I also remember when this was a killer feature for Fedora, and without any > > real way of judging use and demand, I'm hesitant to kill it off. But that's > > definitely plan B. We can point people who are in low-bandwidth situations > > at Silverblue, CoreOS, and Kinoite as the preferred approach. > > Please don't use phrasing like this that implies e.g. CoreOS is distinct from > "Fedora". > But as of right now, it *is*. Perhaps that will change if FCOS realigns with Fedora as part of being promoted to Edition status, but right now, it's different enough content-wise that it's less like Fedora than most of us would want it to be. > A much technically clearer way to say this would be "traditional dnf Fedora" > versus rpm-ostree. > > But even then it's not fully distinct because rpm-ostree also links to libdnf > and whenever you use package layering, it's all the same RPM tools on the > wire. Though...ah right, the deltarpm implementation lives in the dnf Python > code, not the libdnf library, so rpm-ostree doesn't do that. > > Second - and this should be emphasized - a common case at least on Silverblue > is that you run dnf inside a toolbox-style container (or more than one!). So > all bandwidth improvements apply there too. In other words this (implict) > contrast between the two is false because in both cases there are hybrids. > > Now speaking of deltas - really delta implementations are going to benefit > from a stronger "cadence" to releases, exactly much like what we do for > CoreOS (but not Silverblue/IoT) today. The relationship of such a system and > Bodhi is...messy. > ostree deltas are also *much* better than deltarpm in various ways (most > notably the CPU intensive part is bsdiff which we only use selectively > instead of the whole thing). > Cadence only matters if the infrastructure around delta fetching requires it to care. In the case of RPM-OSTree and Flatpak, this is not a problem as long as you're using native OSTree remotes. If you're using OCI image remotes instead, then you *do* have to care about cadence because you have to maintain images and generate deltas based on possible options. The latter option is how we deliver Flatpaks, and so we have the same problem we have with DeltaRPMs. > On the other hand, we really want deltas too for containers; that's > https://github.com/containers/image/pull/902 > > A very tricky case is the intersection of all of these; for my "dev > container"/toolbox on my Silverblue workstation I use a custom container > built on a server with all of my tools, but I do often `yum update` inside it > since that works incrementally and online. (But I do periodically flush and > re-pull) If we implemented container deltas I'd be a lot more likely to use > `podman` to update it instead. Addressing the underlying issue here: container deltas and OSTree deltas are considerably worse for constrained bandwidth than RPM deltas. Outside of the USA (in the general case) and within the USA (in several parts of the country), it is extremely common to have extremely limited bandwidth availability and even more common to have low throughput. As that will basically never change, we have to work with that framework. Regular Fedora variants offer users the ability to pick and choose updates based on their situation. If DeltaRPMs were implemented in our infrastructure correctly, those users would be very well-served by being able to publish a sliding window of the last 30 days of delta RPM content. What's particularly galling here is that we have all the necessary inputs to do it, since Koji keeps everything and Bodhi knows everything that's ever been pushed. It's pretty much a Pungi restriction that we've never been able to do this properly. To be blunt, I would have never done Zchunk metadata if it was going to be used as a tool to kill DeltaRPMs. I firmly believe we need both to have a comprehensive offering that accommodates the needs of Fedora users across the world. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Mon, Jan 4, 2021, at 6:29 PM, Matthew Miller wrote: > > I also remember when this was a killer feature for Fedora, and without any > real way of judging use and demand, I'm hesitant to kill it off. But that's > definitely plan B. We can point people who are in low-bandwidth situations > at Silverblue, CoreOS, and Kinoite as the preferred approach. Please don't use phrasing like this that implies e.g. CoreOS is distinct from "Fedora". A much technically clearer way to say this would be "traditional dnf Fedora" versus rpm-ostree. But even then it's not fully distinct because rpm-ostree also links to libdnf and whenever you use package layering, it's all the same RPM tools on the wire. Though...ah right, the deltarpm implementation lives in the dnf Python code, not the libdnf library, so rpm-ostree doesn't do that. Second - and this should be emphasized - a common case at least on Silverblue is that you run dnf inside a toolbox-style container (or more than one!). So all bandwidth improvements apply there too. In other words this (implict) contrast between the two is false because in both cases there are hybrids. Now speaking of deltas - really delta implementations are going to benefit from a stronger "cadence" to releases, exactly much like what we do for CoreOS (but not Silverblue/IoT) today. The relationship of such a system and Bodhi is...messy. ostree deltas are also *much* better than deltarpm in various ways (most notably the CPU intensive part is bsdiff which we only use selectively instead of the whole thing). On the other hand, we really want deltas too for containers; that's https://github.com/containers/image/pull/902 A very tricky case is the intersection of all of these; for my "dev container"/toolbox on my Silverblue workstation I use a custom container built on a server with all of my tools, but I do often `yum update` inside it since that works incrementally and online. (But I do periodically flush and re-pull) If we implemented container deltas I'd be a lot more likely to use `podman` to update it instead. But anyways, please either explicitly spell out "Fedora CoreOS" to avoid an implicit contrast and making it seem like a separate thing from "Fedora", or go more technical and say "rpm-ostree variant" or so. Thanks! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
> Is it really saving bandwidth, though? The reported savings are > generally very small for me. Downloading the metadata costs something > as well. In F33, mostly so. I generally keep up to date (update once a week), but available deltarpms have been lesser compared to earlier versions. I used deltarpm from the day it was introduced in Fedora, and found it quite useful due to limited internet connection (speed/bandwidth/data limit) — for instance TeXLive package updates (which, afait, doesn’t generate deltarpms in f33) etc. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
* Matthew Miller: > I also remember when this was a killer feature for Fedora, and without any > real way of judging use and demand, I'm hesitant to kill it off. Is it really saving bandwidth, though? The reported savings are generally very small for me. Downloading the metadata costs something as well. Thanks, Florian -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
Dne 05. 01. 21 v 0:29 Matthew Miller napsal(a): > So, the first thing we need to do to fix this is move deltarpm creation out > of the updates process. Right. > Kevin Fenzi tells me this would mean we'd need a > separate delta RPMs repo, Why? You can do that in the same repo. You just need once per X days/hours run createrepo_c --deltas --num-deltas X -- Miroslav Suchy, RHCA Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Mon, Jan 04, 2021 at 06:29:13PM -0500, Matthew Miller wrote: > On Mon, Jan 04, 2021 at 10:21:15PM +, Matthew Almond via devel wrote: > > There's been a lot of interesting talk about the state and future of > > drpm. I'd like to propose we continue the conversation about that with > > a different subject line :) > > Okay, fair. I have a proposal. > > Right now, the problem is that making delta rpms is expensive, and therefore > we aren't making very many, which makes them even less useful. Plus, we're > only making them between updates and for packages where those updates are > frequent, that means you need to keep on top of things, which may be best > practice but is most difficult for low-bandwidth users who might most > benefit in the first place. > > So, the first thing we need to do to fix this is move deltarpm creation out > of the updates process. Kevin Fenzi tells me this would mean we'd need a > separate delta RPMs repo, which doesn't sound like a bad thing to me, but > we're not sure offhand if DNF can handle that without modification. Yeah, I don't recall how dnf looks for drpms. Right now they are in the same repo, using the same repodata. If we moved them to a new repo would they get found correctly? > This would let us make the delta RPMs asynchronously and not block updates. > And, it would also give us the ability to roughly see how important they are > to users, because we could see how popular that repository is compared to > the updates repo. > > I also remember when this was a killer feature for Fedora, and without any > real way of judging use and demand, I'm hesitant to kill it off. But that's > definitely plan B. We can point people who are in low-bandwidth situations > at Silverblue, CoreOS, and Kinoite as the preferred approach. Yeah, I came up with one more possible way we could get more drpms with our current setup, but need to talk to pungi maintainers and see if it's doable. :) After that, it's either split things out or drop drpms I think. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Delta RPMs in Fedora 34
On Mon, Jan 04, 2021 at 10:21:15PM +, Matthew Almond via devel wrote: > There's been a lot of interesting talk about the state and future of > drpm. I'd like to propose we continue the conversation about that with > a different subject line :) Okay, fair. I have a proposal. Right now, the problem is that making delta rpms is expensive, and therefore we aren't making very many, which makes them even less useful. Plus, we're only making them between updates and for packages where those updates are frequent, that means you need to keep on top of things, which may be best practice but is most difficult for low-bandwidth users who might most benefit in the first place. So, the first thing we need to do to fix this is move deltarpm creation out of the updates process. Kevin Fenzi tells me this would mean we'd need a separate delta RPMs repo, which doesn't sound like a bad thing to me, but we're not sure offhand if DNF can handle that without modification. This would let us make the delta RPMs asynchronously and not block updates. And, it would also give us the ability to roughly see how important they are to users, because we could see how popular that repository is compared to the updates repo. I also remember when this was a killer feature for Fedora, and without any real way of judging use and demand, I'm hesitant to kill it off. But that's definitely plan B. We can point people who are in low-bandwidth situations at Silverblue, CoreOS, and Kinoite as the preferred approach. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Delta RPMs in Fedora 34
On Mon, 2021-01-04 at 11:25 -0500, Matthew Miller wrote: > On Sun, Jan 03, 2021 at 03:25:29PM -0800, Kevin Fenzi wrote: > > I remember when drpms landed I heard people say they choose Fedora > > because of them. That may have changed over the years I guess. :) > > and there have been only 2 or 3 reports about how few drpms exist > > in the last few years (ie, most people didn't really notice). > > Hmmm, here's an idea: what if instead of nightly drpms, we made them > only > every two weeks, but always exactly two weeks, so that people > updating on a > specific cadence would get them? There's been a lot of interesting talk about the state and future of drpm. I'd like to propose we continue the conversation about that with a different subject line :) - Matthew ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
