Re: Fedora 40 beta freeze now over
On Tue, 2024-04-02 at 16:55 -0700, Kevin Fenzi wrote: > On Tue, Apr 02, 2024 at 09:28:31PM +0100, Jonathan Dieter wrote: > > * Alternatively, we could update whatever's calling createrepo_c > > to add the `f` prefix to all non-rawhide builds. > > I like this option. ;) > > https://pagure.io/pungi-fedora/pull-request/1269 That looks perfect! :) Jonathan -- ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Fedora 40 beta freeze now over
On Sat, 2024-03-30 at 09:39 -0700, Kevin Fenzi wrote: > On Fri, Mar 29, 2024 at 11:32:10PM +0000, Jonathan Dieter wrote: > > On Wed, 2024-03-27 at 09:12 -0700, Kevin Fenzi wrote: > > > Our next freeze is for Fedora 40 Final, currently scheduled for > > > 2024-04-02, which is NEXT TUESDAY! > > > > Could you please update fedora-repo-zdicts to 2403.1 on the server(s) > > used to generate the metadata? This will reduce the size of the zchunk > > metadata for the fedora repo. > > Yeah, I already updated the rawhide composer the other day... will get > the rest today. > > Thanks for the reminder. Hey Kevin, thanks for looking into this. I've just checked today's compose and it's still not using the dictionaries. Looking at the logs at https://kojipkgs.fedoraproject.org/compose/branched/Fedora-40-20240402.n.0/logs/x86_64/createrepo-Everything.rpm.x86_64.log , it looks like it's not using the expected dictionary path: The dictionaries are in: /usr/share/fedora-repo-zdicts/f40 But createrepo_c is looking in: /usr/share/fedora-repo-zdicts/40 Our options are: * I can push out a new build of fedora-repo-zdicts with paths added that strip out the `f`, but we'll need to get a final freeze exception. * Alternatively, we could update whatever's calling createrepo_c to add the `f` prefix to all non-rawhide builds. * Finally, we can just ignore this and Fedora 40 will have 50% larger zchunk metadata. I'd prefer one of the first two options (whichever is easier), but it's not the end of the world if we go with option 3. I think we're already there with F39. Thanks, Jonathan -- ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Fedora 40 beta freeze now over
On Wed, 2024-03-27 at 09:12 -0700, Kevin Fenzi wrote: > Our next freeze is for Fedora 40 Final, currently scheduled for > 2024-04-02, which is NEXT TUESDAY! Could you please update fedora-repo-zdicts to 2403.1 on the server(s) used to generate the metadata? This will reduce the size of the zchunk metadata for the fedora repo. Thanks, Jonathan -- ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Adding Passim as a Fedora 40 feature?
On Wed, 2023-09-06 at 22:33 +0100, Jonathan Dieter wrote: > On Fri, 2023-08-25 at 12:42 +0100, Richard Hughes wrote: > > The tl;dr: is I want to add a mDNS server that reshares the public > > firmware update metadata from the LVFS on your LAN. The idea is that > > rather than 25 users in an office downloading the same ~2MB file from > > the CDN every day, the first downloads from the CDN and the other 24 > > download from the first machine. All machines still download the > > [tiny] jcat file from the CDN still so we know the SHA256 to search > > for and verify. > > I realize I'm late to the conversation, but what about compressing the > metadata using zchunk, like we do for the DNF metadata? Assuming we > keep a cache of the file locally and that changes (as a percentage of > the whole file) are minimal, this allows you to download only the > differences. The only requirement is that the CDN supports HTTP range > requests. > And, of course, after posting, I realize that I'd missed a chunk of the thread where you explained that you're not a fan of deltas. FWIW, zchunk doesn't do static deltas, so the only file you need to worry about on the server/CDN is the latest one. If this is something you'd be interested in, I'd be happy to help get it working. If not, I'm happy to fade back into the background. :) Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Adding Passim as a Fedora 40 feature?
On Fri, 2023-08-25 at 12:42 +0100, Richard Hughes wrote: > The tl;dr: is I want to add a mDNS server that reshares the public > firmware update metadata from the LVFS on your LAN. The idea is that > rather than 25 users in an office downloading the same ~2MB file from > the CDN every day, the first downloads from the CDN and the other 24 > download from the first machine. All machines still download the > [tiny] jcat file from the CDN still so we know the SHA256 to search > for and verify. I realize I'm late to the conversation, but what about compressing the metadata using zchunk, like we do for the DNF metadata? Assuming we keep a cache of the file locally and that changes (as a percentage of the whole file) are minimal, this allows you to download only the differences. The only requirement is that the CDN supports HTTP range requests. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Orphaning lizardfs
That's wonderful news! I've moved roles since adding it to Fedora and no longer use it in my day job, so I'm more than happy to let you have it! Thanks, Jonathan On Sat, 2023-01-28 at 21:25 -0500, JT wrote: > I can pick this package up if you're stepping back. The lizards devs > are planning on a new release sometime this year, but there's still a > few things they're trying to finish up first before releasing v3.13. > Last I heard they wanted it to be out before summer. > JT > > On January 28, 2023 10:14:14 AM Jonathan Dieter > wrote: > > > I've just orphaned lizardfs. Lizardfs is a clustered network > > filesystem that has very efficient small file / metadata > > performance, > > but hasn't seen any upstream point releases since the end of 2017 > > and > > now FTBFS in the latest mass rebuild. > > > > Jonathan > > ___ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: https://docs.fedoraproject.org/en- > > US/project/code-of-conduct/ > > List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedorapro > > ject.org > > Do not reply to spam, report it: https://pagure.io/fedora- > > infrastructure/new_issue > > > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Orphaning lizardfs
I've just orphaned lizardfs. Lizardfs is a clustered network filesystem that has very efficient small file / metadata performance, but hasn't seen any upstream point releases since the end of 2017 and now FTBFS in the latest mass rebuild. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Thu, 2022-12-01 at 00:41 +, Daniel Alley wrote: > > * zchunk and deltarpm both reimplement / "bundle" multiple different > hashing algorithms zchunk does have bundled versions of various hashing algorithms, but, if it's compiled against OpenSSL (as it is in Fedora), it uses the OpenSSL hashing algorithms rather than the bundled versions. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Intent to retire: novacom-client, novacom-server
On Tue, 2022-08-16 at 22:28 -0500, Maxwell G via devel wrote: > On Tuesday, August 16, 2022 Jonathan Dieter wrote: > > So, unless I hear from someone who wants it within the next week and > > has a plan on how to fix the current FTBFS bug[2], on August 23, I will > > retire novacom-client[3] and novacom-server[4]. > > I would suggest just orphaning the packages. This way, anyone can pick up the > packages themself through the src.fp.o web interface. The packages will get > automatically retired in six weeks if nobody claims them. > Ok, that's done now. I've orphaned both novacom-client and novacom- server. If someone wants them, they're all yours. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Intent to retire: novacom-client, novacom-server
In the deep mists of time, I managed to get a HP Touchpad in the Great Fire Sale of 2011[1] and got to experiment with the Glorious System of Operation that was webOS. Next, naturally, was packaging up novacom, the webOS USB management tool (think the WebOS equivalent of adb), for Fedora. Despite selling my Touchpad a couple of years later, I've continued to maintain novacom, mainly due to the fact that there was very little work involved. Unfortunately, this state of affairs has ended. Libusb, one of novacom's dependencies has been retired, and it's not clear to me that it's worthwhile (or even possible) to keep novacom alive. So, unless I hear from someone who wants it within the next week and has a plan on how to fix the current FTBFS bug[2], on August 23, I will retire novacom-client[3] and novacom-server[4]. Jonathan [1] https://tedium.co/2020/03/31/hp-touchpad-history/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=2113552 [3] https://src.fedoraproject.org/rpms/novacom-client [4] https://src.fedoraproject.org/rpms/novacom-server ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Orphaning deltarpm
Hi everyone, I'm orphaning deltarpm because, as it's currently used in Fedora, it's not very effective, bugs keep getting opened against it because it's not working as well as it should (mostly an infra issue as opposed to a problem with the tool itself), and I no longer have the time or motivation to deal with these tickets when it seems like there's not much benefit. The main problem with deltarpms is that we don't keep them from compose to compose, so we're only getting the previous day's deltarpms when we download updates. This means that the only way you'll be able to truly take advantage of deltarpms is if you get updates after every compose. For more details, see: https://pagure.io/releng/issue/7215 Bugs that need to be looked into: https://bugzilla.redhat.com/show_bug.cgi?id=1873876 - This is mostly two different problems: 1) when the zstd algorithm changes, deltarpms cannot be rebuilt to the exact same RPM, and 2) deltarpms for the kernel package seem to be impossible to rebuild to the exact same RPM due to the compression format used on the kernel RPMs https://bugzilla.redhat.com/show_bug.cgi?id=2058477 - No idea why 100+ MB of packages failed to build correctly (no logs in the ticket), but it's probably related to (1) above. A brief history of how deltarpms came to be added to Fedora: https://www.jdieter.net/posts/2013/05/31/eulogy-for-yum-presto/ Finally, I want to say a huge thank you to Michael Schroeder, who wrote deltarpm. The tool itself still works beautifully well, and the issues we're seeing are all infrastructure related. I also want to say thank you to everyone who worked on getting deltarpms into Fedora in the first place and who have helped keep them running this long. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Failed OpenQA tests for zchunk update - how to troubleshoot?
On Sun, 2022-02-20 at 15:37 -0800, Adam Williamson wrote: > On Sun, 2022-02-20 at 20:26 +0000, Jonathan Dieter wrote: > > I've just pushed zchunk-1.2.0 to all active Fedora branches, and > > it's > > passed the (admittedly non-comprehensive) zchunk test suite, but > > I'm > > seeing 2 OpenQA failed tests in Bodhi: > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2022-e4bcaeea7a > > https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3da9e6213 > > > > Looking through > > https://openqa.fedoraproject.org/tests/1138503#step/_do_install_and_reboot/5 > > it's not clear where the problem is in the screenshots that show as > > failed, especially since zchunk has nothing to do with the UI. > > > > Is this some unrelated OpenQA error, or am I missing something? > > You're missing that I'm an idiot and JSON is awful. Sorry! > > https://pagure.io/fedora-qa/os-autoinst-distri-fedora/c/9bc039b26de4779f74d5da146889779b2a55b111?branch=master > > Should be all good in an hour or so, after I hit a bunch of restart > buttons. Thanks so much for looking into this! That fixed the F34 build, but I'm still seeing a failed test in F35: https://openqa.fedoraproject.org/tests/1138994 Is there some way I can restart it, or does that require special permission? Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Failed OpenQA tests for zchunk update - how to troubleshoot?
I've just pushed zchunk-1.2.0 to all active Fedora branches, and it's passed the (admittedly non-comprehensive) zchunk test suite, but I'm seeing 2 OpenQA failed tests in Bodhi: https://bodhi.fedoraproject.org/updates/FEDORA-2022-e4bcaeea7a https://bodhi.fedoraproject.org/updates/FEDORA-2022-a3da9e6213 Looking through https://openqa.fedoraproject.org/tests/1138503#step/_do_install_and_reboot/5 it's not clear where the problem is in the screenshots that show as failed, especially since zchunk has nothing to do with the UI. Is this some unrelated OpenQA error, or am I missing something? Thanks for any insight, Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: SQLAlchemy integration in Flask
Sorry for taking so long to reply. I'm afraid I don't check this
mailing list as often as I should. :)
On Tue, 2021-12-07 at 08:52 +0100, Aurelien Bompard wrote:
> Thanks for your input!
>
> > 1. We're using a clustered database (CockroachDB, for those who
> > care)
> > that uses optimistic concurrency, so automatic transaction retries
> > are
> > a must, and we need control over how those retries are done.
> >
>
>
> Interesting, we don't use that, but then again we've recently started
> using more funky stuff on the database side (TimescaleDB) so maybe
> one day...
Unfortunately CockroachDB has gone the route of MongoDB in its
licensing, so it's not really open. YugabyteDB looks like it has most
of the same features and is Apache 2.0 licensed, so would probably be a
better fit for Fedora (and, if it wasn't for the fact that it's missing
GIN indexes, we would probably be using it too).
>
> > 2. We are using the same models for a couple of different projects
> > (the
> > API itself and a script that is synchronizing between the old
> > database
> > and the new), and not all the projects are built on Flask.
> > Initially,
> > I was able to get the sync script working with Flask-SQLAlchemy,
> > but
> > things got ugly quickly when I started doing multithreading, so I
> > abandoned it and am now using Flask and SQLAlchemy separately.
> >
>
>
> When I thought about that use case, I supposed it would be OK to
> instantiate the app and start the app context from within the script,
> as it would also give you access to Flask's config file. But I did
> not think about multithreading. Would you recommend against creating
> the app instance and the app context in a command-line script?
Well, that was what I tried to do first, but, as I said, everything
broke down when I tried to do multithreading (and got worse when I
tried to setup multiprocessing). The problem is that Flask-SQLAlchemy
tries to manage the DB session for you, and, since SQLAlchemy sessions
aren't thread-safe, my command-line script kept crashing, and a few
hours of poking around couldn't fix it. If I'd been willing to poke
around more in Flask-SQLAlchemy's, I might have figured something out,
but it just didn't seem to be worth the effort, when manually managing
my sessions fixed the problem completely.
> Is the code you wrote to integrate Flask and SQLAlchemy opensource,
> and available somewhere?
Unfortunately not, but there was actually very little integration code
written.
Our code follows the following pattern (we're using Flask-RESTX, and
I've omitted serializers to keep it simple):
endpoint:
import business
from util import run_transaction
@ns.route("/user/")
class UserLink:
def get(self, id):
return run_transaction(lambda s: business.get_user(s, id))
business:
from database.model import *
def get_user(session, id):
return session.query(User).filter(User.id == id).one()
util:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
import sqlalchemy_cockroachdb
engine = create_engine('postgresql://admin:swordfish@localhost/')
SessionMaker = sessionmaker(engine)
def run_transaction(func):
sqlalchemy_cockroachdb.run_transaction(SessionMaker, func)
The purpose of the run_transaction function is to repeat transactions
if there's a conflict, rather than trying to lock the record, which is
a CockroachDB paradigm.
I hope the above is at least somewhat helpful in explaining how we're
working without Flask-SQLAlchemy
Jonathan
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: SQLAlchemy integration in Flask
On Mon, 2021-12-06 at 18:36 +0100, Aurelien Bompard wrote: > Anyway, this long email is about finding a common ground for > SQLAlchemy integration in Flask, while taking into account our > difficult experiences with webframewoks in the past, but not being > locked in them. Is there something that I misrepresented here? Do you > have opinions? Preferences? So, full disclosure, I'm normally just lurking on this list and am not currently writing or maintaining code for the infrastructure team, so my 2¢ probably isn't worth much more than that. Having said that, in my day job, I've been writing a Flask API to correspond with a massive database restructure using SQLAlchemy. When I started writing the API, I originally used Flask-SQLAlchemy for all the reasons you listed above. However, a couple of months ago I stripped it out for a couple of reasons. 1. We're using a clustered database (CockroachDB, for those who care) that uses optimistic concurrency, so automatic transaction retries are a must, and we need control over how those retries are done. 2. We are using the same models for a couple of different projects (the API itself and a script that is synchronizing between the old database and the new), and not all the projects are built on Flask. Initially, I was able to get the sync script working with Flask-SQLAlchemy, but things got ugly quickly when I started doing multithreading, so I abandoned it and am now using Flask and SQLAlchemy separately. In short, Flask-SQLAlchemy does a great job of tying together Flask and SQLAlchemy if you're 100% sure that your project models will never be required outside of Flask. The minute you step outside of the Flask- SQLAlchemy way of doing things, things start to go very wrong very quickly. Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Freeze break request: Re: Can we update fedora-repo-zdicts on the branched and rawhide composers?
On Wed, 2021-09-08 at 09:14 -0700, Kevin Fenzi wrote: > > I've updated it. > > kevin I can confirm that the latest F35 repodata has the dictionaries now. Thanks so much! Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Can we update fedora-repo-zdicts on the branched and rawhide composers?
Since branching, I've put out a new version of fedora-repo-zdicts with dictionaries for F35 and updated dictionaries for Rawhide. This version (2108.1) is now available in all active Fedora/EPEL branches, I think. Can we update fedora-repo-zdicts on the branched and rawhide composers so they get the latest dictionaries when creating the repodata? Or do we need to wait until the beta freeze ends? Thanks, Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Please don't update zchunk to 1.1.14 on servers where createrepo_c is run
On Wed, 2021-06-02 at 09:35 -0700, Kevin Fenzi wrote: > On Tue, Jun 01, 2021 at 09:13:30PM +0100, Jonathan Dieter wrote: > > A major bug in zchunk-1.1.14 was flagged up to me today. If zchunk- > > 1.1.14 (on a system with zstd 1.5.0+) is used to create a zck file with > > a zdict, the file will be impossible to decompress. Embarrassingly, > > the tests weren't testing this combination. > > > > The good news is that this doesn't affect decompression at all, so this > > is only a problem for the server that's used to generate the zchunked > > metadata, which is using zdicts. > > > > I've just finished building zchunk-1.1.15 which fixes this bug (and > > adds tests to make sure it never happens again), but please make sure > > that zchunk doesn't get updated to 1.1.14 on the servers that generate > > the metadata. > > > > Thanks, and apologies for the inconvenience. > > I updated all of them to 1.1.15 yesterday. > Many of them were on 1.1.14, so I figured it was better to move forward > than move back. :) That sounds good to me! Thanks so much! Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Please don't update zchunk to 1.1.14 on servers where createrepo_c is run
A major bug in zchunk-1.1.14 was flagged up to me today. If zchunk- 1.1.14 (on a system with zstd 1.5.0+) is used to create a zck file with a zdict, the file will be impossible to decompress. Embarrassingly, the tests weren't testing this combination. The good news is that this doesn't affect decompression at all, so this is only a problem for the server that's used to generate the zchunked metadata, which is using zdicts. I've just finished building zchunk-1.1.15 which fixes this bug (and adds tests to make sure it never happens again), but please make sure that zchunk doesn't get updated to 1.1.14 on the servers that generate the metadata. Thanks, and apologies for the inconvenience. Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Can we please update fedora-repo-zdicts on the metadata generation servers for F34 zchunk dictionaries?
On Sat, 2021-04-03 at 21:46 +0100, Jonathan Dieter wrote: > On Sat, 2021-04-03 at 11:09 -0700, Kevin Fenzi wrote: > > ok. I've installed fedora-repo-zdicts on both branched and rawhide > > composers. > > > > Lets see if that works in tomorrow's compose. > > Thanks so much! Fingers crossed. :) I've just checked the latest compose and the repodata now has zdicts. Thanks again Kevin for getting that package into the composers. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Can we please update fedora-repo-zdicts on the metadata generation servers for F34 zchunk dictionaries?
On Sat, 2021-04-03 at 11:09 -0700, Kevin Fenzi wrote: > ok. I've installed fedora-repo-zdicts on both branched and rawhide > composers. > > Lets see if that works in tomorrow's compose. Thanks so much! Fingers crossed. :) Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Can we please update fedora-repo-zdicts on the metadata generation servers for F34 zchunk dictionaries?
Right now, we're not using zdicts for the F34 zchunk metadata because they were only added in fedora-repo-zdicts-2103.1-2 (which should now be in the updates repo in all current Fedora releases). If we could update fedora-repo-zdicts to 2103.1-2 on whichever servers generate the metadata (preferably before the 34 GA metadata is generated), that should significantly reduce the size of the metadata. Thanks, Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: [ipxe-devel] [ipxe/ipxe] Initial support for automatically choosing resolution based on EDID (#48)
Closed #48. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/ipxe/ipxe/pull/48#event-4487171917___ ipxe-devel mailing list ipxe-devel@lists.ipxe.org https://lists.ipxe.org/mailman/listinfo/ipxe-devel
Re: [ipxe-devel] [ipxe/ipxe] Initial support for automatically choosing resolution based on EDID (#48)
Sure. Closing now. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/ipxe/ipxe/pull/48#issuecomment-803655541___ ipxe-devel mailing list ipxe-devel@lists.ipxe.org https://lists.ipxe.org/mailman/listinfo/ipxe-devel
Re: Starting raid-check.timer renders system unusable
On Sun, 2021-03-21 at 17:52 +, Jonathan Dieter wrote: > I'm really hoping that I'm missing something obvious here, but I fear > that a good chunk of our Fedora systems will be unbootable if they're > rebooted without disabling raid-check.timer. Just to follow up on this, it appears that the problem is limited to systems in the Europe/Dublin time zone. Not good, but not the disaster I was fearing. Never been so glad to miss something obvious. ;) Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Starting raid-check.timer renders system unusable
On Sun, 2021-03-21 at 17:52 +, Jonathan Dieter wrote: > There is a workaround: disabling raid-check.timer, but, if you can't > boot due to this bug, you have to boot into single-user mode (which > requires a root password to have been set). As Tom Hughes just pointed out to me, if you're stuck with an unbootable system, you can add systemd.mask=raid-check.timer to the kernel command line. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Starting raid-check.timer renders system unusable
Hey everyone, For reference, a bug report has been filed at https://bugzilla.redhat.com/show_bug.cgi?id=1941335 I just wanted to give a heads up that I came across a bug today that renders Fedora 33 systems unbootable, even after a clean install. If systemd starts raid-check.timer, it gets stuck in what looks like a busy loop, is unable to start new services, systemctl stops responding to commands, and we end up with a lot of zombie processes. The problem is that raid-check.timer (a part of mdadm) is part of the default boot process in Fedora Workstation, and the bug also exists on the version of systemd installed with F33 GA. I've tested on four different systems, each running F33 installed in different ways (some upgrades, some fresh F33 installs), and on each of them, starting raid-check.timer makes the system unusable. There is a workaround: disabling raid-check.timer, but, if you can't boot due to this bug, you have to boot into single-user mode (which requires a root password to have been set). Manually experimenting with dates seems to indicate that this bug is triggered if the current date is after March 3rd, 2021 at 1:00AM, which is why we haven't seen this bug before today. I'm really hoping that I'm missing something obvious here, but I fear that a good chunk of our Fedora systems will be unbootable if they're rebooted without disabling raid-check.timer. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Delta RPMs in Fedora 34
On Tue, 2021-01-05 at 08:49 -0500, Neal Gompa wrote: > To be blunt, I would have never done Zchunk metadata if it was going > to be used as a tool to kill DeltaRPMs. I firmly believe we need both > to have a comprehensive offering that accommodates the needs of > Fedora users across the world. Hey Neal, I'm not sure where you're going with that first sentence, but I think it's pretty obvious that zchunk and deltarpms solve different problems and, following this thread, I don't think anyone has suggested that we should kill deltarpms *because* we have zchunk metadata. When we first brought deltarpms into Fedora, a savings of 60-90% when doing updates was normal. Now that we're losing the deltarpms after each push (as we have been for the last three years), the savings is significantly lower (I normally see less than 10%) and that makes it hard to be motivated to fix the bugs that inevitably arise. It sounds like there might be a plan to keep deltarpms beyond a single push, and, if that happens, I will be more than happy to keep on dealing with deltarpm bugs. :) Thanks, Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Sat, 2021-01-02 at 18:12 +, Jonathan Dieter wrote: > FWIW, I also think it's time for drpms to go. Aside from any potential > issues with the proposed change, they haven't been useful in Fedora for > three years, (see https://pagure.io/releng/issue/7215), and nobody's > been able to put in the time to fix it yet. If that changed and > someone was willing to step up and commit to fixing this, I'd feel very > differently. > > In addition, drpms aren't even working at the moment. Something has > changed during the last week or so that's broken them (see > https://bugzilla.redhat.com/show_bug.cgi?id=1911828). I'll take a > look, but, being honest, there's not much motivation to investigate > this when drpms are of such marginal use in Fedora at the moment. > > Jonathan Apologies for the odd quoting in the previous email; Evolution decided that what you see isn't what you get. :) I've trimmed out everything but my response here. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: DNF/RPM Copy on Write enablement for all variants (System-Wide Change)
On Sat, 2021-01-02 at 13:42 +, Zbigniew Jędrzejewski-Szmek wrote: On Wed, Dec 30, 2020 at 10:10:27AM -0800, Kevin Fenzi wrote: > > This is most likely because we are only making drpms against the most > recent updates. So, we are making very few drpms and only against > things > that recently updated. So... people who actually care about the total download are likely not to update all the time, which also means that drpms will not work for them. > For example: > https://dl.fedoraproject.org/pub/fedora/linux/updates/33/Everything/x86_64/drpms/ > (126 drpms for all of f33 updates). > So... that means that drpms wouldn't even make a difference for people who update often. ...and the proposed Change would require additional contortions to allow drpms to work. It sounds like drpms are not worth the trouble anymore. The effort to make them work properly would be large. I think the crucial bit is that we have more packages and updates than ever, and at the same time more people update at custom schedules, so any reasonable subset of drpms will cover a shrinking subset of upgrades. Zbyszek > > Maybe the time has come to just disable DRPM entirely for F34. > > We could. Or try and make them more usefull again. FWIW, I also think it's time for drpms to go. Aside from any potential issues with the proposed change, they haven't been useful in Fedora for three years, (see https://pagure.io/releng/issue/7215), and nobody's been able to put in the time to fix it yet. If that changed and someone was willing to step up and commit to fixing this, I'd feel very differently. In addition, drpms aren't even working at the moment. Something has changed during the last week or so that's broken them (see https://bugzilla.redhat.com/show_bug.cgi?id=1911828). I'll take a look, but, being honest, there's not much motivation to investigate this when drpms are of such marginal use in Fedora at the moment. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Updated zdicts for F33 zchunk metadata
Thanks Dusty for the pointers on the Freeze Exception process. I've created https://bugzilla.redhat.com/show_bug.cgi?id=1886581 blocked the Final Freeze Exception bug. Jonathan On Thu, Oct 8, 2020 at 8:32 PM Jonathan Dieter wrote: > > I'm afraid I'm late to the party on updating the zstd dictionaries > used by zchunk for F33. I've just built fedora-repo-zdicts-2010.1, > which now includes the F33 dictionaries, but we're obviously past > final freeze. > > I would appreciate karma on the following update so it can go in as a > zero-day update: > https://bodhi.fedoraproject.org/updates/FEDORA-2020-a9a5b96259 > > Additionally, if there's any way we can get the GA metadata generated > on a system with this update installed, the zchunk metadata will be > significantly smaller. Just to be clear, it won't cause any breakage > if we can't, but it would be nice to get the 20-30% reduction in > zchunk metadata size. > > Thanks, and apologies for any inconvenience, > > Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Updated zdicts for F33 zchunk metadata
I'm afraid I'm late to the party on updating the zstd dictionaries used by zchunk for F33. I've just built fedora-repo-zdicts-2010.1, which now includes the F33 dictionaries, but we're obviously past final freeze. I would appreciate karma on the following update so it can go in as a zero-day update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-a9a5b96259 Additionally, if there's any way we can get the GA metadata generated on a system with this update installed, the zchunk metadata will be significantly smaller. Just to be clear, it won't cause any breakage if we can't, but it would be nice to get the 20-30% reduction in zchunk metadata size. Thanks, and apologies for any inconvenience, Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Failed Delta RPMs observed in repo
On Sun, 2020-02-09 at 09:36 -0500, John Mellor wrote: > Question for the repo managers, > > Why do I sometimes see invalid checksums for drpm downloads? E.g: > This excerpt from the update this morning: > > > /var/cache/dnf/updates-7fc4c739b3909d9f/packages/selinux-policy- > > targeted-3.14.4-46.fc31_3.14.4-47.fc31.noarch.drpm: md5 mismatch of > > result > > Some packages were not downloaded. Retrying. > > selinux-policy-targeted-3.14.4-47.fc31.noarch.r 14 MB/s | 13 > > MB 00:00 > > It almost always causes the total download size to be larger than any > savings from using the drpm format: > > > Failed Delta RPMs increased 41.3 MB of updates to 43.3 MB (-4.1% > > wasted) > > > > Is this a repo bug? This is not a repo bug, but rather a (small) bug in how selinux-policy- targeted is packaged. A drpm is assembled by taking the parts of the old installed rpm that haven't changed from the local system and then only downloading the parts that have changed. The problem is that a file in selinux-policy-targeted changes *after* it's installed, but dnf doesn't detect that the old installed rpm has changed until after downloading and attempting to apply the drpm. As you can see on my local system, there are four files that have changed after selinux-policy-targeted was installed: $ sudo rpm -V selinux-policy-targeted S.5T. c /etc/selinux/targeted/contexts/files/file_contexts.local ..5T./var/lib/selinux/targeted/active/commit_num S.5T./var/lib/selinux/targeted/active/file_contexts ...T./var/lib/selinux/targeted/active/homedir_template S.5T./var/lib/selinux/targeted/active/policy.kern .M... g /var/lib/selinux/targeted/active/policy.linked ...T./var/lib/selinux/targeted/active/seusers ...T./var/lib/selinux/targeted/active/users_extra The file in /etc is marked as a config file, so won't cause the deltarpm to fail, but the changes to the other three will. Arguably, any files that may change on the local system should be marked as config files. Jonathan ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Why Must I Do "dnf clean all" Before Updating Will Proceed?
On Tue, 2019-06-04 at 18:54 -0700, Samuel Sieb wrote: > On 6/1/19 5:27 AM, Garry T. Williams wrote: > > On Friday, May 31, 2019 11:05:20 PM EDT Tim via users wrote: > > > On Fri, 2019-05-31 at 17:18 -0400, Garry Williams wrote: > > > > But, of course, the issue is why this happens in the first place. > > > > > > Does your ISP insert a transparent proxy between you and the > > > internet? They're well known to cause caching problems. > > > > Ah, ha! That is a difference between the problem system and the > > others I have that do not experience the problem. > > > > My employer does eavesdrop on everything. > > I think I found the answer to this. I ran into the same problem with my > simple custom proxy. Starting in F30, the repo uses zchunk. This means > that dnf requests lots of byte ranges. If the proxy doesn't support > this, then librepo fails. According to the http specs, a client MUST > support getting more (or less) data than asked for when requesting > ranges. However, librepo does not. I'm about to file a bug for this. Please do, and please file it against zchunk when you do, but please first make sure you've updated to the latest versions if libdnf, librepo and zchunk-libs. librepo is supposed to automatically reduce the number of zchunk byte ranges it requests if there's a failure, so, if it's not, it's most likely a bug. It would also be really helpful to see how your proxy responds to a request for too many byte ranges. And please make sure to attach dnf.librepo.log when you file the bug. Jonathan ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression
On Wed, 2019-05-29 at 18:05 -0400, Neal Gompa wrote: > On Wed, May 29, 2019 at 5:53 PM Josh Boyer wrote: > > > > If we did this, wouldn't it make it very difficult to use tools like > > mock on RHEL / CentOS 7 to build for Fedora 3x? Or does RHEL 7 RPM > > support zstd? > > > > We're pretty much screwed here. Also, since RHEL 8's rpm package does > not have zstd support compiled in, it too cannot handle the RPMs. > > Cf. https://git.centos.org/rpms/rpm/blob/c8/f/SPECS/rpm.spec#_17-18 I just wanted to point out my post[1] in November where I suggested using zchunk as the compression format for rpm. IIRC, the main concern with that proposal was compatibility with RHEL. The one main advantage using zchunk would have over zstd would be the ability to completely eliminated drpms, but, as mentioned in that thread, it would require some changes to the RPM format. Jonathan 1: https://lists.fedorahosted.org/archives/list/devel@lists.fedoraproject.org/thread/YHKXMJHZW3O6EWA2WYMFWOC22KTVTPLB/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression
On Wed, 2019-05-29 at 18:32 -0400, James Cassell wrote: > > Would this help with drpms similar to how it helps with faster yum > repo metadata downloads? My biggest problem with drpms is the slow > rebuild speed which is usually slower than my download bandwidth. It > would be a big win if zstd helps here. Unfortunately not. The drpm rebuild process involves recompressing the rpm, so we'd be affected by the compression speed, not the decompression speed. With zstd compression level > 15, the drpm rebuild speed would actually slow down (possibly quite significantly). Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 31 System-Wide Change proposal: Switch RPMs to zstd compression
On Wed, 2019-05-29 at 20:15 -0600, Chris Murphy wrote: > 'dnf info deltarpm' says > URL : http://gitorious.org/deltarpm/deltarpm > which has an expired certificate, but pushing passed that it says > current version 3.6 is 5 years old. Is this really maintained or > updatabled? Upstream has changed to https://github.com/rpm-software-management/deltarpm. The code is still maintained, but there's not much active development. I can't speak for the upstream maintainer, but I would guess that a PR that adds zstd support would probably be welcomed. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [PATCH] bodhi-backend: Make sure zchunk dicts are installed
On Thu, 2019-05-23 at 17:01 -0400, Randy Barlow wrote: > On Thu, 2019-05-23 at 10:33 -0700, Kevin Fenzi wrote: > > Applied. Thanks. > > One note: The patch to do zchunking is part of Bodhi 4.0.0, which is > not yet in production; we plan to deploy it on Tuesday. Unless I'm mistaken, that patch is specific to updateinfo.xml. The other metadata in updates and updates-testing is currently zchunked, just without a zdict at the moment. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: [PATCH] bodhi-backend: Make sure zchunk dicts are installed
On Sun, 2019-05-19 at 21:25 +0100, Jonathan Dieter wrote: > The zchunk dictionaries used to reduce the size of zchunk metadata seems to > not currently be installed on the bodhi server. This patch makes sure they > are installed. > > Signed-off-by: Jonathan Dieter > --- > roles/bodhi2/backend/tasks/main.yml | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/roles/bodhi2/backend/tasks/main.yml > b/roles/bodhi2/backend/tasks/main.yml > index 32da678db..3ab6ec809 100644 > --- a/roles/bodhi2/backend/tasks/main.yml > +++ b/roles/bodhi2/backend/tasks/main.yml > @@ -18,6 +18,7 @@ >- bodhi-composer >- python3-pyramid_sawing >- sigul > + - fedora-repo-zdicts ># Are these still needed? >- compose-utils >- pungi-utils Just to be clear, I'm not 100% sure this is the right way or the right place to install the zchunk dictionaries, but having them installed should reduce the size of the zchunk metadata by a significant amount. Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
[PATCH] bodhi-backend: Make sure zchunk dicts are installed
The zchunk dictionaries used to reduce the size of zchunk metadata seems to not currently be installed on the bodhi server. This patch makes sure they are installed. Signed-off-by: Jonathan Dieter --- roles/bodhi2/backend/tasks/main.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/roles/bodhi2/backend/tasks/main.yml b/roles/bodhi2/backend/tasks/main.yml index 32da678db..3ab6ec809 100644 --- a/roles/bodhi2/backend/tasks/main.yml +++ b/roles/bodhi2/backend/tasks/main.yml @@ -18,6 +18,7 @@ - bodhi-composer - python3-pyramid_sawing - sigul + - fedora-repo-zdicts # Are these still needed? - compose-utils - pungi-utils -- 2.21.0 ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Can we use SCLs for building for EPEL 6?
On Sun, 2019-04-14 at 16:01 -0400, Stephen John Smoogen wrote: > On Sat, 13 Apr 2019 at 21:06, Todd Zullinger wrote: > > Neal Gompa wrote: > > > If devtoolset is available for EPEL6 (which I think it is?) > > > > I don't believe devtoolset was enabled for el6 in koji. > > When it was added to the mock configs for el6/el7, the > > consensus on the epel list was that it would be added to el6 > > if there was sufficient demand. I've only seen it come up > > once (or maybe twice) since then on the epel list. > > > > I'm not familiar enough with the koji commands to confirm > > it. I can see that rhel7-server-rhscl-7 is listed in the > > external repos, but I don't see a similar rhel6 SCL. > > > > Apologies if I simply missed an announcement on the epel > > lists and am passing on outdated data. > > > > I believe Todd is correct. At the time there was the package needing > SCL's was chromium and the owner had no interest for making the > package in EL6. If zchunk needs it, we can put it in. That's ok. I've gone with Kevin Kofler's suggestion and just fixed the build to work with the old version of GCC. Thanks, all, for the help! Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use SCLs for building for EPEL 6?
On Sat, 2019-04-13 at 13:11 -0700, John Reiser wrote: > > Unfortunately, the gcc in EL6 is too old to build zchunk > > In what specific way(s)? Can the complaints from gcc [which version?], > or other tools in the toolchain, be listed here? > Other developers may have faced the same or similar problems, > and may have tools to help. Sorry, I should have shared that the first time around. The version of gcc that comes with EL6 is 4.4.7. When building zchunk, I get a number of messages that look like: $ ninja-build [1/178] Compiling C object 'src/lib/zck@sha/comp_zstd_zstd.c.o'. FAILED: src/lib/zck@sha/comp_zstd_zstd.c.o cc -Isrc/lib/zck@sha -Isrc/lib -I../src/lib -Iinclude -I../include -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -O0 -g -DZCHUNK_ZSTD -DZCHUNK_OPENSSL -fvisibility=hidden -fPIC -MMD -MQ 'src/lib/zck@sha/comp_zstd_zstd.c.o' -MF 'src/lib/zck@sha/comp_zstd_zstd.c.o.d' -o 'src/lib/zck@sha/comp_zstd_zstd.c.o' -c ../src/lib/comp/zstd/zstd.c In file included from ../src/lib/comp/zstd/zstd.c:34: ../src/lib/zck_private.h:92: error: redefinition of typedef 'zckCtx' include/zck.h:49: note: previous declaration of 'zckCtx' was here ../src/lib/zck_private.h:106: error: redefinition of typedef 'zck_log_type' include/zck.h:47: note: previous declaration of 'zck_log_type' was here ../src/lib/zck_private.h:117: error: redefinition of typedef 'zckHash' include/zck.h:50: note: previous declaration of 'zckHash' was here ../src/lib/zck_private.h:150: error: redefinition of typedef 'zckDL' include/zck.h:54: note: previous declaration of 'zckDL' was here ../src/lib/zck_private.h:165: error: redefinition of typedef 'zckChunk' include/zck.h:51: note: previous declaration of 'zckChunk' was here ../src/lib/zck_private.h:177: error: redefinition of typedef 'zckIndex' include/zck.h:52: note: previous declaration of 'zckIndex' was here ../src/lib/zck_private.h:193: error: redefinition of typedef 'zckRange' include/zck.h:53: note: previous declaration of 'zckRange' was here ../src/lib/zck_private.h:224: error: redefinition of typedef 'zckComp' ../src/lib/zck_private.h:91: note: previous declaration of 'zckComp' was here ../src/lib/zck_private.h:298: error: redefinition of typedef 'zckCtx' ../src/lib/zck_private.h:92: note: previous declaration of 'zckCtx' was here For reference, you can find zck.h.in (which gets processed into zck.h with the version added) at: https://github.com/zchunk/zchunk/blob/master/include/zck.h.in and zck_private.h at: https://github.com/zchunk/zchunk/blob/master/src/lib/zck_private.h As far as I can see, gcc-4.7 doesn't like that I'm typedefing the same struct to the same type twice. Later versions don't see it as a problem at all. (Just to be clear, this still happens if I change zck_private.h to say: typedef struct zckCtx zckCtx;) Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Can we use SCLs for building for EPEL 6?
So, the background is that I'd like to build zchunk for EPEL 6 (it's already built for EPEL 7). Unfortunately, the gcc in EL6 is too old to build zchunk, so I'd prefer to use a newer version from an SCL, rather than rewrite zchunk to be compatible with an ancient version of gcc. I noticed that SCLs are available for EPEL 7 (note the final repository in the list at https://koji.fedoraproject.org/koji/taginfo?tagID=259), but not for EPEL 6 (see https://koji.fedoraproject.org/koji/taginfo?tagID=140). So, is there any way we can use SCLs for building on EPEL 6, or should I stop tilting at windmills and just say EL6 is too old for zchunk? Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [PATCH] bodhi-backend: Add zchunk support to updates and updates-testing repositories
On Thu, 2019-04-11 at 18:08 -0700, Kevin Fenzi wrote:
> On 4/9/19 11:20 AM, Jonathan Dieter wrote:
> > On Tue, 2019-04-09 at 19:14 +0100, Jonathan Dieter wrote:
> > > This re-adds zchunk support for the updates and updates-testing
> > > repositories
> > > for both rpms and modularity.
> > >
> > > Zchunk metadata was turned off due to a broken version of librepo that
> > > made it
> > > out to stable, but a fixed version has been pushed and FESCo has
> > > decided[1] to
> > > go ahead and turn this back on.
> > >
> > > 1: https://pagure.io/fesco/issue/2116
> >
> > In that ticket, we didn't really specify when to turn it back on, so if
> > we want to sit on this patch for a few days, that's fine with me.
> >
> > Once we've decided when this should be applied, I'll send a message to
> > devel-announce with an explanation on how to workaround the segfault
> > for anyone still using librepo-1.9.6-1.
>
> I think we should apply it asap.
>
> However, if I save your email and try and git am it, it doesn't apply at
> all.
>
> Can you resend with the patch as attachment?
>
> I am not sure what thunderbird is doing here. ;(
>
> kevin
Ok, here it is, freshly rebased, as an attachment.
Jonathan
From 4c53d3fba04b1bbbfdb8a7dc1d350e75dd5efd5d Mon Sep 17 00:00:00 2001
From: Jonathan Dieter
Date: Sat, 30 Mar 2019 22:29:33 +
Subject: [PATCH] bodhi-backend: Add zchunk support to updates and
updates-testing repositories
This re-adds zchunk support for the updates and updates-testing repositories
for both rpms and modularity.
Zchunk metadata was turned off due to a broken version of librepo that made it
out to stable, but a fixed version has been pushed and FESCo has decided[1] to
go ahead and turn this back on.
1: https://pagure.io/fesco/issue/2116
Signed-off-by: Jonathan Dieter
---
roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++
roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++
2 files changed, 6 insertions(+)
diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2 b/roles/bodhi2/backend/templates/pungi.module.conf.j2
index 43c6a7e5f..b5bb0c1fb 100644
--- a/roles/bodhi2/backend/templates/pungi.module.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.module.conf.j2
@@ -61,6 +61,9 @@ greedy_method = 'build'
createrepo_c = True
createrepo_checksum = 'sha256'
createrepo_deltas = False
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
#jigdo
create_jigdo = False
diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
index 8d9e9a3f2..020736aee 100644
--- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
@@ -66,6 +66,9 @@ createrepo_deltas = [
('^Everything$', {'*': True})
]
createrepo_database = True
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
# CHECKSUMS
media_checksums = ['sha256']
--
2.21.0
signature.asc
Description: This is a digitally signed message part
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
How to fix dnf segmentation fault (zchunk metadata is re-enabled)
FESCo has given us the go-ahead to turn zchunk metadata on again[1] for the F30 fedora repository after the librepo segmentation fault bug[2] was fixed. An updated librepo[3] was built a week ago and was pushed to stable five days ago, so most beta users should have the new version. If you're using F30 and have librepo-1.9.6-1.fc30, please update to librepo-1.9.6-2.fc30 as soon as possible! If you don't, you will once again get a segmentation fault when running dnf. If you find yourself getting a segmentation fault when running dnf on a system with librepo-1.9.6-1.fc30, you can temporarily work around it by doing the following: 1. Edit /etc/dnf/dnf.conf and add the following line: zchunk=False 2. Run: dnf update librepo Verify that you got librepo-1.9.6-2.fc30 or later in the update 3. Remove the 'zchunk=False' line you added to /etc/dnf/dnf.conf 4. Run: dnf update At this point, dnf should download the repository metadata without any segmentation faults, and any further repository metadata downloads should be significantly smaller. Jonathan 1. https://pagure.io/fesco/issue/2116 2. https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org/thread/RHZ6V6MFVFAIJ6OEXZ5VL7BPPLHJE4GF/ 3. https://bodhi.fedoraproject.org/updates/FEDORA-2019-07c3e09858 ___ devel-announce mailing list -- devel-annou...@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
How to fix dnf segmentation fault (zchunk metadata is re-enabled)
FESCo has given us the go-ahead to turn zchunk metadata on again[1] for the F30 fedora repository after the librepo segmentation fault bug[2] was fixed. An updated librepo[3] was built a week ago and was pushed to stable five days ago, so most beta users should have the new version. If you're using F30 and have librepo-1.9.6-1.fc30, please update to librepo-1.9.6-2.fc30 as soon as possible! If you don't, you will once again get a segmentation fault when running dnf. If you find yourself getting a segmentation fault when running dnf on a system with librepo-1.9.6-1.fc30, you can temporarily work around it by doing the following: 1. Edit /etc/dnf/dnf.conf and add the following line: zchunk=False 2. Run: dnf update librepo Verify that you got librepo-1.9.6-2.fc30 or later in the update 3. Remove the 'zchunk=False' line you added to /etc/dnf/dnf.conf 4. Run: dnf update At this point, dnf should download the repository metadata without any segmentation faults, and any further repository metadata downloads should be significantly smaller. Jonathan 1. https://pagure.io/fesco/issue/2116 2. https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/RHZ6V6MFVFAIJ6OEXZ5VL7BPPLHJE4GF/ 3. https://bodhi.fedoraproject.org/updates/FEDORA-2019-07c3e09858 ___ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org
Re: [PATCH] bodhi-backend: Add zchunk support to updates and updates-testing repositories
On Tue, 2019-04-09 at 19:14 +0100, Jonathan Dieter wrote: > This re-adds zchunk support for the updates and updates-testing repositories > for both rpms and modularity. > > Zchunk metadata was turned off due to a broken version of librepo that made it > out to stable, but a fixed version has been pushed and FESCo has decided[1] to > go ahead and turn this back on. > > 1: https://pagure.io/fesco/issue/2116 In that ticket, we didn't really specify when to turn it back on, so if we want to sit on this patch for a few days, that's fine with me. Once we've decided when this should be applied, I'll send a message to devel-announce with an explanation on how to workaround the segfault for anyone still using librepo-1.9.6-1. Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
[PATCH] bodhi-backend: Add zchunk support to updates and updates-testing repositories
This re-adds zchunk support for the updates and updates-testing repositories
for both rpms and modularity.
Zchunk metadata was turned off due to a broken version of librepo that made it
out to stable, but a fixed version has been pushed and FESCo has decided[1] to
go ahead and turn this back on.
1: https://pagure.io/fesco/issue/2116
Signed-off-by: Jonathan Dieter
---
roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++
roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++
2 files changed, 6 insertions(+)
diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2
b/roles/bodhi2/backend/templates/pungi.module.conf.j2
index 43c6a7e5f..b5bb0c1fb 100644
--- a/roles/bodhi2/backend/templates/pungi.module.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.module.conf.j2
@@ -61,6 +61,9 @@ greedy_method = 'build'
createrepo_c = True
createrepo_checksum = 'sha256'
createrepo_deltas = False
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck',
'--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
#jigdo
create_jigdo = False
diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
index 8d9e9a3f2..020736aee 100644
--- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
@@ -66,6 +66,9 @@ createrepo_deltas = [
('^Everything$', {'*': True})
]
createrepo_database = True
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck',
'--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
# CHECKSUMS
media_checksums = ['sha256']
--
2.21.0
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
PSA: workaround for segfault when running dnf update in F30
librepo-1.9.6 has a major bug that will cause a segfault when a repository has zchunk metadata. To temporarily work around the problem, set zchunk=False in /etc/dnf/dnf.conf or wait until the next updates push comes out About a week ago, we disabled zchunk metadata in the main F30 repository because of arm-specific compose problems[1]. The problems were fixed and we enabled zchunk metadata in F30 updates-testing yesterday, but, in the interval when there were no zchunk-enabled repositories, librepo-1.9.6 was released, and it has a major bug where it segfaults whenever dnf update is run on a repository with zchunk metadata. We've disabled zchunk metadata for F30 updates-testing and doing a new push now. I have submitted a fix[2] for the bug, but for now any F30 users need to do one of the following: * Wait until the next updates push is out. It won't have zchunk metadata, so updates will work normally again. * Set zchunk=False in /etc/dnf/dnf.conf. This will force dnf to fall back to non-zchunk metadata, bypassing the bug. Many apologies for the inconvenience. Jonathan [1] https://pagure.io/dusty/failed-composes/issue/1703 [2] https://github.com/rpm-software-management/librepo/pull/148 ___ devel-announce mailing list -- devel-annou...@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
PSA: workaround for segfault when running dnf update in F30
librepo-1.9.6 has a major bug that will cause a segfault when a repository has zchunk metadata. To temporarily work around the problem, set zchunk=False in /etc/dnf/dnf.conf or wait until the next updates push comes out About a week ago, we disabled zchunk metadata in the main F30 repository because of arm-specific compose problems[1]. The problems were fixed and we enabled zchunk metadata in F30 updates-testing yesterday, but, in the interval when there were no zchunk-enabled repositories, librepo-1.9.6 was released, and it has a major bug where it segfaults whenever dnf update is run on a repository with zchunk metadata. We've disabled zchunk metadata for F30 updates-testing and doing a new push now. I have submitted a fix[2] for the bug, but for now any F30 users need to do one of the following: * Wait until the next updates push is out. It won't have zchunk metadata, so updates will work normally again. * Set zchunk=False in /etc/dnf/dnf.conf. This will force dnf to fall back to non-zchunk metadata, bypassing the bug. Many apologies for the inconvenience. Jonathan [1] https://pagure.io/dusty/failed-composes/issue/1703 [2] https://github.com/rpm-software-management/librepo/pull/148 ___ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Sun, 2019-03-31 at 10:37 -0700, Kevin Fenzi wrote: > On 3/31/19 10:35 AM, Jonathan Dieter wrote: > > On Sun, 2019-03-31 at 10:28 -0700, Kevin Fenzi wrote: > > > On 3/31/19 1:56 AM, Jonathan Dieter wrote: > > > > On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote: > > > > > Due to an unrelated *major* bug in the latest librepo update ( > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to > > > > > request that we disable zchunk metadata generation in updates and > > > > > updates-testing until it's fixed. > > > > > > > > Just to be clear, until either: > > > > * We get a new updates compose out without zchunk metadata, or > > > > * The user sets zchunk=False in /etc/dnf/dnf.conf > > > > > > > > dnf update is broken for anybody using F30 > > > > > > > > Should I send an email to -devel explaining the above? > > > > > > Please do, perhaps devel-announce ? > > > > > > I have reverted things and am working on a new f30-updates-testing push. > > > There was a failed f29-updates-testing last night so I have to finish > > > that first, but hopefully we will have it out in a few hours. > > > > I've sent it out to devel-announce, but it was rejected as I'm not in > > the right group. Will I send it to you and let you forward it? > > Yeah, you have to be subscribed to devel-announce to post there... if > you just subscribe and resend it should go to moderation and I can pass it. > > Or if you want, just send it my way and I can post it... Ok, I've subscribed, sent the message, and it's awaiting moderation. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Sun, 2019-03-31 at 10:28 -0700, Kevin Fenzi wrote: > On 3/31/19 1:56 AM, Jonathan Dieter wrote: > > On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote: > > > Due to an unrelated *major* bug in the latest librepo update ( > > > https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to > > > request that we disable zchunk metadata generation in updates and > > > updates-testing until it's fixed. > > > > Just to be clear, until either: > > * We get a new updates compose out without zchunk metadata, or > > * The user sets zchunk=False in /etc/dnf/dnf.conf > > > > dnf update is broken for anybody using F30 > > > > Should I send an email to -devel explaining the above? > > Please do, perhaps devel-announce ? > > I have reverted things and am working on a new f30-updates-testing push. > There was a failed f29-updates-testing last night so I have to finish > that first, but hopefully we will have it out in a few hours. I've sent it out to devel-announce, but it was rejected as I'm not in the right group. Will I send it to you and let you forward it? Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Sun, 2019-03-31 at 09:09 +0100, Jonathan Dieter wrote: > Due to an unrelated *major* bug in the latest librepo update ( > https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to > request that we disable zchunk metadata generation in updates and > updates-testing until it's fixed. Just to be clear, until either: * We get a new updates compose out without zchunk metadata, or * The user sets zchunk=False in /etc/dnf/dnf.conf dnf update is broken for anybody using F30 Should I send an email to -devel explaining the above? Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Sun, 2019-03-31 at 05:13 +, Peter Robinson wrote: > On Sun, Mar 31, 2019 at 6:01 AM Kevin Fenzi wrote: > > On 3/30/19 9:50 PM, Peter Robinson wrote: > > > > Great, thanks! I'll be keeping an eye on the composes to see if there > > > > are any issues. > > > > > > Wasn't this disabled in the main Fedora branched compose? If so why > > > would we want to enable it only on updates? > > > > There's no updates in f30 indeed, but updates-testing should be there > > and available for testing. Nearer release we will enable updates and if > > we didn't enable this for them now we might well not remember to do so, > > so it seemed like a good idea to just do them both. > > I was referring to commits 6c392f16 and 96adf9a in pungi-fedora, if > it's disabled in the base fedora repo why enable it in > updates/testing? Hey Peter, the zchunk metadata generation was disabled in the base repo because of a bug that popped up in a combination that the compose process happened to hit: using a single baseurl and downloading a zchunk file with tens of thousands of chunks on a slow processor. The bug has been fixed with updates to both zchunk and libcurl (see https://bugzilla.redhat.com/show_bug.cgi?id=1690971) and it shouldn't affect beta users because the number of chunks in updates and updates- testing is a magnitude lower than the base repo. *However* Due to an unrelated *major* bug in the latest librepo update ( https://bugzilla.redhat.com/show_bug.cgi?id=1694411), I'd like to request that we disable zchunk metadata generation in updates and updates-testing until it's fixed. Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Sat, 2019-03-30 at 16:00 -0700, Kevin Fenzi wrote: > On 3/30/19 3:32 PM, Jonathan Dieter wrote: > > On Sat, 2019-03-30 at 15:13 -0700, Kevin Fenzi wrote: > > > On 3/30/19 11:35 AM, Jonathan Dieter wrote: > > > > > > > Stephen and Kevin, thanks so much! > > > > > > Can you rebase and attach the patch? > > > > > > It's not applying cleanly for me... if not I can try and manually > > > poke > > > it later. > > > > > > kevin > > > > I've just rebased and posted the updated patch. There were no > > conflicts when I rebased it against master, so please let me know > > if I > > should be rebasing against a different branch. > > Not sure why it was complaining, but its applied and pushed now. > > kevin Great, thanks! I'll be keeping an eye on the composes to see if there are any issues. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Sat, 2019-03-30 at 15:13 -0700, Kevin Fenzi wrote: > On 3/30/19 11:35 AM, Jonathan Dieter wrote: > > > Stephen and Kevin, thanks so much! > > Can you rebase and attach the patch? > > It's not applying cleanly for me... if not I can try and manually poke > it later. > > kevin I've just rebased and posted the updated patch. There were no conflicts when I rebased it against master, so please let me know if I should be rebasing against a different branch. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
[Freeze Break Request] Add zchunk support to updates and updates-testing repositories
Rebased patch against master ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
[PATCH] Add zchunk support to updates and updates-testing repositories
This adds zchunk support for the updates and updates-testing repositories
for both rpms and modularity
Signed-off-by: Jonathan Dieter
---
roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++
roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++
2 files changed, 6 insertions(+)
diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2
b/roles/bodhi2/backend/templates/pungi.module.conf.j2
index 43c6a7e5f..b5bb0c1fb 100644
--- a/roles/bodhi2/backend/templates/pungi.module.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.module.conf.j2
@@ -61,6 +61,9 @@ greedy_method = 'build'
createrepo_c = True
createrepo_checksum = 'sha256'
createrepo_deltas = False
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck',
'--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
#jigdo
create_jigdo = False
diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
index 8d9e9a3f2..020736aee 100644
--- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
@@ -66,6 +66,9 @@ createrepo_deltas = [
('^Everything$', {'*': True})
]
createrepo_database = True
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck',
'--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
# CHECKSUMS
media_checksums = ['sha256']
--
2.21.0
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Sat, 2019-03-30 at 14:05 -0400, Stephen John Smoogen wrote: > +1 > > On Sat, 30 Mar 2019 at 13:53, Kevin Fenzi wrote: > > On 3/29/19 1:33 PM, Jonathan Dieter wrote: > > > On Mon, 2019-03-11 at 20:23 +, Jonathan Dieter wrote: > > > > On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote: > > > > > On 3/11/19 12:26 AM, Jonathan Dieter wrote: > > > > > > This adds zchunk support for the updates and updates-testing > > > > > > repositories for both rpms and modularity. We already have zchunk > > > > > > metadata being generated for the fedora repository. I'd like to > > > > > > get this in before Beta comes out so Beta users will have zchunk- > > > > > > enabled updates-testing repositories when Beta is released. > > > > > > > > > > yeah, hopefilly not too much pain since it's been in rawhide a while > > > > > now. > > > > > > > > > > > I am making the assumption that a zchunk-enabled createrepo_c > > > > > > (0.12.0-2 > > > > > > or later) is available on the builders (I think I'm safe making that > > > > > > assumption, since zchunk metadata is already being generated for > > > > > > some > > > > > > repos). > > > > > > > > > > Well, bodhi-backend01 (where the updates process/pungi runs for these) > > > > > has a newer one, so yes. It's all run on bodhi-backend01, not > > > > > builders. > > > > > > > > > > > I have *not* tested this patch, because I'm not sure how I'd go > > > > > > about > > > > > > doing so. If we don't have any test builders, my suggestion would > > > > > > be > > > > > > to wait until no compose is running, and then run this play on a > > > > > > builder, verifying that the generated pungi configuration is valid > > > > > > for > > > > > > both f29 and f30, with no createrepo_extra_args in f29. > > > > > > > > > > yeah, we can commit this, run the playbook then examine the results. > > > > > > > > Great. I'm on UTC time right now, so hopefully I'll be off of work and > > > > available if there are any issues whenever we get another +1 and you > > > > run it. I do expect that it will go fine. > > > > > > Since we never got the extra +1 to get this in before Beta, are we at a > > > point where we can turn this on now? > > > > Nope, we are still frozen until the day after beta release. ;( > > > > But I will try and scare up another +1 > > > > kevin > > > > -- > Stephen J Smoogen. Stephen and Kevin, thanks so much! Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Mon, 2019-03-11 at 20:23 +, Jonathan Dieter wrote: > On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote: > > On 3/11/19 12:26 AM, Jonathan Dieter wrote: > > > This adds zchunk support for the updates and updates-testing > > > repositories for both rpms and modularity. We already have zchunk > > > metadata being generated for the fedora repository. I'd like to > > > get this in before Beta comes out so Beta users will have zchunk- > > > enabled updates-testing repositories when Beta is released. > > > > yeah, hopefilly not too much pain since it's been in rawhide a while now. > > > > > I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 > > > or later) is available on the builders (I think I'm safe making that > > > assumption, since zchunk metadata is already being generated for some > > > repos). > > > > Well, bodhi-backend01 (where the updates process/pungi runs for these) > > has a newer one, so yes. It's all run on bodhi-backend01, not builders. > > > > > I have *not* tested this patch, because I'm not sure how I'd go about > > > doing so. If we don't have any test builders, my suggestion would be > > > to wait until no compose is running, and then run this play on a > > > builder, verifying that the generated pungi configuration is valid for > > > both f29 and f30, with no createrepo_extra_args in f29. > > > > yeah, we can commit this, run the playbook then examine the results. > > Great. I'm on UTC time right now, so hopefully I'll be off of work and > available if there are any issues whenever we get another +1 and you > run it. I do expect that it will go fine. Since we never got the extra +1 to get this in before Beta, are we at a point where we can turn this on now? Thanks Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Switching from GCC to llvm-clang for building chromium(vaapi)
On Sat, 2019-03-16 at 12:20 -0500, Michael Cronenworth wrote: > On 3/16/19 7:10 AM, Akarshan Biswas wrote: > > With the above stated reasons, I am willing to know your > > suggestions/opinions on this. > > Building with GCC has become quite a headache for me. If it goes against > > the packaging > > rules, I think we can make an exception for this particular package. > > There is no guideline that says "you must build with gcc" so feel free to > change to the > compiler of your choice. > > I know it's trendy to use clang, and you have little choice in the matter, > but it's sad > that upstreams would rather use an unstable compiler instead of working with > gcc to > implement the features they want. Just to clarify, there actually is https://docs.fedoraproject.org/en-US/packaging-guidelines/#compiler, which states that packages must build with gcc if upstream supports GCC as a compiler. Having said that, if the package doesn't build on GCC, that may be an argument that upstream doesn't support it. Jonathan ___ rpmfusion-developers mailing list -- rpmfusion-developers@lists.rpmfusion.org To unsubscribe send an email to rpmfusion-developers-le...@lists.rpmfusion.org
Re: [Freeze Break Request] Add zchunk support to updates and updates-testing repositories
On Mon, 2019-03-11 at 11:24 -0700, Kevin Fenzi wrote: > On 3/11/19 12:26 AM, Jonathan Dieter wrote: > > This adds zchunk support for the updates and updates-testing > > repositories for both rpms and modularity. We already have zchunk > > metadata being generated for the fedora repository. I'd like to > > get this in before Beta comes out so Beta users will have zchunk- > > enabled updates-testing repositories when Beta is released. > > yeah, hopefilly not too much pain since it's been in rawhide a while now. > > > I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2 > > or later) is available on the builders (I think I'm safe making that > > assumption, since zchunk metadata is already being generated for some > > repos). > > Well, bodhi-backend01 (where the updates process/pungi runs for these) > has a newer one, so yes. It's all run on bodhi-backend01, not builders. > > > I have *not* tested this patch, because I'm not sure how I'd go about > > doing so. If we don't have any test builders, my suggestion would be > > to wait until no compose is running, and then run this play on a > > builder, verifying that the generated pungi configuration is valid for > > both f29 and f30, with no createrepo_extra_args in f29. > > yeah, we can commit this, run the playbook then examine the results. Great. I'm on UTC time right now, so hopefully I'll be off of work and available if there are any issues whenever we get another +1 and you run it. I do expect that it will go fine. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
[Freeze Break Request] Add zchunk support to updates and updates-testing repositories
This adds zchunk support for the updates and updates-testing
repositories for both rpms and modularity. We already have zchunk metadata
being generated for the fedora repository. I'd like to get this in before Beta
comes out so Beta users will have zchunk-enabled updates-testing repositories
when Beta is released.
I am making the assumption that a zchunk-enabled createrepo_c (0.12.0-2
or later) is available on the builders (I think I'm safe making that
assumption, since zchunk metadata is already being generated for some
repos).
I have *not* tested this patch, because I'm not sure how I'd go about
doing so. If we don't have any test builders, my suggestion would be
to wait until no compose is running, and then run this play on a
builder, verifying that the generated pungi configuration is valid for
both f29 and f30, with no createrepo_extra_args in f29.
Signed-off-by: Jonathan Dieter
---
roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++
roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++
2 files changed, 6 insertions(+)
diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2
b/roles/bodhi2/backend/templates/pungi.module.conf.j2
index bb021eb13..7dad35403 100644
--- a/roles/bodhi2/backend/templates/pungi.module.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.module.conf.j2
@@ -59,6 +59,9 @@ greedy_method = 'build'
createrepo_c = True
createrepo_checksum = 'sha256'
createrepo_deltas = False
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck',
'--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
#jigdo
create_jigdo = False
diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
index 8d9e9a3f2..020736aee 100644
--- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
@@ -66,6 +66,9 @@ createrepo_deltas = [
('^Everything$', {'*': True})
]
createrepo_database = True
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck',
'--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
# CHECKSUMS
media_checksums = ['sha256']
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: How do we turn zchunk on for updates-testing for F30?
On Sun, 2019-03-10 at 15:47 +, Peter Robinson wrote: > git send-email so it's inline on the list for easy review. Thanks for the tip! Just sent it using git send-email. Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
[PATCH] Add zchunk support to updates and updates-testing repositories
This adds zchunk support for the updates and updates-testing repositories
for both rpms and modularity
Signed-off-by: Jonathan Dieter
---
roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++
roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++
2 files changed, 6 insertions(+)
diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2
b/roles/bodhi2/backend/templates/pungi.module.conf.j2
index bb021eb13..7dad35403 100644
--- a/roles/bodhi2/backend/templates/pungi.module.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.module.conf.j2
@@ -59,6 +59,9 @@ greedy_method = 'build'
createrepo_c = True
createrepo_checksum = 'sha256'
createrepo_deltas = False
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck',
'--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
#jigdo
create_jigdo = False
diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
index 8d9e9a3f2..020736aee 100644
--- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
@@ -66,6 +66,9 @@ createrepo_deltas = [
('^Everything$', {'*': True})
]
createrepo_database = True
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck',
'--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
# CHECKSUMS
media_checksums = ['sha256']
--
2.20.1
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: How do we turn zchunk on for updates-testing for F30?
On Sat, 2019-03-09 at 21:29 +0100, Mikolaj Izdebski wrote:
> On Sat, Mar 9, 2019 at 2:29 PM Jonathan Dieter
> wrote:
> > Hey, I just noticed that, while we have zchunked metadata for the
> > F30
> > base repository, it's not enabled to for updates-testing.
> >
> > I've looked in the ansible repo and in pungi, but I can't see where
> > createrepo_c is actually called for updates-testing. Can someone
> > please point me in the right direction?
>
> createrepo for updates-testing is ran by pungi. I believe you need to
> enable zchunk in pungi.conf (createrepo_extra_args option). For
> non-modular updates-testing the config is
> roles/bodhi2/backend/templates/pungi.rpm.conf.j2 in ansible.git.
> Similarly, for modular equivalent, pungi config is located at
> roles/bodhi2/backend/templates/pungi.module.conf.j2
Thanks for pointing me in the right direction. I think I've got it,
complete with a conditional so we don't start generating zchunk
metadata for F29 updates.
There doesn't seem to be a way to generate pull requests on
https://infrastructure.fedoraproject.org/cgit/ansible.git, so I'm
attaching the support as a patch. If there's a better way for me to
send it in, please let me know.
Jonathan
P.S. It may be a small and simple patch, but I haven't actually tested
it and am not sure how to go about doing so.
From 17eefaa1cefb624afa0cf95d04e7f337ba70cb42 Mon Sep 17 00:00:00 2001
From: Jonathan Dieter
Date: Sat, 9 Mar 2019 22:52:48 +
Subject: [PATCH] Add zchunk support to updates and updates-testing
repositories
This adds zchunk support for the updates and updates-testing repositories
for both rpms and modularity
Signed-off-by: Jonathan Dieter
---
roles/bodhi2/backend/templates/pungi.module.conf.j2 | 3 +++
roles/bodhi2/backend/templates/pungi.rpm.conf.j2| 3 +++
2 files changed, 6 insertions(+)
diff --git a/roles/bodhi2/backend/templates/pungi.module.conf.j2 b/roles/bodhi2/backend/templates/pungi.module.conf.j2
index bb021eb13..7dad35403 100644
--- a/roles/bodhi2/backend/templates/pungi.module.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.module.conf.j2
@@ -59,6 +59,9 @@ greedy_method = 'build'
createrepo_c = True
createrepo_checksum = 'sha256'
createrepo_deltas = False
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
#jigdo
create_jigdo = False
diff --git a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2 b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
index 8d9e9a3f2..020736aee 100644
--- a/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
+++ b/roles/bodhi2/backend/templates/pungi.rpm.conf.j2
@@ -66,6 +66,9 @@ createrepo_deltas = [
('^Everything$', {'*': True})
]
createrepo_database = True
+[% if release.version_int >= 30 %]
+createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/f[[ release.version_int ]]']
+[% endif %]
# CHECKSUMS
media_checksums = ['sha256']
--
2.20.1
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: How do we turn zchunk on for updates-testing for F30?
On Sat, 2019-03-09 at 09:43 -0500, Neal Gompa wrote: > On Sat, Mar 9, 2019 at 8:28 AM Jonathan Dieter > wrote: > > Hey, I just noticed that, while we have zchunked metadata for the > > F30 > > base repository, it's not enabled to for updates-testing. > > > > I've looked in the ansible repo and in pungi, but I can't see where > > createrepo_c is actually called for updates-testing. Can someone > > please point me in the right direction? > > > > Updates repos are handled by Bodhi, so you'll want to look there. If I'm reading the code correctly looks like Bodhi creates the repodata using pungi. Currently, fedora.conf in the pungi-fedora repo is set to create zchunk metadata for the branches f30 and master. Could updates- testing maybe be using a different branch? Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
How do we turn zchunk on for updates-testing for F30?
Hey, I just noticed that, while we have zchunked metadata for the F30 base repository, it's not enabled to for updates-testing. I've looked in the ansible repo and in pungi, but I can't see where createrepo_c is actually called for updates-testing. Can someone please point me in the right direction? Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Shredding a removable drive (OT)
On Fri, 2019-01-25 at 20:24 -0800, Gordon Messmer wrote: > On 1/25/19 6:45 PM, Jonathan Ryshpan wrote: > > Is there a quicker way to protect my data when I give the drives away, > > other than smashing the drives to bits? > > The quickest would be to encrypt the drives from the beginning. When you > want to discard the drives, you just need to wipe the LUKS header. With > the key wiped out, the rest of the drive is just random bits. This, 100 times over! Jonathan ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Call for zchunk repodata testers
On Thu, 2019-01-17 at 00:48 +0100, Björn 'besser82' Esser wrote: > Am Mittwoch, den 16.01.2019, 22:08 + schrieb Jonathan Dieter: > > Just a heads up Rawhide has had zchunked metadata for almost three > > weeks, and I'd greatly appreciate some more testing on the client side > > before we finish pushing the client changes to Rawhide. > > > > … > > > > If the second day's repository download size for Rawhide is less than > > 54MB, you'll know everything's working correctly. > > I've just tested updating from yesterday's metadata and it took me about > 2.9 MBytes (vs. about 61 MBytes without zchunk support) to download. > > Seems to be working as it should. LGTM =) Thanks for the feedback! That sounds about right for a normal day's updates. Jonathan signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Call for zchunk repodata testers
Just a heads up Rawhide has had zchunked metadata for almost three weeks, and I'd greatly appreciate some more testing on the client side before we finish pushing the client changes to Rawhide. If you're running Rawhide, are willing to test, and have backups of libdnf and librepo, please enable the COPR by running: dnf copr enable jdieter/dnf-zchunk dnf update librepo libdnf At this point all tools using libdnf will default to zchunked metadata if it's available. If the second day's repository download size for Rawhide is less than 54MB, you'll know everything's working correctly. --- Details, feel free to ignore --- I've been testing it on Rawhide for the last three weeks or so, and the minimum I've downloaded has been about 1.2MB (updating every day), while the max today was about 7.9MB (haven't updated for over a week). So far, I haven't hit any bugs, but I've only been regularly testing DNF in the terminal. I did do initial testing with PackageKit and microdnf, and they worked fine too. Currently only primary.xml, filelists.xml and other.xml are zchunked, but I've just finished extending zchunk support to all metadata types in createrepo_c, and the changes should be making their way to Rawhide sometime in the not-to-distant future. Again, a huge thank you to everyone who's helped with this, and especially to Daniel Mach for reviewing the large invasive patches required to make all this work, Neal Gompa for helping push this change, and Kevin Fenzi for flipping the switch to start generating zchunked metadata in Rawhide. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Enabling zchunk metadata generation in F30
On Fri, 2018-12-14 at 15:15 -0500, Randy Barlow wrote: > On Fri, 2018-12-14 at 19:02 +0000, Jonathan Dieter wrote: > > Hey Randy, at the moment the --zck option *only* applies to > > primary.xml, filelists.xml and other.xml. It should be pretty > > straightforward to add it to the others, but I wanted to get those > > three working first. > > Cool, sounds good to me. > > > As for python bindings, they can read zchunk metadata just fine, but I > > don't think I hooked up creating the metadata. Where exactly does it > > generate updateinfo in bodhi? I'd like to see how the function is > > used so I can implement it. > > Bodhi's updateinfo code lives here: > > https://github.com/fedora-infra/bodhi/blob/3.11.3/bodhi/server/metadata.py Thanks for this. I'll take a look at it and see what it will take to make zchunk generation work in there. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Enabling zchunk metadata generation in F30
On Fri, 2018-12-14 at 19:24 +, Jonathan Dieter wrote: > On Fri, 2018-12-14 at 11:13 -0800, Kevin Fenzi wrote: > > On 12/14/18 10:52 AM, Jonathan Dieter wrote: > > > I suspect that the maintainers would like to see this feature tested > > > more before pushing it to F29, but I can ask them, if you'd like. > > > > No, I am not suggesting we implement it now in F29, I am saying that the > > rawhide-composer and bodhi-backend01 machines that run pungi are Fedora > > 29 hosts. The rawhide-composer runs in a chroot, so that should just > > work, but the bodhi-backend01 updates pungi I don't think does, or if it > > does it's a f29 chroot, not rawhide. So we will need a build for it. > > Ok, makes sense. While I'm thinking about it, fedora-repo-zdicts was just added to Fedora yesterday and hasn't even been pushed to the testing repositories yet. It should be available in Rawhide though... Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Enabling zchunk metadata generation in F30
On Fri, 2018-12-14 at 11:13 -0800, Kevin Fenzi wrote: > On 12/14/18 10:52 AM, Jonathan Dieter wrote: > > On Thu, 2018-12-13 at 16:42 -0800, Kevin Fenzi wrote: > > > Cool. > > > > > > I see the new createrepo_c only has a rawhide build... any chance for a > > > f29 update? Or should we look at building a newer in our infra repo? > > > > I suspect that the maintainers would like to see this feature tested > > more before pushing it to F29, but I can ask them, if you'd like. > > No, I am not suggesting we implement it now in F29, I am saying that the > rawhide-composer and bodhi-backend01 machines that run pungi are Fedora > 29 hosts. The rawhide-composer runs in a chroot, so that should just > work, but the bodhi-backend01 updates pungi I don't think does, or if it > does it's a f29 chroot, not rawhide. So we will need a build for it. Ok, makes sense. > > If you do a F29 rebuild for the infra repo, you'll want to make sure to > > pass --with zchunk to rpmbuild, as it defaults to off for anything F29 > > and below. > > Well, we cannot do that with a koji build, but I guess we could just > change the source. It's just a conditional in the spec that's currently set to off for <= F29, so easy enough to change. > Anyhow, perhaps we just target rawhide for now... That works for me. Thanks so much! If there's anything else I can do to help with this, please let me know. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Enabling zchunk metadata generation in F30
On Fri, 2018-12-14 at 11:20 -0500, Randy Barlow wrote: > On Thu, 2018-12-13 at 22:56 +0000, Jonathan Dieter wrote: > > The call to createrepo_c or mergerepo_c > > (whichever is run last to generate the final metadata) would need to > > be > > run with the new zchunk arguments: > > > > --zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/f30 > > Hey Jonathan! > > Bodhi uses createrepo_c both through pungi (to create the bulk of the > repository) and through the createrepo_c Python bindings (to generate > the updateinfo.xml file). Is there a way to ask the Python bindings to > do this? The Fedora 29 updateinfo.xml file looks like it's only about 1 Hey Randy, at the moment the --zck option *only* applies to primary.xml, filelists.xml and other.xml. It should be pretty straightforward to add it to the others, but I wanted to get those three working first. As for python bindings, they can read zchunk metadata just fine, but I don't think I hooked up creating the metadata. Where exactly does it generate updateinfo in bodhi? I'd like to see how the function is used so I can implement it. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Enabling zchunk metadata generation in F30
On Thu, 2018-12-13 at 16:42 -0800, Kevin Fenzi wrote: > On 12/13/18 3:34 PM, Jonathan Dieter wrote: > > On Thu, 2018-12-13 at 15:12 -0800, Kevin Fenzi wrote: > > > pungi calls createrepo_c for us (in both rawhide/branched and updates) > > > so we need a pungi patch (probibly with a config option?) to enable > > > this. If it's added as a optional thing we would need to add that > > > setting to our pungi-fedora config and set it to on. > > > > > > Can you file a pungi issue on that? > > > > I've just checked the pungi issues, and it looks like Lubomír took care > > of this at Flock last summer by adding an arbitrary createrepo_c > > commands option: createrepo_extra_args > > > > I've done a PR for the pungi-fedora config here, but it's untested and > > I'm not sure if I did the variable substitution correctly. > > > > https://pagure.io/pungi-fedora/pull-request/678 > > Cool. > > I see the new createrepo_c only has a rawhide build... any chance for a > f29 update? Or should we look at building a newer in our infra repo? I suspect that the maintainers would like to see this feature tested more before pushing it to F29, but I can ask them, if you'd like. If you do a F29 rebuild for the infra repo, you'll want to make sure to pass --with zchunk to rpmbuild, as it defaults to off for anything F29 and below. Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Orphaning pykka
On Wed, 2018-12-12 at 06:10 +, Raphael Groner wrote: > > I've just orphaned pykka ( > > https://admin.fedoraproject.org/pkgdb/package > > /rpms/pykka/) as I'm no longer using it. > > Hi Jonathan, > what do you use instead? > Regards, Raphael MPD with local music. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Enabling zchunk metadata generation in F30
On Thu, 2018-12-13 at 15:12 -0800, Kevin Fenzi wrote: > pungi calls createrepo_c for us (in both rawhide/branched and updates) > so we need a pungi patch (probibly with a config option?) to enable > this. If it's added as a optional thing we would need to add that > setting to our pungi-fedora config and set it to on. > > Can you file a pungi issue on that? I've just checked the pungi issues, and it looks like Lubomír took care of this at Flock last summer by adding an arbitrary createrepo_c commands option: createrepo_extra_args I've done a PR for the pungi-fedora config here, but it's untested and I'm not sure if I did the variable substitution correctly. https://pagure.io/pungi-fedora/pull-request/678 Jonathan signature.asc Description: This is a digitally signed message part ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Enabling zchunk metadata generation in F30
Createrepo_c in F30 has finally grown zchunk support and I've packaged up some zdicts that we can use for F30/rawhide, so I'd love to see us start building zchunk metadata for F30. To enable zchunk metadata generation, whichever systems are running createrepo_c/mergerepo_c for Rawhide would need createrepo_c-0.12 and fedora-repo-zdicts installed. The call to createrepo_c or mergerepo_c (whichever is run last to generate the final metadata) would need to be run with the new zchunk arguments: --zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/f30 Mergerepo doesn't require zchunk metadata in the source repositories to be able to generate zchunk metadata for the merged repository. I'm not sure who to ask to turn these flags on, so if there's an individual I need to ping, please point me in the right direction. A huge thank you to Daniel Mach for reviewing and merging the createrepo_c zchunk pull request, Jaroslav Mracek for building createrepo_c with zchunk, Robert-André Mauchin for reviewing fedora- repo-zdicts and Neal Gompa for keeping things moving forward with the PR reviews. Thanks, Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Best path for Fedora zchunk dictionaries
I'm currently in the process of packaging up fedora-repo-zdicts[1], a package which will contain the zchunk dictionaries for all active Fedora releases. When running createrepo_c or mergerepo_c with zchunk support, the directory containing the zdicts is passed in and createrepo_c will choose the right zdict for each metadata file. The form of that directory is /usr/share/fedora-repo-dicts/ where is the release that the metadata is being generated for. My question is what should actually look like. This is very Fedora specific, so I want to choose whatever the easiest variable is for infra to pass to createrepo_c or mergerepo_c. Currently is set to PLATFORM_ID in /etc/os-release (so, platform:fedora-30 for Rawhide), but that's probably overly generic. What would be a better pattern for ? fedora-30? f30? Just 30? Jonathan ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Mon, 2018-12-03 at 22:16 +0100, Jan Pokorný wrote: > On 16/11/18 22:07 +0000, Jonathan Dieter wrote: > > The core idea behind zchunk is that a file is split into independently > > compressed chunks and the checksum of each compressed chunk is stored > > in the zchunk header. When downloading a new version of the file, you > > download the zchunk header first, check which chunks you already have, > > and then download the rest. > > Just one more thought I reliazed in hindsight, there are ways to cut down > the installed files in RPM ecosystem, currently with a request to omit > documentation (%doc tagged files, see --nodocs/--excludedocs). > > Indeed, that's a sort of files you can usually omit without hesitation > in containers/VMs. Perhaps there are some more bits that are de facto > optional without losing anything from the functionality. > > So with clever separation, such bits wouldn't even need to be downloaded > when they will not eventually make it to the disk. That might make > things like customizing a base container image tiny bit more swift, > e.g. in CI/CD context without many connectivity guarantees (up to > mirrors anyway). But might not be worth it if the trade-off > is already predictably suboptimal in other aspects. I think this is very interesting and definitely feasible (assuming the core concept of zchunked rpms is reasonable). On a tangent, I realized something else about metadata generation (and I think someone else had picked up on it, but it hadn't quite registered with me). Currently generating metadata for RPMs involves reading the full RPM to calculate the checksum. With zchunk, the header checksum is stored at the beginning of the file and is all you need to verify a zchunk file. Jonathan signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Thu, 2018-11-22 at 11:31 -0500, Josh Boyer wrote: > I'm concerned that this will effectively render EL RPM unable to > handle any Fedora RPMs at all. That's both a practical concern, as > many people develop Fedora using EL and vice versa, and also a broader > ecosystem concern. I would very much like for all of our > distributions to be able to more easily operate together, and this > effectively forks Fedora off into it's own space yet again. For what very little it's worth, a zchunked rpm is still a valid zchunk file, and you would be able to easily extract the payload and the header from a zchunked rpm on an EL system. In fact, because we're not planning to change the rpm header format at all, there could easily be a tool to convert a zchunked rpm into an xz or gzipped rpm (and vice versa). Having said that, there's a huge difference between this and having EL's RPM actually be able to read Fedora's RPMs. > Have we really looked at the wider scope of what a format change like > this would do in the context of some of the larger picture things > we're working on with lifecycle and cross-distro collaboration > efforts? I agree this would be better than delta RPMs when looking at > that *specific* usecase, but improving that (even with compose time > benefits) by doing a format change seems to be inflicting a very high > cost for what is an important but relatively small usecase. This is a really good point and I don't know know the answer. As per Neal's suggestion earlier in the thread, I posted this to the rpm- ecosystem mailing list. Michael from SUSE brought up some very valid concerns about how well zchunk would compare with deltarpm in delta efficiency, but apart from that, it's been pretty quiet there. If it's already obvious that the cost for this proposal isn't worth the gain, I do completely understand. If we're not sure, I'll do a proof- of-concept that converts standard rpms into zchunked rpms, so we can compare sizes and deltas, and hopefully get some data points on speed. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Wed, 2018-11-21 at 14:36 +0100, Kamil Paral wrote: > On Fri, Nov 16, 2018 at 11:13 PM Jonathan Dieter wrote: > > For reference, this is in reply to Paul's email about lifecycle > > objectives, specifically focusing on problem statement #1[1]. > > > > > > Have rpm use zchunk as its compression format, removing the need for > > deltarpms, and thus reducing compose time. This will require changes > > to both the rpm format and new features in the zchunk format. > > > > Hey Jonathan, > > thanks for working on this. The proposed changes sound good to me. > I'm a bit worried that zchunk is not yet a proven format, so it might > be a good idea to use it for metadata first, see whether it works as > expected, and then push it for RPM files. But that's for more > technical people to judge. > > I have some concrete questions, though: > 1. I have noticed that especially with large RPMs (firefox, chrome, > atom, game data like 0ad-data, etc), my PCs are mostly bottlenecked > by CPU when installing them. And that's with a modern 3.5+GHz CPU. > That's because RPM decompression runs in a single thread only, and xz > is just unbelievably slow. I wonder, would zchunk used as an RPM > compression algorithm improve this substantially? Can it decompress > in multiple threads and/or does it have much faster decompression > speeds (and how much)? I don't care about RPM size increase, but I'd > really like to have them installed fast. (That's of course just my > personal preference, but this also affects the speed of mock builds > and such, so I think it's relevant.) The zstd compression that zchunk uses internally is designed to be faster than even gzip at decompression. Currently zchunk is single- threaded, but, given that each chunk is independent, making it multi- threaded should be pretty trivial, and is on the todo list. > 2. In our past QA efforts in Fedora, we had use cases for retrieving > rpm header data without retrieving the actual content (the payload). > That was for cases when we needed to check e.g. dependency issues, > but the rpms were not placed in a repository yet (i.e. no easy access > to their metadata) and it was slow and wasteful to download the whole > rpm just to get the header. Will the new zchunk compression still > make it possible to retrieve just the header without accessing all > the payload data? (It would be great to make this accessible from > Python and not just C, but that's a plea I should direct to rpm > maintainers, I guess). The zchunk format supports the concept of multiple independent streams in a single file. A zchunk rpm would contain two streams, the rpm header and the rpm payload. Since downloading a zchunk file is two steps already (downloading the zchunk header, and then downloading the required chunks), it should be easy enough to download only the chunks needed for the rpm header stream. As for a python API, I would love for zchunk to have that too, but haven't had the time yet. I hope that helps. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Wed, 2018-11-21 at 11:31 -0500, Colin Walters wrote: > After having introduced a new format (OSTree) into the ecosystem here, > as well as working a lot on the Docker/OCI ecosystem, one thing I want > to emphasize is: > > A lot of Red Hat's customers don't connect their systems to the Internet, > they want easy offline mirroring. OSTree supports that, and it's > also possible to do with OCI images of course. > > But, a lot organizations use e.g. https://jfrog.com/artifactory/ > which today doesn't support OSTree (it does support RPM and Docker/OCI). > So any format break for RPM wouldn't be usable until Artifactory gains > support for it. And even after that happened you'd have in some > places a large lag time for it to be deployed. > > In general, any data format break is going to impose a lot higher > costs than you might imagine. Thanks for bringing up these points. You are undoubtedly correct that there's an unknown cost associated with these changes, but hopefully the cost will become a little clearer once we have a POC. > (Also on this topic, I should note that the OSTree data format cleanly > fixes a lot of the issues being discussed here; it has deltas, and also > doesn't make the mistake of checksumming compressed data, > when performing updates only changed files are rewritten, not to mention > a whole transactional update system, etc.) Yep. I've experimented with OSTree and love the concepts behind it. I don't think we're quite ready to ditch the classic rpm systems yet, though. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Tue, 2018-11-20 at 12:45 +, Michael Schroeder wrote: > On Mon, Nov 19, 2018 at 08:30:14PM +0000, Jonathan Dieter wrote: > > Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't > > require extra CPU usage on the client side as they don't go through the > > decompress-recompress cycle that deltarpms do. Re-assembling a zchunk > > file requires no compression or decompression. > > Btw, we can easily do that for deltarpms as well. We only recompress > because we want a rpm that is bit-identical to the remote one. > > Having a '-u' option that makes applydeltarpm write a rpm with an > uncompressed payload and no payload signatures is just a couple of > lines of code. But the problem is that you would lose the signatures. To make this work, we would need to create signatures of both the compressed and uncompressed rpm (which wouldn't be a bad idea). Is there some way we could (ab)use the current rpm format to make this work, or would it be a backwards-incompatible change? Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
Hey, thanks for the detailed analysis. Comments inline. On Tue, 2018-11-20 at 00:47 +0100, s...@gmx.com wrote: > Based on work I've done in this area, I'm somewhat sceptical that this > will work out well. I spent a decent amount of time comparing various > approaches to data saving for both compressed data and regular binaries. > This was done a few months ago, so a few of the algos may have changed a > little since then, but I doubt the changes will be huge. Do you have some examples of the data you tested? > > The various things I tested were, in no particular order: > * xdelta3 (and a few similar VCDIFF based algos) > * zstd > * zchunk > * bsdiff > * courgette > * zucchini > And a few others which aren't interesting enough to mention here. > > xdelta3 is old now, but was the standard binary delta compression tool > for a long time. Of the better performing algos it has reasonably good > generation time, and works reasonably well on compressed data (still a > good way behind the top of the pile), but falls far behind on binaries. > > zstd was generally disappointing for both compressed and binary data. It's > really a compression format, not a delta generator, and while it performed > very well compared to traditional compression formats, it was unable to > compete with other tools here. > > zchunk had basically the same problems as zstd, except that the chunking > overhead made files even larger, and small symbol changes could mean that > a very large number of chunks needed to be transmitted. > > bsdiff is a larve improvement over xdelta3 in terms of final size for > binary data. It is considerably more expensive in terms of memory and cpu > to compute, but is fast to apply[0]. It works much less well on compressed > data, and really needs to work on a binary to be at its best. > > courgette is excellent at reducing binary size, and still very good at > compressed data. By data size alone it's the best of all of these, but is > somewhat complex and expensive, particularly in terms of memory. An > over view of how it works is [1] > > zucchini is an experimental delta generation tool for chromium. I can't > see much that's been published externally about it, but I found it to > compress almost as well as courgette but with greatly reduced memory use. > Code is very much a moving target, but is located at [2]. These seem to be a bit apples-to-oranges. xdelta3, bsdiff, courgette and zucchini are designed to generate deltas between two specific files. zstd is just a compression format, while zchunk is designed to download the smallest difference between two arbitrary files. You are absolutely right about the cost of small symbol changes, one of the reasons that courgette does some amount of disassembly and that bsdiff uses what it calls an "add block". Because of zchunk's design, there's no way it can benefit unless we implement some kind of disassembly, and I'm leery of going down that road. > Overall in my tests zstd/schunk gave massively worse compression compared > to modern delta algorithms (courgette, zucchini), and the total time to > download, apply, install was always slowest for zstd based approaches. I'd love to see your test methodology. I'm not surprised about the difference in delta size (which really isn't the same as compression), but I am a bit surprised by the speed differences you saw, assuming your data was compressed. If your data wasn't, do remember that rpms are compressed, so if you're creating a new delta method, it will have to decompress both the old and new rpms before generating the delta (as deltarpm does). zchunk is able to get around this extra step because it is the compression format. > So, then, the only time this would work is if the changed-chunk detection > feature of zchunk actually works effectively for RPMs. Unfortunately, > when binaries change (and when RPMs as a whole change) it's not unusual > for this to manifest as adding/removing significant chunks of data from > near the beginning of the file, which means that the chunk matching fails > and you end up with a huge and slow download. Thats... not how the chunk matching works. The chunking function uses the buzhash algorithm to try to break on consistent patterns, no matter where data is added or removed. If you remove x bytes from the beginning of the file and then add y bytes, at most one or two chunks should be affected. > If you care only about making the 50th %ile case better, then zchunk is > probably at least not any worse than what we have now. However this > will, I believe, come at the cost of making the 95th %ile case pretty > unpleasant for end users. > > I think it'd be far better to explore Howard's proposal, for per-file > delta generation (as opposed to per-RPM), and use modern delta > generation (probably courgette) to compute the delta. Maybe I misunderstood Howard's proposal, but I understood him to be suggesting that rebuilding a full rpm
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Mon, 2018-11-19 at 16:29 -0500, Simo Sorce wrote: > On Mon, 2018-11-19 at 21:02 +0000, Jonathan Dieter wrote: > > On Mon, 2018-11-19 at 15:18 -0500, Simo Sorce wrote: > > > I do not think you can just trust random metadata somewhere, one of the > > > points of a rpm reinstall is to fix damaged files for example. It does > > > no good if you skip those because some file somewhere says they are > > > "OK". (If I understood your comment about "just downloading changed > > > chunks). > > > > Yes, this is the crux of the problem. As I see it, dnf should verify > > the checksums on the local files before downloading the missing chunks, > > but that doesn't guarantee that the data won't be changed between the > > download step and the install step. RPM would also need to verify the > > checksums before starting the install phase, and would need to bail out > > if the checksums had changed. > > > > My biggest concern, though, is what happens if package A needs a > > specific chunk in /usr/bin/foo and package B changes /usr/bin/foo while > > being installed. The chunk was there when the install phase started, > > but disappeared before package A was actually installed. > > Is this different in a normal install ? > What if package A installs /usr/bin/foo and then package B overwrites > it ? > > Or are you concerned about the case where there may be an identical > chunk in different files ? Are chunks "global" to the host ? This. If we stored the checksums in the rpm database, then, yes they would be global to the host. > This problem could be addressed by copying all uncompressed chunks in a > staging area before installing the rpm, failing in a clean way (ie not > half way through a package install). The penalty is the need for enough > space to copy the uncompressed files though. more clever things could > be done with proper filesystem support and snapshotting and copy-on- > write, but not sure it is worth optimizing for what is normally a > relatively small scratch area (if you do it one package at a time > only). What about just copying any uncompressed chunks required for the current package or any packages still in the install queue? That might reduce the scratch area even further. > > > 2) what are the chunks sizes ? > > > > The chunk sizes vary because you don't want inserting or removing a few > > bytes to completely change all the following chunks. The current > > default average size is 32KB, but that can be adjusted. > > Is this a compromise between compression performance and granularity ? > Anything else went into the decision to settle around 32k ? > Some filesystems seem to gravitate around 64k extents so I am > wondering. Yes, this is just a compromise. The larger the chunk size, the larger the delta you need to download, but the better the compression. We could experiment with this to see if 64k would give us significantly better compression. I would also chunk on file borders in the rpm payload, so we don't end up having a chunk span multiple files. That would get messy fast when trying to rebuild from local files. > > > Finally what signature scheme where you planning to use ? And how do > > > you deal with the data you want to "exclude" from signing, omit it or > > > feed in blank "sectors" ? > > > > I was planning to use GPG signatures, and was planning to just omit the > > data I want excluded. Having said that, while the format supports > > signatures, the code hasn't been written and if either of those answers > > are bad/dangerous, we can change that. > > We use GPG signatures right now, can't be any more dangerous than that > :-) > > The omission vs blanking has no ill effect, but was not explicitly > mentioned, it should. Esp around places where the missing data is in > the middle of a "structure" in your diagrams, or it may be ambiguous > and lead to incompatible implementations if someone is ever going to > build another (and if zchunk is going to be adopted in rpm I bet there > will be some other implementation to do some crazy thing :-) Yep. Let me clarify that in the format definition (and add the new checksum types, I noticed they're missing). Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Mon, 2018-11-19 at 22:18 +0100, Nicolas Mailhot wrote: > Le lundi 19 novembre 2018 à 20:30 +0000, Jonathan Dieter a écrit : > > The zchunk advantage over deltarpms is much less CPU usage, while its > > disadvantages are slightly larger network usage and increased disk > > usage. > > Unfortunately, that's a bad compromise for most limited clients. A > limited client can trade time for CPU or network performance, swap for > memory. What it can absolutely not make more of is storage, both install > and staging storage. Install storage requirements do not depend on rpm > tech level, but will generally go up over a system lifetime, adding > pressure on staging storage. > > So you absolutely need to keep staging storage on par or less than > existing rpm/dnf if you do not want to obsolete classes of Fedora > hardware. > > And Google released a huge quantity of cheap storage-deficient hardware > with its chromebooks. People do install Fedora on those. The only way we can keep staging storage down (and it would actually be less than deltarpm/normal rpm) is to use the local chunks at install time rather than download time. This comes with its own risks though, see the other emails in this thread, specifically the ones following Jan Pokorný's message. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Mon, 2018-11-19 at 21:14 +, Tom Hughes wrote: > On 19/11/2018 20:30, Jonathan Dieter wrote: > > > On the client: > > The zchunk advantage over regular rpm is decreased network usage, while > > its disadvantage is increased disk usage (since the local chunks will > > be uncompressed). > > > > The zchunk advantage over deltarpms is much less CPU usage, while its > > disadvantages are slightly larger network usage and increased disk > > usage. > > > > Note that for most users the increased disk usage is temporary, since > > rpms are deleted after install. > > If they're deleted after install then surely next time there is an > update there won't be any local chunks and everything will have to > be downloaded? > > That's what has been confusing me about this whole thing - as I > understand it the idea is to only download new chunks and to reuse > chunks that are unchanged from earlier revisions, but it seems > that doing that would require keeping a local copy of every > installed rpm which would be a big change that nobody seems to > have mentioned. The idea is to use the locally installed files as the local chunks, the same way that deltarpm does. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Mon, 2018-11-19 at 15:18 -0500, Simo Sorce wrote: > On Mon, 2018-11-19 at 19:58 +0000, Jonathan Dieter wrote: > > That's an interesting thought. I was picturing using the zchunk > > library in the dnf download stage to build a local rpm from the > > verified locally installed files and the downloaded changed chunks, > > but, if I understand your suggestions correctly, you're saying we > > could > > just download the changed chunks and have RPM automatically get the > > rpm-integrity verified chunks during the *install* stage. > > How do you know which chunks to download w/o having a stored (or > recomputed) list of existing chunks ? I thought we should store the chunk checksums of installed files in the rpm database. Something like file, offset, length, checksum type, checksum? > > The advantage of this method is that you don't need to store the local > > data twice, but the danger is that the local files get changed > > elsewhere during the install process. > > > > It's an interesting thought, though, and I wonder if there's a way we > > could work around that danger? > > I do not think you can just trust random metadata somewhere, one of the > points of a rpm reinstall is to fix damaged files for example. It does > no good if you skip those because some file somewhere says they are > "OK". (If I understood your comment about "just downloading changed > chunks). Yes, this is the crux of the problem. As I see it, dnf should verify the checksums on the local files before downloading the missing chunks, but that doesn't guarantee that the data won't be changed between the download step and the install step. RPM would also need to verify the checksums before starting the install phase, and would need to bail out if the checksums had changed. My biggest concern, though, is what happens if package A needs a specific chunk in /usr/bin/foo and package B changes /usr/bin/foo while being installed. The chunk was there when the install phase started, but disappeared before package A was actually installed. > A couple more questions. > I skimmed quickly at the format and I have two questions I did not > immediately see an answer for. > 1) why are you still supporting SHA-1 in a new format ? Zchunk cares about two types of checksums, the chunk checksums, used to determine if two chunks are the same, and the full data checksum (which currently defaults to SHA-25), used to actually validate the data. Originally, SHA-1 was supposed to be used *only* for the chunk checksums, but, somewhere along the way, it was pointed out that using the first 128 bits of a SHA-512 hash would be faster and more secure, so the default for the chunk checksums is now SHA-512/128. The only reason SHA-1 support is still in zchunk is because I don't want to break backwards compatibility for the (probably five) zchunk files created before this change. Having said that, zchunked rpms won't be able to depend on the full data checksum (because the local chunks will be uncompressed), so we'd need to use SHA-256 at minimum for the chunk checksums. > 2) what are the chunks sizes ? The chunk sizes vary because you don't want inserting or removing a few bytes to completely change all the following chunks. The current default average size is 32KB, but that can be adjusted. > Sorry if this is already answered somewhere. > > Finally what signature scheme where you planning to use ? And how do > you deal with the data you want to "exclude" from signing, omit it or > feed in blank "sectors" ? I was planning to use GPG signatures, and was planning to just omit the data I want excluded. Having said that, while the format supports signatures, the code hasn't been written and if either of those answers are bad/dangerous, we can change that. > Thanks for any answer. > Simo. Thank you for looking at this! Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Sun, 2018-11-18 at 13:19 -0500, Stephen John Smoogen wrote: > On Sun, 18 Nov 2018 at 12:49, Neal Gompa wrote: > > On Sun, Nov 18, 2018 at 11:54 AM Jonathan Dieter > > wrote: > > > On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: > > > > Jonathan Dieter wrote: > > > > > My proposal would be to make zchunk the rpm compression > > > > > format for > > > > > Fedora. > > > > > > > > Given that: > > > > 1. zchunk is based on zstd, which is typically less efficient > > > > in terms of > > > >compression ratio than xz, depending on settings > > > >(see, e.g., https://github.com/inikep/lzbench), and > > > > 2. zchunk can by design only compress chunks individually and > > > > not benefit > > > >from the space savings of a solid archive with a global > > > > dictionary, > > > > I fear that this is going to significantly increase the size of > > > > the RPMs, > > > > which matters: > > > > * for the initial downloads, > > > > * for storage (e.g., keepcache=1, local mirrors, etc.), and > > > > * for the people not using deltas for whatever reason. > > > > > > > > I think zchunk makes a lot of sense for the metadata, but I am > > > > not convinced > > > > that it is the right choice for the RPMs themselves. > > > > > > I suspect the first is true, but zchunk does actually allow for a > > > global (per-file) dictionary that can be used to compress each > > > chunk. > > > The difficulty is that the dictionary has to stay the same > > > between file > > > versions, or the chunk checksums won't match. There would have > > > to be > > > some thought put into how to generate and store the dictionaries. > > > > > > As for how much bigger a zchunked rpm will be compared to an xz > > > rpm, at > > > the moment it's a bit hand-wavy. Based on zchunked repodata work > > > I've > > > done, I think we might be looking at a size that's slightly > > > smaller > > > than a gzipped rpm. I won't know for sure until I put together a > > > proof-of-concept, but I want to make sure that there aren't any > > > gaping > > > holes in the proposal before I do that. > > > > > > > I did some work several months ago to evaluate zstd compression for > > RPMs for Fedora, because of the lower memory and CPU usage for > > (de)compression. However, the average size increase from xz was > > pretty > > large (~20% or more on average, and nothing ever was either the > > same > > or smaller), even with heavier compression settings. That might > > have > > changed a bit with newer zstd releases that offer some more > > tunables, > > but I think it'll remain a tough sell on disk space. > > So there are at least 4 legs here: > CPU usage (in both uncompression install and deltarpm) > Memory usage per transaction > Network amount > Disk amount > > I expect that the best we are going to get in any 'improvement' is > going to be 3 out of the 4. The xz compression and delta-rpm has a > cpu/memory tradeoff for disk and network in comparison to gzip but it > is mostly acceptable if you have fairly modern desktops. However for > older hardware or lower power systems that tradeoff may not be good. Just to be clear on this, unlike deltarpm, zchunked rpms shouldn't require extra CPU usage on the client side as they don't go through the decompress-recompress cycle that deltarpms do. Re-assembling a zchunk file requires no compression or decompression. On the client: The zchunk advantage over regular rpm is decreased network usage, while its disadvantage is increased disk usage (since the local chunks will be uncompressed). The zchunk advantage over deltarpms is much less CPU usage, while its disadvantages are slightly larger network usage and increased disk usage. Note that for most users the increased disk usage is temporary, since rpms are deleted after install. In our infrastructure: The zchunk advantage over deltarpms is that they are created in the build stage and shouldn't take any longer than building a normal rpm, while deltarpms take quite a while to build. The disadvantage is that our current rpms use xz compression which is more efficient at compressing a whole file than zchunk is, so zchunked rpms will require more disk space. Hope that helps clarify things. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Mon, 2018-11-19 at 20:16 +0100, Jan Pokorný wrote: > On 19/11/18 13:13 +0100, Nicolas Mailhot wrote: > > Le 2018-11-19 12:28, Martin Kolman a écrit : > > > > > Many people might think RAM would not be an issue in 2018, but in > > > practice there are > > > and likely always will be memory constrained installation targets, > > > such as massive deployments > > > of "small" VMs or the IoT use cases mentioned above. > > > > Sure, that’s the artificial small vm case > > > > The average old/limited hardware is limited in memory, cpu and storage. > > Therefore if you have one factor to sacrifice it's cpu time because you can > > always let the CPU run a little longer, but a limited system won't magically > > grow more memory or more storage. > > > > Storage would not be such a problem is dnf was smart enough to auto > > partition big upgrades in lots of small partial upgrades, before downloading > > gigs of data that do not fit on disk. > > https://bugzilla.redhat.com/show_bug.cgi?id=1609824 > > Also, not familiar with zchunk way of doing things, but couldn't > rpm-integrity-verified installed files be mapped back to "chunks" > to further aleviate space concerns for the machine receiving > updates in some cases? That's an interesting thought. I was picturing using the zchunk library in the dnf download stage to build a local rpm from the verified locally installed files and the downloaded changed chunks, but, if I understand your suggestions correctly, you're saying we could just download the changed chunks and have RPM automatically get the rpm-integrity verified chunks during the *install* stage. The advantage of this method is that you don't need to store the local data twice, but the danger is that the local files get changed elsewhere during the install process. It's an interesting thought, though, and I wonder if there's a way we could work around that danger? Jonathan signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Sun, 2018-11-18 at 07:02 +, Leigh Scott wrote: > +1 to anything to rid me of deltarpms, I currently have to disable > this lame default. The irony is that getting deltarpms into Fedora was largely my fault. ;) Sorry, Leigh. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Mon, 2018-11-19 at 14:57 +0100, Miroslav Suchý wrote: > Dne 16. 11. 18 v 23:07 Jonathan Dieter napsal(a): > >1. Downloading a new release of a zchunked rpm would be larger than > > downloading the equivalent deltarpm. This is offset by the fact > > that the client is able to work out which chunks it needs no matter > > what the original rpm is, rather than needing a specific original > > rpm as deltarpm does. > > Does this means bigger requirements for copies of RPMs too? I mean > for all our repos mirros, for RH Satellites, Spacewalk, Koji, Copr, > Retrace... Yes, and that should actually be item 3 in drawbacks. Zchunked rpms will be larger than the current xz-compressed rpms, but the actual size increase is still unknown. I think we can shoot for roughly the same size as a gzip-compressed rpm, but I'm not sure. Mirror storage *might* actually go down, since we no longer need to store deltarpms, but anything that only stores rpms will definitely require more space. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Sat, 2018-11-17 at 22:30 +0100, Kevin Kofler wrote: > Jonathan Dieter wrote: > > My proposal would be to make zchunk the rpm compression format for > > Fedora. > > Given that: > 1. zchunk is based on zstd, which is typically less efficient in terms of >compression ratio than xz, depending on settings >(see, e.g., https://github.com/inikep/lzbench), and > 2. zchunk can by design only compress chunks individually and not benefit >from the space savings of a solid archive with a global dictionary, > I fear that this is going to significantly increase the size of the RPMs, > which matters: > * for the initial downloads, > * for storage (e.g., keepcache=1, local mirrors, etc.), and > * for the people not using deltas for whatever reason. > > I think zchunk makes a lot of sense for the metadata, but I am not convinced > that it is the right choice for the RPMs themselves. I suspect the first is true, but zchunk does actually allow for a global (per-file) dictionary that can be used to compress each chunk. The difficulty is that the dictionary has to stay the same between file versions, or the chunk checksums won't match. There would have to be some thought put into how to generate and store the dictionaries. As for how much bigger a zchunked rpm will be compared to an xz rpm, at the moment it's a bit hand-wavy. Based on zchunked repodata work I've done, I think we might be looking at a size that's slightly smaller than a gzipped rpm. I won't know for sure until I put together a proof-of-concept, but I want to make sure that there aren't any gaping holes in the proposal before I do that. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Sat, 2018-11-17 at 14:43 -0600, Chris Adams wrote: > Once upon a time, Jonathan Dieter said: > > The benefit of zchunked rpms is that, when downloading an updated rpm, > > you would only need to download the chunks that have changed from > > what's on your system. > > How well do web servers and caches handle range requests? I haven't > really paid attention to range requests in a long time; at one point > IIRC mirrors would often disable them because of "download accelerators" > that would open multiple connections to download parts of the same ISO > in parallel (hogging server resources). When I did the original POC testing, out of Fedora's 150 mirrors, 3 didn't support range requests at all and 3 supported a limited number of ranges in a single http request. Zchunk doesn't open extra connections to the server, but instead combines as many ranges as the server supports into a single request. Currently, if a server doesn't support ranges, zchunk will just download the full file, but this could be changed to try a different server. Jonathan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
On Sat, 2018-11-17 at 14:36 -0500, Neal Gompa wrote: > On Sat, Nov 17, 2018 at 1:15 PM Jonathan Dieter wrote: > > Neal, thanks so much for your thoughts on this. Responses inline: > > > > On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote: > > > > > If we're really considering changing the RPM file format, then we need > > > a proper discussion on rpm-maint@ and rpm-ecosystem@ mailing lists on > > > rpm.org. Can you please start a targeted discussion there? > > > > Sure. > > > > > But addressing the specific concrete suggestion here, there's a few > > > concerns I have: > > > > > > 1. This is a huge format break, which means that for the first time in > > > a _very_ long time, it would not be possible to reuse RHEL for Fedora > > > infrastructure _at all_. That's going to be a difficult problem. > > > There's a large legacy of systems that won't be able to handle that > > > new format, and unfortunately, rpm is not parallel installable in the > > > same manner as something like GCC or Python currently. Making it > > > parallel installable *is* possible (I've done it, and there have been > > > other attempts before), but it's not a supported thing. This is > > > probably the thing that would trigger a major version bump for RPM, > > > since it's a new archive format. > > > > Agreed, that this would be a massive format change and should therefore > > be a major version bump for RPM. New versions of RPM should still be > > able to read and install old-format rpms, but, as you point out, old > > versions of RPM won't be able to read or install new-format rpms. > > Unfortunately, I don't see any way around this. > > > > I don't think there's a way around it either. I just hope we do better > than the last time someone tried to do this... +1 > > > 2. This also means the _entire_ ecosystem of RPM archive parsers will > > > break. This is not particularly insurmountable, actually, as the RPM > > > file format was not particularly well documented, and a new format is > > > an opportunity to revisit some of those old issues and try to do a > > > better job this go around. But it's still a challenge to deal with. > > > > Yes, this is going to be quite a bit of work. > > > > > 3. When you refer to the rpm cpio, I assume you're referring to only > > > the archive payload, right? Typically the payload is what is > > > compressed, and the headers are not. It sounds like you're proposing > > > both aspects to be compressed, and compressed differently. If we made > > > the RPM header an uncompressed zchunk stream and the RPM payload a > > > zstd-compressed zchunk stream, would we be able to support fetching > > > header deltas for retrieving extra information on the fly? Say, for > > > example, attributes like arch color, filecap properties, and so on, > > > that aren't in the rpm-md data for things like transaction tests > > > without the whole RPM? > > > > Yes, I'm referring the the archive payload as the cpio. The zchunk > > format supports the idea of separate data streams, and I was planning > > to use that to put the headers in one stream and the archive payload in > > another. If the header chunks are first in the zchunk file, then they > > could be read without needing to read any of the rest of the file. > > And, yes, we could make the header stream uncompressed if that made it > > easier to parse. > > > > Whether it's compressed or not isn't terribly important, but what is > important is being able to validate the correctness before beginning > any processing, including decompression. Absolutely! This includes both the rpm header and the rpm archive data, and that's why we store both the compressed and uncompressed checksums of the chunks. > > > 4. I'd actually rather make it easier for the header streams to be > > > fetched instead of trying to make specific attributes easier in the > > > header payload. History has shown that any attempt at foresight here > > > tends to fail miserably, and common attributes are already specified > > > in the rpm-md primary.xml anyway, so if you're fetching the header to > > > retrieve an attribute, you *need* to do something weird anyway. > > > > The main purpose of putting separate attributes in the zchunk header is > > so programs like 'file' can determine some basic information about an > > rpm without needing to parse the full rpm header. This data would also > > be in the rpm header, so programs that read t
Re: Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
Neal, thanks so much for your thoughts on this. Responses inline: On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote: > If we're really considering changing the RPM file format, then we need > a proper discussion on rpm-maint@ and rpm-ecosystem@ mailing lists on > rpm.org. Can you please start a targeted discussion there? Sure. > But addressing the specific concrete suggestion here, there's a few > concerns I have: > > 1. This is a huge format break, which means that for the first time in > a _very_ long time, it would not be possible to reuse RHEL for Fedora > infrastructure _at all_. That's going to be a difficult problem. > There's a large legacy of systems that won't be able to handle that > new format, and unfortunately, rpm is not parallel installable in the > same manner as something like GCC or Python currently. Making it > parallel installable *is* possible (I've done it, and there have been > other attempts before), but it's not a supported thing. This is > probably the thing that would trigger a major version bump for RPM, > since it's a new archive format. Agreed, that this would be a massive format change and should therefore be a major version bump for RPM. New versions of RPM should still be able to read and install old-format rpms, but, as you point out, old versions of RPM won't be able to read or install new-format rpms. Unfortunately, I don't see any way around this. > 2. This also means the _entire_ ecosystem of RPM archive parsers will > break. This is not particularly insurmountable, actually, as the RPM > file format was not particularly well documented, and a new format is > an opportunity to revisit some of those old issues and try to do a > better job this go around. But it's still a challenge to deal with. Yes, this is going to be quite a bit of work. > 3. When you refer to the rpm cpio, I assume you're referring to only > the archive payload, right? Typically the payload is what is > compressed, and the headers are not. It sounds like you're proposing > both aspects to be compressed, and compressed differently. If we made > the RPM header an uncompressed zchunk stream and the RPM payload a > zstd-compressed zchunk stream, would we be able to support fetching > header deltas for retrieving extra information on the fly? Say, for > example, attributes like arch color, filecap properties, and so on, > that aren't in the rpm-md data for things like transaction tests > without the whole RPM? Yes, I'm referring the the archive payload as the cpio. The zchunk format supports the idea of separate data streams, and I was planning to use that to put the headers in one stream and the archive payload in another. If the header chunks are first in the zchunk file, then they could be read without needing to read any of the rest of the file. And, yes, we could make the header stream uncompressed if that made it easier to parse. > 4. I'd actually rather make it easier for the header streams to be > fetched instead of trying to make specific attributes easier in the > header payload. History has shown that any attempt at foresight here > tends to fail miserably, and common attributes are already specified > in the rpm-md primary.xml anyway, so if you're fetching the header to > retrieve an attribute, you *need* to do something weird anyway. The main purpose of putting separate attributes in the zchunk header is so programs like 'file' can determine some basic information about an rpm without needing to parse the full rpm header. This data would also be in the rpm header, so programs that read the rpm header wouldn't care about the attributes in the zchunk header. > 5. I'm not exactly sure what you mean by zchunk signing... The zchunk format supports signing, but just for the zchunk header. Because the header contains the checksums for each chunk, this establishes a chain of trust for verifying the whole file. Which brings me to... > 6. I'm wondering why we can't do a perfect reconstruction of the > original RPM, given two RPM sources that are both zchunked? We can > pull it off with repodata, so what's different about RPM that makes > that not doable? The problem is that, unlike the repodata, once an rpm is installed, the package file is deleted and the data is only available on the system in its uncompressed installed form. If we're trying to use that data to rebuild an rpm, we have two options. 1. Compress the data using the same method that was used to create the original rpm. This is what applydeltarpm does, and is why it's so heavy on the CPU. 2. Store the data uncompressed in the rebuilt rpm. This isn't feasible with deltarpm, but, if we store both compressed hashes and uncompressed hashes in the zchunk header, we can do this in zchunk. When running checking the signature, zchunk verifies the header against the signature first, and then checks each chunk to see if it passes *either* the compressed or uncompressed
Proposal: Faster composes by eliminating deltarpms and using zchunked rpms instead
For reference, this is in reply to Paul's email about lifecycle
objectives, specifically focusing on problem statement #1[1].
Have rpm use zchunk as its compression format, removing the need for
deltarpms, and thus reducing compose time. This will require changes
to both the rpm format and new features in the zchunk format.
*deltarpm background*
As part of the compose process, deltarpms are generated between each
new rpm and both the GA version of the rpm and the previous version.
This process is very CPU and memory intensive, especially for large
rpms.
This also means that deltarpms are only useful for an end user if they
are either updating from GA or have been diligent about keeping their
system up-to-date. If a user is updating a package from N-2 to N,
there will be no deltarpm and the full rpm will be downloaded.
*zchunk background*
As some are aware, I've been working on zchunk[2], a compression format
that's designed for highly efficient deltas, and using it minimize
metadata downloads[3].
The core idea behind zchunk is that a file is split into independently
compressed chunks and the checksum of each compressed chunk is stored
in the zchunk header. When downloading a new version of the file, you
download the zchunk header first, check which chunks you already have,
and then download the rest.
*Proposal*
My proposal would be to make zchunk the rpm compression format for
Fedora. This would involve a few additions to the zchunk format[4]
(something the format has been designed to accommodate), and would
require some changes to the rpm file format.
*Benefit*
The benefit of zchunked rpms is that, when downloading an updated rpm,
you would only need to download the chunks that have changed from
what's on your system.
The uncompressed local chunks would be combined with the downloaded
compressed chunks to create a local rpm that will pass signature
verification without needing to recompress the uncompressed local
chunks, making this computationally much faster than rebuilding a
deltarpm, a win for users.
The savings wouldn't be as good as what deltarpm can achieve, but
deltarpms would be redundant and could be removed, completely
eliminating a large step from the compose process.
*Drawbacks*
1. Downloading a new release of a zchunked rpm would be larger than
downloading the equivalent deltarpm. This is offset by the fact
that the client is able to work out which chunks it needs no matter
what the original rpm is, rather than needing a specific original
rpm as deltarpm does.
2. The rebuilt rpm may not be byte-for-byte identical to the original,
but will be able to be validated without decompression, as explained
in the next section
*Changes*
The zchunk format would need to be extended to allow for a zchunked rpm
to contain both the uncompressed chunks that were already on the local
system and the newly downloaded compressed chunks while still passing
signature verification. This would also require moving signature
verification to zchunk.
The rpm file format has to be changed because the zchunk header needs
to be at the beginning of the file in order for the zchunk library
figure out which chunks it needs to download. My suggestions for
changes to the rpm file format are as follows:
1. Signing should be moved to the zchunk format as described at the
beginning of this section
2. The rpm header should be stored in one stream inside the zchunk
file. This allows it to be easily extracted separately from the
data
3. The rpm cpio should be stored in a second stream inside the zchunk
file.
4. At minimum, an optional zchunk element should be set to identify
zchunk rpms as rpms rather than regular zchunk files. If desired,
optional elements could also be set containing %{name}, %[version},
%{release}, %{arch} and %{epoch}. This would allow this information
to be read easily without needing to extract the rpm header stream.
*Final notes*
I realize this is a massive proposal, zchunk is still very young, and
we're still working on getting the dnf zchunk pull requests reviewed.
I do think it's feasible and provides an opportunity to eliminate a
pain point from our compose process while still reducing the download
size for our users.
[1]:
https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements#Challenge_.231:_Faster.2C_more_scalable_composes
[2]: https://github.com/zchunk/zchunk
[3]: https://fedoraproject.org/wiki/Changes/Zchunk_Metadata
[4]: https://github.com/zchunk/zchunk/blob/master/zchunk_format.txt
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
Re: Introducing new koji rpmfusion namespace : rpi
On Thu, 2018-11-08 at 20:35 +0100, Nicolas Chauvet wrote: > Le jeu. 8 nov. 2018 à 19:43, Jonathan Dieter a > écrit : > > On Thu, 2018-11-08 at 16:57 +, Andrew Bauer wrote: > > > I'll create a new package request for the raspberry-vc libs. I'm > > > not > > > as fast as some of you, so give me a few days. I will get this > > > done. > > > > > > I plan to use this specfile as a starting point: > > > https://github.com/fedberry/raspberrypi-vc/blob/master/raspberrypi-vc.spec > > > > Just out of curiosity, does this work with aarch64 or just armhfp? > No it's only armv6hl/armv7hl so far. > (As a side note, I hope aarch64 and fedora side will have better api > such as v4l2-requests and all for the long run). Great, thanks. Jonathan ___ rpmfusion-developers mailing list -- rpmfusion-developers@lists.rpmfusion.org To unsubscribe send an email to rpmfusion-developers-le...@lists.rpmfusion.org
