Re: the mechanics of pushing updates
Luke Macken ([EMAIL PROTECTED]) said: > By the looks of the fedora-release-tools module, there are two scripts > that have been used to sign packages, ftsign and fedorasign, both of > which call /usr/local/bin/rpm-4.1-sign, which is a symlink to > /usr/lib/rpm/rpmk. ... which doesn't interact well with brew/koji. > Koji keeps a sigcache for each package in pkg/ver/rel/data/sigcache/arch/, > although I have no idea at what point in the build process this gets > created. I'm also under the impression that just having this detached > signature isn't enough, and that there still must be some manual > intervention? Is there anything else bodhi needs to do other than make > sure the corresponding .sig exists for each package? The sigcache is a koji feature; you can tell it to write out a signed version of the package on demand. mash, at least, does not do this, and will just take whatever existing version is signed with the best key. (Waiting on it to generate signed versions would be rather slow.) > This isn't as bad as the previous biarch-list-of-doom[0] anymore. Bodhi > imports it into its Multilib database table[1] during initialization, and > doesn't deal with it again. Upon submission of an update, bodhi builds > the list of associated packages, taking care of multilib based on what's > in the db. The multilib table can then be modified with ease using the > TurboGears CatWalk database editor, or a simple command-line tool. Still, it's a list - each new added package would potentially require an addition. > > - rsync the whole repo out > > TODO. At the moment bodhi stages to /mnt/koji/updates-stage -- where > are we going to sync this to? wallace still? Short-term, yes. > > Older updates are cleaned by a cron script later. > > TODO. We need something similar to the fedora-updates-clean script that > is currently in place (but less hackish), or RepoPrune / repomanage. > The TurboGears scheduler[3] is probably a good place for this. I'm > going to try and find some time tonight to throw one together. Well, we *could* go to one-update-only, as older updates will be available directly from the koji web server. > > Disadvantages: > > - multilib. In a world where we continually add new packages, this > > *will not scale*. > > Random idea. Since multilib is handled by mash, which pushed out > rawhide nightly, couldn't we just have mash keep the Multilib table up to > date? Would this solve the scalability issue wrt new packages? rawhide doesn't necessarily mesh with what's available for earlier releases. Bill ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: the mechanics of pushing updates
On Thu, May 24, 2007 at 01:02:35AM -0400, Bill Nottingham wrote: > Mmm, plumbing. bodhi is heading for production soon. To push updates, what > bodhi currently does is, for any update: > > - sign the package Nope, neither bodhi or the current update system sign any packages. The current system mails releng with the proper command to sign the packages, but it has always been done by hand as far as I know. You and Jesse are the only people I know of that have signed updates. By the looks of the fedora-release-tools module, there are two scripts that have been used to sign packages, ftsign and fedorasign, both of which call /usr/local/bin/rpm-4.1-sign, which is a symlink to /usr/lib/rpm/rpmk. I started implementing an XMLRPC server for bodhi so we can eventually do everything from the command line as well as from the browser. Hopefully we can streamline the signing process as much as possible in a command-line sign/push tool, until it can be fully automated with a signing server (when might this happen, btw?). Koji keeps a sigcache for each package in pkg/ver/rel/data/sigcache/arch/, although I have no idea at what point in the build process this gets created. I'm also under the impression that just having this detached signature isn't enough, and that there still must be some manual intervention? Is there anything else bodhi needs to do other than make sure the corresponding .sig exists for each package? > - copy the package to a 'staging' tree of the entirety of updates > - read a static list of packages that should be multilib, act on that This isn't as bad as the previous biarch-list-of-doom[0] anymore. Bodhi imports it into its Multilib database table[1] during initialization, and doesn't deal with it again. Upon submission of an update, bodhi builds the list of associated packages, taking care of multilib based on what's in the db. The multilib table can then be modified with ease using the TurboGears CatWalk database editor, or a simple command-line tool. > - run createrepo > - check deps on the repo FIXME: I need to track down some false positives in bodhi's closure.py[2] (or rewrite it). Mash would obviously resolve this for us. > - rsync the whole repo out TODO. At the moment bodhi stages to /mnt/koji/updates-stage -- where are we going to sync this to? wallace still? > Older updates are cleaned by a cron script later. TODO. We need something similar to the fedora-updates-clean script that is currently in place (but less hackish), or RepoPrune / repomanage. The TurboGears scheduler[3] is probably a good place for this. I'm going to try and find some time tonight to throw one together. > Advantages of this approach: > - it's simple > - it's easy to clean upthings that Go Wrong (just manually remove them > from the repo and re-sync) This also gives bodhi a LOT more control over the repos, as it maintains the extended updateinfo.xml.gz in the repodata as well. If we use mash we will have to maintain this file outside of the tree and re-insert it post-compose. > Disadvantages: > - multilib. In a world where we continually add new packages, this > *will not scale*. Random idea. Since multilib is handled by mash, which pushed out rawhide nightly, couldn't we just have mash keep the Multilib table up to date? Would this solve the scalability issue wrt new packages? > So, we need at least *some* sort of better workflow. > > One alternative - using mash (what we're using to build rawhide.) It > would go something like this: > > - sign the package > - tag the package (for updates-testing, or updates) > - run mash to create a repo of updates/updates-testing, solve it for > multilib > - rsync it out > > Advantages: > - solves multilib > - doesn't require continually keeping a staging tree around > - depcheck is built in when solving multilib > - builds on koji tags to let anyone easily query what updates are > released > > Disadvantages: > - by rebuilding the repo each time, it's going to be slow once > the repo gets large > - harder to clear out other strangeness > - will only have one version of each updated package > > The last of these isn't as *big* of a concern now, as all builds > will be available through the koji web site, space permitting. > > Other ideas for better workflow? What do the extras push scripts do? > Do we want to add a modified version of mash's multilib solver into > bodhi? I think the mash idea is interesting. Although, due to it's overhead, we would probaby have to resort to pushing out a single batch of updates a day, and maybe some smaller batches of security updates. This might become a pain. I'm going to look around at the multilib solver for mash and extrsa tonight and see if bodhi can steal any of it. Michael Schwendt would probably be the expert in the extras world. luke [0]: https://hosted.fedoraproject.org/projects/bodhi/browser/bodhi/deprecated/biarch.py [1]: https://hosted.fedoraproject.org/projects/bodhi/browser/bodhi/mo
Re: the mechanics of pushing updates
Once upon a time Thursday 24 May 2007, Bill Nottingham wrote: > Mmm, plumbing. bodhi is heading for production soon. To push updates, what > bodhi currently does is, for any update: > > - sign the package > - copy the package to a 'staging' tree of the entirety of updates > - read a static list of packages that should be multilib, act on that > - run createrepo > - check deps on the repo > - rsync the whole repo out > > Older updates are cleaned by a cron script later. We plan on adding auto clean up to bodhi, > Other ideas for better workflow? What do the extras push scripts do? > Do we want to add a modified version of mash's multilib solver into > bodhi? Extras keeps the last two versions for releases and last for devel. Extras multilib is all -devel packages and requires with some manual blacklisting/whitelisting we need bodhi to do update announcements etc. So i think that we need to make bodhi's multilib better. probably best to use mash for that. so we have one tool for the job Dennis ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
the mechanics of pushing updates
Mmm, plumbing. bodhi is heading for production soon. To push updates, what bodhi currently does is, for any update: - sign the package - copy the package to a 'staging' tree of the entirety of updates - read a static list of packages that should be multilib, act on that - run createrepo - check deps on the repo - rsync the whole repo out Older updates are cleaned by a cron script later. Advantages of this approach: - it's simple - it's easy to clean upthings that Go Wrong (just manually remove them from the repo and re-sync) Disadvantages: - multilib. In a world where we continually add new packages, this *will not scale*. So, we need at least *some* sort of better workflow. One alternative - using mash (what we're using to build rawhide.) It would go something like this: - sign the package - tag the package (for updates-testing, or updates) - run mash to create a repo of updates/updates-testing, solve it for multilib - rsync it out Advantages: - solves multilib - doesn't require continually keeping a staging tree around - depcheck is built in when solving multilib - builds on koji tags to let anyone easily query what updates are released Disadvantages: - by rebuilding the repo each time, it's going to be slow once the repo gets large - harder to clear out other strangeness - will only have one version of each updated package The last of these isn't as *big* of a concern now, as all builds will be available through the koji web site, space permitting. Other ideas for better workflow? What do the extras push scripts do? Do we want to add a modified version of mash's multilib solver into bodhi? (This is ignoring the process of rsyncing to the mirror master, which will be gross.) Bill ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
