Re: In A World Where...TCs don't exist?
On Sat, Jan 30, 2016 at 09:11:49PM +0100, Dennis Gilmore wrote: > Already added to our next grooming meeting. Nice. Without realizing this discussion already happened here, I asked Dennis the same question as Rich, about the status of the its (virt-builder's) metadata distribution as part of Fedora composes, earlier this afternoon during his Rel Eng talk at DevConf in Brno. Thanks for the talk, Dennis! > On January 29, 2016 7:56:05 PM GMT+01:00, Matthew Miller > wrote: > >On Fri, Jan 29, 2016 at 10:07:14AM -0600, Dennis Gilmore wrote: > >> > > > https://fedorahosted.org/rel-eng/ticket/5805 > >[...] > >> We have none of that info and its not yet on our radar. > >> https://fedoraproject.org/wiki/ReleaseEngineering/PriorityPipeline is > >> the list of things we have coming up. I think there are some not > >> documented as well as they should be. > > > >I think this request just kind of fell through and didn't get included > >in the new prioritization process. Let's get it added in as something > >for _sometime_ in the future, even if it doesn't bump current > >priorities. -- /kashyap -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Re: In A World Where...TCs don't exist?
Already added to our next grooming meeting. On January 29, 2016 7:56:05 PM GMT+01:00, Matthew Miller wrote: >On Fri, Jan 29, 2016 at 10:07:14AM -0600, Dennis Gilmore wrote: >> > > > https://fedorahosted.org/rel-eng/ticket/5805 >[...] >> We have none of that info and its not yet on our radar. >> https://fedoraproject.org/wiki/ReleaseEngineering/PriorityPipeline is >> the list of things we have coming up. I think there are some not >> documented as well as they should be. > >I think this request just kind of fell through and didn't get included >in the new prioritization process. Let's get it added in as something >for _sometime_ in the future, even if it doesn't bump current >priorities. > >-- >Matthew Miller > >Fedora Project Leader >-- >test mailing list >t...@lists.fedoraproject.org >To unsubscribe: >http://lists.fedoraproject.org/admin/lists/t...@lists.fedoraproject.org -- Sent from my Android device with K-9 Mail. Please excuse my brevity.-- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Re: In A World Where...TCs don't exist?
On Fri, Jan 29, 2016 at 10:07:14AM -0600, Dennis Gilmore wrote: > > > > https://fedorahosted.org/rel-eng/ticket/5805 [...] > We have none of that info and its not yet on our radar. > https://fedoraproject.org/wiki/ReleaseEngineering/PriorityPipeline is > the list of things we have coming up. I think there are some not > documented as well as they should be. I think this request just kind of fell through and didn't get included in the new prioritization process. Let's get it added in as something for _sometime_ in the future, even if it doesn't bump current priorities. -- Matthew Miller Fedora Project Leader -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Re: In A World Where...TCs don't exist?
On Friday, January 29, 2016 02:25:49 PM Richard W.M. Jones wrote: > On Fri, Jan 29, 2016 at 03:03:33PM +0100, Stanislav Ochotnicky wrote: > > On Fri 29 Jan 2016 02:51:31 PM CET Richard W.M. Jones wrote: > > > On Fri, Jan 29, 2016 at 04:33:19AM -0800, Adam Williamson wrote: > > >> Hi, folks! I thought this might be about the appropriate time to throw > > >> this out there. > > >> > > >> There hasn't been a big news press on this, but some of you may know > > >> that releng is fairly close to switching over to Pungi 4 for composes. > > >> For those of you who don't know: > > >> > > >> releng is fairly close to switching over to Pungi 4 for composes. > > > > > > [...] > > > > > > Any chance you can publish metadata for these releases? ie. this 2 > > > > > > year old request: > > > https://fedorahosted.org/rel-eng/ticket/5805 > > > > > > We're in the awkward situation now where OpenSUSE and Ubuntu publish > > > machine-readable metadata, but Fedora does not (or if it now does, > > > please point me to it so we can start using it). > > > > > > Many people would test the cloud images and test their software on > > > > > > cloud images if they could do: > > > $ virt-builder fedora-rawhide > > > $ virt-builder fedora-nightly-MMDD > > > > > > or whatever to get them. > > > > I think you might be looking for something like this? > > https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-/compose/ > > metadata/ > > > > See the files in the directory for details, be aware the rpm one is huge > > though :-) > > Possibly. > > Really we're looking for cloud images though (ie. *.qcow2), not > install ISOs or trees. I thought Pungi did both? > > There are a few missing fields we require too: > > - size of the disk image (especially when the image xz-compressed, we >need the uncompressed size in order to plan how to resize it) > > - format of the disk image > > - name of the root filesystem (so we can resize the image intelligently) > > - cryptographically-secure checksum of the image > > - libosinfo database key (so we know what emulated devices to present) > > And the metadata should be GPG signed. > > I've got an example here: > > http://libguestfs.org/download/builder/index.asc > > I'm not hung up on the specific format -- for Ubuntu they use a thing > called "SimpleStreams" which we implemented support for -- but it > needs to contain the same or a subset of that metadata. > > Rich. We have none of that info and its not yet on our radar. https://fedoraproject.org/wiki/ReleaseEngineering/PriorityPipeline is the list of things we have coming up. I think there are some not documented as well as they should be. Dennis signature.asc Description: This is a digitally signed message part. -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Re: In A World Where...TCs don't exist?
On Fri, Jan 29, 2016 at 02:25:49PM +, Richard W.M. Jones wrote: > needs to contain the same or a subset of that metadata. Ummm 'superset' even. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Re: In A World Where...TCs don't exist?
On Fri, Jan 29, 2016 at 03:03:33PM +0100, Stanislav Ochotnicky wrote: > On Fri 29 Jan 2016 02:51:31 PM CET Richard W.M. Jones wrote: > > > On Fri, Jan 29, 2016 at 04:33:19AM -0800, Adam Williamson wrote: > >> Hi, folks! I thought this might be about the appropriate time to throw > >> this out there. > >> > >> There hasn't been a big news press on this, but some of you may know > >> that releng is fairly close to switching over to Pungi 4 for composes. > >> For those of you who don't know: > >> > >> releng is fairly close to switching over to Pungi 4 for composes. > > [...] > > > > Any chance you can publish metadata for these releases? ie. this 2 > > year old request: > > > > https://fedorahosted.org/rel-eng/ticket/5805 > > > > We're in the awkward situation now where OpenSUSE and Ubuntu publish > > machine-readable metadata, but Fedora does not (or if it now does, > > please point me to it so we can start using it). > > > > Many people would test the cloud images and test their software on > > cloud images if they could do: > > > > $ virt-builder fedora-rawhide > > $ virt-builder fedora-nightly-MMDD > > > > or whatever to get them. > > I think you might be looking for something like this? > https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-/compose/metadata/ > > See the files in the directory for details, be aware the rpm one is huge > though :-) Possibly. Really we're looking for cloud images though (ie. *.qcow2), not install ISOs or trees. I thought Pungi did both? There are a few missing fields we require too: - size of the disk image (especially when the image xz-compressed, we need the uncompressed size in order to plan how to resize it) - format of the disk image - name of the root filesystem (so we can resize the image intelligently) - cryptographically-secure checksum of the image - libosinfo database key (so we know what emulated devices to present) And the metadata should be GPG signed. I've got an example here: http://libguestfs.org/download/builder/index.asc I'm not hung up on the specific format -- for Ubuntu they use a thing called "SimpleStreams" which we implemented support for -- but it needs to contain the same or a subset of that metadata. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Re: In A World Where...TCs don't exist?
On Fri 29 Jan 2016 02:51:31 PM CET Richard W.M. Jones wrote: > On Fri, Jan 29, 2016 at 04:33:19AM -0800, Adam Williamson wrote: >> Hi, folks! I thought this might be about the appropriate time to throw >> this out there. >> >> There hasn't been a big news press on this, but some of you may know >> that releng is fairly close to switching over to Pungi 4 for composes. >> For those of you who don't know: >> >> releng is fairly close to switching over to Pungi 4 for composes. > [...] > > Any chance you can publish metadata for these releases? ie. this 2 > year old request: > > https://fedorahosted.org/rel-eng/ticket/5805 > > We're in the awkward situation now where OpenSUSE and Ubuntu publish > machine-readable metadata, but Fedora does not (or if it now does, > please point me to it so we can start using it). > > Many people would test the cloud images and test their software on > cloud images if they could do: > > $ virt-builder fedora-rawhide > $ virt-builder fedora-nightly-MMDD > > or whatever to get them. I think you might be looking for something like this? https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-/compose/metadata/ See the files in the directory for details, be aware the rpm one is huge though :-) -- Stanislav Ochotnicky Business System Analyst, PnT DevOps PMO Team - Brno PGP: 7B087241 Red Hat Inc. http://cz.redhat.com -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Re: In A World Where...TCs don't exist?
On Fri, Jan 29, 2016 at 04:33:19AM -0800, Adam Williamson wrote: > Hi, folks! I thought this might be about the appropriate time to throw > this out there. > > There hasn't been a big news press on this, but some of you may know > that releng is fairly close to switching over to Pungi 4 for composes. > For those of you who don't know: > > releng is fairly close to switching over to Pungi 4 for composes. [...] Any chance you can publish metadata for these releases? ie. this 2 year old request: https://fedorahosted.org/rel-eng/ticket/5805 We're in the awkward situation now where OpenSUSE and Ubuntu publish machine-readable metadata, but Fedora does not (or if it now does, please point me to it so we can start using it). Many people would test the cloud images and test their software on cloud images if they could do: $ virt-builder fedora-rawhide $ virt-builder fedora-nightly-MMDD or whatever to get them. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
In A World Where...TCs don't exist?
Hi, folks! I thought this might be about the appropriate time to throw this out there. There hasn't been a big news press on this, but some of you may know that releng is fairly close to switching over to Pungi 4 for composes. For those of you who don't know: releng is fairly close to switching over to Pungi 4 for composes. This will have various interesting effects on QA and the whole process of building Fedora releases. With the current releng process, TC / RC composes are one beast, and nightly composes are another, very different beast. In fact nightly composes barely really 'exist' at all - when we say 'nightly compose' we really mean 'pungify the rawhide/branched repo, and fire off a bunch of koji tasks'. After the fact, there is no real relationship between any of those bits, which is why I had to write fedfind to go out and synthesize the concept of a 'nightly compose' by finding all the Koji tasks and treating them plus the repository boot.iso's as a single 'compose'. With Pungi 4, all composes will look a lot more similar. 'nightly' composes (which, in point of fact, will probably happen more than once per day - I'm not sure if we came up with a new name yet) look a lot more like current TC/RC composes than current nightly composes. You can see approximately what a Pungi 4 compose currently looks like here: https://kojipkgs.fedoraproject.org/compose/rawhide/ as of right now, the Koji built bits - lives, cloud and ARM disk images, etc - aren't integrated with the installer images, but they *will* be, and they'll all show up in the same location. As you can see it has all the different variants, and a Server DVD image. (A Pungi 4 compose also has a bunch of metadata, which means we can more or less kill off fedfind, thank God). The implication of this I wanted to talk about in this thread is: what does this mean for the release validation process, in terms of what composes we cut and what release validation events we have? So as you probably know, right now, the validation process is built around the milestone 'TC' and 'RC' images. We build a series of Alpha TCs and run a bunch of tests for each of these composes, reporting the results to wiki pages named for the composes. Then we do Alpha RCs, then Beta TCs, and so on through Final RCs. For the last few releases we've added on some 'nightly' validation events, where we create wiki pages named for nightly composes and run the same set of tests on the nightly boot.iso's and Koji images, but these have been framed as kind of an 'early warning system' for use before Alpha TC1 arrives, and once Alpha TC1 arrives we stop doing the nightly validation events. With Pungi 4, I don't think this makes a lot of sense any more. Dennis and I have been talking about this and I think we broadly agree on it. TCs and RCs used to be kinda the only way we *could* do validation testing. For long periods we didn't have reliable nightly builds of Rawhide or Branched at all, certainly not all the Koji-produced images. The process for doing 'real' composes was quite long and painful and required squishy human intervention. If we have automated, more-than-nightly composes that look much like a regular release compose would, there's no clear case for having TCs at all. We could simply stop building them and extend the "nightly" validation process. I think the way to do that would be to keep 'nominating' nightly composes for validation testing all the time, *except* when we're doing RCs. So instead of going something like: 24 Rawhide 20160120 24 Rawhide 20160215 == BRANCH POINT == 24 Branched 20160301 24 Branched 20160315 24 Alpha TC1 24 Alpha TC2 == ALPHA FREEZE == 24 Alpha RC1 24 Alpha RC2 == ALPHA RELEASE == 24 Beta TC1 we'd go something like: 24 Rawhide 20160120 24 Rawhide 20160215 == BRANCH POINT == 24 Branched 20160301 24 Branched 20160315 24 Branched 20160401 24 Alpha RC1 24 Alpha RC2 == ALPHA RELEASE == 24 Branched 20160501 24 Branched 20160515 24 Beta RC1 note: all dates completely made up, this is just for illustration. I think it would be plausible to do this for Fedora 24, if the Pungi 4 switchover happens soon and goes well. There would be some details to pin down in relval and wikitcms and stuff (we might need to tweak the validation event naming approach a bit so that it's possible to identify the sequence of events from the names - i.e. so you know where the RCs fit in), but nothing unsolvable. We'll be talking about a lot of this stuff at DevConf, if anyone's going to be there, pin down me or Dennis or someone else involved in release-y stuff and we'd be happy to discuss it. But I wanted to throw something up on the lists for discussion as well. What do you think? Thanks! One point that's come up already is the way that we manually pull newer packages to fix blocker/FE bugs into TC and RC composes via the 'bleed' repo. We're currently envisaging something like the 'buildroot override
