Re: Release criteria: kickstart
On Wed, 2011-08-31 at 16:42 -0700, Adam Williamson wrote: > Here's a specific wording proposal - proposing we add this as a Beta > criterion, starting with F16 Beta: > > "The installer must be able to successfully complete a scripted > installation, using the installer's preferred scripting system, which > duplicates the default interactive installation as closely as possible". > > sound good? patches? alternative proposals? thanks! As there were no objections, I've gone ahead and added this to the F16 Beta criteria. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On Tue, 2011-08-30 at 16:26 -0700, Adam Williamson wrote: > I propose that to get something in place now we go minimal, and just > implement what there was consensus on: smooge's suggested criterion - > > "Does it take a minimal kickstart and build a default system. The > minimal being the exact stuff that would be created if a person just > clicked through a release." > > as a Beta criterion. i.e. if you take /root/anaconda-ks.cfg from a > click-through install and pass it to anaconda, it should successfully > install. (Maybe with the small wrinkle that you uncomment the > partitioning stuff, so it becomes a true unattended kickstart). > > We can then elaborate the criteria from there based on 'case law', i.e., > we can evaluate actual kickstart bugs as they're proposed as blockers > and propose criteria based on the results of those discussions. How does > that sound? Here's a specific wording proposal - proposing we add this as a Beta criterion, starting with F16 Beta: "The installer must be able to successfully complete a scripted installation, using the installer's preferred scripting system, which duplicates the default interactive installation as closely as possible". sound good? patches? alternative proposals? thanks! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On Tue, 2011-08-23 at 14:13 -0700, Adam Williamson wrote: > Hey, folks. So, another criteria discussion. It came up during Alpha > that we're grievously lacking criteria to ensure kickstart installs > work. So, we should kick that around a bit, I guess... > > kickstart is a very broad area; you can write extremely complex > kickstart files that do a lot of stuff. So broadly what we'd need to do > is define a subset of kickstart functionality that we expect to work, > and then possibly divide that up by release phase (so some stuff must > work by Beta, the rest by Final, for e.g.) > > anaconda devs following this list, do you have any existing expectations > as to what level of kickstart functionality ought to be in place for > releases, and when you think would be appropriate? > > So far it seems everyone more or less agrees that it should be possible > to do at least a basic unattended kickstart install by Beta. So we don't seem to have a real clear consensus here. The last blocker review meeting exposed a few kickstart operations we considered *not* significant enough to be beta blockers, but not really any we did consider significant enough. I propose that to get something in place now we go minimal, and just implement what there was consensus on: smooge's suggested criterion - "Does it take a minimal kickstart and build a default system. The minimal being the exact stuff that would be created if a person just clicked through a release." as a Beta criterion. i.e. if you take /root/anaconda-ks.cfg from a click-through install and pass it to anaconda, it should successfully install. (Maybe with the small wrinkle that you uncomment the partitioning stuff, so it becomes a true unattended kickstart). We can then elaborate the criteria from there based on 'case law', i.e., we can evaluate actual kickstart bugs as they're proposed as blockers and propose criteria based on the results of those discussions. How does that sound? -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On 08/24/2011 02:43 AM, Adam Williamson wrote: > Hey, folks. So, another criteria discussion. It came up during Alpha > that we're grievously lacking criteria to ensure kickstart installs > work. So, we should kick that around a bit, I guess... > > kickstart is a very broad area; you can write extremely complex > kickstart files that do a lot of stuff. So broadly what we'd need to do > is define a subset of kickstart functionality that we expect to work, > and then possibly divide that up by release phase (so some stuff must > work by Beta, the rest by Final, for e.g.) > > anaconda devs following this list, do you have any existing expectations > as to what level of kickstart functionality ought to be in place for > releases, and when you think would be appropriate? > > So far it seems everyone more or less agrees that it should be possible > to do at least a basic unattended kickstart install by Beta. Adam, I like the idea if having a kickstart installs for release criteria. So, I do unattended installs all the time w/ virt-install, but mostly minimal, to manage plenty of kvm guests via shell. For ex, a minimal kickstart I use is below which gives a serial console: install text reboot lang en_US.UTF-8 keyboard us network --bootproto dhcp rootpw redhat firewall --enabled --ssh selinux --enforcing timezone --utc America/New_York #firstboot --disable #NOTE: I have to use rd_NO_PLYMOUTH bootloader --location=mbr --append="console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH" zerombr clearpart --all --initlabel autopart %packages @core %end Here, for instance, plymouth and serial console doesn't play well. Note in the above ks, I have to use the rd_NO_PLYMOUTH flag (this disables plymouth) .If this flag isn't used, serial console access is broken. (I guess I filed a bug for this, which I can't find at the moment. Thankfully, Ray Strode pointed me to the NO_PLYMOUTH flag ) Things like these can be ironed out w/ the kickstart criteria. ps:Link to the above ks: http://kashyapc.fedorapeople.org/virt/fed-minimal.ks thanks, /kashyap -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On 08/26/2011 12:00 AM, Adam Williamson wrote: > On Thu, 2011-08-25 at 23:54 +, "Jóhann B. Guðmundsson" wrote: > >> I'm still on the opinion that we should not be adding an ks testing to >> the alpha criteria. >> >> Cant we just create a set of most commonly ks use cases in ks files for >> releng and they can use pungi to test it with and post the result >> somewhere online with pass/fail more of an autoqa task and say all those >> must pass before beta? >> >> Where does the need for those ks testing suddenly come from and why do >> we need to have those ks test involved in alpha? > sorry, I skipped over Smooge's mention of Alpha. I was aiming for Beta, > not Alpha. I think Smooge's minimal 'does a stock install ks file work' > would be fine as a Beta test. Given that this is something we should be able to automate and more of a task/project for autoqa I dont think we should be limiting ourself with minimal install and aim for more extensive tests than minimal I'm pretty sure Anaconda team and probably various other project members would also like to upload a .ks file somewhere and that file be put in a queue and the test ks process would process that queue on an x interval or simply test give pass or fail if fail, save logs post somewhere, delete file(s) and process next ks . JBG -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On Thu, 2011-08-25 at 23:54 +, "Jóhann B. Guðmundsson" wrote: > I'm still on the opinion that we should not be adding an ks testing to > the alpha criteria. > > Cant we just create a set of most commonly ks use cases in ks files for > releng and they can use pungi to test it with and post the result > somewhere online with pass/fail more of an autoqa task and say all those > must pass before beta? > > Where does the need for those ks testing suddenly come from and why do > we need to have those ks test involved in alpha? sorry, I skipped over Smooge's mention of Alpha. I was aiming for Beta, not Alpha. I think Smooge's minimal 'does a stock install ks file work' would be fine as a Beta test. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On Thu, 2011-08-25 at 16:49 -0600, Stephen John Smoogen wrote: > Well I was trying to suggest something that wasn't handy-wavy. At some > point a set of criteria are going to be needed for what is expected > out anaconda at the minimum. From that criteria a set of test > kickstarts can be built. If they dont' work then you have a blocker. > Is that better? Ah, I see - absolutely, when it comes to *test cases*, indeed the way to test it is going to be to have kickstarts in the test cases. I just tend to think first about defining the criteria, then writing the test cases. Sorry I didn't quite grok you! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On 08/25/2011 10:49 PM, Stephen John Smoogen wrote: > On Thu, Aug 25, 2011 at 16:20, Adam Williamson wrote: >> On Thu, 2011-08-25 at 14:35 -0600, Stephen John Smoogen wrote: >>> On Thu, Aug 25, 2011 at 14:25, Chris Lumens wrote: > kickstart is a very broad area; you can write extremely complex > kickstart files that do a lot of stuff. So broadly what we'd need to do > is define a subset of kickstart functionality that we expect to work, > and then possibly divide that up by release phase (so some stuff must > work by Beta, the rest by Final, for e.g.) > > anaconda devs following this list, do you have any existing expectations > as to what level of kickstart functionality ought to be in place for > releases, and when you think would be appropriate? > > So far it seems everyone more or less agrees that it should be possible > to do at least a basic unattended kickstart install by Beta. You're right, kickstart is incredibly broad. I don't think we could ever hope to come up with criteria to cover all of it. I guess the best we can do is define criteria in terms of something else we already have. >>> I would go for the classical kickstart test for Alpha: >>> >>> Does it take a minimal kickstart and build a default system. The >>> minimal being the exact stuff that would be created if a person just >>> clicked through a release. >> Oh, I like that. That's very good. >> >>> For Beta >>> >>> Take these X broken kickstarts, does it bail at the appropriate places. >>> Take these Y working kickstarts, does it work. >>> Where Y is a set of items that can be tested on say a KVM or a >>> "default" desktop defined somewhere. >> I'm not so keen on that; it's a bit specific. One of my fetishes with >> the criteria is to keep them generic so they don't have to keep being >> changed all the time; I wouldn't want a criterion to rely on some >> specific kickstart file we keep lying around somewhere. > Well I was trying to suggest something that wasn't handy-wavy. At some > point a set of criteria are going to be needed for what is expected > out anaconda at the minimum. From that criteria a set of test > kickstarts can be built. If they dont' work then you have a blocker. > Is that better? > I'm still on the opinion that we should not be adding an ks testing to the alpha criteria. Cant we just create a set of most commonly ks use cases in ks files for releng and they can use pungi to test it with and post the result somewhere online with pass/fail more of an autoqa task and say all those must pass before beta? Where does the need for those ks testing suddenly come from and why do we need to have those ks test involved in alpha? JBG -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On Thu, Aug 25, 2011 at 16:20, Adam Williamson wrote: > On Thu, 2011-08-25 at 14:35 -0600, Stephen John Smoogen wrote: >> On Thu, Aug 25, 2011 at 14:25, Chris Lumens wrote: >> >> kickstart is a very broad area; you can write extremely complex >> >> kickstart files that do a lot of stuff. So broadly what we'd need to do >> >> is define a subset of kickstart functionality that we expect to work, >> >> and then possibly divide that up by release phase (so some stuff must >> >> work by Beta, the rest by Final, for e.g.) >> >> >> >> anaconda devs following this list, do you have any existing expectations >> >> as to what level of kickstart functionality ought to be in place for >> >> releases, and when you think would be appropriate? >> >> >> >> So far it seems everyone more or less agrees that it should be possible >> >> to do at least a basic unattended kickstart install by Beta. >> > >> > You're right, kickstart is incredibly broad. I don't think we could >> > ever hope to come up with criteria to cover all of it. I guess the best >> > we can do is define criteria in terms of something else we already have. >> >> I would go for the classical kickstart test for Alpha: >> >> Does it take a minimal kickstart and build a default system. The >> minimal being the exact stuff that would be created if a person just >> clicked through a release. > > Oh, I like that. That's very good. > >> For Beta >> >> Take these X broken kickstarts, does it bail at the appropriate places. >> Take these Y working kickstarts, does it work. >> Where Y is a set of items that can be tested on say a KVM or a >> "default" desktop defined somewhere. > > I'm not so keen on that; it's a bit specific. One of my fetishes with > the criteria is to keep them generic so they don't have to keep being > changed all the time; I wouldn't want a criterion to rely on some > specific kickstart file we keep lying around somewhere. Well I was trying to suggest something that wasn't handy-wavy. At some point a set of criteria are going to be needed for what is expected out anaconda at the minimum. From that criteria a set of test kickstarts can be built. If they dont' work then you have a blocker. Is that better? -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On Thu, 2011-08-25 at 14:35 -0600, Stephen John Smoogen wrote: > On Thu, Aug 25, 2011 at 14:25, Chris Lumens wrote: > >> kickstart is a very broad area; you can write extremely complex > >> kickstart files that do a lot of stuff. So broadly what we'd need to do > >> is define a subset of kickstart functionality that we expect to work, > >> and then possibly divide that up by release phase (so some stuff must > >> work by Beta, the rest by Final, for e.g.) > >> > >> anaconda devs following this list, do you have any existing expectations > >> as to what level of kickstart functionality ought to be in place for > >> releases, and when you think would be appropriate? > >> > >> So far it seems everyone more or less agrees that it should be possible > >> to do at least a basic unattended kickstart install by Beta. > > > > You're right, kickstart is incredibly broad. I don't think we could > > ever hope to come up with criteria to cover all of it. I guess the best > > we can do is define criteria in terms of something else we already have. > > I would go for the classical kickstart test for Alpha: > > Does it take a minimal kickstart and build a default system. The > minimal being the exact stuff that would be created if a person just > clicked through a release. Oh, I like that. That's very good. > For Beta > > Take these X broken kickstarts, does it bail at the appropriate places. > Take these Y working kickstarts, does it work. > Where Y is a set of items that can be tested on say a KVM or a > "default" desktop defined somewhere. I'm not so keen on that; it's a bit specific. One of my fetishes with the criteria is to keep them generic so they don't have to keep being changed all the time; I wouldn't want a criterion to rely on some specific kickstart file we keep lying around somewhere. > For Final > The above and some subset of obscure items that can be tested reliably > somewhere that development and QA can replicate. again, seems a bit arbitrary; what we really need is the kickstart functionality we actually believe has to work, not an arbitrary set of interesting bugs we happen to be able to test. a minimal answer to this question would be fairly valid, I guess. at least in the short term. sometimes working up criteria in response to real-world cases - i.e. someone finds a kickstart problem, files a bug, proposes it as blocker, we agree it should be one and work up a criterion in response to it - has proved a decent way of doing things in the past. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On Thu, Aug 25, 2011 at 14:25, Chris Lumens wrote: >> kickstart is a very broad area; you can write extremely complex >> kickstart files that do a lot of stuff. So broadly what we'd need to do >> is define a subset of kickstart functionality that we expect to work, >> and then possibly divide that up by release phase (so some stuff must >> work by Beta, the rest by Final, for e.g.) >> >> anaconda devs following this list, do you have any existing expectations >> as to what level of kickstart functionality ought to be in place for >> releases, and when you think would be appropriate? >> >> So far it seems everyone more or less agrees that it should be possible >> to do at least a basic unattended kickstart install by Beta. > > You're right, kickstart is incredibly broad. I don't think we could > ever hope to come up with criteria to cover all of it. I guess the best > we can do is define criteria in terms of something else we already have. I would go for the classical kickstart test for Alpha: Does it take a minimal kickstart and build a default system. The minimal being the exact stuff that would be created if a person just clicked through a release. For Beta Take these X broken kickstarts, does it bail at the appropriate places. Take these Y working kickstarts, does it work. Where Y is a set of items that can be tested on say a KVM or a "default" desktop defined somewhere. For Final The above and some subset of obscure items that can be tested reliably somewhere that development and QA can replicate. -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
> kickstart is a very broad area; you can write extremely complex > kickstart files that do a lot of stuff. So broadly what we'd need to do > is define a subset of kickstart functionality that we expect to work, > and then possibly divide that up by release phase (so some stuff must > work by Beta, the rest by Final, for e.g.) > > anaconda devs following this list, do you have any existing expectations > as to what level of kickstart functionality ought to be in place for > releases, and when you think would be appropriate? > > So far it seems everyone more or less agrees that it should be possible > to do at least a basic unattended kickstart install by Beta. You're right, kickstart is incredibly broad. I don't think we could ever hope to come up with criteria to cover all of it. I guess the best we can do is define criteria in terms of something else we already have. So, how much of the test matrix right now is tested via kickstart files? What other tasks do we have right now that are kickstart-based? I know spin composition uses some pieces of kickstart, and I know the storage tests I wrote for autoqa use kickstart. So right there, we have a couple things we could base criteria on. I guess there's also two ways you can break down kickstart - there's syntax, and semantics. Syntactically, the criteria should be that we recognize all valid kickstart files. Good luck defining what's a valid kickstart file in terms besides "that which is accepted by pykickstart as valid", though. Semantics are the hard part and are what we're really talking about here, though. That's a lot of words to say, "I don't know". - Chris -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Release criteria: kickstart
On 08/23/2011 09:13 PM, Adam Williamson wrote: > Hey, folks. So, another criteria discussion. It came up during Alpha > that we're grievously lacking criteria to ensure kickstart installs > work. So, we should kick that around a bit, I guess... > > kickstart is a very broad area; you can write extremely complex > kickstart files that do a lot of stuff. So broadly what we'd need to do > is define a subset of kickstart functionality that we expect to work, > and then possibly divide that up by release phase (so some stuff must > work by Beta, the rest by Final, for e.g.) > > anaconda devs following this list, do you have any existing expectations > as to what level of kickstart functionality ought to be in place for > releases, and when you think would be appropriate? > > So far it seems everyone more or less agrees that it should be possible > to do at least a basic unattended kickstart install by Beta. For the love of all that is holy do not tie that into the alpha criteria... JBG -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Release criteria: kickstart
Hey, folks. So, another criteria discussion. It came up during Alpha that we're grievously lacking criteria to ensure kickstart installs work. So, we should kick that around a bit, I guess... kickstart is a very broad area; you can write extremely complex kickstart files that do a lot of stuff. So broadly what we'd need to do is define a subset of kickstart functionality that we expect to work, and then possibly divide that up by release phase (so some stuff must work by Beta, the rest by Final, for e.g.) anaconda devs following this list, do you have any existing expectations as to what level of kickstart functionality ought to be in place for releases, and when you think would be appropriate? So far it seems everyone more or less agrees that it should be possible to do at least a basic unattended kickstart install by Beta. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
