On Wed, 2010-12-22 at 17:29 +0000, Adam Williamson wrote: > On Tue, 2010-12-21 at 18:12 -0500, James Laska wrote: > > > > the first isn't particularly specific to this, but it was a prerequisite > > > that I discovered was missing: it's a guide to test case creation in > > > general, explaining the actual practical process of how you create a > > > test case, and the best principles to consider in doing it. > > > > Nice job here, this is something that's difficult to explain if you've > > done it a lot, but I think you've captured the key points. If possible, > > it might be helpful to highlight a few existing examples that stand out > > for the different characteristics you mention (comprehensive, but able > > to stand the test of time). > > Thanks. I'll see if I can find some and add them. > > > Another thought, any reason that we wouldn't want to keep all wiki tests > > in the QA: namespace (and with the prefix QA:Testcase_)? The door is > > left open for other names, I wonder if we want to cut that off ahead of > > time to keep our sanity by having all tests in the same namespace? > > I was a bit unsure on that one. I think I thought of some possible > scenario where you might want to write a test case in a different name > space, but I'm not entirely sure I remember what it was. I can just > change it to say test cases should always go in the qa namespace, I > guess. > > > The page also talks about using [[Category:Test_Cases]]. I worry if we > > are too lax in categorizing new tests we'll end up with a large amount > > of random tests in the main [[Category:Test_Cases]] making it a > > maintenance nightmare to cleanup that category. Should we instead > > direct users to your other page > > (https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_creation) > > for guidance on categorizing test cases? > > This was something I wanted to call out for discussion and forgot - so > far we've put all test cases directly into the Test_Cases category, but > like you, I'm worried that really won't scale. I did wonder whether > others would agree we should stop doing that and instead have them > usually go into a more specific category which in turn would be a > sub-category of Test_Cases, and only have test cases be members of > Test_Cases directly if it really made no sense to have them in a more > specific category.
Agreed ... I think it makes sense to keep Category:Test_Cases as just a container for sub-categories if possible. Mainly for the reasons you note around *trying* to keep content organized. > > > The second is what's really specific to this subject. It describes how > > > to create a set of test cases for a particular package, and a proposed > > > standardized categorization scheme which will allow us to denote test > > > cases as being associated with specific packages, and also denote them > > > as concerning critical path functionality. > > > > I think I mentioned this previously, in the section 'Preparation', I > > appreciate the distinction of 'core' and 'extended'. But I it resonates > > with me better under the context of test "priority". I don't see why we > > can't keep using the terms 'core' and 'extended', but just want to > > clarify their purpose. They're intended to add some sense of execution > > priority to a list of test cases, right? Where critpath comes first, > > then core, then extended, then other? Also, you describe > > categorizing/grouping test cases in more detail below, maybe just link > > to that instead? Was I accurate in my understanding above of your proposed groupings (critpath, core and extended)? Are they intended to convey an execution priority of the tests? > well, the idea is that the two are complementary: if you're going to > separate the test cases into 'core' and 'extended' groups, then why not > identify which functionality is 'core' and which is 'extended' at the > time you're identifying functionality to write test cases for? I'm not > quite sure what your proposal is here - could you draft it up in terms > of an actual change to the page so I can see it more clearly? thanks! I articulated several layouts in previous comments in the ticket. See https://fedorahosted.org/fedora-qa/ticket/154#comment:12 and https://fedorahosted.org/fedora-qa/ticket/154#comment:18. I guess I'm hesitant about introducing new terminology ("core" and "extended") when I'm more familiar with prioritizing test cases using the term "priority". I'm not saying we shouldn't use them, I'm just trying to understand the context. I'm also trying to ensure your project ties in nicely with the work Hurry is doing with regards to scoping out a TCMS (http://fedorahosted.org/fedora-qa/ticket/152). My question (I guess I already re-stated above) was whether you consider the terms "core" and "extended" as a designation of test case priority? Outside of the terminology, I have some concerns whether this is within the scope of the initial project, or something we want to leave as a phase#2 effort. We definitely need to think about it as non-critpath tests will come in, I just hope we don't spend all our collective energy on defining non-critpath tests and then we are still exposed to a lack of test documentation for the critpath. > > In the section, 'Simple (required)', would it help to add a link to > > http://fedoraproject.org/wiki/BugZappers/CorrectComponent#Which_component_is_it.3F > > (or similar page). Something to help testers find the right src.rpm name > > of the component under test? Side note, this might also be a maintenance > > task we can define where I, or anyone interested, could manually scrub (or > > script) finding Categories:Test_Cases searching incorrectly named category > > pages. > > > > Also in 'Simple (required)', we don't tell the author to add their > > 'Category:Package_${sourcename}_test_cases' to 'Category:Test_Cases'. I > > think we want all newly created package categories anchored under > > 'Category:Test_Cases'. > > yup, indeed, oversight - will add it. thanks! > > > General comment. I know we've got an eye towards integrating this work > > with bodhi and/or f-e-k. Until that work is complete, I wonder if those > > notes will introduce confusion/speculation. Should we leave out the > > bits about possible future tool integration until such support is > > active? > > possibly. I was meaning those bits to be read simply as a potential > illustration of programmatic use of the categories to illustrate why > consistent categorization is important, but if you think it's confusing, > we could take it out. No strong opinions here. I thought I learned somewhere that one should avoid future leading statements when documenting process. I could have sworn that was in the Fedora doc guide ... but I could be making it up. Thanks, James
signature.asc
Description: This is a digitally signed message part
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
