Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
On Mon, 2011-01-24 at 23:05 -0800, Adam Williamson wrote: On Fri, 2011-01-21 at 19:02 +, Samuel Greenfeld wrote: I do like litmus! It's a nice evolution from testopia for upstream mozilla. We don't currently have an 'unclear' test result. I'm not opposed to it, but would need better understand how that field is used, and the process around it, in litmus. Agree with James. What I believe Mozilla is doing (since I have not had a chance to work with their QA team yet) is flagging test cases with a form of soft failure in that the result of a testcase neither clearly passed, nor clearly failed. So in addition to Passed, Failed, and any other common states (Blocked, In Progress, etc.) you have an Unclear result state. I didn't receive the mail replied by Samuel. Weird. Hurry is somewhat wrong to say we don't currently have an 'unclear' result; we do have the 'warn' result, which is in some ways similar. We usually use it to indicate when a test turns up some kind of anomalous behaviour which isn't exactly a failure. It depends on what 'unclear' means here and how it reflects the results. If you mean a soft failure or an issue that doesn't block the case run, the 'warn' result is similar to it, then calling it 'unclear' is confusing and not accurate in my opinion. In nitrate system, it has 'blocked', 'failed' and 'error' result status to reflect a problem. User guide suggests 'error' is used for test environment that has problems that prevent Test Case execution. I think we can modify 'error' status to include all soft failures. I've added this to the requirements for further evaluation. Thanks, Hurry -- Contacts Hurry FAS Name: Rhe Timezone: UTC+8 TEL: 86-010-62608141 IRC nick: rhe #fedora-qa #fedora-zh -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
On Fri, 2011-01-21 at 19:02 +, Samuel Greenfeld wrote: I do like litmus! It's a nice evolution from testopia for upstream mozilla. We don't currently have an 'unclear' test result. I'm not opposed to it, but would need better understand how that field is used, and the process around it, in litmus. Agree with James. What I believe Mozilla is doing (since I have not had a chance to work with their QA team yet) is flagging test cases with a form of soft failure in that the result of a testcase neither clearly passed, nor clearly failed. So in addition to Passed, Failed, and any other common states (Blocked, In Progress, etc.) you have an Unclear result state. Hurry is somewhat wrong to say we don't currently have an 'unclear' result; we do have the 'warn' result, which is in some ways similar. We usually use it to indicate when a test turns up some kind of anomalous behaviour which isn't exactly a failure. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
Greetings, Many thanks for Samuel's comments, I found out many valuable things to think, identify and add to use cases/feature comparison. The questions also remind me that many testers so far may be still not familiar with Nitrate, therefore can't give any suggestion upon it. So I'll introduce Nitrate a bit here. Nitrate is a new test case management system which is open-sourced recently. It has advantages on managing, tracking and querying cases,results and is similar to testopia. The introduction pages along with some screenshots can be found at: https://fedorahosted.org/nitrate/ https://fedoraproject.org/wiki/Nitrate One can set up a local nitrate system with the steps in the link, we are also planing to build a testing instance during F-15. On Thu, 2011-01-20 at 15:02 -0500, James Laska wrote: On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote: A few more inline comments on a subset of the questions, and two more thoughts: On Thu, Jan 20, 2011 at 11:54 AM, James Laska jla...@redhat.com wrote: On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote: 1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it? This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability. https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013 From the bug I'm guessing this is because Testopia used Ext-JS, and Ext-JS kept changing licenses. Hopefully there will only be licensing restrictions for Nitrate on the software itself, and not the created work of what one does while actively using it. That was a question I had opened with Hurry. I have no idea how we'd license the content in the system. If anything, I'd hope/expect that to match that of our current wiki. Oh, licensing and license issues. I've listed both of them in feature comparison as requirements. For test days and release validation test events use cases, I identified the steps with different permissions by QA, event host and tester(including anonymous user)[#], and hope this can help to license these instances in Nitrate. Of course, I need to extend more to cover other instances. [#] https://fedoraproject.org/wiki/Rhe/tcms_use_cases#Test_Events_Use_Cases 1. How does Nitrate compare to other open closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its strengths weaknesses? See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results. [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel Are the developers actively monitoring/using this list? I looked at it before, and all I saw were three test posts from July in the archives. I believe they do, but don't yet have a strong upstream presence since there hasn't been a lot of code/progress to share until recently. This will be something I'm sure they'll want to improve as community interest increases. Add it into requirements. But I can confirm that all the nitrate developers are monitoring the list now. When community interest increases, it will become active. :) 1. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans? This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance. With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases. Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term. The reason I bring this up is because OLPC's Spin releases tend to lag behind the official Fedora release. For instance, we just released our hopefully
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
On Fri, Jan 21, 2011 at 11:40 AM, He Rui r...@redhat.com wrote: Also: It might be useful to add an unclear testcase result, similar to how Mozilla's Litmus system does it (https://litmus.mozilla.org). I do like litmus! It's a nice evolution from testopia for upstream mozilla. We don't currently have an 'unclear' test result. I'm not opposed to it, but would need better understand how that field is used, and the process around it, in litmus. Agree with James. What I believe Mozilla is doing (since I have not had a chance to work with their QA team yet) is flagging test cases with a form of soft failure in that the result of a testcase neither clearly passed, nor clearly failed. So in addition to Passed, Failed, and any other common states (Blocked, In Progress, etc.) you have an Unclear result state. This forces the result (including a comment section I presume is forced for all non-passing items) to be evaluated during the release cycle to see if it was an actual failure, or if the test case needs to be updated and/or clarified. See the recently unclear and testcases most frequently marked as unclear links at the bottom of pretty much every page of litmus.mozilla.org for examples. On the former, you can click on the dialog icon beneath each result ID to see the comment without switching pages. Personally I like this approach because management of many projects (in my experience) rarely budget time for test cases to be updated. They prefer that you start testing the next release instead. Projects I have been on have required me to use test cases which were last notably updated a few years ago if not 5-10+ years ago by simply acting as if the inaccurate information is not there. By treating potentially incorrect test cases as a form of failure it forces them to be ideally updated and rerun promptly. The downside is if a tester is unfamiliar with an area or needs hand-holding, they might mark a lot of results unclear which actually are rather straightforward. Or alternatively, no one looks at them, and you have a muddy result at the end of the release. These would put additional effort on the core QA team to remedy the situation. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
Unfortunately while I am familiar with a few test case management systems, I have not been involved with the Fedora project long enough to know its workflows. And a quick search online is not turning up much about Nitrate, with its changelog[1] implying it was first made open source software in July 2010. I might be able to provide some general comments though if I knew more about the product. So for those of us unfamiliar with the history of Nitrate, could you please answer the following: 1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it? 2. Is a sample play/sandbox test instance with more-or-less full access available online? 3. How does Nitrate compare to other open closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its straights weaknesses? 4. Is test case test plan import and export (to XML, etc.) support available? If so is this compatible with any other TCMS's import/export system? 5. Are nested test plan and/or test case hierarchies supported? 6. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans? 7. How long will historical test case results be made available? 8. Is there any plan to tie this into Bodhi and other tools to detect updated packages that may imply test cases need re-running and/or updating? 9. Is this going to be available as a Fedorahosted service like Trac is? If so will all the instances be able to share test cases? 10. Is there any concern that changing test tracking systems may encourage/discourage existing testers to participate? Some of this information may be useful to post on the Trac main page and/or in the Fedora wiki. Thanks in advance for your time. --- SJG [1] https://fedorahosted.org/nitrate/browser/trunk/nitrate/docs/ChangeLog On Tue, Jan 18, 2011 at 6:36 AM, He Rui r...@redhat.com wrote: Greetings! Wiki workflow/use cases are reorganized and grouped by general and main test events(runs) use cases. The general cases cover the basic uses of wiki, and the events cases covers the detailed certain steps for organizing the events. Please review it: https://fedoraproject.org/wiki/Rhe/tcms_use_cases Meanwhile, Wiki and Nitrate comparison is also listed and grouped by above use cases. I've listed as many features as I can think out for comparison. Please have a look and see if any features are missed. You can review it by the use case you're familiar with if reviewing all of them is tough, but be aware that features compared in former use cases are not listed in later cases again to avoid overlaps: https://fedoraproject.org/wiki/Rhe/tcms_Comparison Ticket#152[1] is currently set up for tracking this event. Feel free to add comments or discuss it here. Note: the feedback deadline is Jan 21, the end of this week. After that date, I'll move forward to the next phase. Waiting for your feedback! Cheers, Hurry [1] https://fedorahosted.org/fedora-qa/ticket/152 -- Contacts Hurry FAS Name: Rhe Timezone: UTC+8 TEL: 86-010-62608141 IRC nick: rhe #fedora-qa #fedora-zh ___ test-announce mailing list test-annou...@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/test-announce -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote: Unfortunately while I am familiar with a few test case management systems, I have not been involved with the Fedora project long enough to know its workflows. And a quick search online is not turning up much about Nitrate, with its changelog[1] implying it was first made open source software in July 2010. I might be able to provide some general comments though if I knew more about the product. So for those of us unfamiliar with the history of Nitrate, could you please answer the following: Really good list of questions Samuel, thanks for jumping in. I know Hurry can provide feedback on the questions. However, I'll add my thoughts as well. 1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it? Hurry can touch on the goals for using nitrate. As for history, I can add my experiences ... This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability. https://fedoraproject.org/wiki/QA/Testopia_Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013 After this, several folks got together and decided they would implement a new front-end on top of the testopia db schema. This would resolve the original license incompatibility and address usability issues that were raised with the testopia UI. The project started development internally, and was open-sourced in 2010. 1. Is a sample play/sandbox test instance with more-or-less full access available online? Not yet, I believe that's in plan for sometime during Fedora 15. Hurry and I were discussing the requirements for such an instance earlier this week. If you're interested in helping here, just let us know :) 1. How does Nitrate compare to other open closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its straights weaknesses? See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results. [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel 1. Is test case test plan import and export (to XML, etc.) support available? If so is this compatible with any other TCMS's import/export system? I believe import/export is supported using the testopia.dtd format. http://git.fedorahosted.org/git/?p=nitrate.git;a=blob;f=trunk/nitrate/docs/ChangeLog - Fixed #564258 - [REF] Ability to export/print specified cases - Fixed #612803 - add an export feature for test case runs, can export … 1. Are nested test plan and/or test case hierarchies supported? I don't recall. Perhaps Hurry or the nitrate developers know? I know this feature has been discussed a *lot* with nitrate, and other solutions we've used in the past. I don't think support for nested test plans is something we've had a tremendous need for now, so I don't anticipate this being a MUSTHAVE feature in the near term. 1. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans? This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance. With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases. Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term. 1. How long will historical test case results be made available? I suspect the limiting factor here is database size. I'm not aware of any rules or process that would require removal/archival of old results. However, at some point that could certainly be an issue we'd need to plan for. 1. Is there any plan to tie this into Bodhi and other tools to detect updated packages that may imply test cases need re-running and/or updating? That is certainly possible, but there are currently no
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
A few more inline comments on a subset of the questions, and two more thoughts: On Thu, Jan 20, 2011 at 11:54 AM, James Laska jla...@redhat.com wrote: On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote: 1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it? This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability. https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013 From the bug I'm guessing this is because Testopia used Ext-JS, and Ext-JS kept changing licenses. Hopefully there will only be licensing restrictions for Nitrate on the software itself, and not the created work of what one does while actively using it. 1. How does Nitrate compare to other open closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its strengths weaknesses? See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results. [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel Are the developers actively monitoring/using this list? I looked at it before, and all I saw were three test posts from July in the archives. 1. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans? This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance. With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases. Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term. The reason I bring this up is because OLPC's Spin releases tend to lag behind the official Fedora release. For instance, we just released our hopefully final Fedora 11-based release this past month, and are in the early stages of the Fedora 14-based one. At least some Fedora ARM development work may still be going on with Fedora 13 as well. While decently written test cases will survive somewhat over time, major changes in GUI look feel or other areas can break their backwards compatibility. Ideally Nitrate will default to using the current version of a testcase when making a new plan, preferably following the updates of said testcase until a result is committed which forces the test case version to be needed. 1. How long will historical test case results be made available? I suspect the limiting factor here is database size. I'm not aware of any rules or process that would require removal/archival of old results. However, at some point that could certainly be an issue we'd need to plan for. Again; this would be to help lagging Spins and similar. Right now it looks like Bodhi at least publicly hides information for Fedora versions which have reached end-of-life. (Either that, or I don't know how to find it.) I presume that Nitrate has already been determined to be scalable; otherwise it is a big risk to incorporate it into Fedora. Also: It might be useful to add an unclear testcase result, similar to how Mozilla's Litmus system does it (https://litmus.mozilla.org). -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote: A few more inline comments on a subset of the questions, and two more thoughts: On Thu, Jan 20, 2011 at 11:54 AM, James Laska jla...@redhat.com wrote: On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote: 1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it? This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability. https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013 From the bug I'm guessing this is because Testopia used Ext-JS, and Ext-JS kept changing licenses. Hopefully there will only be licensing restrictions for Nitrate on the software itself, and not the created work of what one does while actively using it. That was a question I had opened with Hurry. I have no idea how we'd license the content in the system. If anything, I'd hope/expect that to match that of our current wiki. 1. How does Nitrate compare to other open closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its strengths weaknesses? See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results. [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel Are the developers actively monitoring/using this list? I looked at it before, and all I saw were three test posts from July in the archives. I believe they do, but don't yet have a strong upstream presence since there hasn't been a lot of code/progress to share until recently. This will be something I'm sure they'll want to improve as community interest increases. 1. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans? This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance. With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases. Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term. The reason I bring this up is because OLPC's Spin releases tend to lag behind the official Fedora release. For instance, we just released our hopefully final Fedora 11-based release this past month, and are in the early stages of the Fedora 14-based one. At least some Fedora ARM development work may still be going on with Fedora 13 as well. While decently written test cases will survive somewhat over time, major changes in GUI look feel or other areas can break their backwards compatibility. Ideally Nitrate will default to using the current version of a testcase when making a new plan, preferably following the updates of said testcase until a result is committed which forces the test case version to be needed. I hope so, yes! :) I'm fairly certain support exists for linking against versioned cases, that is, I know it was present in testopia. Hopefully Hurry, or the nitrate folks can help here. Sounds like we need to add this to our list of requirements. 1. How long will historical test case results be made available? I suspect the limiting factor here is database size. I'm not aware of any rules or process that would require removal/archival of old results. However, at some point that could certainly be an issue we'd need to plan for. Again; this would be to help lagging Spins and similar. Right now it looks like Bodhi at least publicly hides information for Fedora versions which have reached end-of-life. (Either that, or I don't know how to find it.) That or garbage collected. Sounds like we'll need to do some further research on this front. I personally see no reason to purge results for EOL'd releases.
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
On Tue, 2011-01-18 at 19:36 +0800, He Rui wrote: Greetings! Wiki workflow/use cases are reorganized and grouped by general and main test events(runs) use cases. The general cases cover the basic uses of wiki, and the events cases covers the detailed certain steps for organizing the events. Please review it: https://fedoraproject.org/wiki/Rhe/tcms_use_cases Meanwhile, Wiki and Nitrate comparison is also listed and grouped by above use cases. I've listed as many features as I can think out for comparison. Please have a look and see if any features are missed. You can review it by the use case you're familiar with if reviewing all of them is tough, but be aware that features compared in former use cases are not listed in later cases again to avoid overlaps: https://fedoraproject.org/wiki/Rhe/tcms_Comparison Ticket#152[1] is currently set up for tracking this event. Feel free to add comments or discuss it here. Note: the feedback deadline is Jan 21, the end of this week. After that date, I'll move forward to the next phase. Waiting for your feedback! Woah, I like the fancy table layout used for comparing use cases between the wiki and nitrate! Thank you for taking the time to identify our current test workflows and offer a comparison between nitrate and the wiki. You've got a really comprehensive list of use cases that will certainly help us evaluate nitrate. While looking through the comparison wiki, I've noted a few comments/questions below. Hope this helps! I'm unclear what the use case 'Creating A Common Page' is, can you elaborate? Is that when we create a category page to group all test results for a milestone (e.g. https://fedoraproject.org/wiki/Category:Fedora_14_Pre-Alpha_Test_Results)? Under 'creating a common page', the row titled 'Submitting format'. Would that be better named as 'Supports data entry using a form'? Under 'creating a common page', there is a row titled 'Subpage using'. I don't think I understand that use case, can you explain? Under section 'Creating A Test Case', there are some red cells for the wiki for (these are also under 'Creating a Test Plan'). What does the 'red' mean. Does it mean that they are not possible, or that the solution provided is not optimal? * Renaming a case - move to another page * Case draft status - manually add draft category/note * Case review status - approved in ticket or somewhere else Also under section 'Creating A Test Case', the row titled 'Test case re-use (write once, link anywhere)' talks about nitrate support for cloning tests. Does this create a new copy of the original, or does it re-use the existing test? If any changes are made to one, are the propagated to the other clone? Under 'Creating A Test Result Page Template', the row titled 'Group cases (by media)', I imagine nitrate supports other methods of grouping/organizing tests in a test run? Oh wait ... this is grouping, not sorting. How would this be done with nitrate, would a test run be created for each media type? Or would we sort tests of different media types differently? Under 'Creating A Test Result Page', the row titled 'Redirecting links' ... is this when a test run has expired, or been replaced by a newer test run? Like if we get RC2, we'll go and adjust RC1 so people know it's no longer active? Not sure what to call this feature, retire a test run? Under 'Posting Test Results', the row titled 'Opening current links' ... I'm trying to think of a more generic name for that feature. Maybe 'Quick access to active test runs' instead? Under 'Generating A Test Summary', the row titled 'Result summary/report generation'. Haha, you list that as *RED* ... very accurate! :) Under 'Administering', the row titled 'License', is this the license for nitrate and mediawiki, or the license for the content in the cases, plans and results? Under 'Release Validation Test Event', the row titled 'Roadmap for each run' ... can you explain, or link to, what is meant by this? Thanks again! James signature.asc Description: This is a digitally signed message part -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Test-Announce] Call for reviewing TCMS use cases and comparison!
Please ignore the last mail. Hi James, Wow, firstly, many thanks for your careful review of all the items.:) On Wed, 2011-01-19 at 16:08 -0500, James Laska wrote: On Tue, 2011-01-18 at 19:36 +0800, He Rui wrote: Greetings! Wiki workflow/use cases are reorganized and grouped by general and main test events(runs) use cases. The general cases cover the basic uses of wiki, and the events cases covers the detailed certain steps for organizing the events. Please review it: https://fedoraproject.org/wiki/Rhe/tcms_use_cases Meanwhile, Wiki and Nitrate comparison is also listed and grouped by above use cases. I've listed as many features as I can think out for comparison. Please have a look and see if any features are missed. You can review it by the use case you're familiar with if reviewing all of them is tough, but be aware that features compared in former use cases are not listed in later cases again to avoid overlaps: https://fedoraproject.org/wiki/Rhe/tcms_Comparison Ticket#152[1] is currently set up for tracking this event. Feel free to add comments or discuss it here. Note: the feedback deadline is Jan 21, the end of this week. After that date, I'll move forward to the next phase. Waiting for your feedback! Woah, I like the fancy table layout used for comparing use cases between the wiki and nitrate! Thank you for taking the time to identify our current test workflows and offer a comparison between nitrate and the wiki. You've got a really comprehensive list of use cases that will certainly help us evaluate nitrate. While looking through the comparison wiki, I've noted a few comments/questions below. Hope this helps! I'm unclear what the use case 'Creating A Common Page' is, can you elaborate? Is that when we create a category page to group all test results for a milestone (e.g. https://fedoraproject.org/wiki/Category:Fedora_14_Pre-Alpha_Test_Results)? No, it means creating a general new page on wiki, like a documentation page etc, and test cases/plans/categories/results pages are particular cases based on creating a normal page like this. It lists what a user may do when creating a new page. Under 'creating a common page', the row titled 'Submitting format'. Would that be better named as 'Supports data entry using a form'? Ah great, I've changed it. Under 'creating a common page', there is a row titled 'Subpage using'. I don't think I understand that use case, can you explain? Okay, for example on wiki, Test_Days is under the path of QA like: https://fedoraproject.org/wiki/QA/Test_Days So I tried to compare if nitrate can also organize sub-page under its main page. Under section 'Creating A Test Case', there are some red cells for the wiki for (these are also under 'Creating a Test Plan'). What does the 'red' mean. Does it mean that they are not possible, or that the solution provided is not optimal? Oh the first two below are not optimal, the last one is not possible on wiki. * Renaming a case - move to another page I think moving to another page is not a good way to rename a case, there will be two link addresses for this case. * Case draft status - manually add draft category/note It's better to have a case status called draft, or proposed like nitrate does. * Case review status - approved in ticket or somewhere else Nitrate will send an email including the proposed case id to reviewer for approval. Then the reviewer can set 'confirmed', or 'need update' status. For wiki, I remember we used to track the cases status in tickets and after approved in tickets or reviewed in test@, the 'draft' note is removed, but no sign indicates the case has been reviewed and approved. Also under section 'Creating A Test Case', the row titled 'Test case re-use (write once, link anywhere)' talks about nitrate support for cloning tests. Does this create a new copy of the original, or does it re-use the existing test? If any changes are made to one, are the propagated to the other clone? Nitrate creates new copies, so if any change is made to one, the other clone won't change. Under 'Creating A Test Result Page Template', the row titled 'Group cases (by media)', I imagine nitrate supports other methods of grouping/organizing tests in a test run? Oh wait ... this is grouping, not sorting. How would this be done with nitrate, would a test run be created for each media type? Or would we sort tests of different media types differently? AFAIK, except the above two methods, adding tags to tests and filtering them by tags is another choice. But it requires testers to filter by themselves. So.. I need think about them and compare these methods. Under 'Creating A Test Result Page', the row titled 'Redirecting links' ... is this when a test run has expired, or been replaced by a newer test run? Like if we get RC2, we'll go and adjust RC1 so people know
[Test-Announce] Call for reviewing TCMS use cases and comparison!
Greetings! Wiki workflow/use cases are reorganized and grouped by general and main test events(runs) use cases. The general cases cover the basic uses of wiki, and the events cases covers the detailed certain steps for organizing the events. Please review it: https://fedoraproject.org/wiki/Rhe/tcms_use_cases Meanwhile, Wiki and Nitrate comparison is also listed and grouped by above use cases. I've listed as many features as I can think out for comparison. Please have a look and see if any features are missed. You can review it by the use case you're familiar with if reviewing all of them is tough, but be aware that features compared in former use cases are not listed in later cases again to avoid overlaps: https://fedoraproject.org/wiki/Rhe/tcms_Comparison Ticket#152[1] is currently set up for tracking this event. Feel free to add comments or discuss it here. Note: the feedback deadline is Jan 21, the end of this week. After that date, I'll move forward to the next phase. Waiting for your feedback! Cheers, Hurry [1] https://fedorahosted.org/fedora-qa/ticket/152 -- Contacts Hurry FAS Name: Rhe Timezone: UTC+8 TEL: 86-010-62608141 IRC nick: rhe #fedora-qa #fedora-zh ___ test-announce mailing list test-annou...@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/test-announce -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
