Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by rhe): [https://fedoraproject.org/wiki/Rhe/tcms_Comparison wiki vs nitrate page] is updated again. Last time there were some unknown fields in nitrate side. By doing some tests on it, I verified several of them and added some new features. Please refer to it. So far, I think the general use cases have been covered and converted to related features for comparison. -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:15 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by rhe): Replying to [comment:13 xcieja]: After familiarize myself with above comparison, better fits to my own preferences Wiki. It seems there are more useful features, sometimes not availible directly but via plugins, even though it looks more user friendly. My score: 1:0 for Wiki Hehe, thanks for voting for wiki. Note that the comparison is based on the use cases on wiki, so most of the compared features so far are already contained in the wiki and that's why the green fields in the wiki side are more than the nitrate side. But with deeper research on Nitrate TCMS, I'll find more features/advantages in it and add more features for comparison. Also, the red fields in Nitrate side are not permanent, they can be resolved by the tcms team if we decide those features as MUST-HAVE. -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:14 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by xcieja): Hi, I have few questions regarding to red fields by Wiki side: 1) '''Review status - manually add reviewer''' -does it mean that if create test case and perform it, thereafter i want to allow others to see the result i need to add particulars users to list in order to give them rigths to see the status ??? 2) '''Result summary/report - -''' -is it not possible at all to generate the report ??? -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:11 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by rhe): I further updated the use cases page: - https://fedoraproject.org/wiki/Rhe/tcms_use_cases And based on these, the use cases between the two tcms are compared in the link below: - https://fedoraproject.org/wiki/Rhe/tcms_Comparison Feel free to add comments. Thanks! -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:10 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by rhe): Replying to [comment:6 xcieja]: Hi, I have found following table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT4]. In my opinion it can serve as draft of functionality we would like to have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on... Now, i think the question is if we can do it via Wiki or TCMS. Do you mean [https://fedoraproject.org/wiki/User_talk:Johannbg/Draft/QA/Test_Days/Features/EXT4 this example]? AFAIK, for wiki, one have to manually rename the page link when needed. For Nitrate TCMS, you can use 'alias' and 'tag' attitude of each test case to correspond them. -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:7 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by adamwill): Replying to [comment:6 xcieja]: Hi, I have found following table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT4]. In my opinion it can serve as draft of functionality we would like to have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on... Now, i think the question is if we can do it via Wiki or TCMS. Of course we can do it via the Wiki, as that example is done using the Wiki =) we already use a similar table format for test case results for most test days, and the validation testing. The drawbacks of using the wiki to track results are mostly in cleanliness (it's too free-form so users can mess up the tables), consistency (we don't really store all the results for any given test case in one place, instead we store results for *some particular event* which referenced those test cases), flexibility (see what rhe wrote) and comprehensiveness (a more sophisticated system could probably include more detail for each result, perhaps automatically retrieved from the user). -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:9 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by rhe): The [https://fedoraproject.org/wiki/Rhe/tcms_requirements_proposal proposal page] has been updated and written in the table format. To further identify this issue, the following items will be analysed in details one by one: - Identify our current test workflows (or use cases) - Identify problems with our current system and workflows - e.g. what would we like to do but are difficult/impossible with wiki - Objectively compare feature sets of both tools in relation to the important workflows - Identify+prioritize any feature gaps we'd need from nitrate in order to make a smooth transition - Document a plan/roadmap for the transition Where each item will be documented in a separate page, and the use cases page for the 1st one has been drafted: - https://fedoraproject.org/wiki/Rhe/tcms_use_cases Feel free to add comments or tell some uncovered use cases. Thanks. -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:8 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by xcieja): Hi, I have found following table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT4]. In my opinion it can serve as draft of functionality we would like to have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on... Now, i think the question is if we can do it via Wiki or TCMS. -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:6 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by rhe): Replying to [comment:4 aalam]: TCMS is really good idea for Test Creation and Manage Test plan than wiki. Is it necessary to have permission other than view for anonymous user, like Translation team allow only FAS account user to translation on http://translate.fedoraproject.org? I personally consider it as a NICE-TO-HAVE, since it's one big advantage for wiki-based system to allow anonymous user editing pages with certain name space, like 'test day:' and 'Test_Results:'. So far the nitrate system only has read permission for anonymous user, but this can be customized if we review the requirement and decide to use it. -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:5 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
On Fri, 2010-12-03 at 23:33 +0100, Karol Cieśla wrote: Hi, i agree with both of you ,so that rather than just sending e-mails i would like initiate something which could improve the current situation. So the steps I see now look as follows: I stage 1) someone could sketch out/outline/describe the functionality of Wiki/TCMS we would like to have for the test cases (if possible also provide graphical form) -i.e table, page, buttons Hurry and I discussed this topic yesterday on IRC. I suspect she'll take a similar approach, but the general idea will be to first identify what our QA community needs are when it comes to test documentation/results. For example, what works well, what isn't working well, where do we want to improve when it comes to our current test management. 2) discuss with the whole community such form -pros,cons As always, I expect the process will be transparent and there will be plenty of opportunity for anyone interested to get involved and help move things forward. 3) reach out to people maintaining the Wiki and ask them if they can create such form wit help of Wiki/TCMS and can sustain bigger number of test cases (i.e up to 300) If the analysis shows that investing in wiki-based solution is the most sustainable option forward, there are several solutions available (http://www.mediawiki.org/wiki/Extension:Semantic_Forms and possibly http://www.mediawiki.org/wiki/Extension:StructuredInput). My impression with Semantic was that while it may scratch the wiki form-input itch for us, it's certainly not without a startup cost and I'm still unclear whether that's the right tool for the job. It's a fun experiment, but without knowing what where our current gaps are, and where we want to improve, it's hard to make a decision to invest in semantic-mediawiki over another actively maintained upstream project. II stage To ponder as the whole community the process of adding/creating/deleting/marking/prioritizing test cases. While it would be an interesting discussion, I don't know if we'd be able to move forward with actionable work after a general community ponder session. I'd fear that would revolve too much around ponies [1]. I'm inclined to put emphasis on reviewing what we're doing now, understand why we're doing it, document the pro's con's and prioritize the MUSTHAVE features. P.S -i can try to prepare for the middle of the next week some draft to point 1 -it will be something simple but reflecting the idea. Always welcome additional hands to help move the discussion forward! Thanks, James W dniu 2010-12-03 17:36, James Laska pisze: Just a few extra thoughts on the subject ... On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote: On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote: Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories. That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result). I think we could organise them better -i.e create test category and put all of them instead of many places. We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases. Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all. We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability- better overall opinion. Sure, we can always do with more test cases. More test cases/plans would certainly change the conversation a bit. I think we all want to increase the value that the Fedora QA team can offer to the project. One way to increase our value is by improving our test coverage by way of test documentation (procedures, plans and cases). There are plenty of other ways ... but we can save those for other threads. I've always been hesitant to add tests for the sake of adding tests. Test plans/cases are just like software. If the tests aren't addressing a priority issue, they won't be used as much, and like unused software, will suffer from bit rot. The best test cases/plans are the ones frequently used, referenced and have maintainer buy-in. Meaning, if the tests fail, the maintainer cares. I want to grow the library of tests we maintain and run, but hopefully grow in a manner and pace that we, as a community, can sustain.
Re: Re: Re: [Fedora QA] #152: Test Cases Management
Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories. I think we could organise them better -i.e create test category and put all of them instead of many places. Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all. We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability- better overall opinion. Please don`t get me wrong -if it works fine like now i don`t want to create something new.Please treat as as comments of newcomer in the project. Regards, Karol W dniu 20:59, Adam Williamson pisze: On Thu, 2010-12-02 at 23:12 +0100, xcieja wrote: Hi, i can say i am new to Fedora in general as project but i have participated in many testing campaigns professionaly, so honestly i see this TCMS(?) or any other way to organise test cases as big add value. It would be really great if we had a list of test cases for whole system (of course everyone can contibute) and mark each of them as successful/failed, track the status etc... The one of the advantages is that i am looking at such list of test cases and i am able to easily figure out what has been already tested and wht remaining is. If have missed something and such 'thing exists please correct me an forget the idea, otherwise what do you think ?? We already do this, for most of our programmed testing, via the Wiki, using matrices. See: https://fedoraproject.org/wiki/Test_Results:Current_Desktop_Test https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test for instance. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Re: Re: [Fedora QA] #152: Test Cases Management
On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote: Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories. That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result). I think we could organise them better -i.e create test category and put all of them instead of many places. We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases. Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all. We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability- better overall opinion. Sure, we can always do with more test cases. Please don`t get me wrong -if it works fine like now i don`t want to create something new.Please treat as as comments of newcomer in the project. Not at all :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
Hi, i agree with both of you ,so that rather than just sending e-mails i would like initiate something which could improve the current situation. So the steps I see now look as follows: *I stage* _1)_ someone could sketch out/outline/describe the functionality of Wiki/TCMS we would like to have for the test cases (if possible also provide graphical form) -i.e table, page, buttons _2)_ discuss with the whole community such form -pros,cons _3)_ reach out to people maintaining the Wiki and ask them if they can create such form wit help of Wiki/TCMS and can sustain bigger number of test cases (i.e up to 300) *II stage* To ponder as the whole community the process of adding/creating/deleting/marking/prioritizing test cases. P.S -i can try to prepare for the middle of the next week some draft to point 1 -it will be something simple but reflecting the idea. Regards, Karol W dniu 2010-12-03 17:36, James Laska pisze: Just a few extra thoughts on the subject ... On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote: On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote: Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories. That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result). I think we could organise them better -i.e create test category and put all of them instead of many places. We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases. Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all. We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability- better overall opinion. Sure, we can always do with more test cases. More test cases/plans would certainly change the conversation a bit. I think we all want to increase the value that the Fedora QA team can offer to the project. One way to increase our value is by improving our test coverage by way of test documentation (procedures, plans and cases). There are plenty of other ways ... but we can save those for other threads. I've always been hesitant to add tests for the sake of adding tests. Test plans/cases are just like software. If the tests aren't addressing a priority issue, they won't be used as much, and like unused software, will suffer from bit rot. The best test cases/plans are the ones frequently used, referenced and have maintainer buy-in. Meaning, if the tests fail, the maintainer cares. I want to grow the library of tests we maintain and run, but hopefully grow in a manner and pace that we, as a community, can sustain. With the test plans that Adam points to, I'm pretty confident in our ability to develop, discuss/debate and execute desktop and installation tests as a community. We've ironed out the kinks in the workflow, increased community engagement and developed good test plans as a result. My impression is we are ready for additional test areas. That's what's exciting to me about the proventesters effort. As you can tell from recent (and old) devel@ list threads, testing proposed updates is important work that's needed, requested by package maintainers and well under-documentated. I don't worry as much that tests written for frequent proventester use will go stale given it's been a long-standing exposure in the project. Also, given the huge number of components in Fedora, there is room for just about every contributor to participate and carve out a niche. But which tests do we prioritize first, where do we write the tests, where to review+discuss them, how to run them etc... (more on this later). For me, these are two separate (but related) efforts. TCMS is tool designed to address specific workflow/tracking needs. We also need to determine how to best to sustainably expand the test coverage we can offer to the project. We have a wiki-based TCMS now. It has met our needs for the current set of organized test efforts. It's not perfect, but the return on the investment has been huge. The questions I'd like to see answered in ticket#152, is (1) whether the wiki can continue to scale as our test management needs grow, and (2) what aspects of our wiki-based TCMS are good/bad? Thanks, James -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by adamwill): I know James has been talking about this with some RH people. We discussed it on the phone and agreed that we like the idea in principle but the ease and flexibility of the current wiki system is something we don't want to lose, so we thought we should set some quite high barriers: any tcms system must be as easy to use (both for those creating the test cases and those running them) as the wiki is. -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:1 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: [Fedora QA] #152: Test Cases Management
#152: Test Cases Management --+- Reporter: rhe | Owner: rhe Type: enhancement | Status: new Priority: major| Milestone: Fedora 15 Component: Wiki | Version: Resolution: |Keywords: retrospective --+- Comment (by jlaska): I love the idea of doing a test event with a pilot instance of nitrate/TCMS. It was fun doing that with the semantic-mediawiki, and I think we learn a lot with that approach. We've all talked about TCMS and how best to integrate it with our workflow. There are definite shortcomings to our current wiki-based approach that limit our ability to growth, but there are plenty of positive lessons learned that I think we should recognize and prioritize. Hurry: Would you be willing to start drafting some wiki content so we can capture this? Either a requirements or technology comparison document? I don't want this to be a free-for-all 'I want a pony' list of features we want of a TCMS solution. But perhaps starting with documenting the pro's and con's with our current wiki-based workflow? The goal for me would be to identify what our MUST-HAVE and NICE-TO-HAVE's are from any solution. Ideally, we can use that to determine the best route forward (whether it's filing bugs/tickets/rfe's against TCMS, or something else). -- Ticket URL: https://fedorahosted.org/fedora-qa/ticket/152#comment:2 Fedora QA http://fedorahosted.org/fedora-qa Fedora Quality Assurance -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Re: [Fedora QA] #152: Test Cases Management
Hi,
i can say i am new to Fedora in general as project but i have
participated in many testing campaigns professionaly, so honestly i see
this TCMS(?) or any other way to organise test cases as big add value.
It would be really great if we had a list of test cases for whole system
(of course everyone can contibute) and mark each of them as
successful/failed, track the status etc...
The one of the advantages is that i am looking at such list of test
cases and i am able to easily figure out what has been already tested
and wht remaining is.
If have missed something and such 'thing exists please correct me an
forget the idea, otherwise what do you think ??
Regards,
Karol
W dniu 20:59, Fedora QA pisze:
ם�z�BjǬ1�ځ鞞ߑz�+���㰝秅��zg��+Z��'{늊�)���̊W�h��y
���w��h��W��*'E�(��b�r��
+v��,��-��
��z{[ɧZ����'�Z��!j��z{Z�H�a���(��G��)��v+ �� v+h��^�'y��j
�y�aj� �) ��u梞��ȩ^y� jw_��bn)b�* ����ޞ��+2�צ��(��a�x0y�'����%�Ǭ��� �.�� ��zb�ڮ���Ƨ���̬�馺�[y� j̭�� n�a~�톋 r���)��z�\jǬjwm�� ���x-�隲�^�)���rG�Q��l���v�چ�-y�+���v�ک��� ��y��Wm���y�+j,��h��y�+j��y�+i ��+r�.���e===
--
test mailing list
test@lists.fedoraproject.org
To unsubscribe:
https://admin.fedoraproject.org/mailman/listinfo/test
