Re: Re: Re: [Fedora QA] #152: Test Cases Management
Just a few extra thoughts on the subject ... On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote: > On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote: > > Hi, > > yes, you are right there are tests, but in my opinion they are in few > > different places under different categories. > > That wasn't what I meant: I meant we already use the Wiki for the > purposes you identified as an advantage of a TCMS (listing the tests > that need to be performed in relation to some specific process, and > whether they have already been performed, by whom, and with which > result). > > > I think we could organise them better -i.e create test category and put > > all of them instead of many places. > > We sure could, but we don't necessarily need a TCMS to do this. :) Note > that we do try to keep them all within one Wiki namespace and we do use > Wiki categories to organize some test cases. > > > Moreover, i just have taken a look briefly and i see there are round 100 > > test cases in total (please correct me if i am wrong).I think that for > > such project/system it is not enough at all. > > > > We have big community, let`s assume everyone from QA create one test, we > > will have quite huge number of tests and obviously more faults detected > > before main release, less corrections after=better stability,usability-> > > better overall opinion. > > Sure, we can always do with more test cases. More test cases/plans would certainly change the conversation a bit. I think we all want to increase the value that the Fedora QA team can offer to the project. One way to increase our value is by improving our test coverage by way of test documentation (procedures, plans and cases). There are plenty of other ways ... but we can save those for other threads. I've always been hesitant to add tests for the sake of adding tests. Test plans/cases are just like software. If the tests aren't addressing a priority issue, they won't be used as much, and like unused software, will suffer from bit rot. The best test cases/plans are the ones frequently used, referenced and have maintainer buy-in. Meaning, if the tests fail, the maintainer cares. I want to grow the library of tests we maintain and run, but hopefully grow in a manner and pace that we, as a community, can sustain. With the test plans that Adam points to, I'm pretty confident in our ability to develop, discuss/debate and execute desktop and installation tests as a community. We've ironed out the kinks in the workflow, increased community engagement and developed good test plans as a result. My impression is we are ready for additional test areas. That's what's exciting to me about the proventesters effort. As you can tell from recent (and old) devel@ list threads, testing proposed updates is important work that's needed, requested by package maintainers and well under-documentated. I don't worry as much that tests written for frequent proventester use will go stale given it's been a long-standing exposure in the project. Also, given the huge number of components in Fedora, there is room for just about every contributor to participate and carve out a niche. But which tests do we prioritize first, where do we write the tests, where to review+discuss them, how to run them etc... (more on this later). For me, these are two separate (but related) efforts. TCMS is tool designed to address specific workflow/tracking needs. We also need to determine how to best to sustainably expand the test coverage we can offer to the project. We have a wiki-based "TCMS" now. It has met our needs for the current set of organized test efforts. It's not perfect, but the return on the investment has been huge. The questions I'd like to see answered in ticket#152, is (1) whether the wiki can continue to scale as our test management needs grow, and (2) what aspects of our wiki-based TCMS are good/bad? Thanks, James signature.asc Description: This is a digitally signed message part -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Re: Re: [Fedora QA] #152: Test Cases Management
On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote: > Hi, > yes, you are right there are tests, but in my opinion they are in few > different places under different categories. That wasn't what I meant: I meant we already use the Wiki for the purposes you identified as an advantage of a TCMS (listing the tests that need to be performed in relation to some specific process, and whether they have already been performed, by whom, and with which result). > I think we could organise them better -i.e create test category and put > all of them instead of many places. We sure could, but we don't necessarily need a TCMS to do this. :) Note that we do try to keep them all within one Wiki namespace and we do use Wiki categories to organize some test cases. > Moreover, i just have taken a look briefly and i see there are round 100 > test cases in total (please correct me if i am wrong).I think that for > such project/system it is not enough at all. > > We have big community, let`s assume everyone from QA create one test, we > will have quite huge number of tests and obviously more faults detected > before main release, less corrections after=better stability,usability-> > better overall opinion. Sure, we can always do with more test cases. > Please don`t get me wrong -if it works fine like now i don`t want to > create something new.Please treat as as comments of newcomer in the project. Not at all :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Re: Re: [Fedora QA] #152: Test Cases Management
Hi, yes, you are right there are tests, but in my opinion they are in few different places under different categories. I think we could organise them better -i.e create test category and put all of them instead of many places. Moreover, i just have taken a look briefly and i see there are round 100 test cases in total (please correct me if i am wrong).I think that for such project/system it is not enough at all. We have big community, let`s assume everyone from QA create one test, we will have quite huge number of tests and obviously more faults detected before main release, less corrections after=better stability,usability-> better overall opinion. Please don`t get me wrong -if it works fine like now i don`t want to create something new.Please treat as as comments of newcomer in the project. Regards, Karol W dniu 20:59, Adam Williamson pisze: > On Thu, 2010-12-02 at 23:12 +0100, xcieja wrote: >> Hi, >> i can say i am new to Fedora in general as project but i have >> participated in many testing campaigns professionaly, so honestly i see >> this TCMS(?) or any other way to organise test cases as big add value. >> >> It would be really great if we had a list of test cases for whole system >> (of course everyone can contibute) and mark each of them as >> successful/failed, track the status etc... >> >> The one of the advantages is that i am looking at such list of test >> cases and i am able to easily figure out what has been already tested >> and wht remaining is. >> >> If have missed something and such 'thing" exists please correct me an >> forget the idea, otherwise what do you think ?? > We already do this, for most of our programmed testing, via the Wiki, > using matrices. See: > > https://fedoraproject.org/wiki/Test_Results:Current_Desktop_Test > https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test > > for instance. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Re: [Fedora QA] #152: Test Cases Management
On Thu, 2010-12-02 at 23:12 +0100, xcieja wrote: > Hi, > i can say i am new to Fedora in general as project but i have > participated in many testing campaigns professionaly, so honestly i see > this TCMS(?) or any other way to organise test cases as big add value. > > It would be really great if we had a list of test cases for whole system > (of course everyone can contibute) and mark each of them as > successful/failed, track the status etc... > > The one of the advantages is that i am looking at such list of test > cases and i am able to easily figure out what has been already tested > and wht remaining is. > > If have missed something and such 'thing" exists please correct me an > forget the idea, otherwise what do you think ?? We already do this, for most of our programmed testing, via the Wiki, using matrices. See: https://fedoraproject.org/wiki/Test_Results:Current_Desktop_Test https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test for instance. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Re: [Fedora QA] #152: Test Cases Management
Hi, i can say i am new to Fedora in general as project but i have participated in many testing campaigns professionaly, so honestly i see this TCMS(?) or any other way to organise test cases as big add value. It would be really great if we had a list of test cases for whole system (of course everyone can contibute) and mark each of them as successful/failed, track the status etc... The one of the advantages is that i am looking at such list of test cases and i am able to easily figure out what has been already tested and wht remaining is. If have missed something and such 'thing" exists please correct me an forget the idea, otherwise what do you think ?? Regards, Karol W dniu 20:59, Fedora QA pisze: > ם�z�BjǬ1�ځ鞞ߑz�+���㰝秅��zg��+Z��'{늊�)���̊W�h��y > ���w��h��W��*'E�(��b�r�� > +v��,��-�� > ��z{[ɧZ����'�Z��!j��z{Z�H�a��"�(��G��)��v+��v+h��^�'y��j > �y�aj��)��u梞��ȩ^y�jw_��bn)b�*����ޞ�"�+2�צ��(��a�x0y�'����%�Ǭ����.����&zb�ڮ���Ƨ��&�̬�馺�[y�j̭��n�a~�톋r���)��z�\jǬjwm�����x-�隲�^�)"���rG�Q��l���v�چ�-y�+���v�ک�����y��&Wm���y�+j,��h��y�+j��y�+i��+r�.���e=== -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test