Re: [Libreoffice-qa] Creating test cases for Litmus server
Rimas Kudelis píše v Po 14. 11. 2011 v 21:53 +0200: > > I have described how to contribute a test case at > > https://wiki.documentfoundation.org/QA/Testing/Test_Cases_Contribution > > > > It allows both ways: wiki page and mailing list. > > Petr, did you account for the fact the the initial assumption (only > super admins can add testcases) is now invalid? Is current granularity > sufficient, or is there a demand for even more control over who can do what? My understanding is that people can't edit test cases out of box. Someone needs to add them to the special group. I think that we should not give these rights automatically. It should work the same way like the commit rights for git LO sources. People need to send few patches first. If they are good, they get commit rights. Does it make sense? Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Creating test cases for Litmus server
2011.11.15 10:42, Petr Mladek rašė: > Rimas Kudelis píše v Po 14. 11. 2011 v 21:53 +0200: >>> I have described how to contribute a test case at >>> https://wiki.documentfoundation.org/QA/Testing/Test_Cases_Contribution >>> >>> It allows both ways: wiki page and mailing list. >> Petr, did you account for the fact the the initial assumption (only >> super admins can add testcases) is now invalid? Is current granularity >> sufficient, or is there a demand for even more control over who can do what? > My understanding is that people can't edit test cases out of box. > Someone needs to add them to the special group. > > I think that we should not give these rights automatically. It should > work the same way like the commit rights for git LO sources. People need > to send few patches first. If they are good, they get commit rights. > > Does it make sense? Sure it does! :) Rimas ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
[Libreoffice-qa] FOSDEM: LibreOffice dev-toom talks ...
Hi guys, The FOSDEM organisers have kindly agreed to dedicate a developer room to LibreOffice development at the conference ( book now for 4-5 Feb 2012: http://fosdem.org/2012/ :-) So - that means we're eager to get -technical- talks (FOSDEM really is a true hackers conference), about LibreOffice its development, code structure, how to get involved with easy hacks, QA tooling, l10n improvements, etc. If you've done something cool, but would only want five minutes to talk about it, that's fine too: we can have a session putting together a number of smaller more focused talks. Perhaps you want to persuade people that it is in fact possible to get code into LibreOffice by show-casing your easy-hack; whatever - please submit a proposal :-) You can do that in the wiki, just copy the blank session template there and fill it out for yourself (as Fridrich has done): http://wiki.documentfoundation.org/Marketing/Events/Fosdem2012 And the TSC will decide on the program closer to the time. I suspect we'll try to have a team meal there too one evening (if we can find a big enough room); we did last year, and in general - it should be great fun: celebrating the (immanent) release of 3.5.0 at that date I hope. Hope to see you there, Regards, Michael. -- michael.me...@suse.com <><, Pseudo Engineer, itinerant idiot ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Hi guys, first, I am sorry that this mail does react on the last mail from Rimas. I started to write this replay in the morning. I had to interrupt it for some reasons. I think that the ideas mentioned below still make sense. Though, I might change my preference after reading Rimas's reply. Yifan Jiang píše v Út 15. 11. 2011 v 13:34 +0800: > On Mon, Nov 14, 2011 at 10:25:28PM +0200, Rimas Kudelis wrote: > > Well, as an alternative, branches/groups/subgroups could be reviewed > > again. :) > > The idea is actually brilliant! Fully agree! > The following is something they would look like, assume we don't change a lot > the whole structure of current organization. > > Case 1. Carry priority information with subgroups name > Case 2. Carry priority information with groups name > Case 3. Carry priority information with branches > I am slightly inclined to the Case 3, it still keeps most of original things > (groups and subroups), which have already been divided in a clear logic and > not that easy to get familiar with. Also Case 3 is the most easy way if we > want to create different runs in different phase of release. I think that creating test run is only one scenario. I think that we need to think about all use cases: A. Maintenance -- The are few actions: + localize test - has two subactions: + creating the structure: If we could copy the whole group, solution 1 is fastest because there are less groups and branches. Solution 2 is good because you need not switch between branches. If we need to create the whole structure from scratch, it does not matter because the total number of subgroups is always the same + checking differences and localizing If you select another branch in the search dialog, you need to select group and subgroup again => Solution 1 is better than 2 and 2 is better than 3 because if is easier to switch between subgroups; it is easier to between groups than branches + adding/updating test cases: + IMHO, it is similar to checking and localizing => 1 is better than 2 and 2 is better than 3 because it is easier to switch subgroups than groups and switch groups than branches B. Creating Branches for new release + 1 or 2 are easier than 3 because they have less branches C. Creating Test Runs - + I do not know how this works. You say that 3 is the easiest way because you just create run from a branch? + IMHO, 1 would be nightmare because you would need to select to many subgroups manually if you want to test only P1 test. D. Testing -- This is very important because many people will do this many time!!! I think that we have 2 actions: + doing the test People need to find easily what tests are not finished. You need to enter a test run to see the statistic => we should have as few active test runs as possible if 3 produces more test runs, I am against it => 1 or 2 is better + analyzing results: I am not sure how we check the state of testing; the best solution would be to check on page and see how many tests are finished; how many tests succeded and failed I guess that the solution 3 is bad from this point of view. Summary: 3 is bad for testing. 1 is bad for creating test runs. 2 is the best compromise in most situations. => I prefer 2. Note that 2 has better granularity. We will have many branches in the long term for released products. Solution 1 produces too many subgroups in a single group. = Hmm, when I think about it. I am somehow afraid that we would have too many subgroups in the l10n branch. It does not mater how we split it but it will be hard to maintain. Do we really expect that many l10n-specific test cases for Draw, Impress, Base, or Calc? Do we really need the subgroups? What about the following structure: + function tests: + P1 + General + Base + Calc + ... + P2 + General + Base + ... + l10n test: + P1 + en + de + fr + it + P2 + en + de + fr
Re: [Libreoffice-qa] Test case naming
Rimas Kudelis píše v Út 15. 11. 2011 v 13:43 +0200: > Hi all, > > I took a quick look at how testruns are created and what we can and > cannot choose when creating a testrun. My suggestion is based on the > fact that each testrun comprises of > > * all subgroups in > * one or more testgroups in > * exactly one branch in > * exactly one product. > > That is, a testrun cannot span through multiple branches or products, > nor can you specify which subgroups to add to it. You can only choose > groups. This is important information. Thanks for looking at it. > > Overall satistics: > > > > We now have 3 branches: > > > > master regression branch > > - 1 group > > - 7 subgroups in each group > > master l10n branch > > - 4 groups > > - 7 subgroups in each group > > master feature branch > > - we may not consider case priority here? > > Current scheme does not allow creating a catch-all testrun comprising > all tests for a single product/branch. I don't know whether or not such > ability is desired, but it looks logical to me. You are right. 1 run is the best solution. Well, 2 or three test runs for a release still might be acceptable. > Also, maybe I shouldn't look that far into future, but I hope that > there will come a day when proper localization of testcases will be > possible (that is, instead of creating a clone of testcase X in > another language, we would actually be able to translate testcase X > into that language). With that in mind, current testgroups (which > represent different locales) would become unneeded. I am not sure if the l10n tests will be pure word-to-word translation. Some things are done completely different in different languages, for example the layout of the letter template, right-to-left language features, decimal number delimiter, dates. I am sure that some languages would need special tests. BTW: How is the localization used during test run? I know that I select locale in the "Run Tests - Your Chosen Test Run" dialog, see https://wiki.documentfoundation.org/Litmus/Litmus_User_Guide#Enter_Test_Configuration If I select "de", will I see only the "de" test cases from the l10n branch? I am not sure if it is important but it would be nice. > > Case 1. Carry priority information with subgroups name > This scheme would make it impossible to create a testrun for e.g. just > P1 tests. Or just writer tests. I guess this is pretty much the same as > what we have now, so all criticism from above paragraph applies here too. Good point. We can't use this. > > Case 2. Carry priority information with groups name > This is probably OK if you don't mind having to create three testruns > for each version It is not ideal but acceptable. > and be unable to create testruns for separate components IMHO, this is fine. > . I think you forgot locales here though. I assume that for > each locale/priority there would be a separate testgroup? I guess so. Hmm, that would create too many test groups. See my other mail. > > Case 3. Carry priority information with branches > IMO, Case 3 is the worst case, because for each version of LibreOffice, > you would have to create 4*3=12 testruns – one for each priority/test > type combination. I am not sure if it is the worst case because the solution 1 was bad as well :-) Anyway, I think that 3 is not a way to go. > My suggestion is to have a single branch, but carry Priority, Locale and > Component information in the Testgroup, and to represent test type by a > subgroup. That is, what is now a branch, would become a subgroup, and > everything else would become a testgroup (see the attached PDF file). Hmm, I am afraid that we would get too many testgroups. It produces 7 (General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one language. We have 109 localizations in sources => we could end up with more than 700 test groups. > This allows to: > * create testruns based on priority, locale, component, or any > combination of these The question is what test runs we will want to create. > * create a single catch-all testrun for a single version of LibO would be nice > * share the same General Functional Tests subgroup between testgroups > designated for different locales (that is, you would create this > subgroup once, and add it to all locale groups) I am not sure what you mean with this. Could we share subgroups between test groups in Litmus? Is it clearly visible or is hard to maintain and see? BTW: What do you mean with the "Basis Functional Test"? We do not have this subgroup in the current structure. > * drop 75% of the testgroups (there would be about 28 of them initially) > if/when proper testcase localization is implemented, still not rendering > the remaining testgroups useless. I am not sure if we really could drop them. Well, the question is if we need to split between lang-dependent and lang-independent on the t
Re: [Libreoffice-qa] Test case naming
Hi Petr, 2011.11.15 17:30, Petr Mladek rašė: > Rimas Kudelis píše v Út 15. 11. 2011 v 13:43 +0200: >> Also, maybe I shouldn't look that far into future, but I hope that >> there will come a day when proper localization of testcases will be >> possible (that is, instead of creating a clone of testcase X in >> another language, we would actually be able to translate testcase X >> into that language). With that in mind, current testgroups (which >> represent different locales) would become unneeded. > I am not sure if the l10n tests will be pure word-to-word translation. > Some things are done completely different in different languages, for > example the layout of the letter template, right-to-left language > features, decimal number delimiter, dates. I am sure that some > languages would need special tests. I didn't say they have to be translated word-to-word. They should be *localized*, and I would expect a localized testcase to suggest localized number and date formats and other stuff. RTL, on the other hand, might probably need a few additional testcases. Though not many, I would guess. > BTW: How is the localization used during test run? It's not. It's just shown in the test results. > I know that I select locale in the "Run Tests - Your Chosen Test Run" > dialog, see > https://wiki.documentfoundation.org/Litmus/Litmus_User_Guide#Enter_Test_Configuration > > If I select "de", will I see only the "de" test cases from the l10n > branch? I am not sure if it is important but it would be nice. No, currently you will still see all testcases. My idea with localizable testcases though is pretty much about this. If testcase X was localized instead of duplicated, it would only be shown once, in user's preferred locale (I don't know yet how exactly the user would "prefer" that locale though and what a user who would want to test more than one locale would do). >> My suggestion is to have a single branch, but carry Priority, Locale and >> Component information in the Testgroup, and to represent test type by a >> subgroup. That is, what is now a branch, would become a subgroup, and >> everything else would become a testgroup (see the attached PDF file). > Hmm, I am afraid that we would get too many testgroups. It produces 7 > (General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one > language. We have 109 localizations in sources => we could end up with > more than 700 test groups. Which I guess is not realistic, at least in the short term. ;) Right now we only have four languages into which testcases are translated. With 50, it's of course a different story, but with 4 to 10... it's probably still manageable enough. >> This allows to: >> * create testruns based on priority, locale, component, or any >> combination of these > The question is what test runs we will want to create. > >> * create a single catch-all testrun for a single version of LibO > would be nice > >> * share the same General Functional Tests subgroup between testgroups >> designated for different locales (that is, you would create this >> subgroup once, and add it to all locale groups) > I am not sure what you mean with this. Could we share subgroups between > test groups in Litmus? Is it clearly visible or is hard to maintain and > see? Like I mentioned before, when you create/edit a testgroup, you can add the subgroups you want to it. So subgroups can be shared, yes. > BTW: What do you mean with the "Basis Functional Test"? We do not have > this subgroup in the current structure. Don't we? https://tcm.documentfoundation.org/manage_testgroups.cgi?testgroup_id=58 >> * drop 75% of the testgroups (there would be about 28 of them initially) >> if/when proper testcase localization is implemented, still not rendering >> the remaining testgroups useless. > I am not sure if we really could drop them. Well, the question is if we > need to split between lang-dependent and lang-independent on the top > level. I don't think there are many lang-dependent testcases. Desired result may depend on the language, yes, but the testcase itself? > Heh, it seems that it is quite complex problem. I am going to write > another mail where I will try to summarize some ideas :-) Good luck with that. :) Rimas ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
[Libreoffice-qa] Test Structure in Litmus
Hi, I am going to summarize the ideas from the mail "Test case naming". I am sorry for the long mail. The problem is not easy. There are many ideas. I tried to sort them from different point of view. I hope that I did not miss anything important and it will help to find the final conclusion. -- Let's start by a short summary. Litmus allows to split test cases by: + product + branch + testgroup + subgroup We want to split test cases by: + priority (P1,P2,P3,P4) + component (General,Base,Calc,Draw,Impress,Math,Writer) + meaning (functional, l10n, feature) + l10n tests in languages (en,de,fr,it,ja,.) + platform (Linux,Windows,MAC) We need to take care of the following scenarios: + maintenance: + creating test cases + updating test cases + localization + branch for new release + test run for new release/build + doing tests + analyzing results --- Let's look at some conditions: 1. priority: + why + reduced test run for bugfix releases + overal prioritization of the work + main requirements: + allow to create test run for given priority + easily run tests according to priority + easily see test results by priority 2. component: + why + reduced test run if only writer is modified + helps to split work by expertize + main requirements: + allow to create test run for given component + allow to run tests by component 3. meaning: + why + functional tests do not depend on localization + the list of language dependent tests might be different for each language + feature tests are not well described; are done only once by one or few dedicated people + requirements: + function test appears only once in test run; can be approved by any user with any locale and any platform + language dependent tests must be approved more times in all locales + test results must clearly show what tests were done; functional tests are important by priority; l10n tests are important by priority and language; e.g. German is more important than Czech because there are more active users 4. platform: + why: + some tests might depend on platform; for example the open/save dialogs look different way on Linux,Windows, and MAX; also KDE vs. GNOME + main requirements: + these tests must be done on all affected platforms + test results must clearly show where it was tested 5. Maintenance: + problems: + searching for a particular test case + put test cases into the right position + comparing en-US template with localized test cases + localizing the whole bunch of test cases into new language + requirements: + balance the number of branches, testgroups, subgroups, and test cases in subgroup + must be easy to copy test cases for new language + often switched criteria must be in high level; for example, do people switch more often between priority or category or language when they work with the test cases? 6. Branch for new release: + requirements: + must be easy to do the branch + must be easy to see all tests for the given release 7. Test run creation: + requirements: + 1 or very small number of active test runs because it is compilcated to switch between them and analyze results from many test runs + create test run for a given priority (basic vs. full test) + create test run for a given component (only Writer modified) 8. Doing tests: + requirements: + must be easy to search what needs testing + must be possible to sort testing by priority and component + priority is more important than component because it does not help to have perfectly tested Writer when Base is not tested at all 9. Test analyze: + requirements: + must be easy to see what was tested by priority, co
Re: [Libreoffice-qa] Test case naming
Rimas Kudelis píše v Út 15. 11. 2011 v 18:21 +0200: > 2011.11.15 17:30, Petr Mladek rašė: > I didn't say they have to be translated word-to-word. They should be > *localized*, and I would expect a localized testcase to suggest > localized number and date formats and other stuff. I see. Well another question is how it appears in the test run. The language independent tests are proceed once. The language dependent tests need to be proceed for every localization. > RTL, on the other hand, might probably need a few additional testcases. > Though not many, I would guess. I agree. > > BTW: How is the localization used during test run? > > > > If I select "de", will I see only the "de" test cases from the l10n > > branch? I am not sure if it is important but it would be nice. > > No, currently you will still see all testcases. > > My idea with localizable testcases though is pretty much about this. If > testcase X was localized instead of duplicated, it would only be shown > once, in user's preferred locale (I don't know yet how exactly the user > would "prefer" that locale though and what a user who would want to test > more than one locale would do). Yes, it sounds great. > >> My suggestion is to have a single branch, but carry Priority, Locale and > >> Component information in the Testgroup, and to represent test type by a > >> subgroup. That is, what is now a branch, would become a subgroup, and > >> everything else would become a testgroup (see the attached PDF file). > > Hmm, I am afraid that we would get too many testgroups. It produces 7 > > (General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one > > language. We have 109 localizations in sources => we could end up with > > more than 700 test groups. > > Which I guess is not realistic, at least in the short term. ;) Right > now we only have four languages into which testcases are translated. > With 50, it's of course a different story, but with 4 to 10... it's > probably still manageable enough. I hope that 10+ is quite realistic. If people provide translation. They might want to do also testing. I think that we have more than 10 active l10n teams. Maybe, they just do not know how to work with Litmus. > >> * share the same General Functional Tests subgroup between testgroups > >> designated for different locales (that is, you would create this > >> subgroup once, and add it to all locale groups) > > I am not sure what you mean with this. Could we share subgroups between > > test groups in Litmus? Is it clearly visible or is hard to maintain and > > see? > > Like I mentioned before, when you create/edit a testgroup, you can add > the subgroups you want to it. So subgroups can be shared, yes. Heh, it is interesting feature. I need to think if we could somehow use it. I am slightly afraid to use it because you will see the same text in different context. You will not know how your changes affect other locations. It is hard to visualize. Heh, I just might be too tired today ;-) > > BTW: What do you mean with the "Basis Functional Test"? We do not have > > this subgroup in the current structure. > > Don't we? > https://tcm.documentfoundation.org/manage_testgroups.cgi?testgroup_id=58 I guess that it comes from the older structure and is used only in the obsolete branches. In each case, it does not fit the current approach. "Basic tests" is something that should go the "P1" or "P2" group of the functional or l10n tests. > >> * drop 75% of the testgroups (there would be about 28 of them initially) > >> if/when proper testcase localization is implemented, still not rendering > >> the remaining testgroups useless. > > I am not sure if we really could drop them. Well, the question is if we > > need to split between lang-dependent and lang-independent on the top > > level. > > I don't think there are many lang-dependent testcases. Desired result > may depend on the language, yes, but the testcase itself? I see the point. The logic should be the same for all languages. The Right-to-Left stuff will be tested only in some locales but it still can be described in all languages. Well, we still do not have this support, so we need to split the languages in the group names. Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Hi Rimas and Petr, Thanks for the great discussion :) I appreciate Rimas' new scheme with some questions remaining. Let me take the opportunity to give a mid term summary and share my thoughts based on the new scheme. 1. Firstly through the complicated mail thread, I feel it is necessary to clarify the definition of test types helping us to have a common sense: - Function Regression test (Language Neutral Tests) It is language neutral and we only need to run it once in any of the languages. The localization for this kind of tests could be word-by-word. - L10N Regression test (Language Dependent Tests) It is language dependent and we need to run them in all language. The localization for this kind of tests could be slightly different language versions based on the libreoffice features. - Feature test It contains new features testing for each release of libreoffice. Test cases are added on demand in each new release, and the test cases are not reusable in the next release. 2. In the new scheme pdf, it looks feature tests are not included while a legacy Basic Function Test is there. Just to make things more clear, I am udpating Rimas' chart a bit: Basic Function Test -> Feature Tests General Function Test -> Function Regression Tests I believe we didn't really have misunderstanding on this but the legacy problem, right? 3. Do we want components split in L10n test? I don't have a definite answer for this yet but prone to keep the components. On one hand, at least for impress and calc, we have some specific areas to cover as tempalte, slide show in multiple language, calc cell format etc. On the other hand, the consistent structure may make people's learning curve more mild. 4. The amount of how many localization is another concern. In the new scheme, for each priority we will have "7 components * 4 locale * 4 priority" (112) groups. If I understand it right, when there are 10 locales, the number will be 2800 ?! Image we have 112 test groups in the first section when running test: http://wiki.documentfoundation.org/images/7/77/Litmus_submit-de.png While another choice to be considered could be switching the position of test types and components, something like: / p1-en-Feature Tests - 7 components (en) / p1-xx-Feature Tests - 7 components (xx) / p1-en-L10n Regresstion Tests - 7 components (en) / p1-xx-L10n Regresstion Tests - 7 components (xx) master - p1-en-Function Regresstion Tests - 7 components (ATM: multi-language case) \ p1-xx-Function Regresstion Tests / Besides it will reduce the groups size to "3*locale*priority", by the fact we can't select subgroups when creating test run (I didn't notice it before, thanks Rimas), we can create a test run for particular test types, which might be more likely to happen than creating test runs for particular components. It would be more clear of the strategy by comments Rimas' origin plan: >* create testruns based on priority, locale, component, or any >combination of these -> create testruns based on priority, locale, test type, or any combination of these >* create a single catch-all testrun for a single version of LibO >* share the same General Functional Tests subgroup between testgroups >designated for different locales (that is, you would create this >subgroup once, and add it to all locale groups) -> share the same components subgroup in Function Regression Tests testgroups. >* drop 75% of the testgroups (there would be about 28 of them initially) >if/when proper testcase localization is implemented, still not rendering >the remaining testgroups useless. -> This is the thing I am not pretty sure. Will this switch cause unhandlable problems for our potential translation system ? 5. Another critical question raised about different testing execution times of Function and L10N regression. >> 2011.11.15 17:30, Petr Mladek rašė: >> I didn't say they have to be translated word-to-word. They should be >> *localized*, and I would expect a localized testcase to suggest >> localized number and date formats and other stuff. > > I see. > > Well another question is how it appears in the test run. The language > independent tests are proceed once. The language dependent tests need to > be proceed for every localization. The test case transaltion can be considered into 2 layers in current test strategy: - For Function Regression Tests (Language Neutral Tests) Review the definition on the top of the mail, the translation version case id is supposed to be the same with its English version. So all of the people run the same test cases but read them in different language. - For L10n Regression Tests (Language Dependent Tests) Review the definition on the top of the mail, the translation version case id is not the same one as its English version. So all of th
Re: [Libreoffice-qa] Test case naming
Hi, > 4. The amount of how many localization is another concern. In the new scheme, > for each priority we will have "7 components * 4 locale * 4 priority" (112) > groups. If I understand it right, when there are 10 locales, the number will > be 2800 ?! Sorry for bring shocking, s/2800/280 :) Best wishes, Yifan ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test Structure in Litmus
Hi Petr, Hehe, I missed this mail before I started summarizing today. I'll have a review for sure :) Best wishes, Yifan On Tue, Nov 15, 2011 at 07:44:59PM +0100, Petr Mladek wrote: > Hi, > > I am going to summarize the ideas from the mail "Test case naming". > > I am sorry for the long mail. The problem is not easy. There are many > ideas. I tried to sort them from different point of view. I hope that I > did not miss anything important and it will help to find the final > conclusion. > > -- > > Let's start by a short summary. > > Litmus allows to split test cases by: > > + product > + branch > + testgroup > + subgroup > > > We want to split test cases by: > > + priority (P1,P2,P3,P4) > + component (General,Base,Calc,Draw,Impress,Math,Writer) > + meaning (functional, l10n, feature) > + l10n tests in languages (en,de,fr,it,ja,.) > + platform (Linux,Windows,MAC) > > We need to take care of the following scenarios: > > + maintenance: > + creating test cases > + updating test cases > + localization > + branch for new release > + test run for new release/build > + doing tests > + analyzing results > > --- > > Let's look at some conditions: > > 1. priority: > > + why > + reduced test run for bugfix releases > + overal prioritization of the work > > + main requirements: > > + allow to create test run for given priority > + easily run tests according to priority > + easily see test results by priority > > > 2. component: > > + why > > + reduced test run if only writer is modified > + helps to split work by expertize > > + main requirements: > > + allow to create test run for given component > + allow to run tests by component > > 3. meaning: > > + why > > + functional tests do not depend on localization > + the list of language dependent tests might be > different for each language > + feature tests are not well described; are done only > once by one or few dedicated people > > + requirements: > > + function test appears only once in test run; can be > approved by any user with any locale and any platform > + language dependent tests must be approved more times > in all locales > + test results must clearly show what tests were > done; functional tests are important by priority; > l10n tests are important by priority and language; > e.g. German is more important than Czech because > there are more active users > > > 4. platform: > > + why: > > + some tests might depend on platform; for example the > open/save dialogs look different way on Linux,Windows, > and MAX; also KDE vs. GNOME > > + main requirements: > > + these tests must be done on all affected platforms > + test results must clearly show where it was tested > > > 5. Maintenance: > > + problems: > > + searching for a particular test case > + put test cases into the right position > + comparing en-US template with localized test cases > + localizing the whole bunch of test cases into new > language > > + requirements: > > + balance the number of branches, testgroups, > subgroups, and test cases in subgroup > + must be easy to copy test cases for new language > + often switched criteria must be in high level; > for example, do people switch more often between > priority or category or language when they work > with the test cases? > > > 6. Branch for new release: > > + requirements: > > + must be easy to do the branch > + must be easy to see all tests for the given release > > > 7. Test run creation: > > + requirements: > > + 1 or very small number of active test runs because it > is compilcated to switch between them and analyze > results from many test runs > + create test run for a given priority (basic vs. full > test) > + create test run for a given component (only Writer > modified) > > 8. Doing tests: > > + requirements: > > + must be easy to search what needs testing > + must be possible to sort testi
