Rimas Kudelis píše v Út 15. 11. 2011 v 13:43 +0200: > Hi all, > > I took a quick look at how testruns are created and what we can and > cannot choose when creating a testrun. My suggestion is based on the > fact that each testrun comprises of > > * all subgroups in > * one or more testgroups in > * exactly one branch in > * exactly one product. > > That is, a testrun cannot span through multiple branches or products, > nor can you specify which subgroups to add to it. You can only choose > groups.
This is important information. Thanks for looking at it. > > Overall satistics: > > > > We now have 3 branches: > > > > master regression branch > > - 1 group > > - 7 subgroups in each group > > master l10n branch > > - 4 groups > > - 7 subgroups in each group > > master feature branch > > - we may not consider case priority here? > > Current scheme does not allow creating a catch-all testrun comprising > all tests for a single product/branch. I don't know whether or not such > ability is desired, but it looks logical to me. You are right. 1 run is the best solution. Well, 2 or three test runs for a release still might be acceptable. > Also, maybe I shouldn't look that far into future, but I hope that > there will come a day when proper localization of testcases will be > possible (that is, instead of creating a clone of testcase X in > another language, we would actually be able to translate testcase X > into that language). With that in mind, current testgroups (which > represent different locales) would become unneeded. I am not sure if the l10n tests will be pure word-to-word translation. Some things are done completely different in different languages, for example the layout of the letter template, right-to-left language features, decimal number delimiter, dates. I am sure that some languages would need special tests. BTW: How is the localization used during test run? I know that I select locale in the "Run Tests - Your Chosen Test Run" dialog, see https://wiki.documentfoundation.org/Litmus/Litmus_User_Guide#Enter_Test_Configuration If I select "de", will I see only the "de" test cases from the l10n branch? I am not sure if it is important but it would be nice. > > Case 1. Carry priority information with subgroups name > This scheme would make it impossible to create a testrun for e.g. just > P1 tests. Or just writer tests. I guess this is pretty much the same as > what we have now, so all criticism from above paragraph applies here too. Good point. We can't use this. > > Case 2. Carry priority information with groups name > This is probably OK if you don't mind having to create three testruns > for each version It is not ideal but acceptable. > and be unable to create testruns for separate components IMHO, this is fine. > . I think you forgot locales here though. I assume that for > each locale/priority there would be a separate testgroup? I guess so. Hmm, that would create too many test groups. See my other mail. > > Case 3. Carry priority information with branches > IMO, Case 3 is the worst case, because for each version of LibreOffice, > you would have to create 4*3=12 testruns – one for each priority/test > type combination. I am not sure if it is the worst case because the solution 1 was bad as well :-) Anyway, I think that 3 is not a way to go. > My suggestion is to have a single branch, but carry Priority, Locale and > Component information in the Testgroup, and to represent test type by a > subgroup. That is, what is now a branch, would become a subgroup, and > everything else would become a testgroup (see the attached PDF file). Hmm, I am afraid that we would get too many testgroups. It produces 7 (General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one language. We have 109 localizations in sources => we could end up with more than 700 test groups. > This allows to: > * create testruns based on priority, locale, component, or any > combination of these The question is what test runs we will want to create. > * create a single catch-all testrun for a single version of LibO would be nice > * share the same General Functional Tests subgroup between testgroups > designated for different locales (that is, you would create this > subgroup once, and add it to all locale groups) I am not sure what you mean with this. Could we share subgroups between test groups in Litmus? Is it clearly visible or is hard to maintain and see? BTW: What do you mean with the "Basis Functional Test"? We do not have this subgroup in the current structure. > * drop 75% of the testgroups (there would be about 28 of them initially) > if/when proper testcase localization is implemented, still not rendering > the remaining testgroups useless. I am not sure if we really could drop them. Well, the question is if we need to split between lang-dependent and lang-independent on the top level. Heh, it seems that it is quite complex problem. I am going to write another mail where I will try to summarize some ideas :-) Thanks a lot for the valuable input. Best Regards, Petr _______________________________________________ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/