[Libreoffice-qa] Test case naming
Hi, The list of existing test cases looks the following way in Litmus: id # testcase summary # subgroup --- 1066 # g001 - Installing LibreOffice # General 1061 # g002 - Uninstalling LibreOffice # General 1053 # g003 - First launch of LibreOffice# General 1063 # g004 - Creating a new document# General 1048 # w002 - Importing MS Word documents# Writer 1052 # w003 - Exporting ODF text document to MS Word formats # Writer 1049 # w004 - PDF export of text docs# Writer 1054 # w001 - Creating a new text document # Writer See also the screenshot at http://download.go-oo.org/qa/test-case-list.png I see two problems here: 1. The test cases are ordered by the id, so the first writer test case is the last in the list. I wonder how complicated would be to hack Litmus to sort it by the testcase summary. 2. I am not sure what is the meaning of the numbers 001, 002, 003. It looks like they define the order in which we should process the test cases. If this is true, it does not look ideal: + if we do another important test case, we will need to rename all less important test cases to keep the right order + test cases will be checked by many people; we can't force them to do it in an exact order; the result would be that all people will test the same test case in parallel Hmm, we need to encourage people to do the test cases in random order. We still should somehow prioritize the test cases. I suggest to split test cases into several levels by priorities: P1 - highest: used for very basic tests, e.g. app can be installed; it starts; is able to load/save some test documents; so it a kind of smoketest P2 - high: test very common functionality that is used by most users. e.g. able to write text, insert picture; draw elements; create table; use function in calc; create graph, run presentation P3 - medium: test common functionality that is used by typical a bit experienced office user, e.g. create borders around tables; do animation between slides; modify text style; modify master slide page; P4 - low: test functionality for hi-tech users, e.g. writing macros, using Calc solver, complex operations with data bases I suggest to use the names: p1g - p1w - p2g - p2w - Then we will have all p1 test cases listed before p2 test cases. What do you think? Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Hello Petr, and everyone, 2011.11.11 22:14, Petr Mladek rašė: > The list of existing test cases looks the following way in Litmus: > > id # testcase summary # subgroup > --- > 1066 # g001 - Installing LibreOffice # General > 1061 # g002 - Uninstalling LibreOffice # General > 1053 # g003 - First launch of LibreOffice# General > 1063 # g004 - Creating a new document# General > 1048 # w002 - Importing MS Word documents# Writer > 1052 # w003 - Exporting ODF text document to MS Word formats # Writer > 1049 # w004 - PDF export of text docs# Writer > 1054 # w001 - Creating a new text document # Writer > > See also the screenshot at http://download.go-oo.org/qa/test-case-list.png > > > I see two problems here: > > 1. The test cases are ordered by the id, so the first writer test case >is the last in the list. No, they are not ordered by id. Actually, I've just digged, and the order is as specified for subgroups inside a testgroup, and then as specified for testcases inside each subgroup. If you look at what you pasted (and at your image) again, you'll notice that #w001 is the only testcase that wasn't shown in its expected position, and that is because it wasn't positioned correctly inside the subgroup. I've just corrected its position and it's now shown where expected. >I wonder how complicated would be to hack Litmus to sort it by >the testcase summary. This is thus irrelevant, cause there's nothing to modify. :) > 2. I am not sure what is the meaning of the numbers 001, 002, 003. > >It looks like they define the order in which we should process the >test cases. If this is true, it does not look ideal: > > + if we do another important test case, we will need to rename > all less important test cases to keep the right order > > + test cases will be checked by many people; we can't force them > to do it in an exact order; the result would be that all > people will test the same test case in parallel > > Hmm, we need to encourage people to do the test cases in > random order. We still should somehow prioritize the test > cases. I personally don't like current naming with ugly prefices at all, but it's Yifan's call, and I suppose he has a good reason for that naming scheme. However, I'm afraid we can't randomize testcase order by default. Currently, this would probably have to be done manually each time a relevant subgroup is updated. That would be a PITA. On the other hand, it's not really that bad. You can still run the tests in random order, but you will always see them in the specified order. > I suggest to split test cases into several levels by priorities: > > P1 - highest: used for very basic tests, e.g. app can be > installed; it starts; is able to load/save some test > documents; so it a kind of smoketest > > P2 - high: test very common functionality that is used by most > users. e.g. able to write text, insert picture; draw > elements; create table; use function in calc; create graph, > run presentation > > P3 - medium: test common functionality that is used by typical > a bit experienced office user, e.g. create borders around > tables; do animation between slides; modify text style; > modify master slide page; > > P4 - low: test functionality for hi-tech users, e.g. writing > macros, using Calc solver, complex operations with data > bases > > I suggest to use the names: > > p1g - > p1w - > p2g - > p2w - > > Then we will have all p1 test cases listed before p2 test cases. > > > What do you think? Prioritizing is probably a good idea, but like I said, random order would require some Litmus modification. While certainly possible (and probably quite easy), I'm not sure that is what we indeed want. Rimas ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Hi Rimas, Petr and all, Thanks for the nice discussions, my ideas as follows :) On Sat, Nov 12, 2011 at 10:46:48PM +0200, Rimas Kudelis wrote: > This is thus irrelevant, cause there's nothing to modify. :) > > > 2. I am not sure what is the meaning of the numbers 001, 002, 003. > > > >It looks like they define the order in which we should process the > >test cases. If this is true, it does not look ideal: > > > > + if we do another important test case, we will need to rename > > all less important test cases to keep the right order > > > > + test cases will be checked by many people; we can't force them > > to do it in an exact order; the result would be that all > > people will test the same test case in parallel > > > > Hmm, we need to encourage people to do the test cases in > > random order. We still should somehow prioritize the test > > cases. > > I personally don't like current naming with ugly prefices at all, but > it's Yifan's call, and I suppose he has a good reason for that naming > scheme. However, I'm afraid we can't randomize testcase order by > default. Currently, this would probably have to be done manually each > time a relevant subgroup is updated. That would be a PITA. On the other > hand, it's not really that bad. You can still run the tests in random > order, but you will always see them in the specified order. >From my understanding, I neither mean to encourage users to run cases in a specific order. The ids were originally being there since I got involved. I think it still makes sense to kept them there mainly because of the "syncing" problem of test cases in different languages. For example: #EN - w001 xxx is supposed to have the same content with (but in different version of language): #FR - w001 xxx #DE - w001 xxx #pt-BR - w001 xxx These give us reasonable information showing which cases are supposed to be "synced" to each other (they may not have exact same steps of testing because of the diversity of language settings, but they should test the same areas). So for current testing organization, I think these ids are still playing their role in L10N test branches. Otherwise, syncing of cases could be painful. Meanwhile in Function Regression testing branch, by the fact we are now using a single case to host all language versions of test case, it may not make sense to keep the id any more. > > I suggest to split test cases into several levels by priorities: > > > > P1 - highest: used for very basic tests, e.g. app can be > > installed; it starts; is able to load/save some test > > documents; so it a kind of smoketest > > > > P2 - high: test very common functionality that is used by most > > users. e.g. able to write text, insert picture; draw > > elements; create table; use function in calc; create graph, > > run presentation > > > > P3 - medium: test common functionality that is used by typical > > a bit experienced office user, e.g. create borders around > > tables; do animation between slides; modify text style; > > modify master slide page; > > > > P4 - low: test functionality for hi-tech users, e.g. writing > > macros, using Calc solver, complex operations with data > > bases > > > > I suggest to use the names: > > > > p1g - > > p1w - > > p2g - > > p2w - > > > > Then we will have all p1 test cases listed before p2 test cases. > > > > > > What do you think? > > Prioritizing is probably a good idea, but like I said, random order > would require some Litmus modification. While certainly possible (and > probably quite easy), I'm not sure that is what we indeed want. Actually it is a great idea to have priority here, at least they are helpful for us to define subset of test runs. For example, we can create "smoke test runs" by select P1 only test cases when creating a test run from a full regression branch containing all cases. That is to say, even before we sort out how order of the test cases could be implemented, we can always create specific test runs on demand via the information of the priority "tags". i.e We can define "smoke test" runs, "basic test" runs, "full regression test" run by selecting cases, in which all the test cases are divided physically in test runs and sorting becomes trivial. But it still makes sense if we can do some hacking in ordering, which could save much of the test cases selection effort and makes the system more flexible. :) At the same time, as stated before, for L10N test case branch, we probably still want test case id until we have some better solution for translation syncing problem. Best wishes, Yifan ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Ch
Re: [Libreoffice-qa] Test case naming
Rimas Kudelis píše v So 12. 11. 2011 v 22:46 +0200: > 2011.11.11 22:14, Petr Mladek rašė: > > The list of existing test cases looks the following way in Litmus: > > > > id # testcase summary # subgroup > > --- > > 1066 # g001 - Installing LibreOffice # General > > 1061 # g002 - Uninstalling LibreOffice # General > > 1053 # g003 - First launch of LibreOffice# General > > 1063 # g004 - Creating a new document# General > > 1048 # w002 - Importing MS Word documents# Writer > > 1052 # w003 - Exporting ODF text document to MS Word formats # Writer > > 1049 # w004 - PDF export of text docs# Writer > > 1054 # w001 - Creating a new text document # Writer > > If you look at what you pasted (and at your image) again, you'll notice > that #w001 is the only testcase that wasn't shown in its expected > position, and that is because it wasn't positioned correctly inside the > subgroup. I've just corrected its position and it's now shown where > expected. Interesting. So, the test cases are not sorted alphabetically. Am I right? How do you define the sorting inside the group? Are you able to do it from the UI? I do not see this possibility in the "Manage Testcases" interface :-) > However, I'm afraid we can't randomize testcase order by > default. Currently, this would probably have to be done manually each > time a relevant subgroup is updated. That would be a PITA. On the other > hand, it's not really that bad. You can still run the tests in random > order, but you will always see them in the specified order. I agree. We do not need to randomize the test cases in the view. It might be enough to ask people to run it randomly. Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Yifan Jiang píše v Ne 13. 11. 2011 v 18:46 +0800: > On Sat, Nov 12, 2011 at 10:46:48PM +0200, Rimas Kudelis wrote: > For example: > > #EN - w001 xxx > > is supposed to have the same content with (but in different version of > language): > > #FR - w001 xxx > #DE - w001 xxx > #pt-BR - w001 xxx > > These give us reasonable information showing which cases are supposed > to be "synced" to each other (they may not have exact same steps of > testing because of the diversity of language settings, but they should > test the same areas). So for current testing organization, I think > these ids are still playing their role in L10N test > branches. Otherwise, syncing of cases could be painful. So, the number 001, 002, 003, 004 is a l10n test case number (something like bugzilla number). Would be enough to mention it in brackets at the end of the test case summary? I mean something like: p1 - test case summary (w#1,en) p1 - another test case summary (w#2,en) and localized p1 - test case summary (w#1,en) p1 - popis testu (w#1,cs) p1 - Testfall Zusammenfassung (w#1,cs) I know that it is not ideal because it wont be that easy to sort the test cases by the id and compare the list. On the other hand, syncing localized test cases will not be easy anyway. I think that the bug priority is more important sorting criteria Note that p1 #EN - w001 test case summary looks confusing to me. There are just too many identifiers in the prefix. And it does not help with sorting as well. > Meanwhile in Function Regression testing branch, by the fact we are > now using a single case to host all language versions of test case, it > may not make sense to keep the id any more. This way, it would look the same for function regression test and localization regression tests. The localization regression test will just have some extra identification in the brackets. > > > I suggest to split test cases into several levels by priorities: > Actually it is a great idea to have priority here, at least they are > helpful for us to define subset of test runs. For example, we can > create "smoke test runs" by select P1 only test cases when creating a > test run from a full regression branch containing all cases. Exactly > That is to say, even before we sort out how order of the test cases > could be implemented, we can always create specific test runs on > demand via the information of the priority "tags". BTW: How do you suggest to create the priority "tag"? Is there any better solution than to put it into prefix of the test case summary? Rimas, Ti Fan, thank you both for looking at it. Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
2011.11.14 12:07, Petr Mladek rašė: > Rimas Kudelis píše v So 12. 11. 2011 v 22:46 +0200: >> 2011.11.11 22:14, Petr Mladek rašė: >>> The list of existing test cases looks the following way in Litmus: >>> >>> id # testcase summary # subgroup >>> --- >>> 1066 # g001 - Installing LibreOffice # General >>> 1061 # g002 - Uninstalling LibreOffice # General >>> 1053 # g003 - First launch of LibreOffice# General >>> 1063 # g004 - Creating a new document# General >>> 1048 # w002 - Importing MS Word documents# Writer >>> 1052 # w003 - Exporting ODF text document to MS Word formats # Writer >>> 1049 # w004 - PDF export of text docs# Writer >>> 1054 # w001 - Creating a new text document # Writer >> If you look at what you pasted (and at your image) again, you'll notice >> that #w001 is the only testcase that wasn't shown in its expected >> position, and that is because it wasn't positioned correctly inside the >> subgroup. I've just corrected its position and it's now shown where >> expected. > Interesting. So, the test cases are not sorted alphabetically. Am I > right? How do you define the sorting inside the group? Are you able to > do it from the UI? I do not see this possibility in the "Manage > Testcases" interface :-) When you are creating/editing a subgroup and adding testcases to it, you can sort testcases as you like. Similarly, when you are creating/editing a testgroup and adding subgroups to it, you can sort subgroups as you like. So yes, this is available in the UI. >> However, I'm afraid we can't randomize testcase order by >> default. Currently, this would probably have to be done manually each >> time a relevant subgroup is updated. That would be a PITA. On the other >> hand, it's not really that bad. You can still run the tests in random >> order, but you will always see them in the specified order. > I agree. We do not need to randomize the test cases in the view. It > might be enough to ask people to run it randomly. Good. :) Rimas ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
On 2011.11.14 12:28, Petr Mladek wrote: > Yifan Jiang píše v Ne 13. 11. 2011 v 18:46 +0800: >> For example: >> >> #EN - w001 xxx >> >> is supposed to have the same content with (but in different version of >> language): >> >> #FR - w001 xxx >> #DE - w001 xxx >> #pt-BR - w001 xxx >> >> These give us reasonable information showing which cases are supposed >> to be "synced" to each other (they may not have exact same steps of >> testing because of the diversity of language settings, but they should >> test the same areas). So for current testing organization, I think >> these ids are still playing their role in L10N test >> branches. Otherwise, syncing of cases could be painful. Ah, this makes sense. > So, the number 001, 002, 003, 004 is a l10n test case number (something > like bugzilla number). Would be enough to mention it in brackets at the > end of the test case summary? I mean something like: > > p1 - test case summary (w#1,en) > p1 - another test case summary (w#2,en) > > and localized > > p1 - test case summary (w#1,en) > p1 - popis testu (w#1,cs) > p1 - Testfall Zusammenfassung (w#1,cs) > > I know that it is not ideal because it wont be that easy to sort the > test cases by the id and compare the list. On the other hand, syncing > localized test cases will not be easy anyway. I think that the bug > priority is more important sorting criteria > > Note that > > p1 #EN - w001 test case summary looks confusing to me. There are just > too many identifiers in the prefix. And it does not help with sorting as > well. P1 W01EN would be shorter. Still admittedly quite ugly though. >> Meanwhile in Function Regression testing branch, by the fact we are >> now using a single case to host all language versions of test case, it >> may not make sense to keep the id any more. Note the testcase still has its real id (used in the database). If needed, it could be made more visible. > This way, it would look the same for function regression test and > localization regression tests. The localization regression test will > just have some extra identification in the brackets. Like you said, this would make different testcases harder to associate with each other. OTOH, I guess only the admins often see them all in the same place. I suggest to split test cases into several levels by priorities: >> Actually it is a great idea to have priority here, at least they are >> helpful for us to define subset of test runs. For example, we can >> create "smoke test runs" by select P1 only test cases when creating a >> test run from a full regression branch containing all cases. > Exactly > >> That is to say, even before we sort out how order of the test cases >> could be implemented, we can always create specific test runs on >> demand via the information of the priority "tags". > BTW: How do you suggest to create the priority "tag"? Is there any > better solution than to put it into prefix of the test case summary? Well, as an alternative, branches/groups/subgroups could be reviewed again. :) Also, Litmus allows marking certain test runs as recommended and shows them on top. This means that separate P1 testruns could be created and promoted on Litmus homepage. Rimas ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Hi Petr, Rimas and all, On Mon, Nov 14, 2011 at 10:25:28PM +0200, Rimas Kudelis wrote: > >> That is to say, even before we sort out how order of the test cases > >> could be implemented, we can always create specific test runs on > >> demand via the information of the priority "tags". > > BTW: How do you suggest to create the priority "tag"? Is there any > > better solution than to put it into prefix of the test case summary? > > Well, as an alternative, branches/groups/subgroups could be reviewed > again. :) The idea is actually brilliant! I also thought a bit in this way, but didn't conclude anything cenrtain at the moment. The problems I can think by leveraging branch/group/subgroup is 1. The test cases priority heavily depend on Litmus tool that it doesn't carry the information with their summaries. Instead, the group/subgroup/branch where they are in tell us the priority. 2. The maintainence effort and process may be increased a bit, but in a acceptable amount (at least for me). The benifits are: 1. Creating test runs can be easier that higher level definition would allow admins to create a specific priority test run simply by selecting corresponding subgroup, group or branch of test cases, instead of reviwing each test case's title. 2. The maintenence of test cases is assumed to be more elegant and intuitive. For example, we can change a test case's priority by change its corresponding subgroup, group or branch, instead of changing test case's title. The following is something they would look like, assume we don't change a lot the whole structure of current organization. Overall satistics: We now have 3 branches: master regression branch - 1 group - 7 subgroups in each group master l10n branch - 4 groups - 7 subgroups in each group master feature branch - we may not consider case priority here? Case 1. Carry priority information with subgroups name Prototype: Branch Master Function Regression Test Group: Function test Subgroup p1 writer Subgroup p2 writer Subgroup p3 writer Subgroup p4 writer Subgroup p1 calc Subgroup p2 calc Subgroup p3 calc ... Case 2. Carry priority information with groups name Prototype: Branch Master Function Regression Test Group: Function test p1 Subgroup writer Subgroup calc ... Group: Function test p2 Subgroup writer Subgroup calc ... Group: Function test p3 Subgroup writer Subgroup calc ... Case 3. Carry priority information with branches Branch Master Function Regression Test p1 Branch Master Function Regression Test p2 Branch Master Function Regression Test p3 Branch Master Function Regression Test p4 All the structure inside different priority branches are the same. Branch Master L10n Regression Test p1 Branch Master L10n Regression Test p2 Branch Master L10n Regression Test p3 Branch Master L10n Regression Test p4 All the structure inside different priority branches are the same. I am slightly inclined to the Case 3, it still keeps most of original things (groups and subroups), which have already been divided in a clear logic and not that easy to get familiar with. Also Case 3 is the most easy way if we want to create different runs in different phase of release. Intuitively, we can "feel" all the three cases by describe a particular l10n test case in each: Case 1 - subgroup. This is a P1 writer test case in English locale for regression testing Case 2 - group This is a P1 English locale writer test case for regression testing Case 3 - branch This is a P1 regression test case for writer in an English locale I feel Case 2 is hard to describe :) Case 1 is okay, but it doesn't really take advantage against Case 3. It would be interesting if you share some your ideas of cons/prons of them. :) Best wishes, Yifan ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Hi guys, first, I am sorry that this mail does react on the last mail from Rimas. I started to write this replay in the morning. I had to interrupt it for some reasons. I think that the ideas mentioned below still make sense. Though, I might change my preference after reading Rimas's reply. Yifan Jiang píše v Út 15. 11. 2011 v 13:34 +0800: > On Mon, Nov 14, 2011 at 10:25:28PM +0200, Rimas Kudelis wrote: > > Well, as an alternative, branches/groups/subgroups could be reviewed > > again. :) > > The idea is actually brilliant! Fully agree! > The following is something they would look like, assume we don't change a lot > the whole structure of current organization. > > Case 1. Carry priority information with subgroups name > Case 2. Carry priority information with groups name > Case 3. Carry priority information with branches > I am slightly inclined to the Case 3, it still keeps most of original things > (groups and subroups), which have already been divided in a clear logic and > not that easy to get familiar with. Also Case 3 is the most easy way if we > want to create different runs in different phase of release. I think that creating test run is only one scenario. I think that we need to think about all use cases: A. Maintenance -- The are few actions: + localize test - has two subactions: + creating the structure: If we could copy the whole group, solution 1 is fastest because there are less groups and branches. Solution 2 is good because you need not switch between branches. If we need to create the whole structure from scratch, it does not matter because the total number of subgroups is always the same + checking differences and localizing If you select another branch in the search dialog, you need to select group and subgroup again => Solution 1 is better than 2 and 2 is better than 3 because if is easier to switch between subgroups; it is easier to between groups than branches + adding/updating test cases: + IMHO, it is similar to checking and localizing => 1 is better than 2 and 2 is better than 3 because it is easier to switch subgroups than groups and switch groups than branches B. Creating Branches for new release + 1 or 2 are easier than 3 because they have less branches C. Creating Test Runs - + I do not know how this works. You say that 3 is the easiest way because you just create run from a branch? + IMHO, 1 would be nightmare because you would need to select to many subgroups manually if you want to test only P1 test. D. Testing -- This is very important because many people will do this many time!!! I think that we have 2 actions: + doing the test People need to find easily what tests are not finished. You need to enter a test run to see the statistic => we should have as few active test runs as possible if 3 produces more test runs, I am against it => 1 or 2 is better + analyzing results: I am not sure how we check the state of testing; the best solution would be to check on page and see how many tests are finished; how many tests succeded and failed I guess that the solution 3 is bad from this point of view. Summary: 3 is bad for testing. 1 is bad for creating test runs. 2 is the best compromise in most situations. => I prefer 2. Note that 2 has better granularity. We will have many branches in the long term for released products. Solution 1 produces too many subgroups in a single group. = Hmm, when I think about it. I am somehow afraid that we would have too many subgroups in the l10n branch. It does not mater how we split it but it will be hard to maintain. Do we really expect that many l10n-specific test cases for Draw, Impress, Base, or Calc? Do we really need the subgroups? What about the following structure: + function tests: + P1 + General + Base + Calc + ... + P2 + General + Base + ... + l10n test: + P1 + en + de + fr + it + P2 + en + de + fr
Re: [Libreoffice-qa] Test case naming
Rimas Kudelis píše v Út 15. 11. 2011 v 13:43 +0200: > Hi all, > > I took a quick look at how testruns are created and what we can and > cannot choose when creating a testrun. My suggestion is based on the > fact that each testrun comprises of > > * all subgroups in > * one or more testgroups in > * exactly one branch in > * exactly one product. > > That is, a testrun cannot span through multiple branches or products, > nor can you specify which subgroups to add to it. You can only choose > groups. This is important information. Thanks for looking at it. > > Overall satistics: > > > > We now have 3 branches: > > > > master regression branch > > - 1 group > > - 7 subgroups in each group > > master l10n branch > > - 4 groups > > - 7 subgroups in each group > > master feature branch > > - we may not consider case priority here? > > Current scheme does not allow creating a catch-all testrun comprising > all tests for a single product/branch. I don't know whether or not such > ability is desired, but it looks logical to me. You are right. 1 run is the best solution. Well, 2 or three test runs for a release still might be acceptable. > Also, maybe I shouldn't look that far into future, but I hope that > there will come a day when proper localization of testcases will be > possible (that is, instead of creating a clone of testcase X in > another language, we would actually be able to translate testcase X > into that language). With that in mind, current testgroups (which > represent different locales) would become unneeded. I am not sure if the l10n tests will be pure word-to-word translation. Some things are done completely different in different languages, for example the layout of the letter template, right-to-left language features, decimal number delimiter, dates. I am sure that some languages would need special tests. BTW: How is the localization used during test run? I know that I select locale in the "Run Tests - Your Chosen Test Run" dialog, see https://wiki.documentfoundation.org/Litmus/Litmus_User_Guide#Enter_Test_Configuration If I select "de", will I see only the "de" test cases from the l10n branch? I am not sure if it is important but it would be nice. > > Case 1. Carry priority information with subgroups name > This scheme would make it impossible to create a testrun for e.g. just > P1 tests. Or just writer tests. I guess this is pretty much the same as > what we have now, so all criticism from above paragraph applies here too. Good point. We can't use this. > > Case 2. Carry priority information with groups name > This is probably OK if you don't mind having to create three testruns > for each version It is not ideal but acceptable. > and be unable to create testruns for separate components IMHO, this is fine. > . I think you forgot locales here though. I assume that for > each locale/priority there would be a separate testgroup? I guess so. Hmm, that would create too many test groups. See my other mail. > > Case 3. Carry priority information with branches > IMO, Case 3 is the worst case, because for each version of LibreOffice, > you would have to create 4*3=12 testruns – one for each priority/test > type combination. I am not sure if it is the worst case because the solution 1 was bad as well :-) Anyway, I think that 3 is not a way to go. > My suggestion is to have a single branch, but carry Priority, Locale and > Component information in the Testgroup, and to represent test type by a > subgroup. That is, what is now a branch, would become a subgroup, and > everything else would become a testgroup (see the attached PDF file). Hmm, I am afraid that we would get too many testgroups. It produces 7 (General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one language. We have 109 localizations in sources => we could end up with more than 700 test groups. > This allows to: > * create testruns based on priority, locale, component, or any > combination of these The question is what test runs we will want to create. > * create a single catch-all testrun for a single version of LibO would be nice > * share the same General Functional Tests subgroup between testgroups > designated for different locales (that is, you would create this > subgroup once, and add it to all locale groups) I am not sure what you mean with this. Could we share subgroups between test groups in Litmus? Is it clearly visible or is hard to maintain and see? BTW: What do you mean with the "Basis Functional Test"? We do not have this subgroup in the current structure. > * drop 75% of the testgroups (there would be about 28 of them initially) > if/when proper testcase localization is implemented, still not rendering > the remaining testgroups useless. I am not sure if we really could drop them. Well, the question is if we need to split between lang-dependent and lang-independent on the t
Re: [Libreoffice-qa] Test case naming
Hi Petr, 2011.11.15 17:30, Petr Mladek rašė: > Rimas Kudelis píše v Út 15. 11. 2011 v 13:43 +0200: >> Also, maybe I shouldn't look that far into future, but I hope that >> there will come a day when proper localization of testcases will be >> possible (that is, instead of creating a clone of testcase X in >> another language, we would actually be able to translate testcase X >> into that language). With that in mind, current testgroups (which >> represent different locales) would become unneeded. > I am not sure if the l10n tests will be pure word-to-word translation. > Some things are done completely different in different languages, for > example the layout of the letter template, right-to-left language > features, decimal number delimiter, dates. I am sure that some > languages would need special tests. I didn't say they have to be translated word-to-word. They should be *localized*, and I would expect a localized testcase to suggest localized number and date formats and other stuff. RTL, on the other hand, might probably need a few additional testcases. Though not many, I would guess. > BTW: How is the localization used during test run? It's not. It's just shown in the test results. > I know that I select locale in the "Run Tests - Your Chosen Test Run" > dialog, see > https://wiki.documentfoundation.org/Litmus/Litmus_User_Guide#Enter_Test_Configuration > > If I select "de", will I see only the "de" test cases from the l10n > branch? I am not sure if it is important but it would be nice. No, currently you will still see all testcases. My idea with localizable testcases though is pretty much about this. If testcase X was localized instead of duplicated, it would only be shown once, in user's preferred locale (I don't know yet how exactly the user would "prefer" that locale though and what a user who would want to test more than one locale would do). >> My suggestion is to have a single branch, but carry Priority, Locale and >> Component information in the Testgroup, and to represent test type by a >> subgroup. That is, what is now a branch, would become a subgroup, and >> everything else would become a testgroup (see the attached PDF file). > Hmm, I am afraid that we would get too many testgroups. It produces 7 > (General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one > language. We have 109 localizations in sources => we could end up with > more than 700 test groups. Which I guess is not realistic, at least in the short term. ;) Right now we only have four languages into which testcases are translated. With 50, it's of course a different story, but with 4 to 10... it's probably still manageable enough. >> This allows to: >> * create testruns based on priority, locale, component, or any >> combination of these > The question is what test runs we will want to create. > >> * create a single catch-all testrun for a single version of LibO > would be nice > >> * share the same General Functional Tests subgroup between testgroups >> designated for different locales (that is, you would create this >> subgroup once, and add it to all locale groups) > I am not sure what you mean with this. Could we share subgroups between > test groups in Litmus? Is it clearly visible or is hard to maintain and > see? Like I mentioned before, when you create/edit a testgroup, you can add the subgroups you want to it. So subgroups can be shared, yes. > BTW: What do you mean with the "Basis Functional Test"? We do not have > this subgroup in the current structure. Don't we? https://tcm.documentfoundation.org/manage_testgroups.cgi?testgroup_id=58 >> * drop 75% of the testgroups (there would be about 28 of them initially) >> if/when proper testcase localization is implemented, still not rendering >> the remaining testgroups useless. > I am not sure if we really could drop them. Well, the question is if we > need to split between lang-dependent and lang-independent on the top > level. I don't think there are many lang-dependent testcases. Desired result may depend on the language, yes, but the testcase itself? > Heh, it seems that it is quite complex problem. I am going to write > another mail where I will try to summarize some ideas :-) Good luck with that. :) Rimas ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Rimas Kudelis píše v Út 15. 11. 2011 v 18:21 +0200: > 2011.11.15 17:30, Petr Mladek rašė: > I didn't say they have to be translated word-to-word. They should be > *localized*, and I would expect a localized testcase to suggest > localized number and date formats and other stuff. I see. Well another question is how it appears in the test run. The language independent tests are proceed once. The language dependent tests need to be proceed for every localization. > RTL, on the other hand, might probably need a few additional testcases. > Though not many, I would guess. I agree. > > BTW: How is the localization used during test run? > > > > If I select "de", will I see only the "de" test cases from the l10n > > branch? I am not sure if it is important but it would be nice. > > No, currently you will still see all testcases. > > My idea with localizable testcases though is pretty much about this. If > testcase X was localized instead of duplicated, it would only be shown > once, in user's preferred locale (I don't know yet how exactly the user > would "prefer" that locale though and what a user who would want to test > more than one locale would do). Yes, it sounds great. > >> My suggestion is to have a single branch, but carry Priority, Locale and > >> Component information in the Testgroup, and to represent test type by a > >> subgroup. That is, what is now a branch, would become a subgroup, and > >> everything else would become a testgroup (see the attached PDF file). > > Hmm, I am afraid that we would get too many testgroups. It produces 7 > > (General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one > > language. We have 109 localizations in sources => we could end up with > > more than 700 test groups. > > Which I guess is not realistic, at least in the short term. ;) Right > now we only have four languages into which testcases are translated. > With 50, it's of course a different story, but with 4 to 10... it's > probably still manageable enough. I hope that 10+ is quite realistic. If people provide translation. They might want to do also testing. I think that we have more than 10 active l10n teams. Maybe, they just do not know how to work with Litmus. > >> * share the same General Functional Tests subgroup between testgroups > >> designated for different locales (that is, you would create this > >> subgroup once, and add it to all locale groups) > > I am not sure what you mean with this. Could we share subgroups between > > test groups in Litmus? Is it clearly visible or is hard to maintain and > > see? > > Like I mentioned before, when you create/edit a testgroup, you can add > the subgroups you want to it. So subgroups can be shared, yes. Heh, it is interesting feature. I need to think if we could somehow use it. I am slightly afraid to use it because you will see the same text in different context. You will not know how your changes affect other locations. It is hard to visualize. Heh, I just might be too tired today ;-) > > BTW: What do you mean with the "Basis Functional Test"? We do not have > > this subgroup in the current structure. > > Don't we? > https://tcm.documentfoundation.org/manage_testgroups.cgi?testgroup_id=58 I guess that it comes from the older structure and is used only in the obsolete branches. In each case, it does not fit the current approach. "Basic tests" is something that should go the "P1" or "P2" group of the functional or l10n tests. > >> * drop 75% of the testgroups (there would be about 28 of them initially) > >> if/when proper testcase localization is implemented, still not rendering > >> the remaining testgroups useless. > > I am not sure if we really could drop them. Well, the question is if we > > need to split between lang-dependent and lang-independent on the top > > level. > > I don't think there are many lang-dependent testcases. Desired result > may depend on the language, yes, but the testcase itself? I see the point. The logic should be the same for all languages. The Right-to-Left stuff will be tested only in some locales but it still can be described in all languages. Well, we still do not have this support, so we need to split the languages in the group names. Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Hi Rimas and Petr, Thanks for the great discussion :) I appreciate Rimas' new scheme with some questions remaining. Let me take the opportunity to give a mid term summary and share my thoughts based on the new scheme. 1. Firstly through the complicated mail thread, I feel it is necessary to clarify the definition of test types helping us to have a common sense: - Function Regression test (Language Neutral Tests) It is language neutral and we only need to run it once in any of the languages. The localization for this kind of tests could be word-by-word. - L10N Regression test (Language Dependent Tests) It is language dependent and we need to run them in all language. The localization for this kind of tests could be slightly different language versions based on the libreoffice features. - Feature test It contains new features testing for each release of libreoffice. Test cases are added on demand in each new release, and the test cases are not reusable in the next release. 2. In the new scheme pdf, it looks feature tests are not included while a legacy Basic Function Test is there. Just to make things more clear, I am udpating Rimas' chart a bit: Basic Function Test -> Feature Tests General Function Test -> Function Regression Tests I believe we didn't really have misunderstanding on this but the legacy problem, right? 3. Do we want components split in L10n test? I don't have a definite answer for this yet but prone to keep the components. On one hand, at least for impress and calc, we have some specific areas to cover as tempalte, slide show in multiple language, calc cell format etc. On the other hand, the consistent structure may make people's learning curve more mild. 4. The amount of how many localization is another concern. In the new scheme, for each priority we will have "7 components * 4 locale * 4 priority" (112) groups. If I understand it right, when there are 10 locales, the number will be 2800 ?! Image we have 112 test groups in the first section when running test: http://wiki.documentfoundation.org/images/7/77/Litmus_submit-de.png While another choice to be considered could be switching the position of test types and components, something like: / p1-en-Feature Tests - 7 components (en) / p1-xx-Feature Tests - 7 components (xx) / p1-en-L10n Regresstion Tests - 7 components (en) / p1-xx-L10n Regresstion Tests - 7 components (xx) master - p1-en-Function Regresstion Tests - 7 components (ATM: multi-language case) \ p1-xx-Function Regresstion Tests / Besides it will reduce the groups size to "3*locale*priority", by the fact we can't select subgroups when creating test run (I didn't notice it before, thanks Rimas), we can create a test run for particular test types, which might be more likely to happen than creating test runs for particular components. It would be more clear of the strategy by comments Rimas' origin plan: >* create testruns based on priority, locale, component, or any >combination of these -> create testruns based on priority, locale, test type, or any combination of these >* create a single catch-all testrun for a single version of LibO >* share the same General Functional Tests subgroup between testgroups >designated for different locales (that is, you would create this >subgroup once, and add it to all locale groups) -> share the same components subgroup in Function Regression Tests testgroups. >* drop 75% of the testgroups (there would be about 28 of them initially) >if/when proper testcase localization is implemented, still not rendering >the remaining testgroups useless. -> This is the thing I am not pretty sure. Will this switch cause unhandlable problems for our potential translation system ? 5. Another critical question raised about different testing execution times of Function and L10N regression. >> 2011.11.15 17:30, Petr Mladek rašė: >> I didn't say they have to be translated word-to-word. They should be >> *localized*, and I would expect a localized testcase to suggest >> localized number and date formats and other stuff. > > I see. > > Well another question is how it appears in the test run. The language > independent tests are proceed once. The language dependent tests need to > be proceed for every localization. The test case transaltion can be considered into 2 layers in current test strategy: - For Function Regression Tests (Language Neutral Tests) Review the definition on the top of the mail, the translation version case id is supposed to be the same with its English version. So all of the people run the same test cases but read them in different language. - For L10n Regression Tests (Language Dependent Tests) Review the definition on the top of the mail, the translation version case id is not the same one as its English version. So all of th
Re: [Libreoffice-qa] Test case naming
Hi, > 4. The amount of how many localization is another concern. In the new scheme, > for each priority we will have "7 components * 4 locale * 4 priority" (112) > groups. If I understand it right, when there are 10 locales, the number will > be 2800 ?! Sorry for bring shocking, s/2800/280 :) Best wishes, Yifan ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
Re: [Libreoffice-qa] Test case naming
Yifan Jiang píše v St 16. 11. 2011 v 14:48 +0800: > 4. The amount of how many localization is another concern. In the new scheme, > for each priority we will have "7 components * 4 locale * 4 priority" (112) > groups. If I understand it right, when there are 10 locales, the number will > be 280 ?! IMHO, anything more than 30 testgroups is not usable. It will be hard to analyze results. It will be hard to create test run for P1 tests... => I think that we can't have language name in the split test groups by languages. We have 4 languages now. IMHO, it is realistic to have 20 languages in not that far future. => we can have language in the subgroup names only. > While another choice to be considered could be switching the position > of test types and components, something like: > >/ p1-en-Feature Tests - 7 components (en) >/ p1-xx-Feature Tests - 7 components (xx) >/ p1-en-L10n Regresstion Tests - 7 components (en) >/ p1-xx-L10n Regresstion Tests - 7 components (xx) > master - p1-en-Function Regresstion Tests - 7 components (ATM: multi-language > case) >\ p1-xx-Function Regresstion Tests / > > Besides it will reduce the groups size to "3*locale*priority", by the fact we > can't select subgroups when creating test run (I didn't notice it before, > thanks Rimas), we can create a test run for particular test types, which might > be more likely to happen than creating test runs for particular components. > > It would be more clear of the strategy by comments Rimas' origin plan: > > >* create testruns based on priority, locale, component, or any > >combination of these > > -> create testruns based on priority, locale, test type, or any > combination of these I am afraid that it is not the effort. We need a compromise. We need to estimate what test runs will make sense. I am afraid that priority is the most important criteria. Anything else would crate too many test groups and is not worth the effort. Please, see the mail "Test Structure in Litmus" for more details. Best Regards, Petr ___ List Name: Libreoffice-qa mailing list Mail address: Libreoffice-qa@lists.freedesktop.org Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
