Re: [Libreoffice-qa] db test VM

[Stephan Bergmann]

On 10/27/2011 09:08 PM, Petr Mladek wrote:

Stefan Bergman has recently enabled subsequent test[1] that need the
installed LO. They could be written in Java and C++. I guess that they
are supposed to replace the problematic testtool. I wonder if you could
write the Base test this way as well.


Originally, the subsequenttests framework was designed to clean up the 
already existing qa/unoapi and qa/complex tests, and the smoke test.  It 
works by having C++/Java test code that drives an soffice instance via 
remote UNO, while testtool tests are written in BASIC and have more 
intimate access to the soffice instance's UI.  (The smoke test is kind 
of a mixture, driving soffice via remote UNO from a C++ test, but the 
main test logic is included in a BASIC script that is run within 
soffice.  That's more for historical reasons than a clean design, and I 
would not necessarily recommend modelling new tests after that pattern.)


So, it remains to be seen whether subsequenttests can replace testtool; 
migrating the existing testtool tests would definitely not be trivial.


Stephan
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] db test VM

[Petr Mladek]

Michael Meeks píše v Čt 27. 10. 2011 v 16:10 +0100:
> > Now for today I would like to actually get testtool running in
> > interactive mode - i.e start testtool, load one of the .bas test and
> > have it execute (as of right now that last step fails for every test
> > script I've tried. It gives an error323 when it jumps into the first
> > include file in the script...)
> 
>   The general consensus I've heard in development is that maintaining
> testtool (particularly as we change the UI) is going to be increasingly
> problematic. It is also rather a creaky pile.

IMHO, nobody has maintained the testtool scripts in Libreoffice and
nobody ran them for LO-3.4. They probably can't even be started, see:
https://bugs.freedesktop.org/show_bug.cgi?id=37566
https://bugs.freedesktop.org/show_bug.cgi?id=37059

LibreOffice developers prefer to write unit tests that are proceed
during compilation.

Stefan Bergman has recently enabled subsequent test[1] that need the
installed LO. They could be written in Java and C++. I guess that they
are supposed to replace the problematic testtool. I wonder if you could
write the Base test this way as well.

>   :-) Noel is prolly a good contact there.

IFAIK, Noel tried to fix the two above mentioned bugs. Then he put it
aside as nobody was really interested into it. IMHO, it does not make
sense to have working testtool if nobody maintains the test scripts.

Debugging the testtool scripts is time consuming and a kind of
nightmare. This is why developers prefer the unit and subsequent tests.

>   I (for one) was hoping that with openQA we could do a similar level of
> UI testing, but based on what actually shows up rendered on the screen
> (OCR'd) and that this might be more robust, and provide a fresh start
> here that doesn't have testtool's problems.

Yi Fan did some investigation. It seems that openQA does not support any
OCR. It probably just compares md5-sums of screenshot of the whole
screen. I am afraid that this approach is not usable for LO because it
is hard to maintain. Any different pixel will invalidate the test.

Reference:
http://wiki.documentfoundation.org/QA/Testing/Subsequenttests

Best Regards,
Petr

___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] Updating Litmus testing organization.

[Petr Mladek]

Yifan Jiang píše v St 26. 10. 2011 v 23:50 +0800:
> Branch Master Function Regression Test
> ==
> 
> Function Regression Test Branch stores language neutral function cases
> regardless in which locale the tests will be executed.

[...]

> Branch Master L10N Regression Test
> ==
> 
> L10N Regression Test Branch stores language specific test cases which should
> all be language sensitive. That is to say, we only put those cases really need
> test in multiple languages into this branch. For example: dictionary, spell
> check, hyphenazing, menu/dialog translation, asian language layout etc.

Great change. It helps to split the test cases for the
language-independent functionality and test cases that need to be run in
every localization. It saves resources and allows us to do more complex
testing.

> The English version is still there because we also need to provide a
> "synchronizing center" to share information between different
> languages. Ideally it is preferrable to have an English version of cases for
> each language specific test cases, so that QA who knows English can be
> enlighten when he/she creates test cases in his/her own language. Naturally,
> it is easy to figure out the test cases in English group will not be added
> into any test runs since they are supposed to be only usable in L10N
> environment.

In fact, we should run English version as well. We want to check, for
example, English spell checker, templates, typos in strings, optional
help content ;-)

>Current Progress of Updating
> =

> While I didn't remove any old cases at the moment, they are all
> currently in
> "Branch Master L10N Regression Test", since some of the test cases are
> re-usable and just there waiting for updating. I just do not have
> enough knowledge to update all of them :)

Many people do not read documentation. They just follow what they see. I
suggest to remove test cases that were moved to "Function Regression
Test" branch. If you are not comfortable with removing them, you might
temporary move them to a branch called "obsolete" or "bin" :-)

Also, it would be great if authors of the existing test cases could help
to clean up the rest.

Great work!

Best Regards,
Petr

___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

[Libreoffice-qa] Updating Litmus testing organization.

[Yifan Jiang]

Dear Libreoffice QA fellows,

Looking through the past year's knowledge of our Libreoffice Release QA and
people I talked in Libreoffice Conference, I was working on some change of
Litmus test cases organization these days.

The target of these works is mainly to make the test case categorized in a way
of more clear and efficient to run. The change is not significant to break our
previous experience but make them more clean.

** Concretely, in short, we are going to divide the single Regression test
branch into Function Regression Test branch and L10N Regression Test branch.

The example of test runs I created from this updated method can be seen:

https://tcm.documentfoundation.org/run_tests.cgi

Please don't be scared about the following long description, you will only
need to review them if you want to know more details about contributing
writing test cases. On the other hand, you don't have to read them if you are
only interested to run test cases as you did before.

Old test case organization
==

The old strategy of orgnizing cases in Litmus can be described as easy as:

+ Branch Master Regression Test
- Group: en/de/fr/pt-BR
- Subgroup: base/calc/writer/impress/draw
- Test Cases (described in the corresponding native language of 
Group)

So in the master branch of test cases repository, we have 4 Groups, in each of
the group we have 5 subgroups, which contains the substantial test cases. It
is easy to understand why the subgroups are divided by Libreoffice function
components. On the second level of the hierachy, we also have created the
lanugage Groups for the purpose of describing test cases in different
languages so that:

1. QA from non-english speaking environment can understand the tests in
his/her native language

2. Test cases can be modified for a particular language l10n test. For
example, considering the same test coverage of translation, the wording
and expecting results canbe slightly different between fr and pt-BR.

Why Divide
==

Typically, the old strategy brings us a "duplicate effort of execution
problem" when function tests are populated into Litmus. Let me show a good
example of a Writer test cases (w001):

The case in en Group:
#EN-w001 - Creating a new text document
https://tcm.documentfoundation.org/show_test.cgi?id=7

The case in fr Group:
#FR-w001 - Création d'un nouveau document texte
https://tcm.documentfoundation.org/show_test.cgi?id=46

The case in de Group:
#DE-w001 - Ein neues Textdokument erstellen
https://tcm.documentfoundation.org/show_test.cgi?id=87

It can be easily seen they are all describing exaclty same tests of "Creating
a new text document", and they were tested for several times in the different
language QA people (thanks for the effort to test!). However the test itself
is irrelevant with languages. That is to say, for this kind of tests, no
matter in which language, we only need to spend one time (not three or more
times) to execute for each release candidate build.

So it would be nice to treat those cases in another way, which brings our new
branch:

Branch Master Function Regression Test
==

Function Regression Test Branch stores language neutral function cases
regardless in which locale the tests will be executed.

For the sake of language diversity of QA people, it would be best to have them
described in different languages as well. Ideally the same case can be shown
in different language based on QA machine's locale or QA's flavor, but it
needs some deep hackers to expand Litmus capacity (see: point 4 -
http://wiki.documentfoundation.org/Litmus_TODO). So as a rude workaround,
currently we just merged all of the language of test description into a single
test case. Therefore no mattter who executes the case, all other people can
see its latest status now.

This is how #w001 looks like now:

https://tcm.documentfoundation.org/show_test.cgi?id=1054

Specifically the organization of the branch canbe described as:

+ Branch Master Function Regression Test
- Group: Function Test
- Subgroup: base/calc/writer/impress/draw
- Test Cases (each testcase is described in multiple languages)

As you can see we have only one Group called function test here, we do not
have functional overlapped test cases describing in different language anymore
in this branches.

Meanwhile, we can not deny the considerable needs of testing Language specific
features and translations. Actually the old way of organization fits these
requirement very well, so we can just keep what we have in a separate branch:

Branch Master L10N Regression Test
==

L10N Regression Test Branch stores language specific test cases which should
all be language sensitive. That is to say, we only put those cases really need
test in multiple languages into this branch. For example: dictionary, spell
check, hyphenazing, 

Re: [Libreoffice-qa] db test VM

[Michael Meeks]

Hi Drew,

On Thu, 2011-10-27 at 09:07 -0400, drew wrote:
> published that appliance VM this morning.

Nice :-)

> Now for today I would like to actually get testtool running in
> interactive mode - i.e start testtool, load one of the .bas test and
> have it execute (as of right now that last step fails for every test
> script I've tried. It gives an error323 when it jumps into the first
> include file in the script...)

The general consensus I've heard in development is that maintaining
testtool (particularly as we change the UI) is going to be increasingly
problematic. It is also rather a creaky pile.

Clearly if we just use it for base testing - perhaps it still makes
some sense (if we can isolate that stuff), but even so keeping the
infrastructure working is quite a burden.

> I suppose at this point what makes sense is to start hanging out on IRC
> and asking for help there as I try to finger out what I'm missing in
> getting testtool to run.

:-) Noel is prolly a good contact there.

I (for one) was hoping that with openQA we could do a similar level of
UI testing, but based on what actually shows up rendered on the screen
(OCR'd) and that this might be more robust, and provide a fresh start
here that doesn't have testtool's problems.

But of course, that prolly merits wider discussion.

Thanks !

Michael.

-- 
michael.me...@suse.com  <><, Pseudo Engineer, itinerant idiot

___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

[Libreoffice-qa] Minutes - QA related - TSC call 2011-10-27

[Rainer Bielefeld]

Hi,

you can find a summery (from my point of view) concerning QA related 
discussion of the Engineering Steering Committee on



Kind regards

Rainer
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] db test VM

[drew]

On Wed, 2011-10-26 at 16:00 +0200, Petr Mladek wrote:


> drew píše v Po 24. 10. 2011 v 10:16 -0400:
> > Howdy all,
> > 
> > Well, thought I'd let folks know what I've been up to with the
> > SuseSTUDIO service...errr stuff. 
> > 
> > Creating a VM with the daily build files turned out to be really
> > straightforward.
> 
> nice
> 
> > -  OK, so it's not automated for daily updates, but doing so by hand now
> > is really quite simple and fairly quick (3 clicks and ~10 minutes), and
> > not that big a deal for what I personally had in mind.
> 
> 3 clicks to create the image sounds perfectly fine. I think that more
> important is to automate the real testing.
> 
> > Currently there is no menu integration (should be there tomorrow :)
> 
> I am afraid that the plain daily build does not provide the desktop
> integration. Well, it should not be really needed for the Base testing.
> 
> > Anyway - I'll update the wiki page with more details this afternoon.
> 
> Where is the wiki page?

Well, I didn't really update this yet but the wiki page is at
http://wiki.documentfoundation.org/User:Drew/baseQA_VM

(which is now also listed on the information page for the appliance)

Alright so yesterday I added a couple of new files to:
http://susegallery.com/a/EvUJ20/lo-client-daily-libreoffice_master

1) Added all files from the daily release_configuration build, including
the testtool, binfilters, etc.

2) Pulled the testautomation scripts from git and added them to the
appliance at:
  /opt/lo-dev/program/qatesttool

3) added the files from
http://dev-builds.libreoffice.org/daily/losmoketest-0.2.tar.bz2
to the custom repository (drewjensen2: OpenSuse 11.4) and the appliance,
installed at:
  /tmp/losmoketest-0.2

4) pulled the updated losmoketest.py file from git and added at:
  /tmp

published that appliance VM this morning.

Now for today I would like to actually get testtool running in
interactive mode - i.e start testtool, load one of the .bas test and
have it execute (as of right now that last step fails for every test
script I've tried. It gives an error323 when it jumps into the first
include file in the script...)

I suppose at this point what makes sense is to start hanging out on IRC
and asking for help there as I try to finger out what I'm missing in
getting testtool to run.

Thanks

//drew


___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/