Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hey,

On Fri, Mar 8, 2019 at 6:33 PM Miklos Vajna  wrote:

> Hi,
>
> The current practice is: if 'make check' passes (which is more or less
> enforced by Jenkins) and the change looks good to a reviewer, the change
> goes in. And then releases are based on time, so it's really rare that
> there are "blocker" bugs which would delay a release.
>
> The ideal for any new kind of testing (including accessibility) is that
> it's integrated into 'make check', and whatever those tests cover are
> not OK to be broken anytime.
>
> If the proposed a11y tests are part of make check, then it's easy to
> promise that they won't be broken; otherwise it's just a best effort
> thing without any guarantees.
>
> At least that's how I understood what Markus wrote, and I agree with
> that.
>

Exactly that. With the added comment about the reliability necessary or
developers will start ignoring test failures or even worse disable failing
tests.

Kind regards,
Markus


> Regards,
>
> Miklos
>
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hey,

On Fri, Mar 8, 2019 at 6:33 PM Miklos Vajna  wrote:

> Hi,
>
> The current practice is: if 'make check' passes (which is more or less
> enforced by Jenkins) and the change looks good to a reviewer, the change
> goes in. And then releases are based on time, so it's really rare that
> there are "blocker" bugs which would delay a release.
>
> The ideal for any new kind of testing (including accessibility) is that
> it's integrated into 'make check', and whatever those tests cover are
> not OK to be broken anytime.
>
> If the proposed a11y tests are part of make check, then it's easy to
> promise that they won't be broken; otherwise it's just a best effort
> thing without any guarantees.
>
> At least that's how I understood what Markus wrote, and I agree with
> that.
>

Exactly that. With the added comment about the reliability necessary or
developers will start ignoring test failures or even worse disable failing
tests.

Kind regards,
Markus


> Regards,
>
> Miklos
>
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Extending subsequent tests with dogtail tests?

[Alex ARNAUD]

Le 07/03/2019 à 19:32, Markus Mohrhard a écrit :
> Most likely no. If we have tests for something it is more likely that it
> will be fixed after it is discovered by in the end an accessibility
> regression is similar to any other regression. That means that a test
> started failing you are expected to inspect the test failure before
> commiting but we will most likely not stop a release if a new test
> discovers that something is broken. In general while we do much to avoid
> regressions and having tests helps we don't treat regressions as
> complete release blockers. There is always a case-by-case analysis
> neccessary.

Do you mean someone will break something in LibreOffice and will fix it 
in the future or do you mean someone break accessibility and Hypra has 
to fix it itself in the future ?


In the last scenario, I don't see any reason to have non-regression test 
as we already know that people usually don't take care of accessibility 
and it's why we're forced to keep LibreOffice 4.2 for now. If we expect 
to have one developer fixing bug of thousand of others (on many free 
software) especially because it's accessibility I don't think it's reliable.


Best regards,
Alex.
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Extending subsequent tests with dogtail tests?

[Miklos Vajna]

Hi,

The current practice is: if 'make check' passes (which is more or less
enforced by Jenkins) and the change looks good to a reviewer, the change
goes in. And then releases are based on time, so it's really rare that
there are "blocker" bugs which would delay a release.

The ideal for any new kind of testing (including accessibility) is that
it's integrated into 'make check', and whatever those tests cover are
not OK to be broken anytime.

If the proposed a11y tests are part of make check, then it's easy to
promise that they won't be broken; otherwise it's just a best effort
thing without any guarantees.

At least that's how I understood what Markus wrote, and I agree with
that.

Regards,

Miklos


signature.asc
Description: Digital signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Alex ARNAUD]

Le 07/03/2019 à 19:32, Markus Mohrhard a écrit :
> Most likely no. If we have tests for something it is more likely that it
> will be fixed after it is discovered by in the end an accessibility
> regression is similar to any other regression. That means that a test
> started failing you are expected to inspect the test failure before
> commiting but we will most likely not stop a release if a new test
> discovers that something is broken. In general while we do much to avoid
> regressions and having tests helps we don't treat regressions as
> complete release blockers. There is always a case-by-case analysis
> neccessary.

Do you mean someone will break something in LibreOffice and will fix it 
in the future or do you mean someone break accessibility and Hypra has 
to fix it itself in the future ?


In the last scenario, I don't see any reason to have non-regression test 
as we already know that people usually don't take care of accessibility 
and it's why we're forced to keep LibreOffice 4.2 for now. If we expect 
to have one developer fixing bug of thousand of others (on many free 
software) especially because it's accessibility I don't think it's reliable.


Best regards,
Alex.
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hey,

let me at least add some comments.

On Thu, Mar 7, 2019 at 5:42 PM Jean-Philippe MENGUAL <
jean-philippe.meng...@libreoffice.org> wrote:

> Hi,
>
>
> Le 06/03/2019 à 20:52, Samuel Thibault a écrit :
> > Hello,
> >
> > Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:
> >> On a slightly related note I think that we have already quite a few
> tests for
> >> the accessibility UNO layer but as that layer is full of bugs many of
> the tests
> >> are disabled. It might be a good idea to work on these tests before
> actually
> >> trying to implement more complex tests that depend on lower layers
> working
> >> correctly.
> >
> > I'm not sure which piece you are referring to.  Is that the AWB?  I
> > indeed see some source code in toolkit/test/accessibility but no
> > reference to it.
>

There should be accessibility UNO API tests in most top level modules. E.g.
from a quick git grep Accessible -- qadevOOo/Jar_OOoRunner.mk:

qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleAction \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleContext \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleEditableText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleEventBroadcaster \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleExtendedComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleImage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleSelection \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleValue \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_basctl/AccessibleShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_HeaderFooter \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_PreviewCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sch/AccessibleDocumentView \
qadevOOo/Jar_OOoRunner.mk:qadevOOo/tests/java/mod/_sc/ScAccessibleCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvGrid \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvRuler \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleDocument \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleDocumentPagePreview \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeader \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeaderArea \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewHeaderCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleSpreadsheet \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleDrawDocumentView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleOutlineView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleSlideView \
qadevOOo/Jar_OOoRunner.mk:qadevOOo/tests/java/mod/_sm/SmEditAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sm/SmGraphicAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTableCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrl \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrlEntry \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPageList \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBoxEntry \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessibleControlShape \
qadevOOo/Jar_OOoRunner.mk:

Re: Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hey,

let me at least add some comments.

On Thu, Mar 7, 2019 at 5:42 PM Jean-Philippe MENGUAL <
jean-philippe.meng...@libreoffice.org> wrote:

> Hi,
>
>
> Le 06/03/2019 à 20:52, Samuel Thibault a écrit :
> > Hello,
> >
> > Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:
> >> On a slightly related note I think that we have already quite a few
> tests for
> >> the accessibility UNO layer but as that layer is full of bugs many of
> the tests
> >> are disabled. It might be a good idea to work on these tests before
> actually
> >> trying to implement more complex tests that depend on lower layers
> working
> >> correctly.
> >
> > I'm not sure which piece you are referring to.  Is that the AWB?  I
> > indeed see some source code in toolkit/test/accessibility but no
> > reference to it.
>

There should be accessibility UNO API tests in most top level modules. E.g.
from a quick git grep Accessible -- qadevOOo/Jar_OOoRunner.mk:

qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleAction \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleContext \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleEditableText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleEventBroadcaster \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleExtendedComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleImage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleSelection \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleValue \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_basctl/AccessibleShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_HeaderFooter \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_PreviewCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sch/AccessibleDocumentView \
qadevOOo/Jar_OOoRunner.mk:qadevOOo/tests/java/mod/_sc/ScAccessibleCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvGrid \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvRuler \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleDocument \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleDocumentPagePreview \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeader \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeaderArea \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewHeaderCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleSpreadsheet \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleDrawDocumentView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleOutlineView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleSlideView \
qadevOOo/Jar_OOoRunner.mk:qadevOOo/tests/java/mod/_sm/SmEditAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sm/SmGraphicAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTableCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrl \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrlEntry \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPageList \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBoxEntry \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessibleControlShape \
qadevOOo/Jar_OOoRunner.mk:

Re: Extending subsequent tests with dogtail tests?

[Jean-Philippe MENGUAL]

Hi,


Le 06/03/2019 à 20:52, Samuel Thibault a écrit :

Hello,

Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:

On a slightly related note I think that we have already quite a few tests for
the accessibility UNO layer but as that layer is full of bugs many of the tests
are disabled. It might be a good idea to work on these tests before actually
trying to implement more complex tests that depend on lower layers working
correctly.


I'm not sure which piece you are referring to.  Is that the AWB?  I
indeed see some source code in toolkit/test/accessibility but no
reference to it.


The focus handling can be easily integrated into the existing UI
testing infrastructure and might benefit there from some of the concepts that
should make them more stable


Good :) So could you plan to work on it?


How could such plan be scheduled in LO qa? Should we report a bug? Or 
open a wiki roadmap?


The problem now is that it is not possible to fix accessibility bugs, 
fix regressions from 4.2, integrating in the code non-regression tests, 
and do the same for the three major programs in free software. It is 
less a will problem than a resource problem, because even with funds, we 
do not have enough persons to work on this with skills related to 
Libreoffice and accessibility in general. As you know more and more 
persons go the web or backend techno, less in programming for desktop 
software.


So while we are ready to fix accessibility bugs, we need non-regression 
tests. And it is difficult to do both. The tool on which we are working, 
dogtail, is interesting because enables to test via the same framework 
different programs, without needing to know the code of each of one. I 
think gateway is possible between Libreoffice framework test and such 
tool. We also could imagine a dedicated machine with dogtail to test, 
but should be acceptd. Also, the thing is to know if LO is ready to 
prevent a release or a commit because introduces a regression in 
accessibility.


Well to sum up, beyond our effort, that we try general and 
cross-software, we would need to know how we can together set a kind of 
roadmap to implement accessibility in the existing frameworks, and add a 
layer to make common scenarios between our general tool and LO's one. 
Having that may be in the easy hacks to have contributions? Or b a 
specific TDF tender just like we did for labels?


Regards


Samuel
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice



--
Logo Hypra  JEAN-PHILIPPE MENGUAL
DIRECTEUR TECHNIQUE ET QUALITÉ
102, rue des poissonniers, 75018, Paris
Tel : +331 84 73 06 61  Mob : +336 76 34 93 37

jpmeng...@hypra.fr 
www.hypra.fr 
Facebook Hypra  Twitter Hypra
 Linkedin Jean-Philippe



___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Jean-Philippe MENGUAL]

Hi,


Le 06/03/2019 à 20:52, Samuel Thibault a écrit :

Hello,

Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:

On a slightly related note I think that we have already quite a few tests for
the accessibility UNO layer but as that layer is full of bugs many of the tests
are disabled. It might be a good idea to work on these tests before actually
trying to implement more complex tests that depend on lower layers working
correctly.


I'm not sure which piece you are referring to.  Is that the AWB?  I
indeed see some source code in toolkit/test/accessibility but no
reference to it.


The focus handling can be easily integrated into the existing UI
testing infrastructure and might benefit there from some of the concepts that
should make them more stable


Good :) So could you plan to work on it?


How could such plan be scheduled in LO qa? Should we report a bug? Or 
open a wiki roadmap?


The problem now is that it is not possible to fix accessibility bugs, 
fix regressions from 4.2, integrating in the code non-regression tests, 
and do the same for the three major programs in free software. It is 
less a will problem than a resource problem, because even with funds, we 
do not have enough persons to work on this with skills related to 
Libreoffice and accessibility in general. As you know more and more 
persons go the web or backend techno, less in programming for desktop 
software.


So while we are ready to fix accessibility bugs, we need non-regression 
tests. And it is difficult to do both. The tool on which we are working, 
dogtail, is interesting because enables to test via the same framework 
different programs, without needing to know the code of each of one. I 
think gateway is possible between Libreoffice framework test and such 
tool. We also could imagine a dedicated machine with dogtail to test, 
but should be acceptd. Also, the thing is to know if LO is ready to 
prevent a release or a commit because introduces a regression in 
accessibility.


Well to sum up, beyond our effort, that we try general and 
cross-software, we would need to know how we can together set a kind of 
roadmap to implement accessibility in the existing frameworks, and add a 
layer to make common scenarios between our general tool and LO's one. 
Having that may be in the easy hacks to have contributions? Or b a 
specific TDF tender just like we did for labels?


Regards


Samuel
___
LibreOffice mailing list
libreoff...@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice



--
Logo Hypra  JEAN-PHILIPPE MENGUAL
DIRECTEUR TECHNIQUE ET QUALITÉ
102, rue des poissonniers, 75018, Paris
Tel : +331 84 73 06 61  Mob : +336 76 34 93 37

jpmeng...@hypra.fr 
www.hypra.fr 
Facebook Hypra  Twitter Hypra
 Linkedin Jean-Philippe



___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: Extending subsequent tests with dogtail tests?

[Samuel Thibault]

Hello,

Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:
> On a slightly related note I think that we have already quite a few tests for
> the accessibility UNO layer but as that layer is full of bugs many of the 
> tests
> are disabled. It might be a good idea to work on these tests before actually
> trying to implement more complex tests that depend on lower layers working
> correctly.

I'm not sure which piece you are referring to.  Is that the AWB?  I
indeed see some source code in toolkit/test/accessibility but no
reference to it.

> The focus handling can be easily integrated into the existing UI
> testing infrastructure and might benefit there from some of the concepts that
> should make them more stable

Good :) So could you plan to work on it?

Samuel
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Extending subsequent tests with dogtail tests?

[Samuel Thibault]

Noel Grandin, le jeu. 28 févr. 2019 15:38:25 +0200, a ecrit:
> On Sun, 24 Feb 2019 at 22:04, Samuel Thibault <[1]sthiba...@hypra.fr> wrote:
> 
> The fact that even the C UI strings may change is a concern indeed. We
> however don't really have another way to identify widgets, do we? (we
> don't want to identify them structurally, that'd be even less stable)
> 
> The LibreOffice python UI tests use IDs that are maintained in the .ui files
> (and are just strings), which are relatively stable.

Right. AIUI they are not exposed to the accessibility interfaces
however, we'll have to fix that.

Samuel
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Noel Grandin]

On Sun, 24 Feb 2019 at 22:04, Samuel Thibault  wrote:

> The fact that even the C UI strings may change is a concern indeed. We
> however don't really have another way to identify widgets, do we? (we
> don't want to identify them structurally, that'd be even less stable)
>
>
The LibreOffice python UI tests use IDs that are maintained in the .ui
files (and are just strings), which are relatively stable.
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: Extending subsequent tests with dogtail tests?

[Noel Grandin]

On Sun, 24 Feb 2019 at 22:04, Samuel Thibault  wrote:

> The fact that even the C UI strings may change is a concern indeed. We
> however don't really have another way to identify widgets, do we? (we
> don't want to identify them structurally, that'd be even less stable)
>
>
The LibreOffice python UI tests use IDs that are maintained in the .ui
files (and are just strings), which are relatively stable.
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Extending subsequent tests with dogtail tests?

[Samuel Thibault]

Hello,

Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:
> On Sat, Feb 23, 2019 at 6:24 PM Samuel Thibault <[1]sthiba...@hypra.fr> wrote:
> 
> > That said, we could as well make tests work at both layers. Run them
> > along uitests, thus very frequently, and run them periodically through
> > the accessibility layer on a system which has it.
> 
> I think you most likely want to test them independently if possible
> as I think you'll discover that combining them will lead to hard to diagnose
> problems.

That's why I mentioned trying with both layers.

> Additionally, the target of any test framework has to be that it
> minimizes the number of random test failures and requires as little adoption
> for unrelated code changes as possible.

Sure. That's why we're proposing to follow the blind user testcases,
which we usually don't want to change without some thinking.

> I noticed after quickly looking at your proposed tests is that you use UI
> strings in your tests which I think you want to avoid as much as possible.
> There are several problems with UI strings that make them a bad property 
> during
> testing: they change often, often not even by developers and are localized.
> Especially the second point means that your tests suddenly only work in en-US
> which surely limits a bit the usefulness of the tests. Even more if you plan 
> to
> generate test cases automatically as it means that the tests can only be
> generated with an en-US locale.

Sure, both testing and generating testcases needs to be done in the C
locale.

The fact that even the C UI strings may change is a concern indeed. We
however don't really have another way to identify widgets, do we? (we
don't want to identify them structurally, that'd be even less stable)

That was actually mentioned in

https://gitlab.gnome.org/GNOME/atk/issues/4

we'd need stable identifiers for the expected fields.

> On a slightly related note, please make sure that you are developing against
> python 3. I did not check if you are actually using any pure python2 features
> but at least "#!/usr/bin/python" will get you a python 2 interpreter on most
> systems.

Yes, that's on purpose because Debian doesn't have the python3 package
for dogtail yet :)

Samuel
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hello Samuel,

let me add a few comments based on many years implementing testing
frameworks for LibreOffice and most recently working on the UI testing.

On Sat, Feb 23, 2019 at 6:24 PM Samuel Thibault  wrote:

> Noel Grandin, le sam. 23 févr. 2019 12:19:19 +0200, a ecrit:
> > However, the current python UI test stuff talks directly to the vcl/
> widgets.
> > But between the vcl/ widgets and an actual accessibility user lies at
> least two
> > major chunks of code - the generic accessibility/ stuff, and the
> operating
> > system specific bridges.
>
> Yes, that's my concern with testing only at the vcl layer.
>
> That said, we could as well make tests work at both layers. Run them
> along uitests, thus very frequently, and run them periodically through
> the accessibility layer on a system which has it.
>
>
If my understanding of your idea is correct you want to test actually two
things that are required for working accessibility for users. On one hand
you need to test the accessibility APIs and on the other hand correct focus
handling. I think you most likely want to test them independently if
possible as I think you'll discover that combining them will lead to hard
to diagnose problems. Additionally, the target of any test framework has to
be that it minimizes the number of random test failures and requires as
little adoption for unrelated code changes as possible.

I noticed after quickly looking at your proposed tests is that you use UI
strings in your tests which I think you want to avoid as much as possible.
There are several problems with UI strings that make them a bad property
during testing: they change often, often not even by developers and are
localized. Especially the second point means that your tests suddenly only
work in en-US which surely limits a bit the usefulness of the tests. Even
more if you plan to generate test cases automatically as it means that the
tests can only be generated with an en-US locale.

On a slightly related note I think that we have already quite a few tests
for the accessibility UNO layer but as that layer is full of bugs many of
the tests are disabled. It might be a good idea to work on these tests
before actually trying to implement more complex tests that depend on lower
layers working correctly. The focus handling can be easily integrated into
the existing UI testing infrastructure and might benefit there from some of
the concepts that should make them more stable (deadlock detection,
addressing UI elements through ID instead of UI visible strings, mostly
working async dialog and action handling) and I think it makes sense to
check how much of the accessibility part can not be covered with a
combination of UI testing and accessibility UNO API testing. It might even
be possible to get accessibility event interception into the UI testing to
check that events are generated correctly during actions. I would need to
look into how our accessibility handling works in the VCL layer to be sure
that this actually works.

For the dependencies we surely don't want to include the source code itself
into LibreOffice's external dependency handling. At least for the beginning
the easiest approach might be something like an
--enable-accessibility-tests flag which can be checked in configure.ac
where we make sure all required dependencies are around. If we ever decide
that they should be run by default we can still switch the default value
and make it possible for people to override the setting in their autogen
settings.

On a slightly related note, please make sure that you are developing
against python 3. I did not check if you are actually using any pure
python2 features but at least "#!/usr/bin/python" will get you a python 2
interpreter on most systems.

Kind regards,
Markus
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hello Samuel,

let me add a few comments based on many years implementing testing
frameworks for LibreOffice and most recently working on the UI testing.

On Sat, Feb 23, 2019 at 6:24 PM Samuel Thibault  wrote:

> Noel Grandin, le sam. 23 févr. 2019 12:19:19 +0200, a ecrit:
> > However, the current python UI test stuff talks directly to the vcl/
> widgets.
> > But between the vcl/ widgets and an actual accessibility user lies at
> least two
> > major chunks of code - the generic accessibility/ stuff, and the
> operating
> > system specific bridges.
>
> Yes, that's my concern with testing only at the vcl layer.
>
> That said, we could as well make tests work at both layers. Run them
> along uitests, thus very frequently, and run them periodically through
> the accessibility layer on a system which has it.
>
>
If my understanding of your idea is correct you want to test actually two
things that are required for working accessibility for users. On one hand
you need to test the accessibility APIs and on the other hand correct focus
handling. I think you most likely want to test them independently if
possible as I think you'll discover that combining them will lead to hard
to diagnose problems. Additionally, the target of any test framework has to
be that it minimizes the number of random test failures and requires as
little adoption for unrelated code changes as possible.

I noticed after quickly looking at your proposed tests is that you use UI
strings in your tests which I think you want to avoid as much as possible.
There are several problems with UI strings that make them a bad property
during testing: they change often, often not even by developers and are
localized. Especially the second point means that your tests suddenly only
work in en-US which surely limits a bit the usefulness of the tests. Even
more if you plan to generate test cases automatically as it means that the
tests can only be generated with an en-US locale.

On a slightly related note I think that we have already quite a few tests
for the accessibility UNO layer but as that layer is full of bugs many of
the tests are disabled. It might be a good idea to work on these tests
before actually trying to implement more complex tests that depend on lower
layers working correctly. The focus handling can be easily integrated into
the existing UI testing infrastructure and might benefit there from some of
the concepts that should make them more stable (deadlock detection,
addressing UI elements through ID instead of UI visible strings, mostly
working async dialog and action handling) and I think it makes sense to
check how much of the accessibility part can not be covered with a
combination of UI testing and accessibility UNO API testing. It might even
be possible to get accessibility event interception into the UI testing to
check that events are generated correctly during actions. I would need to
look into how our accessibility handling works in the VCL layer to be sure
that this actually works.

For the dependencies we surely don't want to include the source code itself
into LibreOffice's external dependency handling. At least for the beginning
the easiest approach might be something like an
--enable-accessibility-tests flag which can be checked in configure.ac
where we make sure all required dependencies are around. If we ever decide
that they should be run by default we can still switch the default value
and make it possible for people to override the setting in their autogen
settings.

On a slightly related note, please make sure that you are developing
against python 3. I did not check if you are actually using any pure
python2 features but at least "#!/usr/bin/python" will get you a python 2
interpreter on most systems.

Kind regards,
Markus
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Extending subsequent tests with dogtail tests?

[Samuel Thibault]

Noel Grandin, le sam. 23 févr. 2019 12:19:19 +0200, a ecrit:
> However, the current python UI test stuff talks directly to the vcl/ widgets.
> But between the vcl/ widgets and an actual accessibility user lies at least 
> two
> major chunks of code - the generic accessibility/ stuff, and the operating
> system specific bridges.

Yes, that's my concern with testing only at the vcl layer.

That said, we could as well make tests work at both layers. Run them
along uitests, thus very frequently, and run them periodically through
the accessibility layer on a system which has it.

Samuel
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Noel Grandin]

So yes, the focus information would be fairly easy to expose to the python
UI test infrastructure.

And yes, it would be good to limit the number of different frameworks we
use for testing (we already have 4 - C++ direct, C++ via UNO, Java via UNO,
python via UNO)

However, the current python UI test stuff talks directly to the vcl/
widgets. But between the vcl/ widgets and an actual accessibility user lies
at least two major chunks of code - the generic accessibility/ stuff, and
the operating system specific bridges.

And it would be good to test those too, which this dogtail-based stuff
would do.

 just laying out the pros and cons... :-)
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: Extending subsequent tests with dogtail tests?

[Noel Grandin]

So yes, the focus information would be fairly easy to expose to the python
UI test infrastructure.

And yes, it would be good to limit the number of different frameworks we
use for testing (we already have 4 - C++ direct, C++ via UNO, Java via UNO,
python via UNO)

However, the current python UI test stuff talks directly to the vcl/
widgets. But between the vcl/ widgets and an actual accessibility user lies
at least two major chunks of code - the generic accessibility/ stuff, and
the operating system specific bridges.

And it would be good to test those too, which this dogtail-based stuff
would do.

 just laying out the pros and cons... :-)
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Extending subsequent tests with dogtail tests?

[Miklos Vajna]

Hi Samuel,

On Thu, Feb 21, 2019 at 05:06:37PM +0100, Samuel Thibault  
wrote:
> Indeed, but AIUI the keyboard event synthesis is directly sent to the
> expected ui object, so it doesn't check that e.g. when pressing F6
> several times in writer, the proper objects get focus. Or would it be
> possible to extend it to be able to check the focus?

CC Markus if focus testing is (or could be) possible with ui testing.

Regards,

Miklos


signature.asc
Description: Digital signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Extending subsequent tests with dogtail tests?

[Samuel Thibault]

Miklos Vajna, le jeu. 21 févr. 2019 09:08:48 +0100, a ecrit:
> Are you aware of
> ?

Sure :)

> Seems there is some overlap between dogtail and uitests.

Indeed, but AIUI the keyboard event synthesis is directly sent to the
expected ui object, so it doesn't check that e.g. when pressing F6
several times in writer, the proper objects get focus. Or would it be
possible to extend it to be able to check the focus?

Samuel
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Re: Extending subsequent tests with dogtail tests?

[Miklos Vajna]

Hi Samuel,

Are you aware of
?

Seems there is some overlap between dogtail and uitests.

Regards,

Miklos


signature.asc
Description: Digital signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice

Extending subsequent tests with dogtail tests?

[Samuel Thibault]

Hello,

Libreoffice currently uses unit tests to check precise functions, and
junit tests to check the behavior at the programmatic interface.

For accessibility testing, we would like to introduce tests to check the
behavior at the user interface itself, by using dogtail (developped by
Fedora, based on at-spi) to simulate keypresses, mouse clicks, and check
the results.

The idea is that at Hypra, we have a well-established documentation
of what keyboard shortcut sequences blind users would employ to use
libreoffice, and which we should thus make sure always work. The key
sequences might have to be changed if that's for a good reason, but at
least we would know about it and update the documentation accordingly.

I have put a prototype of what we are thinking of on 
https://git.hypra.fr/hypra/regression-testing/tree/master

An example of what it would look like is:

  press('F6')
  menu = app.child('File', 'menu')
  waitFocus(menu)
  
  press('F6')
  new = app.child('New', 'push button')
  waitFocus(new)
  
  press('F6')
  style = app.child('Paragraph Style', 'text')
  waitFocus(style)

etc.

which here allows to check that the F6 shortcut properly gets the menu,
the toolbar, the style, to be focused, etc. 

I have written a "log" script that can generate such testcase code:
a user can run libreoffice and perform a usage scenario, and "log"
will get the interesting events from at-spi (keypresses, focus change,
checkbox status change, etc.) and emit the python code which simulates
keypresses and waits for the results. That way we can easily produce
testcases from our documentation, by just running the use cases
described in it, which are representative of what a blind user needs to
be able to do.

Dogtail depends on atspi, but also on python-cairo, python-gi,
gir1.2-gtk-3.0.  I don't think we really want to add them to external/
just for testing?  Just like, AIUI, junit tests don't run if you don't
have java already available?  So we would just run the tests along junit
tests, on a system which is known to have the required dependencies,
right?  We (Hypra) could then feed libreoffice with such tests, from
our documentation.  We can probably produce dozens of them, which can
probably be tested within something like a few minutes.

Samuel
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice