Re: Reorganization of Firefox-UI tests in mozilla-central

2017-10-31 Thread Martijn 
Update MDN docs when moving Marionette tests all over the place.

On Wed, Nov 9, 2016 at 11:16 AM, Henrik Skupin  wrote:
> Henrik Skupin wrote on 01/09/16 17:37:
>
> After a longer time without a response I would like to give a follow-up
> on where we stand at the moment and what the future will be for those tests.
>
> But first, I want to say thanks to everyone who participated in this
> thread. The received feedback was very helpful and showed that a couple
> of topics are questionable and needed further discussion.
>
> With all that in mind we decided to get away with the original idea of
> moving the firefox-ui tests to the appropriate components in the state
> they are in right now. This will ensure that:
>
> * no other subdirectory for a new kind of test has to be added
> * don't make it harder for developers to decide which harness to use
>
> Instead our team decided to work towards the goal in transforming the
> firefox-ui-functional tests into plain Marionette tests. The work as
> such is not that trivial given that a couple of things have to be
> obeyed, and not everything is clear yet - especially not how we want to
> continue in testing nightly and release builds via mozmill-ci. But that
> shouldn't stop us to make writing Marionette tests for Firefox easier.
>
> Just to give an overview, here are the sub-projects I have to work on:
>
> * Decouple the Firefox Puppeteer package from FirefoxTestCase and make
> it an optional (mixin) class, which then could even be used with
> MarionetteTestCase if wanted.
>
> * Get rid of large portions of the firefox-ui harness except for update
> tests which would still require a separate harness due to their own
> command line arguments.
>
> * Refactor Marionette tests in domain specific jobs. As you know we run
> all the existing tests (unit, browser, layout, toolkit) in a single job
> and report as `Mn` to Treeherder. This should be split up into chunks.
> There will be a follow-up email on this topic soon.
>
> * Ensure that those jobs which report as Tier-1 will not use remote
> testcase data, and consider a new Tier-2 group for others (necessary for
> lot of the fx-ui security tests)
>
> * Refactor existing firefox-ui-functional tests into plain Marionette
> tests and get browser peer review before moving them to the appropriate
> tests folders.
>
> * Keep automation (mozharness, mozmill-ci) intact in parallel so we do
> not loose test coverage as needed by QE.
>
>
> I'm sure that the above outlined goals will be in a better alignment
> with you all. If not, or if there are questions, please let me know.
>
> Thanks,
>
> --
> Henrik Skupin
> Senior Software Engineer
> Mozilla Corporation
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-11-09 Thread Henrik Skupin 
Henrik Skupin wrote on 01/09/16 17:37:

After a longer time without a response I would like to give a follow-up
on where we stand at the moment and what the future will be for those tests.

But first, I want to say thanks to everyone who participated in this
thread. The received feedback was very helpful and showed that a couple
of topics are questionable and needed further discussion.

With all that in mind we decided to get away with the original idea of
moving the firefox-ui tests to the appropriate components in the state
they are in right now. This will ensure that:

* no other subdirectory for a new kind of test has to be added
* don't make it harder for developers to decide which harness to use

Instead our team decided to work towards the goal in transforming the
firefox-ui-functional tests into plain Marionette tests. The work as
such is not that trivial given that a couple of things have to be
obeyed, and not everything is clear yet - especially not how we want to
continue in testing nightly and release builds via mozmill-ci. But that
shouldn't stop us to make writing Marionette tests for Firefox easier.

Just to give an overview, here are the sub-projects I have to work on:

* Decouple the Firefox Puppeteer package from FirefoxTestCase and make
it an optional (mixin) class, which then could even be used with
MarionetteTestCase if wanted.

* Get rid of large portions of the firefox-ui harness except for update
tests which would still require a separate harness due to their own
command line arguments.

* Refactor Marionette tests in domain specific jobs. As you know we run
all the existing tests (unit, browser, layout, toolkit) in a single job
and report as `Mn` to Treeherder. This should be split up into chunks.
There will be a follow-up email on this topic soon.

* Ensure that those jobs which report as Tier-1 will not use remote
testcase data, and consider a new Tier-2 group for others (necessary for
lot of the fx-ui security tests)

* Refactor existing firefox-ui-functional tests into plain Marionette
tests and get browser peer review before moving them to the appropriate
tests folders.

* Keep automation (mozharness, mozmill-ci) intact in parallel so we do
not loose test coverage as needed by QE.


I'm sure that the above outlined goals will be in a better alignment
with you all. If not, or if there are questions, please let me know.

Thanks,

-- 
Henrik Skupin
Senior Software Engineer
Mozilla Corporation
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-05 Thread Henrik Skupin 
L. David Baron wrote on 09/04/2016 10:02 AM:

> Recently, sheriffing practices have changed so that intermittent
> failure bugs that show up in different tests now have a separate
> bugzilla bug for each test they occur in.  This causes:
> 
>  1. a large number of rarely-occurring bugs -- enough that
>  intermittent-orange bugs are a significant fraction of the bugs
>  reported, but most of the bugs document only a single failure (in
>  many cases, a problem that has also occurred on other tests,
>  tracked in different bugs).  This encourages ignoring the bugs and
>  just looking at orangefactor for what's interesting/relevant.
> 
>  2. spreading out a single problem across >100 bugs [1] leads to the
>  severity of that problem being ignored for extended periods of
>  time.

We are all aware of this and lately the following bug got filed:
https://bugzilla.mozilla.org/show_bug.cgi?id=1299274

So maybe a solution is not that far away.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-04 Thread L. David Baron 
On Friday 2016-09-02 11:49 -0700, Wes Kocher wrote:
> I don't believe that's a workable situation. At the moment, the policy is 
> that every new intermittent failure gets a bug filed for the purpose of 
> tracking it. There's talk (and has been for at least a year or two, now) that 
> work will begin on OrangeFactor version 2, where intermittent failures could 
> get tracked within OrangeFactor's own database, and we only file bugs for the 
> failures that get too noisy, but no one has had time to work on that in the 
> last two years, so would need prioritization from higher ups for someone(s) 
> to dedicate some time to work on it.

In the past, a single problem that showed up across multiple tests
would be covered by a single bug, as it should be.

Recently, sheriffing practices have changed so that intermittent
failure bugs that show up in different tests now have a separate
bugzilla bug for each test they occur in.  This causes:

 1. a large number of rarely-occurring bugs -- enough that
 intermittent-orange bugs are a significant fraction of the bugs
 reported, but most of the bugs document only a single failure (in
 many cases, a problem that has also occurred on other tests,
 tracked in different bugs).  This encourages ignoring the bugs and
 just looking at orangefactor for what's interesting/relevant.

 2. spreading out a single problem across >100 bugs [1] leads to the
 severity of that problem being ignored for extended periods of
 time.

-David

[1] https://bugzilla.mozilla.org/showdependencytree.cgi?id=1285531

-- 
턞   L. David Baron http://dbaron.org/   턂
턢   Mozilla  https://www.mozilla.org/   턂
 Before I built a wall I'd ask to know
 What I was walling in or walling out,
 And to whom I was like to give offense.
   - Robert Frost, Mending Wall (1914)


signature.asc
Description: PGP signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Matthew N. 

On 2016-09-02 2:29 AM, Gijs Kruitbosch wrote:

On 02/09/2016 00:15, Matthew N. wrote:

On 2016-09-01 9:24 AM, Gijs Kruitbosch wrote:

On 01/09/2016 16:37, Henrik Skupin wrote:

Do those locations sound good? I have heard at least once that
"firefox_ui" might not be the best choice as folder name, but that's
how
the harness is called, and corresponds to what we have for other
harnesses too.


As I did over IRC, I would like to strongly object to the continued use
of per-test-type subfolders in our test directories. You can already use
a specific mach command per test type, and the tests are listed in
different manifests, *and* there's all the different filename
conventions (browser_, test_html, test_xul, .js) that
further point out what type of test you're looking at. The subfolders
add nothing useful.


As someone who has been adding the directory levels to
toolkit/components/passwordmgr/test/ recently, I disagree with this
since they add a grouping of relevant files making it more obvious which
files go with which test suites.


But in general you don't need to know this? When does it matter and is
it an unknown? You can just pass a filename to ./mach test and it'll do
the right thing.


* refactoring tests and related helper functions (e.g. for e10s). e.g. 
pwmgr_common.js or notification_common.js could belong to any of the 
four suites used by pwmgr. Same for a head.js file.
* when working on a test and the directory doesn't take a super long 
time to run I often want to run everything in the directory for the 
suite of the I'm working on, especially as I'm putting the final touches 
on the test. This is mostly to make sure I'm doing proper cleanup in the 
test.
* knowing which of the ini files to add a new test to (and actually 
noticing the ini files if there are lots of files in the directory).
* when writing a new test I want to look at what helpers relevant to the 
component are available for that suite



* Chrome mochitests, plain mochitests and xpcshell share the same prefix
(test_) so they are interleaved together in directory listings.


Again, not sure why that's a problem. If there's too many files in a
directory, that's a problem, but I'd rather we split them out by
function/subject than by test suite.


See above


* Files that accompany tests have no prefix convention.


Yes, but if that is a problem (which I'm not sure about) it continues to
be a problem when you don't mix test types into one directory.


If you're not mixing then you know that accompanying files are relevant 
to the tests in that directory.



* head.js files have no indication of what suite they're for (i.e. no
prefix)


But there's no need for the files to be called head.js. All of this is
also already true without adding the firefox-ui tests.


Right (for xpcshell) but that's the standard name that is commonly used, 
second is to have head_(name of thing being tested).js which would still 
conflict with another head_(name of thing being tested).js for a 
different suite in the same directory if they're testing the same 
module/component.



* `mach test` doesn't support specifying a flavor of mochitest.


I'm sure that could be added (in fact, I thought there were plans for
this already) - in the meantime, ./mach mochitest works.


I would really like this but it's still a reason to not flatten test 
directories at the moment.



* Changing the subdirectory of my `mach mochitest` command is usually
faster than adding the flavor argument since the path is usually at the
end of the command. Since `mach test` doesn't support the flavor
argument I don't have to remember whether to use the argument or change
the path as I can always change the path when directories are used.


Again, we should just fix ./mach test. But also, this isn't just about
the mach command, but also about editing, and running individual tests,
where the subdirectory "system" forces me to do useless extra typing to
run/edit "browser/browser_foo.js". It gets even worse when you write a
new test and realize halfway through that you need to switch from
mochitest-plain to mochitest-browser because mochitest-plain doesn't
have enough privileges to determine whatever you need to determine.


In the rare cases where you need to move a test between suites I don't 
see this a big deal.



Out of curiosity: if you're not running a single test, why isn't just
running all the mochitests sufficient? Why are you wanting to run a
specific suite but not the others? And is that really the majority case?


My most common experience is with password manager which uses three 
flavors of mochitest (it really should only need 2 but the last "chrome" 
test hasn't been rewritten yet). When I'm developing a new test which 
may involve refactoring shared suite helpers for the directory and I 
already know that my application code change is passing all existing 
test suites I don't want to have to wait for the unrelated mochitest 
flavors to run while developing.

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Andrew McCreight 
On Fri, Sep 2, 2016 at 12:11 PM, Ryan VanderMeulen  wrote:

> On 9/2/2016 2:56 PM, Andrew McCreight wrote:
>
>> In DOM triage, we've just set up our list to not include bugs with the
>> keyword intermittent failure. Ideally, we'd see intermittent failures that
>> have a high rate, but given that sheriffs keep an eye on oranges, they can
>> look for somebody to fix frequent ones as needed.
>>
>
> I guess that explains why trunk's OrangeFactor is pushing 30 these days.


No, that is unrelated. We have only recently started systematically
triaging DOM bugs at all, so it isn't like we stopped triaging intermittent
oranges.


>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

RE: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Wes Kocher 
I don't believe that's a workable situation. At the moment, the policy is that 
every new intermittent failure gets a bug filed for the purpose of tracking it. 
There's talk (and has been for at least a year or two, now) that work will 
begin on OrangeFactor version 2, where intermittent failures could get tracked 
within OrangeFactor's own database, and we only file bugs for the failures that 
get too noisy, but no one has had time to work on that in the last two years, 
so would need prioritization from higher ups for someone(s) to dedicate some 
time to work on it.

KW

From: e...@mozilla.com
Date: Fri, 2 Sep 2016 11:35:26 -0700
Subject: Re: Reorganization of Firefox-UI tests in mozilla-central
To: m...@hskupin.info
CC: dev-platform@lists.mozilla.org; kwie...@gmail.com

Henrik, 

It's not clear to me if this is just moving the location of existing tests, or 
if you're bringing in new tests. 

If it's the later, I'm concerned that we're going to get more intermittents 
filed in Bugzilla, which adds noise to the triage process. 

If you are adding more tests, then would it be possible to back off from filing 
intermittent bugs unless they become problematic, failing tests more than a 
quarter to half the time?

Thanks!

Emma

On Thu, Sep 1, 2016 at 8:37 AM, Henrik Skupin <m...@hskupin.info> wrote:
Hello,



Via bug 1272145 we want to move our existing Firefox-UI tests from

/testing/firefox-ui/ closer to the code which these are testing, so that

the former location only contains the test harness and appropriate unit

tests.



Link to the current set of tests:

https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests



For those of you who haven't heard about these tests yet, they are

basically Marionette tests with an additional layer (firefox-puppeteer)

on top to ease the test creation for ui specific checks. Benefits we

have are restarting and quitting the browser, and running the tests in

any localized build of Firefox.



As listed below you can find the current type of tests and a proposed

location:



* locationbar tests:/browser/base/content/test/urlbar/firefox_ui/

* private browsing tests:

/browser/components/privatebrowsing/test/firefox_ui/

* safe browsing tests:

/browser/components/safebrowsing/content/test/firefox_ui/

* session store tests:  
/browser/components/sessionstore/test/firefox_ui/

* update tests: /toolkit/mozapps/update/tests/firefox_ui/



Do those locations sound good? I have heard at least once that

"firefox_ui" might not be the best choice as folder name, but that's how

the harness is called, and corresponds to what we have for other

harnesses too.



Locations for the following tests are not clear yet:



* l10n tests (shortcuts):   not clear yet (maybe under

/browser/base/content/test/)

* security tests:   not clear yet



L10n related tests mostly cover shortcuts and that the correct command

is invoked to find failures like bug 1173735.



Nearly all of the security tests are running checks against a real

server with various SSL certificates (DV, OV, EV) and protocol versions.

We will have a meeting soon to determine which of those tests are needed

and which ones can be removed. So we might also be able to find the

correct location for the tests.



For now I would like to know if the proposal is fine or if we should go

a completely different route.



Thanks



--

Henrik

___

dev-platform mailing list

dev-platform@lists.mozilla.org

https://lists.mozilla.org/listinfo/dev-platform


  
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Emma Humphries 
Henrik,

It's not clear to me if this is just moving the location of existing tests,
or if you're bringing in new tests.

If it's the later, I'm concerned that we're going to get more intermittents
filed in Bugzilla, which adds noise to the triage process.

If you are adding more tests, then would it be possible to back off from
filing intermittent bugs unless they become problematic, failing tests more
than a quarter to half the time?

Thanks!

Emma

On Thu, Sep 1, 2016 at 8:37 AM, Henrik Skupin  wrote:

> Hello,
>
> Via bug 1272145 we want to move our existing Firefox-UI tests from
> /testing/firefox-ui/ closer to the code which these are testing, so that
> the former location only contains the test harness and appropriate unit
> tests.
>
> Link to the current set of tests:
> https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests
>
> For those of you who haven't heard about these tests yet, they are
> basically Marionette tests with an additional layer (firefox-puppeteer)
> on top to ease the test creation for ui specific checks. Benefits we
> have are restarting and quitting the browser, and running the tests in
> any localized build of Firefox.
>
> As listed below you can find the current type of tests and a proposed
> location:
>
> * locationbar tests:/browser/base/content/test/
> urlbar/firefox_ui/
> * private browsing tests:
> /browser/components/privatebrowsing/test/firefox_ui/
> * safe browsing tests:
> /browser/components/safebrowsing/content/test/firefox_ui/
> * session store tests:  /browser/components/
> sessionstore/test/firefox_ui/
> * update tests: /toolkit/mozapps/update/tests/firefox_ui/
>
> Do those locations sound good? I have heard at least once that
> "firefox_ui" might not be the best choice as folder name, but that's how
> the harness is called, and corresponds to what we have for other
> harnesses too.
>
> Locations for the following tests are not clear yet:
>
> * l10n tests (shortcuts):   not clear yet (maybe under
> /browser/base/content/test/)
> * security tests:   not clear yet
>
> L10n related tests mostly cover shortcuts and that the correct command
> is invoked to find failures like bug 1173735.
>
> Nearly all of the security tests are running checks against a real
> server with various SSL certificates (DV, OV, EV) and protocol versions.
> We will have a meeting soon to determine which of those tests are needed
> and which ones can be removed. So we might also be able to find the
> correct location for the tests.
>
> For now I would like to know if the proposal is fine or if we should go
> a completely different route.
>
> Thanks
>
> --
> Henrik
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Gijs Kruitbosch 

On 02/09/2016 13:49, Henrik Skupin wrote:

Gijs Kruitbosch wrote on 09/02/2016 11:37 AM:

We should be doing the same for firefox-ui tests. Not just to avoid
duplication of files in archives, but because otherwise, if we want to
add new tests somewhere else, we both have to add the manifests to the
build system and then modify this build system python file to make sure
they get included in the test archive. That would be wrong.


No, this is not what we want to do. Anything related to the obj dir
causes a lot of operations and cpu cycles to build those archives (see
bug 1283919 comment 1).


This comment has no information at all about operations or cpu cycles, 
so I don't follow.



With having the manifests in place we can also
run the tests directly from the source tree.


I don't understand what this sentence actually means. We have manifests 
for all our test suites - it doesn't seem to have any direct bearing on 
whether they can be run from the source tree or not (mochitests have 
manifests and can't be run from the source tree today, AIUI). What point 
are you trying to make?



It's true that the
situation with a master manifest is not ideal right now, and especially
for marionette we have it located in
testing/marionette/harness/marionette/test/. Anyway for firefox-ui tests
I want to have testing/firefox-ui/tests/(functional|update)-tests.ini.

From here we can reference everything.


So what's the proposal? This is no longer clear to me. You want to keep 
a single manifest somewhere under testing/, and then have the actual 
test files spread around the tree?


It would seem to make more sense to have individual manifests for the 
different bits. That's certainly more scalable and maintainable (ie have 
it be obvious why and when which tests run, and not have a motley 
collection in a single manifest file) and helps preserve infra resources 
as well, because we make decisions about what builds/tests to run based 
on the paths of files modified. It's not sensible that whenever I would 
want to write a patch and accompanying test, I have to modify files in 
testing/marionette/harness or testing/firefox-ui/tests/ .


It's not clear to me why marionette and firefox-ui don't use the same 
moz.build infrastructure to collect test manifests and thereby test 
files as mochitest/xpcshell/reftests, and why we couldn't use exactly 
that infrastructure to determine what to stick in the test archives as 
far as the tests themselves are concerned. The mixing of information 
storage here (in both directory naming/structure and manifest files) 
only leads to duplication and confusion/bugs.


~ Gijs
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Gijs Kruitbosch wrote on 09/02/2016 01:22 PM:
> AIUI, if we can run tests directly from a checkout, we no longer need 
> the common.tests.zip archive to exist (or at least not to have the tests 
> in them), so the problem is moot, no?

Those files are used by test machines for all of our test suites. That
is so they do not have to clone the full tree. I'm not aware of any
planned changes here. But build people might know more.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Gijs Kruitbosch wrote on 09/02/2016 01:22 PM:

> AIUI, if we can run tests directly from a checkout, we no longer need 
> the common.tests.zip archive to exist (or at least not to have the tests 
> in them), so the problem is moot, no?

I think so. There should not be a need to have to run "mach build
package-tests" before running any test from the source tree. mach itself
takes care about copying necessary bits to the obj dir if necessary.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Gijs Kruitbosch wrote on 09/02/2016 11:37 AM:
> I am not familiar with this bit of our build architecture, but as far as 
> I can tell from a quick look it builds bits of the zipfile off the 
> objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest, 
> where (again, AFAICT) things get installed via manifests.

This is only happening for old test suites but not for more recent ones.
But also for reftests we do no longer have to copy all the files to the
obj dir first. We package them directly from the source tree into the
various *.tests.zip files. But this is all related to directories and
not individual files.

> We should be doing the same for firefox-ui tests. Not just to avoid 
> duplication of files in archives, but because otherwise, if we want to 
> add new tests somewhere else, we both have to add the manifests to the 
> build system and then modify this build system python file to make sure 
> they get included in the test archive. That would be wrong.

No, this is not what we want to do. Anything related to the obj dir
causes a lot of operations and cpu cycles to build those archives (see
bug 1283919 comment 1). With having the manifests in place we can also
run the tests directly from the source tree. It's true that the
situation with a master manifest is not ideal right now, and especially
for marionette we have it located in
testing/marionette/harness/marionette/test/. Anyway for firefox-ui tests
I want to have testing/firefox-ui/tests/(functional|update)-tests.ini.
>From here we can reference everything.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Gijs Kruitbosch 

On 02/09/2016 10:54, James Graham wrote:

On 02/09/16 10:37, Gijs Kruitbosch wrote:

On 02/09/2016 08:08, Henrik Skupin wrote:



The problematic piece here will be the package-tests step which
currently picks complete subfolders. It would mean if we mix-up tests
for firefox-ui-tests and eg. mochitests all would end-up twice in the
common.tests.zip archive. If we want to get away from using subfolders
we would have to improve the test archiver
(https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py)


first to not only collect the directories of referenced manifests, but
also only pick those tests which are referenced and leave all others
behind. This would apply to all test suites currently covered by this
mozbuild action.


I am not familiar with this bit of our build architecture, but as far as
I can tell from a quick look it builds bits of the zipfile off the
objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest,
where (again, AFAICT) things get installed via manifests.

We should be doing the same for firefox-ui tests. Not just to avoid
duplication of files in archives, but because otherwise, if we want to
add new tests somewhere else, we both have to add the manifests to the
build system and then modify this build system python file to make sure
they get included in the test archive. That would be wrong.


In the medium term we are trying to move away from requiring the
package-tests step in favour of being able to run tests directly from a
checkout. Therefore I suggest we avoid adding unnecessary dependencies
on the objdir.


AIUI, if we can run tests directly from a checkout, we no longer need 
the common.tests.zip archive to exist (or at least not to have the tests 
in them), so the problem is moot, no?


~ Gijs
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread James Graham 

On 02/09/16 10:37, Gijs Kruitbosch wrote:

On 02/09/2016 08:08, Henrik Skupin wrote:



The problematic piece here will be the package-tests step which
currently picks complete subfolders. It would mean if we mix-up tests
for firefox-ui-tests and eg. mochitests all would end-up twice in the
common.tests.zip archive. If we want to get away from using subfolders
we would have to improve the test archiver
(https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py)

first to not only collect the directories of referenced manifests, but
also only pick those tests which are referenced and leave all others
behind. This would apply to all test suites currently covered by this
mozbuild action.


I am not familiar with this bit of our build architecture, but as far as
I can tell from a quick look it builds bits of the zipfile off the
objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest,
where (again, AFAICT) things get installed via manifests.

We should be doing the same for firefox-ui tests. Not just to avoid
duplication of files in archives, but because otherwise, if we want to
add new tests somewhere else, we both have to add the manifests to the
build system and then modify this build system python file to make sure
they get included in the test archive. That would be wrong.


In the medium term we are trying to move away from requiring the 
package-tests step in favour of being able to run tests directly from a 
checkout. Therefore I suggest we avoid adding unnecessary dependencies 
on the objdir.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Gijs Kruitbosch 

On 02/09/2016 08:08, Henrik Skupin wrote:

Gijs Kruitbosch wrote on 09/01/2016 06:24 PM:


As I did over IRC, I would like to strongly object to the continued use
of per-test-type subfolders in our test directories. You can already use
a specific mach command per test type, and the tests are listed in
different manifests, *and* there's all the different filename
conventions (browser_, test_html, test_xul, .js) that
further point out what type of test you're looking at. The subfolders
add nothing useful.


The problematic piece here will be the package-tests step which
currently picks complete subfolders. It would mean if we mix-up tests
for firefox-ui-tests and eg. mochitests all would end-up twice in the
common.tests.zip archive. If we want to get away from using subfolders
we would have to improve the test archiver
(https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py)
first to not only collect the directories of referenced manifests, but
also only pick those tests which are referenced and leave all others
behind. This would apply to all test suites currently covered by this
mozbuild action.


I am not familiar with this bit of our build architecture, but as far as 
I can tell from a quick look it builds bits of the zipfile off the 
objdir. So it collects mochitests from $OBJDIR/_tests/testing/mochitest, 
where (again, AFAICT) things get installed via manifests.


We should be doing the same for firefox-ui tests. Not just to avoid 
duplication of files in archives, but because otherwise, if we want to 
add new tests somewhere else, we both have to add the manifests to the 
build system and then modify this build system python file to make sure 
they get included in the test archive. That would be wrong.


In case I'm wrong (always possible!): As I noted in my other email, 
there are other cases where we mix tests, so if the archiver is really 
that dumb then maybe we should fix the archiver. :-)


~ Gijs
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Gijs Kruitbosch 

On 02/09/2016 00:15, Matthew N. wrote:

On 2016-09-01 9:24 AM, Gijs Kruitbosch wrote:

On 01/09/2016 16:37, Henrik Skupin wrote:

Do those locations sound good? I have heard at least once that
"firefox_ui" might not be the best choice as folder name, but that's how
the harness is called, and corresponds to what we have for other
harnesses too.


As I did over IRC, I would like to strongly object to the continued use
of per-test-type subfolders in our test directories. You can already use
a specific mach command per test type, and the tests are listed in
different manifests, *and* there's all the different filename
conventions (browser_, test_html, test_xul, .js) that
further point out what type of test you're looking at. The subfolders
add nothing useful.


As someone who has been adding the directory levels to
toolkit/components/passwordmgr/test/ recently, I disagree with this
since they add a grouping of relevant files making it more obvious which
files go with which test suites.


But in general you don't need to know this? When does it matter and is 
it an unknown? You can just pass a filename to ./mach test and it'll do 
the right thing.



* Chrome mochitests, plain mochitests and xpcshell share the same prefix
(test_) so they are interleaved together in directory listings.


Again, not sure why that's a problem. If there's too many files in a 
directory, that's a problem, but I'd rather we split them out by 
function/subject than by test suite.



* Files that accompany tests have no prefix convention.


Yes, but if that is a problem (which I'm not sure about) it continues to 
be a problem when you don't mix test types into one directory.



* head.js files have no indication of what suite they're for (i.e. no
prefix)


But there's no need for the files to be called head.js. All of this is 
also already true without adding the firefox-ui tests.



* `mach test` doesn't support specifying a flavor of mochitest.


I'm sure that could be added (in fact, I thought there were plans for 
this already) - in the meantime, ./mach mochitest works.



* Changing the subdirectory of my `mach mochitest` command is usually
faster than adding the flavor argument since the path is usually at the
end of the command. Since `mach test` doesn't support the flavor
argument I don't have to remember whether to use the argument or change
the path as I can always change the path when directories are used.


Again, we should just fix ./mach test. But also, this isn't just about 
the mach command, but also about editing, and running individual tests, 
where the subdirectory "system" forces me to do useless extra typing to 
run/edit "browser/browser_foo.js". It gets even worse when you write a 
new test and realize halfway through that you need to switch from 
mochitest-plain to mochitest-browser because mochitest-plain doesn't 
have enough privileges to determine whatever you need to determine.


Out of curiosity: if you're not running a single test, why isn't just 
running all the mochitests sufficient? Why are you wanting to run a 
specific suite but not the others? And is that really the majority case?


Personally, I would prefer to have per-subject directories, and to have 
"mach test" and friends allow specifying particular suites if we're 
hunting down other-test-dependent intermittents in a particular suite, 
or something. The most common situation right now is that you want to 
run either a single test or want to ensure that the code changes you 
just made didn't break any of the relevant tests. In the latter case 
"relevant tests" is related to *what the tests are testing*, not to what 
framework the tests use. So ultimately, the grouping by test framework 
makes it hard to run only the tests you care about. I know that some 
test frameworks (last I checked, not including marionette or fx-ui) 
support running tests by "tags", but most tests aren't annotated with 
them, and in any case I feel like it ends up being a hack for the fact 
that our directory layout is not well-suited for that usecase.



* xpcshell and browser-chrome both use "head.js" as the filename for
helpers though they run in very different contexts.


xpcshell lets you use whatever you like, I think, with the head/tail 
annotations in the .ini file, for cases where this would actually be 
confusing.



For a new contributor, having each suite in their own directory is much
less confusing/overwhelming for the above reasons.


I will buy that it is a more obvious signal (than filename and content) 
if you don't know much about our testing infrastructure and need to know 
what kind of test you're looking at while you're looking at the 
directory view. But again, that doesn't feel like it's the most common 
case. Wanting to write a new test (where you'd need advice anyway, 
especially given that our naming (chrome/browser-chrome) is so 
confusing) or editing a specific existing test (e.g. one that fails on 
try) don't actually fall in this

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Matthew N. wrote on 09/02/2016 01:15 AM:

>>> /browser/components/sessionstore/test/firefox_ui/
>>> * update tests:/toolkit/mozapps/update/tests/firefox_ui/
> 
> Is there a plan to merge those with 
> /toolkit/mozapps/update/tests/marionette? It seems unusual to use both 
> when this new test suite is basically a layer on top of Marionette as 
> you said.

Those marionette tests have been created specifically for B2G:
https://dxr.mozilla.org/mozilla-central/source/testing/marionette/harness/marionette/tests/update-tests.ini

But the firefox-ui harness can only be used for Firefox. Reason is that
we have our own testcase classes and several enhancements to the
harness. An integration of those into Marionette base was denied to not
make Marionette that bloated. As result we have to keep our layering for
now.

> * Chrome mochitests, plain mochitests and xpcshell share the same prefix 
> (test_) so they are interleaved together in directory listings.

And exactly this makes it hard to only collect the relevant files
(tests, support files) via test_archive.py as mentioned in my other email.

> "firefox_ui", "ui", and "integration" all overlap with what 
> mochitest-browser-chrome is about IMO and I think naming the suite 
> "Firefox-UI" was confusing from the beginning. If I was a new 
> contributor working on a UI feature and decided I wanted to write tests, 
> I wouldn't want to be misled into thinking I should write a "Firefox-UI" 
> test when mochitest-browser-chrome is actually the desired suite. I 
> would suggest "puppeteer" or "marionette" for directory names to avoid 
> confusion.

As of now we only have puppeteer available for Firefox. And it will not
be a hard requirement to use hopefully soon once we decoupled the
harness from it. Also we just got Marionette support for Fennec. It
means beside testing/puppeteer/firefox it might be good to get shared
helper code for Fennec specific tests into testing/puppeteer/fennec.

Using marionette here would also be confusing because those tests won't
work with the marionette command.

Maybe at the end we should try again to get our testcase classes and
harness extensions into Marionette again. But depending on the future
that we may want to use Selenium for our tests directly, I would wait
and not spend my time with such a refactoring.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-02 Thread Henrik Skupin 
Gijs Kruitbosch wrote on 09/01/2016 06:24 PM:

> As I did over IRC, I would like to strongly object to the continued use 
> of per-test-type subfolders in our test directories. You can already use 
> a specific mach command per test type, and the tests are listed in 
> different manifests, *and* there's all the different filename 
> conventions (browser_, test_html, test_xul, .js) that 
> further point out what type of test you're looking at. The subfolders 
> add nothing useful.

The problematic piece here will be the package-tests step which
currently picks complete subfolders. It would mean if we mix-up tests
for firefox-ui-tests and eg. mochitests all would end-up twice in the
common.tests.zip archive. If we want to get away from using subfolders
we would have to improve the test archiver
(https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py)
first to not only collect the directories of referenced manifests, but
also only pick those tests which are referenced and leave all others
behind. This would apply to all test suites currently covered by this
mozbuild action.

> Finally, "firefox_ui" (as well as "ui") as a name for a directory is 
> going to cause all kinds of confusion for people exploring the repo 
> without detailed knowledge of what's going on. Additionally, it's not 
> like most of the mochitest-browser tests aren't tests of the Firefox 
> UI... If we absolutely must have some kind of subdirectory because of 
> reasons I have yet to hear, I think "integration" would be a better 
> choice of name as far as subdirs of "test" go (as juxtaposed to "unit" 
> for our xpcshell tests).

I would totally be fine with integration as name, unless we don't hit
the above mentioned problem that tests for other harnesses end-up in the
same folder.

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-01 Thread Matthew N. 

On 2016-09-01 9:24 AM, Gijs Kruitbosch wrote:

(CC'ing firefox-dev which doesn't seem to have had the original email
though it should have done, please follow up to m.d.platform.)

On 01/09/2016 16:37, Henrik Skupin wrote:

* locationbar tests:/browser/base/content/test/urlbar/firefox_ui/
* private browsing tests:
/browser/components/privatebrowsing/test/firefox_ui/
* safe browsing tests:
/browser/components/safebrowsing/content/test/firefox_ui/
* session store tests:
/browser/components/sessionstore/test/firefox_ui/
* update tests:/toolkit/mozapps/update/tests/firefox_ui/


Is there a plan to merge those with 
/toolkit/mozapps/update/tests/marionette? It seems unusual to use both 
when this new test suite is basically a layer on top of Marionette as 
you said.



Do those locations sound good? I have heard at least once that
"firefox_ui" might not be the best choice as folder name, but that's how
the harness is called, and corresponds to what we have for other
harnesses too.


As I did over IRC, I would like to strongly object to the continued use
of per-test-type subfolders in our test directories. You can already use
a specific mach command per test type, and the tests are listed in
different manifests, *and* there's all the different filename
conventions (browser_, test_html, test_xul, .js) that
further point out what type of test you're looking at. The subfolders
add nothing useful.


As someone who has been adding the directory levels to 
toolkit/components/passwordmgr/test/ recently, I disagree with this 
since they add a grouping of relevant files making it more obvious which 
files go with which test suites.


* Chrome mochitests, plain mochitests and xpcshell share the same prefix 
(test_) so they are interleaved together in directory listings.

* Files that accompany tests have no prefix convention.
* head.js files have no indication of what suite they're for (i.e. no 
prefix)

* `mach test` doesn't support specifying a flavor of mochitest.
* Changing the subdirectory of my `mach mochitest` command is usually 
faster than adding the flavor argument since the path is usually at the 
end of the command. Since `mach test` doesn't support the flavor 
argument I don't have to remember whether to use the argument or change 
the path as I can always change the path when directories are used.
* xpcshell and browser-chrome both use "head.js" as the filename for 
helpers though they run in very different contexts.


For a new contributor, having each suite in their own directory is much 
less confusing/overwhelming for the above reasons.


Password Manager may be special in that it's using four different suites 
so I'm not suggesting that every component needs to put their tests in a 
subdirectory named after the test suite but I don't think we should be 
dumping tests of different suites in one directory unless the 
distinction between files would be clear to a newcomer.



Furthermore, only the toolkit case is currently meaningfully split out
into subdirs. The sessionstore test/ dir has a subdir (but also has a
bundle of tests directly in that dir)


Sure, but there isn't a mix of multiple suites in one directory here.


, and the privatebrowsing one has
no leafs and only a subdir ("browser"). None of the others have any
subdirs at all.


That just seems like good planning for the future when other suites are 
used for that code. The paths of the tests would need to change.



Getting back to the toolkit case, the subdirs are
actually confusing because only some of the subdirs have tests (as a
counterexample, "data" just has random helper files) and the root
testing dir also has .cpp files in it (I guess for gtests?).


Nobody is saying that directories under a "test"/"tests" directory 
should only include test file themselves. Having files to support tests 
in organized directories doesn't seem like a problem to me.



IOW, in the general case, I think that in most of those directories
there's no precedent to do what you propose.


Having the new subdirectory in these specific cases may not fit but I 
disagree that it's the wrong approach in general.



Finally, "firefox_ui" (as well as "ui") as a name for a directory is
going to cause all kinds of confusion for people exploring the repo
without detailed knowledge of what's going on. Additionally, it's not
like most of the mochitest-browser tests aren't tests of the Firefox
UI... If we absolutely must have some kind of subdirectory because of
reasons I have yet to hear, I think "integration" would be a better
choice of name as far as subdirs of "test" go (as juxtaposed to "unit"
for our xpcshell tests).


"firefox_ui", "ui", and "integration" all overlap with what 
mochitest-browser-chrome is about IMO and I think naming the suite 
"Firefox-UI" was confusing from the beginning. If I was a new 
contributor working on a UI feature and decided I wanted to write tests, 
I wouldn't want to be misled into thinking I should write

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-01 Thread Andreas Tolfsen 
Henrik Skupin  writes:

> Do those locations sound good? I have heard at least once that
> "firefox_ui" might not be the best choice as folder name, but that's
> how the harness is called, and corresponds to what we have for other
> harnesses too.

I would suggest s/firefox_ui/ui/g, or to eliminate the need for a
sub-directory overall if it is obvious from the file names what they do.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Reorganization of Firefox-UI tests in mozilla-central

2016-09-01 Thread Gijs Kruitbosch 
(CC'ing firefox-dev which doesn't seem to have had the original email 
though it should have done, please follow up to m.d.platform.)


On 01/09/2016 16:37, Henrik Skupin wrote:

* locationbar tests:/browser/base/content/test/urlbar/firefox_ui/
* private browsing tests:
/browser/components/privatebrowsing/test/firefox_ui/
* safe browsing tests:  
/browser/components/safebrowsing/content/test/firefox_ui/
* session store tests:  
/browser/components/sessionstore/test/firefox_ui/
* update tests: /toolkit/mozapps/update/tests/firefox_ui/

Do those locations sound good? I have heard at least once that
"firefox_ui" might not be the best choice as folder name, but that's how
the harness is called, and corresponds to what we have for other
harnesses too.


As I did over IRC, I would like to strongly object to the continued use 
of per-test-type subfolders in our test directories. You can already use 
a specific mach command per test type, and the tests are listed in 
different manifests, *and* there's all the different filename 
conventions (browser_, test_html, test_xul, .js) that 
further point out what type of test you're looking at. The subfolders 
add nothing useful.


Furthermore, only the toolkit case is currently meaningfully split out 
into subdirs. The sessionstore test/ dir has a subdir (but also has a 
bundle of tests directly in that dir), and the privatebrowsing one has 
no leafs and only a subdir ("browser"). None of the others have any 
subdirs at all. Getting back to the toolkit case, the subdirs are 
actually confusing because only some of the subdirs have tests (as a 
counterexample, "data" just has random helper files) and the root 
testing dir also has .cpp files in it (I guess for gtests?).


IOW, in the general case, I think that in most of those directories 
there's no precedent to do what you propose.


Finally, "firefox_ui" (as well as "ui") as a name for a directory is 
going to cause all kinds of confusion for people exploring the repo 
without detailed knowledge of what's going on. Additionally, it's not 
like most of the mochitest-browser tests aren't tests of the Firefox 
UI... If we absolutely must have some kind of subdirectory because of 
reasons I have yet to hear, I think "integration" would be a better 
choice of name as far as subdirs of "test" go (as juxtaposed to "unit" 
for our xpcshell tests).


~ Gijs

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Reorganization of Firefox-UI tests in mozilla-central

2016-09-01 Thread Henrik Skupin 
Hello,

Via bug 1272145 we want to move our existing Firefox-UI tests from
/testing/firefox-ui/ closer to the code which these are testing, so that
the former location only contains the test harness and appropriate unit
tests.

Link to the current set of tests:
https://dxr.mozilla.org/mozilla-central/source/testing/firefox-ui/tests

For those of you who haven't heard about these tests yet, they are
basically Marionette tests with an additional layer (firefox-puppeteer)
on top to ease the test creation for ui specific checks. Benefits we
have are restarting and quitting the browser, and running the tests in
any localized build of Firefox.

As listed below you can find the current type of tests and a proposed
location:

* locationbar tests:/browser/base/content/test/urlbar/firefox_ui/
* private browsing tests:
/browser/components/privatebrowsing/test/firefox_ui/
* safe browsing tests:  
/browser/components/safebrowsing/content/test/firefox_ui/
* session store tests:  
/browser/components/sessionstore/test/firefox_ui/
* update tests: /toolkit/mozapps/update/tests/firefox_ui/

Do those locations sound good? I have heard at least once that
"firefox_ui" might not be the best choice as folder name, but that's how
the harness is called, and corresponds to what we have for other
harnesses too.

Locations for the following tests are not clear yet:

* l10n tests (shortcuts):   not clear yet (maybe under
/browser/base/content/test/)
* security tests:   not clear yet

L10n related tests mostly cover shortcuts and that the correct command
is invoked to find failures like bug 1173735.

Nearly all of the security tests are running checks against a real
server with various SSL certificates (DV, OV, EV) and protocol versions.
We will have a meeting soon to determine which of those tests are needed
and which ones can be removed. So we might also be able to find the
correct location for the tests.

For now I would like to know if the proposal is fine or if we should go
a completely different route.

Thanks

-- 
Henrik
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform