Re: firefox-ui-tests are now in mozilla-central with TaskCluster support
Matthew N. wrote on 03/01/2016 06:18 PM: > Could you give an overview of what is tested by this suite and how it > differs from mochitest-browser-chrome? It seems one difference is that > some(?) tests run on non-en-US. So Andrew already told a lot, and just to emphasize here we really do not want to work with any JS globals etc when navigating through the ui but always use ui elements directly. The reason is that we also run our tests with a couple of other locales and for each of those the command, access keys and others can differ. The before mentioned firefox-puppeteer library is a helper to make test creation even more easier. So you don't have to remember the real DOM id, label, anon attribute or what-ever when accessing such elements. We have a nice hierarchy like: self.browser.navbar.back_button.click() The number of tests are still low given that we haven't had the time to port more tests from mozmill. That means as of now we have converted only the most important tests including remote security checks for various protocols and some locationbar and privatebrowsing tests. We would like to get more tests added especially those which would require a restart of Firefox, or those for testing shortcuts etc across locales. So we hope that you all might want to create one in case you have a need for. > Could you also add a description to > https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing? Yes I will and this is actually one of my remaining goals this quarter. The work for that will be tracked on bug 1237552. In case you want to know more about the code right now, you can find everything here: http://mxr.mozilla.org/mozilla-central/source/testing/puppeteer/firefox/ http://mxr.mozilla.org/mozilla-central/source/testing/firefox-ui/ Let me know if there are still open questions. -- Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: firefox-ui-tests are now in mozilla-central with TaskCluster support
They're more like end-to-end tests than browser-chrome. E.g instead of calling Gecko APIs, they'll send a mouse event to a button, send key events to a textbox, etc. So they'll theoretically perform actions closer to what a user might do. Since they're written in python, they can also do things like restart the browser within the test (for e.g testing updates). It uses a library called Firefox Puppeteer that's built on top of marionette-client: http://firefox-puppeteer.readthedocs.org/en/latest/ (which lives in testing/puppeteer despite what the docs say) Henrik can probably go into more details on what is tested. -Andrew On 01/03/16 12:18 PM, Matthew N. wrote: CC: firefox-dev On 2016-03-01 5:17 AM, Henrik Skupin wrote: Over the last years the formerly known Mozmill tests and now Firefox ui tests have been located in their own repository. That meant that we always got regressions due to changes developers have been made in Firefox. Hunting down those regressions and fixing them was always a time consuming job. Beside all that we were also responsible for merging our branches accordingly to our 6 week cycle. Hi Henrik, thanks for sharing. Could you give an overview of what is tested by this suite and how it differs from mochitest-browser-chrome? It seems one difference is that some(?) tests run on non-en-US. Could you also add a description to https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing? Thanks, Matthew N. (:MattN) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: firefox-ui-tests are now in mozilla-central with TaskCluster support
CC: firefox-dev On 2016-03-01 5:17 AM, Henrik Skupin wrote: Over the last years the formerly known Mozmill tests and now Firefox ui tests have been located in their own repository. That meant that we always got regressions due to changes developers have been made in Firefox. Hunting down those regressions and fixing them was always a time consuming job. Beside all that we were also responsible for merging our branches accordingly to our 6 week cycle. Hi Henrik, thanks for sharing. Could you give an overview of what is tested by this suite and how it differs from mochitest-browser-chrome? It seems one difference is that some(?) tests run on non-en-US. Could you also add a description to https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing? Thanks, Matthew N. (:MattN) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
firefox-ui-tests are now in mozilla-central with TaskCluster support
Over the last years the formerly known Mozmill tests and now Firefox ui tests have been located in their own repository. That meant that we always got regressions due to changes developers have been made in Firefox. Hunting down those regressions and fixing them was always a time consuming job. Beside all that we were also responsible for merging our branches accordingly to our 6 week cycle. To finally get rid of that I was working all the last weeks to make the following changes: * Added mozharness scripts for executing firefox-ui-tests and enhanced base mozharness modules to cope with various other feature requirements we had. * Moved our tests into mozilla-central under testing/firefox-ui/ and updated test packaging. Backports happened down to Firefox 45.0 so that we will benefit from the upcoming ESR release too. * Updated mozmill-ci (which now runs our firefox-ui-tests) to use test packages across all supported branches down to current Firefox 45.0ESR. With that we can finally get rid of our Github repository. It turned out to not be an easy process given that we do not only run our tests against en-US but also a couple of other locales for nightly and release builds. So a couple of blockers had to be fixed side-by-side in packaging, build notifications via Mozilla Pulse etc. * Added support for TaskCluster to run our tests once the build has been finished. Similar to other tests this only works for linux64 debug right now. Test results can be found on Treeherder in the Fxfn group. Keep in mind that those are still Tier-3, so you will have to make those tests visible. If you have to make changes to our tests and want to push to try, simply use '-u firefox-ui-functional,firefox-ui-functional-e10s' in the try comment (trychooser will be updated soon). With all the above close to be done now, I'm starting to write documentation. This will enable us to push results as Tier-2, and also have a base set for contributors to get started hacking on our tests. If you have questions please ask or join our #automation channel on IRC. Cheers, Henrik ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
