Summary: architecture, design, hosting, and APIs for the WMF automated
browser test project are in place, useful testing is happening now, and we
are beginning to build out the test suite for both desktop and mobile
browsers.
About a year ago the WMF created a mandate to implement browser-level
automated testing. I did the preliminary work, and in October 2012 Željko
Filipin joined WMF to be the technical lead for this project. This is an
excellent time to describe the current status of the project and to discuss
where it goes next.
We are using the Ruby programming language for this project because the
tools available for this sort of testing are more sophisticated and better
supported in Ruby than in other languages. While the entire testing
"stack" is remarkable, three aspects are particularly worth noting:
A project called webdriver is in the process of being approved by the W3C
as an internet standard for browser automation. Selenium is the
client-side implementation of the webdriver standard. The Ruby
implementation of selenium-webdriver is built and tested automatically
according to the webdriver standard. As the webdriver standard changes, so
selenium-webdriver in Ruby changes also, automatically. [1]
But selenium is a low-level API for driving browsers, essentially a limited
set of tools to build with. Ruby is unique in also having a standard
high-level API. Today the watir-webdriver project provides a powerful,
well-designed, well-supported high-level API for selenium-webdriver based
on the HTML5 standard.[2]
Cucumber is a high-level API for doing Acceptance Test Driven Development,
or ATDD. Cucumber allows us to create executable specifications in the
form of Given/When/Then statements. For example, here is a working
Cucumber specification for a login test:
Scenario: Log in without entering credentials
Given I am at test2 Log in page
When I log in without entering credentials
Then Log in page should open
And feedback should be You have not specified a valid username.
Cucumber is valuable because it creates a communication channel that does
not usually exist for software development projects. For one thing, casual
users and the greater community can read the Cucumber specifications[3] and
understand what is being tested (and what is failing!). In a similar way,
Cucumber allows non-programming members of the community to contribute to
the test automation project by submitting test scenarios to be automated.
We have created a test backlog page on mediawiki.org[4] for such
scenarios.
The tests themselves are controlled by an instance of the Jenkins
continuous integration server provided by a hosting service called
Cloudbees. The tests are run on hosts managed by Sauce Labs, and are
currently running on Firefox, Chrome, IE6/7/8/9/10, android, iphone, and
ipad, using test2wiki, beta labs, and the mobile web site as targets. As
of today, the status of all the tests are available publicly[5].
Using Cloudbees to host browser tests provides a number of advantages.
Cloudbees provides free and seamless integration with Sauce Labs and all
of their tools. With Cloudbees we have fine control over versions of the
software we use, and we can easily update versions of Ruby gems and Ruby
itself at will. It is possible to publish information between the
Cloudbees Jenkins instance and the WMF Jenkins also, although we have not
yet done this. Cloudbees is economical. Finally, although managed by a
commercial company, every aspect of the integrated service is open source
software. There is little risk in using Cloudbees. The source code for
the tests themselves resides in WMF's gerrit system[6]. If Cloudbees were
to disappear tomorrow, we could re-create the services it provides on WMF
hosts or other hosts, although at some expense both financially and in
terms of maintenance.
Željko and I spent significant time working on the design and
implementation of our first few tests. After that tweaking was essentially
done, the tests we have came fully online November 26th. Since that time
even our few existing tests have identified the following issues:
UploadWizard broken for IE9 only:
https://bugzilla.wikimedia.org/show_bug.cgi?id=42512
UploadWizard mistakenly reverted to broken version on test2, in deployment
pipeline. (no BZ ticket, fixed on the fly by Ryan Kaldari)
AFTv5 user experience issue with IE9:
https://bugzilla.wikimedia.org/show_bug.cgi?id=42551
AFTv5 user experience issue with all IEs, and later breakage in the same
code: https://bugzilla.wikimedia.org/show_bug.cgi?id=42517
The tests have also exposed important issues with our test environments
themselves: https://bugzilla.wikimedia.org/show_bug.cgi?id=42736
And seen just today, UW broken for IE6/IE7:
https://bugzilla.wikimedia.org/show_bug.cgi?id=4292
This week I will be publishing detailed technical design and architecture
documentation for those who might like to contribute beyond simply adding
to the backlog of tests to be implemented. It is worth noting that browser
test automation projects are expensive and often fail, and even WMF has at
least one example of such a failure in its history. In my experience, such
failures are almost always caused by poor design. Željko and I have been
specialists in browser test automation with open source tools since such
tools came into existence, and we intend for this project to continue to be
a world class reference implementation of such a project as it grows.
We will be presenting a walkthrough of the project this Thursday[7].
[1]
https://speakerdeck.com/jarib/automating-130-browser-platform-and-language-combinations-without-going-insane
[2] http://rdoc.info/gems/watir-webdriver/#API_docs
[3] a sophisticated Cucumber test specification in use right now:
https://github.com/wikimedia/qa-browsertests/blob/master/features/uploadwizard.feature
[4] http://www.mediawiki.org/wiki/Qa/test_backlog
[5] https://wmf.ci.cloudbees.com/ NOTE: current Firefox failures are due to
a bug in selenium to be fixed in next release; IE6/IE7 failure due to
https://bugzilla.wikimedia.org/show_bug.cgi?id=42922 (new behavior today),
IE6 failure because IE6 is not supported for AFTv5 those test should be
turned off.
[6] https://gerrit.wikimedia.org/r/#/admin/projects/qa/browsertests also
https://github.com/wikimedia/qa-browsertests
[7] https://www.mediawiki.org/wiki/Meetings/2012-12-13
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
