I attended a W3C Testing Strategy meeting yesterday and the delta between our
test frameworks and testharness.js was discussed. The W3C is very interested in
getting our tests into their harness, and I shared some of the shortcomings
that I gathered from Jonas:
* It requires that all code is wrapped in: test(function() { <...> })
* For asynchronous testing, you'll need the following boiler plate:
obj = async_test(); obj.step(function() { <...> }); obj.done();
* In some cases gets even more complicated and needs additional obj.step_func
calls.
* Exceptions thrown outside of a test/test_step function doesn't cause test
failure
* Test functions have much longer name "assert_equals" vs. "is" and
"assert_true" vs. "ok"
* Can't run server-side code.
I agreed to participate in a task force to solve the issues related to bringing
our tests to W3C (and bringing W3C tests downstream to us.) The W3C is willing
to make changes to their test infrastructure to accommodate our tests. Let's
make our requirements as crisp as possible: Should we be pushing for Marionette
(vs. mochi) as the basis for the W3C framework changes? What changes do we need
on our side to get interop between W3C tests and our frameworks and enable
synching tests to/from our trunk and the W3C's?
--Jet
----- Original Message -----
From: "David Burns" <dbu...@mozilla.com>
To: "Jonas Sicking" <jo...@sicking.cc>
Cc: "dev-platform" <dev-platform@lists.mozilla.org>
Sent: Wednesday, January 30, 2013 6:11:05 AM
Subject: Re: Bringing Marionette and Mochitest closer
I am all for moving towards Marionette being the basis of all
frameworks. It will allow us to share tests between other vendors
because Marionette is based on the W3C Browser Automation Spec[1].
One of the things that this brings up is how we can share between other
vendors. Testharness.js has become the W3 in browser standard so if we
are going for interop between vendors then using testharness.js is what
we will need to look at that or look at pure Marionette calls into the
browser if we need to get out of the sandbox.
One thing to note, and might not be obvious from Marionette
documentation, is that it is easy to switch between Browser Chrome and
Browser Content and execute commands (including pure JS).
Opera use their equivalent of Marionette called OperaDriver[3] to load
the browser and set the browser capabilities and visit a server that
loads the in-browser test and runs testharness.js or calling into Chrome
since it accesses the browser via its Scope Protocol. If I remember
correctly, Chromium is moving to that model too but through their remote
debugger like us.
I have also put some in line messages below for more specific things.
If you want to discuss this further via vidyo/skype/irc let me know. I
am keep to fix any potential shortcomings in Marionette.
David Burns
AutomatedTester
[1] https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html
[2] https://github.com/mozilla/mozbase/tree/master/mozprofile
[3] https://github.com/operasoftware/operadriver
[4] [2] https://github.com/mozilla/mozbase/tree/master/mozhttpd
[5] https://wiki.mozilla.org/Auto-tools/Projects/Mozbase
On 30/01/2013 07:13, Jonas Sicking wrote:
> We'd also need to improve the web server which is started by
> marionette so that we can satisfy some of the use cases that we're
> currently using .sjs files to solve in mochitest.
I am sure that we update MozHTTPd[4] to have a similar feature, might be
exact and might not be trivial to do but might be worth the effort at
for a later stage. MozHTTPd is part of MozBase[5], the ATeam project to
standardise tools we use in testing.
> Instead we suggest that we use a simpler python based solution.
Marionette's python client will be great at this. We are already using
this extensively.
> This means that we can't immediately migrate the existing body of
> mochitests to the new http server. But we only have a total of 184
> .sjs files, so it seems like something that should be doable
> eventually.
>
> So for now this means that we'll have to set up a new test harness.
> But I'm personally hoping that we can eventually let this harness
> cover also our /mochitests which would be an all-around win.
So a quick win for this sounds like Marionette + (MozProfile[2] +
Special Powers) to get what we want but there might be more to it.
> Would very much like to hear feedback on these plans.
>
> Something else we talked about very briefly was to improve the test
> environment which we use for running mochitests on B2G such that we
> enable the same "adjust test, press reload" flow that is commonly used
> on desktop when writing mochitests.
Since there is a python package we can easily use the REPL for python to
do things but I don't see how Marionette would stop this. Marionette
just "drives" the browser. If you are just wanting to load a page then
Marionette's navigate() method would just load that.
> This will hopefully both help with
> getting more tests authored for B2G-specific features (tests which can
> hopefully be migrated to cross-platform as the features become
> cross-platform), and help with getting the current body of mochitests
> running on B2G.
>
> / Jonas
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
