A few of us got together last week and had a quick brainstorming session about how to leverage the combined power from Mochitest and Marionette better. The following issues were raised:
* Marionette has support for a http server, but one that is much less feature-full than the one used by Mochitest. * Mochitest is great as long as you are only needing to interact with things that are accessible from inside the browser sandbox. But as soon as you need to interact with browser UI, such as security dialogs, or other processes, it's often impossible to use mochitest. Especially if you want your test to work across desktop/android/b2g. * Mochitest relies on SpecialPowers which is unlikely to ever become available cross-browser. So unlikely that we'll be able to submit tests that use SpecialPowers to W3C. * Marionette requires a lot more boilerplate in order to run JS code in the context of a webpage. * Unclear where the line goes for when to write mochitest vs. marionette tests. Something that sounds like it could solve, or at least alleviate a lot of these problems is if we were able to start a mochitest-like test driver from marionette. But set up marionette such that it listened for commands from this test driver and then used those commands to interact with browser UI or other processes (such as app processes). We'd also need to improve the web server which is started by marionette so that we can satisfy some of the use cases that we're currently using .sjs files to solve in mochitest. In particular things like responses which are generated programmatically, state-full responses, and asynchronous responses. We could technically reuse the existing http server that we are using in mochitest for this. However there are a couple of disadvantages with this: * It's unlikely that we can standardize on that server. I.e. it's unlikely that we can submit tests which rely on .sjs files to W3c. If for no other reason, this because .sjs relies on a fair number of XPCOM-isms * The fact that we are using gecko as the http server means a whole lot of complications on mobile. In short, we currently need gecko both compiled for android, which is the version of gecko being tested, and compiled for a desktop platform, which acts as host of the web server. This is currently a drain on our infrastructure and means that we're running tests slower than we otherwise could. Instead we suggest that we use a simpler python based solution. This means that we can't immediately migrate the existing body of mochitests to the new http server. But we only have a total of 184 .sjs files, so it seems like something that should be doable eventually. So for now this means that we'll have to set up a new test harness. But I'm personally hoping that we can eventually let this harness cover also our mochitests which would be an all-around win. Would very much like to hear feedback on these plans. Something else we talked about very briefly was to improve the test environment which we use for running mochitests on B2G such that we enable the same "adjust test, press reload" flow that is commonly used on desktop when writing mochitests. This will hopefully both help with getting more tests authored for B2G-specific features (tests which can hopefully be migrated to cross-platform as the features become cross-platform), and help with getting the current body of mochitests running on B2G. / Jonas _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
