Re: Bringing Marionette and Mochitest closer
Jonas Sicking wrote: The fact that we are using gecko as the http server means a whole lot of complications on mobile. In short, we currently need gecko both compiled for android, which is the version of gecko being tested, and compiled for a desktop platform, which acts as host of the web server. This is currently a drain on our infrastructure and means that we're running tests slower than we otherwise could. Aren't we already compiling and packaging tests for gecko for desktop? Is android compiling them all over again instead of using existing compiled packages? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Bringing Marionette and Mochitest closer
On 1/30/2013 5:56 AM, Neil wrote: Jonas Sicking wrote: The fact that we are using gecko as the http server means a whole lot of complications on mobile. In short, we currently need gecko both compiled for android, which is the version of gecko being tested, and compiled for a desktop platform, which acts as host of the web server. This is currently a drain on our infrastructure and means that we're running tests slower than we otherwise could. Aren't we already compiling and packaging tests for gecko for desktop? Is android compiling them all over again instead of using existing compiled packages? Android tests are using pre-built desktop binaries to run httpd.js, but it's an extra hassle. Also developers who want to run Mochitests for mobile platforms need to have a desktop build laying around, which is one more hoop to jump through. -Ted ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Bringing Marionette and Mochitest closer
I am all for moving towards Marionette being the basis of all frameworks. It will allow us to share tests between other vendors because Marionette is based on the W3C Browser Automation Spec[1]. One of the things that this brings up is how we can share between other vendors. Testharness.js has become the W3 in browser standard so if we are going for interop between vendors then using testharness.js is what we will need to look at that or look at pure Marionette calls into the browser if we need to get out of the sandbox. One thing to note, and might not be obvious from Marionette documentation, is that it is easy to switch between Browser Chrome and Browser Content and execute commands (including pure JS). Opera use their equivalent of Marionette called OperaDriver[3] to load the browser and set the browser capabilities and visit a server that loads the in-browser test and runs testharness.js or calling into Chrome since it accesses the browser via its Scope Protocol. If I remember correctly, Chromium is moving to that model too but through their remote debugger like us. I have also put some in line messages below for more specific things. If you want to discuss this further via vidyo/skype/irc let me know. I am keep to fix any potential shortcomings in Marionette. David Burns AutomatedTester [1] https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html [2] https://github.com/mozilla/mozbase/tree/master/mozprofile [3] https://github.com/operasoftware/operadriver [4] [2] https://github.com/mozilla/mozbase/tree/master/mozhttpd [5] https://wiki.mozilla.org/Auto-tools/Projects/Mozbase On 30/01/2013 07:13, Jonas Sicking wrote: We'd also need to improve the web server which is started by marionette so that we can satisfy some of the use cases that we're currently using .sjs files to solve in mochitest. I am sure that we update MozHTTPd[4] to have a similar feature, might be exact and might not be trivial to do but might be worth the effort at for a later stage. MozHTTPd is part of MozBase[5], the ATeam project to standardise tools we use in testing. Instead we suggest that we use a simpler python based solution. Marionette's python client will be great at this. We are already using this extensively. This means that we can't immediately migrate the existing body of mochitests to the new http server. But we only have a total of 184 .sjs files, so it seems like something that should be doable eventually. So for now this means that we'll have to set up a new test harness. But I'm personally hoping that we can eventually let this harness cover also our /mochitests which would be an all-around win. So a quick win for this sounds like Marionette + (MozProfile[2] + Special Powers) to get what we want but there might be more to it. Would very much like to hear feedback on these plans. Something else we talked about very briefly was to improve the test environment which we use for running mochitests on B2G such that we enable the same adjust test, press reload flow that is commonly used on desktop when writing mochitests. Since there is a python package we can easily use the REPL for python to do things but I don't see how Marionette would stop this. Marionette just drives the browser. If you are just wanting to load a page then Marionette's navigate() method would just load that. This will hopefully both help with getting more tests authored for B2G-specific features (tests which can hopefully be migrated to cross-platform as the features become cross-platform), and help with getting the current body of mochitests running on B2G. / Jonas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Bringing Marionette and Mochitest closer
On 01/30/2013 06:33 PM, Benjamin Smedberg wrote: On 1/30/2013 2:13 AM, Jonas Sicking wrote: A few of us got together last week and had a quick brainstorming session about how to leverage the combined power from Mochitest and Marionette better. The following issues were raised: What is Marionette? From the discussion, it sounds like a W3C testing framework, but it's not clear whether it's something we're currently using, or something we're discussing using. I don't remember having seen it on TBPL, for example. --BDS https://developer.mozilla.org/en-US/docs/Marionette It's definitely on TBPL. Cheers, Josh ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Bringing Marionette and Mochitest closer
Marionette is one of the core frameworks for testing FirefoxOS but works pretty much everywhere since it is part Gecko. You can see it on TBPL with https://tbpl.mozilla.org/?tree=Mozilla-Inboundjobname=marionette for example. David On 30/01/2013 18:33, Benjamin Smedberg wrote: On 1/30/2013 2:13 AM, Jonas Sicking wrote: A few of us got together last week and had a quick brainstorming session about how to leverage the combined power from Mochitest and Marionette better. The following issues were raised: What is Marionette? From the discussion, it sounds like a W3C testing framework, but it's not clear whether it's something we're currently using, or something we're discussing using. I don't remember having seen it on TBPL, for example. --BDS ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Bringing Marionette and Mochitest closer
A few of us got together last week and had a quick brainstorming session about how to leverage the combined power from Mochitest and Marionette better. The following issues were raised: * Marionette has support for a http server, but one that is much less feature-full than the one used by Mochitest. * Mochitest is great as long as you are only needing to interact with things that are accessible from inside the browser sandbox. But as soon as you need to interact with browser UI, such as security dialogs, or other processes, it's often impossible to use mochitest. Especially if you want your test to work across desktop/android/b2g. * Mochitest relies on SpecialPowers which is unlikely to ever become available cross-browser. So unlikely that we'll be able to submit tests that use SpecialPowers to W3C. * Marionette requires a lot more boilerplate in order to run JS code in the context of a webpage. * Unclear where the line goes for when to write mochitest vs. marionette tests. Something that sounds like it could solve, or at least alleviate a lot of these problems is if we were able to start a mochitest-like test driver from marionette. But set up marionette such that it listened for commands from this test driver and then used those commands to interact with browser UI or other processes (such as app processes). We'd also need to improve the web server which is started by marionette so that we can satisfy some of the use cases that we're currently using .sjs files to solve in mochitest. In particular things like responses which are generated programmatically, state-full responses, and asynchronous responses. We could technically reuse the existing http server that we are using in mochitest for this. However there are a couple of disadvantages with this: * It's unlikely that we can standardize on that server. I.e. it's unlikely that we can submit tests which rely on .sjs files to W3c. If for no other reason, this because .sjs relies on a fair number of XPCOM-isms * The fact that we are using gecko as the http server means a whole lot of complications on mobile. In short, we currently need gecko both compiled for android, which is the version of gecko being tested, and compiled for a desktop platform, which acts as host of the web server. This is currently a drain on our infrastructure and means that we're running tests slower than we otherwise could. Instead we suggest that we use a simpler python based solution. This means that we can't immediately migrate the existing body of mochitests to the new http server. But we only have a total of 184 .sjs files, so it seems like something that should be doable eventually. So for now this means that we'll have to set up a new test harness. But I'm personally hoping that we can eventually let this harness cover also our mochitests which would be an all-around win. Would very much like to hear feedback on these plans. Something else we talked about very briefly was to improve the test environment which we use for running mochitests on B2G such that we enable the same adjust test, press reload flow that is commonly used on desktop when writing mochitests. This will hopefully both help with getting more tests authored for B2G-specific features (tests which can hopefully be migrated to cross-platform as the features become cross-platform), and help with getting the current body of mochitests running on B2G. / Jonas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
