Re: Running mochitests from a copy of the objdir?
Gregory Szorc wrote: Well, mach seems to be working for people doing m-c development. [Still needs a working build environment, while python runtests.py just used to need an objdir.] -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Neil wrote: Gregory Szorc wrote: Well, mach seems to be working for people doing m-c development. [Still needs a working build environment, while python runtests.py just used to need an objdir.] In fact there was a time where I could cross-compile and copy the objdir onto the target OS and run tests. -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/20/2014 4:16 AM, Neil wrote: Neil wrote: Gregory Szorc wrote: Well, mach seems to be working for people doing m-c development. [Still needs a working build environment, while python runtests.py just used to need an objdir.] In fact there was a time where I could cross-compile and copy the objdir onto the target OS and run tests. I'm sorry if we've made this use case more difficult, but it's not one we're intentionally supporting. It may be that we can keep it working with small changes, but our supported use cases are run from mach and run from the test package. -Ted ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Ted Mielczarek wrote: our supported use cases are run from mach and run from the test package. How does a test package differ from an objdir? Do you still need an entire build environment to run a test package? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Ted Mielczarek wrote: it requires you to specify all of the relevant options. And where are these options documented? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/20/2014 12:22 PM, L. David Baron wrote: (I estimated that it was going to be faster to get that working than to try to figure out how to use the packaged tests, since it was possible to reverse-engineer from mochitest run inside mach, though if there had been instructions on how to use packaged tests that somebody had actually used before I'd likely have gone the other way.) Building packaged tests is easy (make package for the installer, make package-tests for the tests); running them is a little harder since you have to build the python runtests.py command line yourself. Or you can open up a tbpl log and grab the exact command line there. Certainly far easier than trying to work out how to run mozharness on a local system... -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
I have done this sort of thing a number of times too (sometimes successfully, sometimes not), and it's been multiple hours each time. I also recall hearing from several other people who have tried to do this and ended up giving up. I love it when mach just works and runs exactly what I want, but even then it's usually just the first step -- next I need to restrict the set of tests run, or run under a debugger, or leave the browser open after the test, or strace something, or recompile and use a different binary, or whatever. It's good when mach expands to cover more of these things, but it seems like I always fall off of the prepared path at some point. It helps when it prints out command lines, though often they're buried in a bunch of other output and/or must be run within some modified environment that isn't recorded or isn't obvious. mozharness scripts suffer from the same sorts of problems. They work in automation, but often aren't very usable on a developer's box (eg they'll insist on downloading binaries and test packages instead of being able to run on a local checkout or binary.) And even when they work, you still end up needing to dig out the actual commands run to repeat them. Paths that are tested via automation work. Anything else is the wild wild west. It would be nice to have a developer-friendly path tested in automation, even if it's not the way the real jobs work. On 08/20/2014 10:22 AM, L. David Baron wrote: On Wednesday 2014-08-20 09:14 +0100, Neil wrote: Gregory Szorc wrote: Well, mach seems to be working for people doing m-c development. [Still needs a working build environment, while python runtests.py just used to need an objdir.] I ran into this problem not long ago when I needed to run mochitests on a different machine from the machine I was building on, because I needed to run the mochitests in a configuration that supported OMTC. (OMTC is not supported over VNC, so I couldn't run them on the machine I was building on.) I ended up repeatedly using the command: /full/path/to/build/obj/firefox-debugopt/_virtualenv/bin/python ./obj/firefox-debugopt/_tests/testing/mochitest/runtests.py --timeout 100 --autorun --close-when-done --console-level=INFO --testing-modules-dir=/full/path/to/build/obj/firefox-debugopt/_tests/modules --extra-profile-file=obj/firefox-debugopt/dist/plugins --test-path=layout/style/test/test_animations_omta_start.html --setpref=layers.offmainthreadcomposition.async-animations=true --setpref=layers.acceleration.force-enabled=true after rsyncing the objdir using the following rsync filter file: + /dist/ + /dist/bin/ H /dist/bin/core H /dist/bin/core.* H /dist/bin/test_* H /dist/bin/Test* H /dist/bin/*Test H /dist/bin/*_unittest H /dist/bin/*_unittests H /dist/bin/*-tests + /dist/plugins/ H /dist/* + /_tests/ H /_tests/testing/mochitest/core - /_virtualenv/ + /mozinfo.json + /config.status H /* (which took me a few hours to figure out, if I recall). Doing the equivalent with reftest is substantially easier. (I estimated that it was going to be faster to get that working than to try to figure out how to use the packaged tests, since it was possible to reverse-engineer from mochitest run inside mach, though if there had been instructions on how to use packaged tests that somebody had actually used before I'd likely have gone the other way.) This is part of why I dislike more and more layers being added to the way in which we run tests; development often requires running things in different-from-normal ways, and the more layers we have, the harder that gets. -David ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 20/08/14 18:38, Joshua Cranmer wrote: On 8/20/2014 12:22 PM, L. David Baron wrote: (I estimated that it was going to be faster to get that working than to try to figure out how to use the packaged tests, since it was possible to reverse-engineer from mochitest run inside mach, though if there had been instructions on how to use packaged tests that somebody had actually used before I'd likely have gone the other way.) Building packaged tests is easy (make package for the installer, make package-tests for the tests); running them is a little harder since you have to build the python runtests.py command line yourself. Or you can open up a tbpl log and grab the exact command line there. Certainly far easier than trying to work out how to run mozharness on a local system... Running mozharness on a local system is actually documented [1], although I think that makes it sound harder than it actually is. I have a run.sh script in the root of my mozharness checkout that looks like #!/bin/bash cd scripts CONFIG_FILE=../configs/web_platform_tests/test_config.py BUILD_ROOT=/home/jgraham/develop/gecko-dev/obj-x86_64-unknown-linux-gnu/dist INSTALLER_URL=file://$BUILD_ROOT/firefox-34.0a1.en-US.linux-x86_64.tar.bz2 TEST_URL=file://$BUILD_ROOT/firefox-34.0a1.en-US.linux-x86_64.tests.zip ./web_platform_tests.py --no-read-buildbot-config --config-file $CONFIG_FILE --installer-url $INSTALLER_URL --test-url $TEST_URL Obviously for different testsuites you need different config files or command lines, e.g. mochitest-plain would have something like CONFIG_FILE=../configs/unittests/linux_unittest.py and a final command like: ./desktop_unittest.py --no-read-buildbot-config --mochitest-suite plain --config-file $CONFIG_FILE --installer-url $INSTALLER_URL --test-url $TEST_URL --download-symbols true [1] https://wiki.mozilla.org/ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/20/14 10:58 AM, James Graham wrote: On 20/08/14 18:38, Joshua Cranmer wrote: On 8/20/2014 12:22 PM, L. David Baron wrote: (I estimated that it was going to be faster to get that working than to try to figure out how to use the packaged tests, since it was possible to reverse-engineer from mochitest run inside mach, though if there had been instructions on how to use packaged tests that somebody had actually used before I'd likely have gone the other way.) Building packaged tests is easy (make package for the installer, make package-tests for the tests); running them is a little harder since you have to build the python runtests.py command line yourself. Or you can open up a tbpl log and grab the exact command line there. Certainly far easier than trying to work out how to run mozharness on a local system... Running mozharness on a local system is actually documented [1], although I think that makes it sound harder than it actually is. I have a run.sh script in the root of my mozharness checkout that looks like #!/bin/bash cd scripts CONFIG_FILE=../configs/web_platform_tests/test_config.py BUILD_ROOT=/home/jgraham/develop/gecko-dev/obj-x86_64-unknown-linux-gnu/dist INSTALLER_URL=file://$BUILD_ROOT/firefox-34.0a1.en-US.linux-x86_64.tar.bz2 TEST_URL=file://$BUILD_ROOT/firefox-34.0a1.en-US.linux-x86_64.tests.zip ./web_platform_tests.py --no-read-buildbot-config --config-file $CONFIG_FILE --installer-url $INSTALLER_URL --test-url $TEST_URL Obviously for different testsuites you need different config files or command lines, e.g. mochitest-plain would have something like CONFIG_FILE=../configs/unittests/linux_unittest.py and a final command like: ./desktop_unittest.py --no-read-buildbot-config --mochitest-suite plain --config-file $CONFIG_FILE --installer-url $INSTALLER_URL --test-url $TEST_URL --download-symbols true [1] https://wiki.mozilla.org/ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer It sounds like the copy build to remote machine so we can run tests there is somewhat common and needs to be better supported by the tools. I think it makes sense to leverage test archives and build packages for this. Since mozharness already supports running things from there, I guess the missing piece is some glue code that invokes mozharness with a sane set of default options. That missing piece sounds a lot like what mach does. We talked about bundling mach in the tests archive to facilitate things like this. Perhaps we should move forward with that? Or is there a better way? The problem is at the unholy intersection of build, test, and automation responsibilities. Who's the owner? A-Team? ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
From: Gregory Szorc g...@mozilla.com To: James Graham ja...@hoppipolla.co.uk, dev-platform@lists.mozilla.org Sent: Wednesday, August 20, 2014 11:24:00 AM Subject: Re: Running mochitests from a copy of the objdir? It sounds like the copy build to remote machine so we can run tests there is somewhat common and needs to be better supported by the tools. +1 ...+10 I think it makes sense to leverage test archives and build packages for this. Since mozharness already supports running things from there, I guess the missing piece is some glue code that invokes mozharness with a sane set of default options. That missing piece sounds a lot like what mach does. We talked about bundling mach in the tests archive to facilitate things like this. Perhaps we should move forward with that? Or is there a better way? The problem is at the unholy intersection of build, test, and automation responsibilities. Who's the owner? A-Team? ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Gregory Szorc wrote: On 8/18/2014 4:45 PM, Neil wrote: Time was that you could just python runtests.py to run mochitests. Then we needed modules that you don't get in the default python, so you had to invoke python from the virtualenv instead. Now that doesn't work either, because it's trying to run .mozconfig, so my questions are a) why and b) how do I stop it? (I found --binary but that didn't seem to be enough on its own.) Can you please describe your workflow It's quite simple: python runtests.py had all the right defaults for me, while the Makefile targets pass a bunch of parameters that I don't want, and mach doesn't work in comm-central yet, so I don't know how badly that would work out. (as opposed to giving you hints on how to subtly hack around the existing implementation) I'd prefer hints on how to fix the build system, but I'll take hacks too. -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/19/14 1:53 AM, Neil wrote: Gregory Szorc wrote: On 8/18/2014 4:45 PM, Neil wrote: Time was that you could just python runtests.py to run mochitests. Then we needed modules that you don't get in the default python, so you had to invoke python from the virtualenv instead. Now that doesn't work either, because it's trying to run .mozconfig, so my questions are a) why and b) how do I stop it? (I found --binary but that didn't seem to be enough on its own.) Can you please describe your workflow It's quite simple: python runtests.py had all the right defaults for me, while the Makefile targets pass a bunch of parameters that I don't want, and mach doesn't work in comm-central yet, so I don't know how badly that would work out. (as opposed to giving you hints on how to subtly hack around the existing implementation) I'd prefer hints on how to fix the build system, but I'll take hacks too. Well, mach seems to be working for people doing m-c development. I think the underlying bug here is mach mochitest doesn't work for comm-central. It should. I'm sure there is a bug on file somewhere. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Hi Neil, Can you show us the command-line you're using? Jonathan On 8/19/2014 1:53 AM, Neil wrote: Gregory Szorc wrote: On 8/18/2014 4:45 PM, Neil wrote: Time was that you could just python runtests.py to run mochitests. Then we needed modules that you don't get in the default python, so you had to invoke python from the virtualenv instead. Now that doesn't work either, because it's trying to run .mozconfig, so my questions are a) why and b) how do I stop it? (I found --binary but that didn't seem to be enough on its own.) Can you please describe your workflow It's quite simple: python runtests.py had all the right defaults for me, while the Makefile targets pass a bunch of parameters that I don't want, and mach doesn't work in comm-central yet, so I don't know how badly that would work out. (as opposed to giving you hints on how to subtly hack around the existing implementation) I'd prefer hints on how to fix the build system, but I'll take hacks too. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Gregory Szorc wrote: I think the underlying bug here is mach mochitest doesn't work for comm-central. It should. I'm sure there is a bug on file somewhere. Fair enough, but for when it does work, what's the mach version of python runtests.py? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Jonathan Griffin wrote: Can you show us the command-line you're using? ../path/to/python runtests.py -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/19/14 2:11 PM, Neil wrote: Gregory Szorc wrote: I think the underlying bug here is mach mochitest doesn't work for comm-central. It should. I'm sure there is a bug on file somewhere. Fair enough, but for when it does work, what's the mach version of python runtests.py? mach mochitest? mach mochitest-plain? Under the scenes mach is using straight Python to construct the mochitest harness/runner and invoke it directly. No extra processes necessary. You can run |mach mach-debug-commands| to locate where each command is defined in source. Or, load https://hg.mozilla.org/mozilla-central/file/149d3ce6e020/testing/mochitest/mach_commands.py for the mochitest commands. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Gregory Szorc wrote: On 8/19/14 2:11 PM, Neil wrote: Gregory Szorc wrote: I think the underlying bug here is mach mochitest doesn't work for comm-central. It should. I'm sure there is a bug on file somewhere. Fair enough, but for when it does work, what's the mach version of python runtests.py? mach mochitest? mach mochitest-plain? Under the scenes mach is using straight Python to construct the mochitest harness/runner and invoke it directly. No extra processes necessary. You can run |mach mach-debug-commands| to locate where each command is defined in source. Or, load https://hg.mozilla.org/mozilla-central/file/149d3ce6e020/testing/mochitest/mach_commands.py for the mochitest commands. Right, but http://hg.mozilla.org/mozilla-central/file/149d3ce6e020/testing/mochitest/mach_commands.py#l290 suggests that the mach defaults aren't the same as the runtest.py defaults, which are the ones that I want. -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Running mochitests from a copy of the objdir?
Time was that you could just python runtests.py to run mochitests. Then we needed modules that you don't get in the default python, so you had to invoke python from the virtualenv instead. Now that doesn't work either, because it's trying to run .mozconfig, so my questions are a) why and b) how do I stop it? (I found --binary but that didn't seem to be enough on its own.) -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
You should use ./mach mochitest-* to run mochitests. Gavin On Mon, Aug 18, 2014 at 4:45 PM, Neil n...@parkwaycc.co.uk wrote: Time was that you could just python runtests.py to run mochitests. Then we needed modules that you don't get in the default python, so you had to invoke python from the virtualenv instead. Now that doesn't work either, because it's trying to run .mozconfig, so my questions are a) why and b) how do I stop it? (I found --binary but that didn't seem to be enough on its own.) -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/18/2014 4:45 PM, Neil wrote: Time was that you could just python runtests.py to run mochitests. Then we needed modules that you don't get in the default python, so you had to invoke python from the virtualenv instead. Now that doesn't work either, because it's trying to run .mozconfig, so my questions are a) why and b) how do I stop it? (I found --binary but that didn't seem to be enough on its own.) Can you please describe your workflow so we can consider a solution for your needs (as opposed to giving you hints on how to subtly hack around the existing implementation). i.e. please report user stories instead of trying to reverse engineer the build and automation systems :) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
