Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On Wed, Aug 20, 2014 at 4:24 PM, Jeff Gilbert wrote: > I have been asked in the past if we really need to run WebGL tests on > Android, if they have coverage on Desktop platforms. > And then again later, why B2G if we have Android. > > There seems to be enough belief in test-once-run-everywhere that I feel the > need to *firmly* establish that this is not acceptable, at least for the code > I work with. > I'm happy I'm not alone in this. I'm a firm believer that we ultimately need to run basically all combinations of tests and platforms before allowing code to reach mozilla-central. There's lots of platform specific code paths, and it's hard to track which tests trigger them, and which don't. It would however be really cool if we were able to pull data on which tests tend to fail in a way that affects all platforms, and which ones tend to fail on one platform only. If we combine this with the ability of having tbpl (or treeherder) "fill in the blanks" whenever a test fails, it seems like we could run many of our tests only one one platform for most checkins to mozilla-inbound. / Jonas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
> From: "Ehsan Akhgari" > To: "Jeff Gilbert" > Cc: "Chris AtLee" , "Jonathan Griffin" > , dev-platform@lists.mozilla.org > Sent: Wednesday, August 20, 2014 4:00:15 PM > Subject: Re: Experiment with running debug tests less often on > mozilla-inbound the week of August 25 > > On 2014-08-20, 6:29 PM, Jeff Gilbert wrote: > > If running debug tests on a single platform is generally sufficient for > > non-graphics bugs, > > It is not. That is the point I was trying to make. :-) > > > it might be useful to have the Graphics branch run debug tests on all > platforms, for use with graphics checkins. (while running a decreased > number of debug tests on the main branches) It's still possible for > non-graphics code to expose platform-specific bugs, but it's less > likely, so maybe larger regression windows are acceptable for > platform-specific bugs in non-graphics code. > > I don't really understand how graphics is special here. We do have > platform specific code outside of graphics as well, so we don't need to > solve this problem for gfx specifically. > Maybe Graphics isn't that special, but this stuff hits really close to home for us. I have been asked in the past if we really need to run WebGL tests on Android, if they have coverage on Desktop platforms. And then again later, why B2G if we have Android. There seems to be enough belief in test-once-run-everywhere that I feel the need to *firmly* establish that this is not acceptable, at least for the code I work with. I'm happy I'm not alone in this. -Jeff ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
> From: "Gregory Szorc" > To: "James Graham" , dev-platform@lists.mozilla.org > Sent: Wednesday, August 20, 2014 11:24:00 AM > Subject: Re: Running mochitests from a copy of the objdir? > > It sounds like the "copy build to remote machine so we can run tests > there" is somewhat common and needs to be better supported by the tools. +1 ...+10 > > I think it makes sense to leverage test archives and build packages for > this. Since mozharness already supports running things from there, I > guess the missing piece is some glue code that invokes mozharness with a > sane set of default options. That missing piece sounds a lot like what > mach does. We talked about bundling mach in the tests archive to > facilitate things like this. Perhaps we should move forward with that? > Or is there a better way? > > The problem is at the unholy intersection of build, test, and automation > responsibilities. Who's the owner? A-Team? > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On 2014-08-20, 6:29 PM, Jeff Gilbert wrote: If running debug tests on a single platform is generally sufficient for non-graphics bugs, It is not. That is the point I was trying to make. :-) > it might be useful to have the Graphics branch run debug tests on all platforms, for use with graphics checkins. (while running a decreased number of debug tests on the main branches) It's still possible for non-graphics code to expose platform-specific bugs, but it's less likely, so maybe larger regression windows are acceptable for platform-specific bugs in non-graphics code. I don't really understand how graphics is special here. We do have platform specific code outside of graphics as well, so we don't need to solve this problem for gfx specifically. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
If running debug tests on a single platform is generally sufficient for non-graphics bugs, it might be useful to have the Graphics branch run debug tests on all platforms, for use with graphics checkins. (while running a decreased number of debug tests on the main branches) It's still possible for non-graphics code to expose platform-specific bugs, but it's less likely, so maybe larger regression windows are acceptable for platform-specific bugs in non-graphics code. -Jeff - Original Message - From: "Ehsan Akhgari" To: "Jeff Gilbert" , "Chris AtLee" Cc: "Jonathan Griffin" , dev-platform@lists.mozilla.org Sent: Wednesday, August 20, 2014 3:16:31 PM Subject: Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25 On 2014-08-20, 5:46 PM, Jeff Gilbert wrote: > Graphics in particular is plagued by non-cross-platform code. Debug coverage > on Linux gives us no practical coverage for our windows, mac, android, or b2g > code. Maybe this is better solved with reviving the Graphics branch, however. Having more branches doesn't necessarily help with consuming less infra resources, unless if the builds will be run with a lower frequency or something. Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On 2014-08-20, 5:46 PM, Jeff Gilbert wrote: Graphics in particular is plagued by non-cross-platform code. Debug coverage on Linux gives us no practical coverage for our windows, mac, android, or b2g code. Maybe this is better solved with reviving the Graphics branch, however. Having more branches doesn't necessarily help with consuming less infra resources, unless if the builds will be run with a lower frequency or something. Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
Graphics in particular is plagued by non-cross-platform code. Debug coverage on Linux gives us no practical coverage for our windows, mac, android, or b2g code. Maybe this is better solved with reviving the Graphics branch, however. -Jeff - Original Message - From: "Chris AtLee" To: "Ehsan Akhgari" Cc: "Jonathan Griffin" , "Jeff Gilbert" , dev-platform@lists.mozilla.org Sent: Wednesday, August 20, 2014 9:02:14 AM Subject: Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25 On 18:25, Tue, 19 Aug, Ehsan Akhgari wrote: >On 2014-08-19, 5:49 PM, Jonathan Griffin wrote: >>On 8/19/2014 2:41 PM, Ehsan Akhgari wrote: >>>On 2014-08-19, 3:57 PM, Jeff Gilbert wrote: I would actually say that debug tests are more important for continuous integration than opt tests. At least in code I deal with, we have a ton of asserts to guarantee behavior, and we really want test coverage with these via CI. If a test passes on debug, it should almost certainly pass on opt, just faster. The opposite is not true. "They take a long time and then break" is part of what I believe caused us to not bother with debug testing on much of Android and B2G, which we still haven't completely fixed. It should be unacceptable to ship without CI on debug tests, but here we are anyways. (This is finally nearly fixed, though there is still some work to do) I'm not saying running debug tests less often is on the same scale of bad, but I would like to express my concerns about heading in that direction. >>> >>>I second this. I'm curious to know why you picked debug tests for >>>this experiment. Would it not make more sense to run opt tests on >>>desktop platforms on every other run? >>> >>Just based on the fact that they take longer and thus running them less >>frequently would have a larger impact. If there's a broad consensus >>that debug runs are more valuable, we could switch to running opt tests >>less frequently instead. > >Yep, the debug tests indeed take more time, mostly because they run >more checks. :-) The checks in opt builds are not exactly a subset >of the ones in debug builds, but they are close. Based on that, I >think running opt tests on every other push is a more conservative >one, and I support it more. That being said, for this one week >limited trial, given that the sheriffs will help backfill the skipped >tests, I don't care very strongly about this, as long as it doesn't >set the precedence that we can ignore debug tests! I'd like to highlight that we're still planning on running debug linux64 tests for every build. This is based on the assumption that debug-specific failures are generally cross-platform failures as well. Does this help alleviate some concern? Or is that assumption just plain wrong? Cheers, Chris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On Wed, Aug 20, 2014 at 03:58:55PM +0100, Ed Morley wrote: > On 19/08/2014 21:55, Benoit Girard wrote: > >I completely agree with Jeff Gilbert on this one. > > > >I think we should try to coalesce -better-. I just checked the current > >state of mozilla-inbound and it doesn't feel any of the current patch > >really need their own set of tests because they're are not time > >sensitive or sufficiently complex. Right now developers are asked to > >create bugs for their own change with their own patch. This leads to a > >lot of little patches being landed by individual developers which > >seems to reflect the current state of mozilla-inbound. > > > >Perhaps we should instead promote checkin-needed (or a similar simple) > >to coalesce simple changes together. Opting into this means that your > >patch may take significantly longer to get merged if it's landed with > >another bad patch and should only be used when that's acceptable. > >Right now developers with commit access are not encouraged to make use > >of checkin-needed AFAIK. If we started recommending against individual > >landings for simple changes, and improved the process, we could > >probably significantly cut the number of tests jobs by cutting the > >number of pushes. > > I agree we should try to coalesce better - however doing this via a manual > "let's get someone to push a bunch of checkin-needed patches in one go" is > suboptimal: > 1) By tweaking coalescing in buildbot & pushing patches individually, we > could get the same build+test job per commit ratio as doing checkin-neededs, > but with the bonus of being able to backfill jobs where needed. This isn't > possible when say 10-20 checkin-neededs are landed in one push, since our > tooling can only trigger (and more importantly display the results of) jobs > on a per push level. It would have been useful on several occasions to be able to trigger builds at changeset level instead of push level, independently of checkin-needed. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On 2014-08-20, 12:02 PM, Chris AtLee wrote: On 18:25, Tue, 19 Aug, Ehsan Akhgari wrote: On 2014-08-19, 5:49 PM, Jonathan Griffin wrote: On 8/19/2014 2:41 PM, Ehsan Akhgari wrote: On 2014-08-19, 3:57 PM, Jeff Gilbert wrote: I would actually say that debug tests are more important for continuous integration than opt tests. At least in code I deal with, we have a ton of asserts to guarantee behavior, and we really want test coverage with these via CI. If a test passes on debug, it should almost certainly pass on opt, just faster. The opposite is not true. "They take a long time and then break" is part of what I believe caused us to not bother with debug testing on much of Android and B2G, which we still haven't completely fixed. It should be unacceptable to ship without CI on debug tests, but here we are anyways. (This is finally nearly fixed, though there is still some work to do) I'm not saying running debug tests less often is on the same scale of bad, but I would like to express my concerns about heading in that direction. I second this. I'm curious to know why you picked debug tests for this experiment. Would it not make more sense to run opt tests on desktop platforms on every other run? Just based on the fact that they take longer and thus running them less frequently would have a larger impact. If there's a broad consensus that debug runs are more valuable, we could switch to running opt tests less frequently instead. Yep, the debug tests indeed take more time, mostly because they run more checks. :-) The checks in opt builds are not exactly a subset of the ones in debug builds, but they are close. Based on that, I think running opt tests on every other push is a more conservative one, and I support it more. That being said, for this one week limited trial, given that the sheriffs will help backfill the skipped tests, I don't care very strongly about this, as long as it doesn't set the precedence that we can ignore debug tests! I'd like to highlight that we're still planning on running debug linux64 tests for every build. This is based on the assumption that debug-specific failures are generally cross-platform failures as well. Does this help alleviate some concern? Or is that assumption just plain wrong? well, yes, most of our code is cross platform, but there are debug only checks in our platform specific code as well, so if we're talking about something more permanent than that week long experiment, then running debug tests on Linux64 doesn't alleviate all concerns. But it's fine for this short experiment. Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On 18:25, Tue, 19 Aug, Ehsan Akhgari wrote: On 2014-08-19, 5:49 PM, Jonathan Griffin wrote: On 8/19/2014 2:41 PM, Ehsan Akhgari wrote: On 2014-08-19, 3:57 PM, Jeff Gilbert wrote: I would actually say that debug tests are more important for continuous integration than opt tests. At least in code I deal with, we have a ton of asserts to guarantee behavior, and we really want test coverage with these via CI. If a test passes on debug, it should almost certainly pass on opt, just faster. The opposite is not true. "They take a long time and then break" is part of what I believe caused us to not bother with debug testing on much of Android and B2G, which we still haven't completely fixed. It should be unacceptable to ship without CI on debug tests, but here we are anyways. (This is finally nearly fixed, though there is still some work to do) I'm not saying running debug tests less often is on the same scale of bad, but I would like to express my concerns about heading in that direction. I second this. I'm curious to know why you picked debug tests for this experiment. Would it not make more sense to run opt tests on desktop platforms on every other run? Just based on the fact that they take longer and thus running them less frequently would have a larger impact. If there's a broad consensus that debug runs are more valuable, we could switch to running opt tests less frequently instead. Yep, the debug tests indeed take more time, mostly because they run more checks. :-) The checks in opt builds are not exactly a subset of the ones in debug builds, but they are close. Based on that, I think running opt tests on every other push is a more conservative one, and I support it more. That being said, for this one week limited trial, given that the sheriffs will help backfill the skipped tests, I don't care very strongly about this, as long as it doesn't set the precedence that we can ignore debug tests! I'd like to highlight that we're still planning on running debug linux64 tests for every build. This is based on the assumption that debug-specific failures are generally cross-platform failures as well. Does this help alleviate some concern? Or is that assumption just plain wrong? Cheers, Chris signature.asc Description: Digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reordering opened windows
On 20/08/14 17:50, Ehsan Akhgari wrote: >> A hide/show pair after that might also be sufficient, but I assume >> something that simple was tried already. > > This is also interesting. This maps to nsWindow::Show(false/true). Yes, I believe that this would work, but I didn't try it in practice, as I'm pretty sure it would cause flickering. -- David Rajchenbach-Teller, PhD Performance Team, Mozilla signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/20/14 10:58 AM, James Graham wrote: On 20/08/14 18:38, Joshua Cranmer 🐧 wrote: On 8/20/2014 12:22 PM, L. David Baron wrote: (I estimated that it was going to be faster to get that working than to try to figure out how to use the packaged tests, since it was possible to reverse-engineer from mochitest run inside mach, though if there had been instructions on how to use packaged tests that somebody had actually used before I'd likely have gone the other way.) Building packaged tests is easy (make package for the installer, make package-tests for the tests); running them is a little harder since you have to build the python runtests.py command line yourself. Or you can open up a tbpl log and grab the exact command line there. Certainly far easier than trying to work out how to run mozharness on a local system... Running mozharness on a local system is actually documented [1], although I think that makes it sound harder than it actually is. I have a run.sh script in the root of my mozharness checkout that looks like #!/bin/bash cd scripts CONFIG_FILE=../configs/web_platform_tests/test_config.py BUILD_ROOT=/home/jgraham/develop/gecko-dev/obj-x86_64-unknown-linux-gnu/dist INSTALLER_URL=file://$BUILD_ROOT/firefox-34.0a1.en-US.linux-x86_64.tar.bz2 TEST_URL=file://$BUILD_ROOT/firefox-34.0a1.en-US.linux-x86_64.tests.zip ./web_platform_tests.py --no-read-buildbot-config --config-file $CONFIG_FILE --installer-url $INSTALLER_URL --test-url $TEST_URL Obviously for different testsuites you need different config files or command lines, e.g. mochitest-plain would have something like CONFIG_FILE=../configs/unittests/linux_unittest.py and a final command like: ./desktop_unittest.py --no-read-buildbot-config --mochitest-suite plain --config-file $CONFIG_FILE --installer-url $INSTALLER_URL --test-url $TEST_URL --download-symbols true [1] https://wiki.mozilla.org/ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer It sounds like the "copy build to remote machine so we can run tests there" is somewhat common and needs to be better supported by the tools. I think it makes sense to leverage test archives and build packages for this. Since mozharness already supports running things from there, I guess the missing piece is some glue code that invokes mozharness with a sane set of default options. That missing piece sounds a lot like what mach does. We talked about bundling mach in the tests archive to facilitate things like this. Perhaps we should move forward with that? Or is there a better way? The problem is at the unholy intersection of build, test, and automation responsibilities. Who's the owner? A-Team? ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 20/08/14 18:38, Joshua Cranmer 🐧 wrote: > On 8/20/2014 12:22 PM, L. David Baron wrote: >> (I estimated that it was going to be faster to get that working than >> to try to figure out how to use the packaged tests, since it was >> possible to reverse-engineer from mochitest run inside mach, though >> if there had been instructions on how to use packaged tests that >> somebody had actually used before I'd likely have gone the other >> way.) > > Building packaged tests is easy (make package for the installer, make > package-tests for the tests); running them is a little harder since you > have to build the python runtests.py command line yourself. Or you can > open up a tbpl log and grab the exact command line there. Certainly far > easier than trying to work out how to run mozharness on a local system... > Running mozharness on a local system is actually documented [1], although I think that makes it sound harder than it actually is. I have a run.sh script in the root of my mozharness checkout that looks like #!/bin/bash cd scripts CONFIG_FILE=../configs/web_platform_tests/test_config.py BUILD_ROOT=/home/jgraham/develop/gecko-dev/obj-x86_64-unknown-linux-gnu/dist INSTALLER_URL=file://$BUILD_ROOT/firefox-34.0a1.en-US.linux-x86_64.tar.bz2 TEST_URL=file://$BUILD_ROOT/firefox-34.0a1.en-US.linux-x86_64.tests.zip ./web_platform_tests.py --no-read-buildbot-config --config-file $CONFIG_FILE --installer-url $INSTALLER_URL --test-url $TEST_URL Obviously for different testsuites you need different config files or command lines, e.g. mochitest-plain would have something like CONFIG_FILE=../configs/unittests/linux_unittest.py and a final command like: ./desktop_unittest.py --no-read-buildbot-config --mochitest-suite plain --config-file $CONFIG_FILE --installer-url $INSTALLER_URL --test-url $TEST_URL --download-symbols true [1] https://wiki.mozilla.org/ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
I have done this sort of thing a number of times too (sometimes successfully, sometimes not), and it's been multiple hours each time. I also recall hearing from several other people who have tried to do this and ended up giving up. I love it when mach just works and runs exactly what I want, but even then it's usually just the first step -- next I need to restrict the set of tests run, or run under a debugger, or leave the browser open after the test, or strace something, or recompile and use a different binary, or whatever. It's good when mach expands to cover more of these things, but it seems like I always fall off of the prepared path at some point. It helps when it prints out command lines, though often they're buried in a bunch of other output and/or must be run within some modified environment that isn't recorded or isn't obvious. mozharness scripts suffer from the same sorts of problems. They work in automation, but often aren't very usable on a developer's box (eg they'll insist on downloading binaries and test packages instead of being able to run on a local checkout or binary.) And even when they work, you still end up needing to dig out the actual commands run to repeat them. Paths that are tested via automation work. Anything else is the wild wild west. It would be nice to have a developer-friendly path tested in automation, even if it's not the way the "real" jobs work. On 08/20/2014 10:22 AM, L. David Baron wrote: > On Wednesday 2014-08-20 09:14 +0100, Neil wrote: >> Gregory Szorc wrote: >> >>> Well, mach seems to be working for people doing m-c development. >> [Still needs a working build environment, while python runtests.py >> just used to need an objdir.] > I ran into this problem not long ago when I needed to run mochitests > on a different machine from the machine I was building on, because I > needed to run the mochitests in a configuration that supported OMTC. > (OMTC is not supported over VNC, so I couldn't run them on the > machine I was building on.) > > I ended up repeatedly using the command: > > /full/path/to/build/obj/firefox-debugopt/_virtualenv/bin/python > ./obj/firefox-debugopt/_tests/testing/mochitest/runtests.py --timeout 100 > --autorun --close-when-done --console-level=INFO > --testing-modules-dir=/full/path/to/build/obj/firefox-debugopt/_tests/modules > --extra-profile-file=obj/firefox-debugopt/dist/plugins > --test-path=layout/style/test/test_animations_omta_start.html > --setpref=layers.offmainthreadcomposition.async-animations=true > --setpref=layers.acceleration.force-enabled=true > > after rsyncing the objdir using the following rsync filter file: > > + /dist/ > + /dist/bin/ > H /dist/bin/core > H /dist/bin/core.* > H /dist/bin/test_* > H /dist/bin/Test* > H /dist/bin/*Test > H /dist/bin/*_unittest > H /dist/bin/*_unittests > H /dist/bin/*-tests > + /dist/plugins/ > H /dist/* > + /_tests/ > H /_tests/testing/mochitest/core > - /_virtualenv/ > + /mozinfo.json > + /config.status > H /* > > (which took me a few hours to figure out, if I recall). > > Doing the equivalent with reftest is substantially easier. > > > (I estimated that it was going to be faster to get that working than > to try to figure out how to use the packaged tests, since it was > possible to reverse-engineer from mochitest run inside mach, though > if there had been instructions on how to use packaged tests that > somebody had actually used before I'd likely have gone the other > way.) > > > This is part of why I dislike more and more layers being added to > the way in which we run tests; development often requires running > things in different-from-normal ways, and the more layers we have, > the harder that gets. > > -David > > > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/20/2014 12:22 PM, L. David Baron wrote: (I estimated that it was going to be faster to get that working than to try to figure out how to use the packaged tests, since it was possible to reverse-engineer from mochitest run inside mach, though if there had been instructions on how to use packaged tests that somebody had actually used before I'd likely have gone the other way.) Building packaged tests is easy (make package for the installer, make package-tests for the tests); running them is a little harder since you have to build the python runtests.py command line yourself. Or you can open up a tbpl log and grab the exact command line there. Certainly far easier than trying to work out how to run mozharness on a local system... -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On Wednesday 2014-08-20 09:14 +0100, Neil wrote: > Gregory Szorc wrote: > > >Well, mach seems to be working for people doing m-c development. > > [Still needs a working build environment, while python runtests.py > just used to need an objdir.] I ran into this problem not long ago when I needed to run mochitests on a different machine from the machine I was building on, because I needed to run the mochitests in a configuration that supported OMTC. (OMTC is not supported over VNC, so I couldn't run them on the machine I was building on.) I ended up repeatedly using the command: /full/path/to/build/obj/firefox-debugopt/_virtualenv/bin/python ./obj/firefox-debugopt/_tests/testing/mochitest/runtests.py --timeout 100 --autorun --close-when-done --console-level=INFO --testing-modules-dir=/full/path/to/build/obj/firefox-debugopt/_tests/modules --extra-profile-file=obj/firefox-debugopt/dist/plugins --test-path=layout/style/test/test_animations_omta_start.html --setpref=layers.offmainthreadcomposition.async-animations=true --setpref=layers.acceleration.force-enabled=true after rsyncing the objdir using the following rsync filter file: + /dist/ + /dist/bin/ H /dist/bin/core H /dist/bin/core.* H /dist/bin/test_* H /dist/bin/Test* H /dist/bin/*Test H /dist/bin/*_unittest H /dist/bin/*_unittests H /dist/bin/*-tests + /dist/plugins/ H /dist/* + /_tests/ H /_tests/testing/mochitest/core - /_virtualenv/ + /mozinfo.json + /config.status H /* (which took me a few hours to figure out, if I recall). Doing the equivalent with reftest is substantially easier. (I estimated that it was going to be faster to get that working than to try to figure out how to use the packaged tests, since it was possible to reverse-engineer from mochitest run inside mach, though if there had been instructions on how to use packaged tests that somebody had actually used before I'd likely have gone the other way.) This is part of why I dislike more and more layers being added to the way in which we run tests; development often requires running things in different-from-normal ways, and the more layers we have, the harder that gets. -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla https://www.mozilla.org/ 𝄂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914) signature.asc Description: Digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On 14-08-20 11:48 AM, Ehsan Akhgari wrote: > On 2014-08-20, 10:58 AM, Ed Morley wrote: >> On 19/08/2014 21:55, Benoit Girard wrote: >>> I completely agree with Jeff Gilbert on this one. >>> >>> I think we should try to coalesce -better-. I just checked the current >>> state of mozilla-inbound and it doesn't feel any of the current patch >>> really need their own set of tests because they're are not time >>> sensitive or sufficiently complex. Right now developers are asked to >>> create bugs for their own change with their own patch. This leads to a >>> lot of little patches being landed by individual developers which >>> seems to reflect the current state of mozilla-inbound. >>> >>> Perhaps we should instead promote checkin-needed (or a similar simple) >>> to coalesce simple changes together. Opting into this means that your >>> patch may take significantly longer to get merged if it's landed with >>> another bad patch and should only be used when that's acceptable. >>> Right now developers with commit access are not encouraged to make use >>> of checkin-needed AFAIK. If we started recommending against individual >>> landings for simple changes, and improved the process, we could >>> probably significantly cut the number of tests jobs by cutting the >>> number of pushes. >> >> I agree we should try to coalesce better - however doing this via a >> manual "let's get someone to push a bunch of checkin-needed patches in >> one go" is suboptimal: >> 1) By tweaking coalescing in buildbot & pushing patches individually, we >> could get the same build+test job per commit ratio as doing >> checkin-neededs, but with the bonus of being able to backfill jobs where >> needed. This isn't possible when say 10-20 checkin-neededs are landed in >> one push, since our tooling can only trigger (and more importantly >> display the results of) jobs on a per push level. >> 2) Tooling can help make these decisions much more effectively and >> quickly than someone picking through bugs - ie we should expand the >> current "only schedule job X if directory Y changed" buildbotcustom >> logic further. >> 3) Adding a human in the workflow increases r+-to-committed cycle times, >> uses up scarce sheriff time, and also means the person who wrote the >> patch is not the one landing it, and so someone unfamiliar with the code >> often ends up being the one to resolve conflicts. We should be using >> tooling, not human cycles to lands patches in a repo (ie the >> long-promised autoland). > > Another approach to this would be to identify more patterns that allow > us to skip some jobs. We already do this for very simple things > (changes to a file in browser/ won't trigger b2g and Android builds, for > example), but I'm sure we could do more. For example, changes to files > under widget/ may only affect one platform, depending on which > directories you touch. Another idea that I have had is adding some > smarts to make it possible to parse the test manifest files, and > recognize things such as skip-if, to figure out what platforms a test > only change for example might not affect, and skip the builds and tests > on those platforms. > > One thing to note is that there is going to be a *long* tail of these > types of heuristics that we could come up with, so it would be nice to > try to only address the ones that provide the most benefits. For that, > someone needs to look at the recent N commits on a given branch and > figure out what jobs we _could_ have safely skipped for each one. If someone wants to have a look at doing this more intelligently, the relevant code is reasonably isolated in https://github.com/mozilla/build-buildbotcustom/blob/master/misc.py#L127. The object received there is a Buildbot Change object, which contains most (all) of the metadata about a revision: https://mxr.mozilla.org/build-central/source/buildbot/master/buildbot/changes/changes.py#11 I believe this is called once for every revision in a push. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reordering opened windows
On 2014-08-20, 10:54 AM, Katelyn Gadd wrote: Has anyone tried SetWindowPos? It has a mechanism for altering the windows z-order, which determines the order in which windows are enumerated. That should affect ordering in the taskbar. Hmm, interesting. SetWindowPos is what nsWindow::PlaceBehind uses, so perhaps you could try that, David? A hide/show pair after that might also be sufficient, but I assume something that simple was tried already. This is also interesting. This maps to nsWindow::Show(false/true). ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On 2014-08-20, 10:58 AM, Ed Morley wrote: On 19/08/2014 21:55, Benoit Girard wrote: I completely agree with Jeff Gilbert on this one. I think we should try to coalesce -better-. I just checked the current state of mozilla-inbound and it doesn't feel any of the current patch really need their own set of tests because they're are not time sensitive or sufficiently complex. Right now developers are asked to create bugs for their own change with their own patch. This leads to a lot of little patches being landed by individual developers which seems to reflect the current state of mozilla-inbound. Perhaps we should instead promote checkin-needed (or a similar simple) to coalesce simple changes together. Opting into this means that your patch may take significantly longer to get merged if it's landed with another bad patch and should only be used when that's acceptable. Right now developers with commit access are not encouraged to make use of checkin-needed AFAIK. If we started recommending against individual landings for simple changes, and improved the process, we could probably significantly cut the number of tests jobs by cutting the number of pushes. I agree we should try to coalesce better - however doing this via a manual "let's get someone to push a bunch of checkin-needed patches in one go" is suboptimal: 1) By tweaking coalescing in buildbot & pushing patches individually, we could get the same build+test job per commit ratio as doing checkin-neededs, but with the bonus of being able to backfill jobs where needed. This isn't possible when say 10-20 checkin-neededs are landed in one push, since our tooling can only trigger (and more importantly display the results of) jobs on a per push level. 2) Tooling can help make these decisions much more effectively and quickly than someone picking through bugs - ie we should expand the current "only schedule job X if directory Y changed" buildbotcustom logic further. 3) Adding a human in the workflow increases r+-to-committed cycle times, uses up scarce sheriff time, and also means the person who wrote the patch is not the one landing it, and so someone unfamiliar with the code often ends up being the one to resolve conflicts. We should be using tooling, not human cycles to lands patches in a repo (ie the long-promised autoland). Another approach to this would be to identify more patterns that allow us to skip some jobs. We already do this for very simple things (changes to a file in browser/ won't trigger b2g and Android builds, for example), but I'm sure we could do more. For example, changes to files under widget/ may only affect one platform, depending on which directories you touch. Another idea that I have had is adding some smarts to make it possible to parse the test manifest files, and recognize things such as skip-if, to figure out what platforms a test only change for example might not affect, and skip the builds and tests on those platforms. One thing to note is that there is going to be a *long* tail of these types of heuristics that we could come up with, so it would be nice to try to only address the ones that provide the most benefits. For that, someone needs to look at the recent N commits on a given branch and figure out what jobs we _could_ have safely skipped for each one. Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Opening windows hidden, showing windows lowered
On 2014-08-20, 10:50 AM, David Rajchenbach-Teller wrote: TL;DR I'm looking for: 1. an API to open a XUL window but keep it invisible; You should be able to show/hide windows using nsIWidget::Show. 2. an API to make an invisible XUL window visible, but behind the currently active window; The visibility can be controlled as per above. The ordering can be controlled using nsIWidget::PlaceBehind. 3. an API to open a XUL window but keep it in the background. Again, PlaceBehind is what you want, I think. Note that none of these APIs can open the window for you, you need to open it yourself, get the widget for the window and then perform these operations. Also, note that I just looked at the code to find out these answers, and I have not tested them myself, so please take this with a grain of salt. Cheers, Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Ted Mielczarek wrote: it requires you to specify all of the relevant options. And where are these options documented? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Opening windows hidden, showing windows lowered
Jan Varga wrote: On 20/08/14 10:50, David Rajchenbach-Teller wrote: I'm looking for: 1. an API to open a XUL window but keep it invisible; 2. an API to make an invisible XUL window visible, but behind the currently active window; I remember that the Composer in Mozilla Suite was doing something like 1. and 2. for new mail windows. Sorry, but it only makes existing windows invisible and makes invisible windows active and visible. -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On 19/08/2014 21:55, Benoit Girard wrote: I completely agree with Jeff Gilbert on this one. I think we should try to coalesce -better-. I just checked the current state of mozilla-inbound and it doesn't feel any of the current patch really need their own set of tests because they're are not time sensitive or sufficiently complex. Right now developers are asked to create bugs for their own change with their own patch. This leads to a lot of little patches being landed by individual developers which seems to reflect the current state of mozilla-inbound. Perhaps we should instead promote checkin-needed (or a similar simple) to coalesce simple changes together. Opting into this means that your patch may take significantly longer to get merged if it's landed with another bad patch and should only be used when that's acceptable. Right now developers with commit access are not encouraged to make use of checkin-needed AFAIK. If we started recommending against individual landings for simple changes, and improved the process, we could probably significantly cut the number of tests jobs by cutting the number of pushes. I agree we should try to coalesce better - however doing this via a manual "let's get someone to push a bunch of checkin-needed patches in one go" is suboptimal: 1) By tweaking coalescing in buildbot & pushing patches individually, we could get the same build+test job per commit ratio as doing checkin-neededs, but with the bonus of being able to backfill jobs where needed. This isn't possible when say 10-20 checkin-neededs are landed in one push, since our tooling can only trigger (and more importantly display the results of) jobs on a per push level. 2) Tooling can help make these decisions much more effectively and quickly than someone picking through bugs - ie we should expand the current "only schedule job X if directory Y changed" buildbotcustom logic further. 3) Adding a human in the workflow increases r+-to-committed cycle times, uses up scarce sheriff time, and also means the person who wrote the patch is not the one landing it, and so someone unfamiliar with the code often ends up being the one to resolve conflicts. We should be using tooling, not human cycles to lands patches in a repo (ie the long-promised autoland). Best wishes, Ed ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Opening windows hidden, showing windows lowered
I remember that the Composer in Mozilla Suite was doing something like 1. and 2. for new mail windows. On 20/08/14 10:50, David Rajchenbach-Teller wrote: TL;DR I'm looking for: 1. an API to open a XUL window but keep it invisible; 2. an API to make an invisible XUL window visible, but behind the currently active window; 3. an API to open a XUL window but keep it in the background. Do we have such things? Otherwise, how hard would it be to add them? Long version As discussed previously on this list, I'd like to tweak the order in which we restore windows on startup to make sure that recently used windows are restored earlier than older windows, and to make the experience less janky for users of multiple windows. Unfortunately, we apparently cannot change the order in which windows are opened, as this messes with the Windows taskbar (and possibly equivalent tools on other platforms), so my fallback plan would be to: 1/ open hidden blank windows (to keep the taskbar happy without wasting too much startup time); 2/ make the window that we consider useful visible and start loading it; 3/ make other windows progressively visible, with a lower z-order. So far, I haven't been able to find any such API in or around . Does anyone have any suggestions as to where I could poke? Cheers, David ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reordering opened windows
Has anyone tried SetWindowPos? It has a mechanism for altering the windows z-order, which determines the order in which windows are enumerated. That should affect ordering in the taskbar. A hide/show pair after that might also be sufficient, but I assume something that simple was tried already. On Wed, Aug 20, 2014 at 7:41 AM, David Rajchenbach-Teller wrote: > No success, I take it? > > On 04/07/14 20:11, Ehsan Akhgari wrote: >> I'm not aware of any such APIs, but CCing some other folks who know more >> about Windows than I do. >> > > > -- > David Rajchenbach-Teller, PhD > Performance Team, Mozilla > > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Opening windows hidden, showing windows lowered
TL;DR I'm looking for: 1. an API to open a XUL window but keep it invisible; 2. an API to make an invisible XUL window visible, but behind the currently active window; 3. an API to open a XUL window but keep it in the background. Do we have such things? Otherwise, how hard would it be to add them? Long version As discussed previously on this list, I'd like to tweak the order in which we restore windows on startup to make sure that recently used windows are restored earlier than older windows, and to make the experience less janky for users of multiple windows. Unfortunately, we apparently cannot change the order in which windows are opened, as this messes with the Windows taskbar (and possibly equivalent tools on other platforms), so my fallback plan would be to: 1/ open hidden blank windows (to keep the taskbar happy without wasting too much startup time); 2/ make the window that we consider useful visible and start loading it; 3/ make other windows progressively visible, with a lower z-order. So far, I haven't been able to find any such API in or around . Does anyone have any suggestions as to where I could poke? Cheers, David -- David Rajchenbach-Teller, PhD Performance Team, Mozilla signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Reordering opened windows
No success, I take it? On 04/07/14 20:11, Ehsan Akhgari wrote: > I'm not aware of any such APIs, but CCing some other folks who know more > about Windows than I do. > -- David Rajchenbach-Teller, PhD Performance Team, Mozilla signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Web APIs documentation meeting Friday at 10 AM PDT
The Web APIs documentation meeting is Friday at 10 AM Pacific Time (see http://bit.ly/APIDocsMeeting for your time zone). Everyone's welcome to attend; if you're interested in ensuring that all Web APIs are properly documented, we'd love your input. We have an agenda, as well as details on how to join, here: https://etherpad.mozilla.org/WebAPI-docs-2014-08-22. If you have topics you wish to discuss, please feel free to add them to the agenda. We look forward to seeing you there! -- Eric Shepherd Developer Documentation Lead Mozilla Blog: http://www.bitstampede.com/ Twitter: @sheppy ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/20/2014 7:24 AM, Neil wrote: > Ted Mielczarek wrote: > >> our supported use cases are "run from mach" and "run from the test >> package". >> >> > How does a test package differ from an objdir? Do you still need an > entire build environment to run a test package? > The test package is the result of "make package-tests", it's what we run all our tests from on TBPL. The key difference here is that it doesn't expect the Python scripts to have usable defaults, it requires you to specify all of the relevant options. -Ted ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
CHANGE: Next merge date is Tue, Sep 2, 2014
(Cross posting for visibility. Followups to dev-planning please.) The next merge was scheduled for Mon, Sep 1, 2014, which is labour day in Canada and the US. In order to accommodate holiday schedules, the merge will instead occur on Tue, Sep 2, 2014. This change will have no impact on the Firefox Desktop and Firefox for Android 32 releases, which will proceed with release activities as originally scheduled. Lawrence ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On 8/20/2014 3:07 AM, Mike Hommey wrote: Optimized builds have been the default for a while, if not ever[1]. Bug 54828 made optimized builds the default in 2004 right before we released Firefox 1.0. It only took four years to make that decision ;-) --BDS ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Ted Mielczarek wrote: our supported use cases are "run from mach" and "run from the test package". How does a test package differ from an objdir? Do you still need an entire build environment to run a test package? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
On 8/20/2014 4:16 AM, Neil wrote: > Neil wrote: > >> Gregory Szorc wrote: >> >>> Well, mach seems to be working for people doing m-c development. >> >> [Still needs a working build environment, while python runtests.py >> just used to need an objdir.] > > In fact there was a time where I could cross-compile and copy the > objdir onto the target OS and run tests. > I'm sorry if we've made this use case more difficult, but it's not one we're intentionally supporting. It may be that we can keep it working with small changes, but our supported use cases are "run from mach" and "run from the test package". -Ted ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
PSA: TabParents now also live in content processes
Hi all, As a part of the ongoing nested content process work[1][2], the TabParent now could be created in a content process so please don't assume it's in the chrome process. Passing PBrowser around in ipdl protocol is not safe either, because the PBrowser may live in a different protocol tree. If you absolutely need to pass a PBrowser in the protocol, please use PBrowserOrId (bug 1041419). Before the testing infrastructure is ready[3] we could only track this issue by hand. Many bugs has already been filed and tracked in [1]. The feature is currently pref'ed off but reviewers please be aware of this and suggest alternative way to implement if they use TabParent/PBrowser incorrectly. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=nested-oop [2]: https://wiki.mozilla.org/Nested_Content_Processes [3]: https://bugzilla.mozilla.org/show_bug.cgi?id=1038620 Kanru ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Neil wrote: Gregory Szorc wrote: Well, mach seems to be working for people doing m-c development. [Still needs a working build environment, while python runtests.py just used to need an objdir.] In fact there was a time where I could cross-compile and copy the objdir onto the target OS and run tests. -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Running mochitests from a copy of the objdir?
Gregory Szorc wrote: Well, mach seems to be working for people doing m-c development. [Still needs a working build environment, while python runtests.py just used to need an objdir.] -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Experiment with running debug tests less often on mozilla-inbound the week of August 25
On Tue, Aug 19, 2014 at 11:26:42PM -0700, Jeff Gilbert wrote: > I was just going to ask about this. I glanced through the mozconfigs > in the tree for at least Linux debug, but it looks like it only has > --enable-debug, not even -O1. Maybe it's buried somewhere in there, > but I didn't find it with a quick look. > > I took a look at the build log for WinXP debug, and --enable-opt is > only present on the configure line for nspr, whereas --enable-debug is > in a number of other places. Optimized builds have been the default for a while, if not ever[1]. So unless you add an explicit --disable-optimize, you still get an optimized build, whether you use --enable-debug or not. As a matter of fact, we *did* have --disable-optimize in the debug build mozconfigs, but that was removed 3 years ago, in bug 669953. Mike 1. At least, it was the case in the oldest tree we have in mercurial. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
