Hi All, We keep having problems with various of the platforms that our developers are using aren't working. The problems range from that application launching doesn't work, all the way to that the builds don't even compile.
This is a really bad problem and one we have to fix *now*. Long term solutions aren't interesting since this is actively preventing people from writing code and fixing blockers. To fix this, we need to identify which platforms that developers are actually relying on, and then ensure that we have tests that are at least covering the basics of a running build. As much as possible these tests should be running on every single checkin to any of the relevant source trees so that when something breaks, it's detected and immediately backed out. For now, the best tool that we have for doing this is TBPL. Other test suites are simply not run when for example someone checks in a gecko change. So TBPL has to be our main test runner for automated tests. A group of developers that work on various parts of the code (gaia, gecko and gonk) got together this week to try to bootstrap this conversation. This email is intended to broaden that conversation. A few ground rules: * We have very limited resources as far as getting automated tests up and running. So we should be very picky about which platforms that we choose to maintain. And we should prioritize them for maximum bang for our buck. * We can't rely on things that are planning to be fixed a few months out. We need to get in good shape with this *now*. Discussions about what to do long term are off topic for this thread. But please do start separate thread about them. * We should first focus on standing up tests on mozilla-central+gaia-master. When we have that done, we should look at adding support for other branches that are still being actively developed. The platforms that we concluded are currently relevant to support are: ***B2G-desktop out-of-process*** Used by gecko devs that are testing and developing out-of-process APIs. I.e. most of our gecko devs. Need to support both optimized and debug builds. ***B2G-desktop in-process*** Used by 3rd party devs since our B2G-desktop OOP builds aren't working well enough. This is what what the simulator product is built on. Possibly used by some gaia devs. The firefox developer tools should (as of recently?) be working well with this build. Since this is only used by non-gecko and non-gonk developers, we can probably get away without support for debug builds here. ***Firefox + js-API-shims*** This runs gaia inside of a standard Firefox-Desktop build. The gaia build here is somewhat special since it creates non-packaged gaia apps. This is used by gaia developers becuase it provides the best debugger support. Including support for firebug. Since this is only used by non-gecko and non-gonk developers, we can probably get away without support for debug builds here. ***Hardware Nexus4*** This is something that everyone needs. I.e. all types of developers need the ability to build for and test on hardware. We chose Nexus 4 because it's the closest thing we have to a reference platform right now. The real reference platform might be different, but we can adjust to that once the reference platform is picked. Need to support both optimized and debug builds. ***ARM Emulator JB+ICS*** We need this mostly for automated testing. Automated testing on hardware is *very* challenging, and while it's something we're working on improving, the emulator builds is what enables the best testing *now*. Debug builds might run so slow here that it's not really worth testing. Though maybe doing some very basic tests with debug builds would be a good idea. So how do we test these? ***B2G-desktop in-process/out-of-process*** Testing-wise the B2G desktop in-process and out-of-process platforms are basically the same. We already have a few testsuites (mochitest, reftest, gaia UI?) running for in-process B2G-desktop though they do not appear to be enabled on TBPL front page. The gaia integration test suite should be coming online soon. We need to ensure that UI-events are getting properly tested. That means that we need to dispatch the same events as when a developer loads one of these builds and starts interacting with them using a mouse. We need to ensure that that is what the marionette test tool does. Apparently we have some addon that are loaded into these builds which changes the mouse events that gecko dispatches into touch events that gaia will understand. It's unclear if this addon is always added to these builds or if that only happens under some circumstances. We need to ensure that this addon is always loaded. And we need to ensure that when marionette does testing, that the events go through this addon the same way as when a developer is running a build. How well are the automated gaia UI tests covering things like having the basics up and running? Mochitest and reftest mostly test gecko, so it doesn't actually ensure that gaia isn't completely busted. The main thing preventing out-of-process tests from moving forward is that the out-of-process builds simply aren't working well enough. GFX and UI-events have traditionally been the blockers here. We should immediately start fixing this as to get the out-of-process builds tested and onto TBPL. Given the GFX issues, it's important that we have some way of testing that the right pixels are actually ending up on screen. I'm not sure how well the reftests are testing this? *** Hardware *** Automated testing here is unfortunately very challenging. The hardware tends to crap out every few times that we flash it and someone needs to manually pull out the battery to un-crap them. We also don't yet have the ability to test things like telephony/bluetooth or other radio hardware, possibly with exception of WiFi. So all in all, we have very low bang for the buck here. I suggest we for now only automatically test building for hardware, but not actually do automated testing on hardware. Eventually we should improve here, but that's too far out to be in scope for this thread. *** ARM Emulator JB+ICS *** We already have reftests, mochitests, crashtests and WebAPI tests up and running here for ICS. So we need to stand up the same tests for JB. We also need to expand the set of mochitests (and maybe reftests/crashtests?) that we are running. The current set is horribly small. Another problem here is that debug builds are just too damn slow. So we're currently only running optimized builds. This is a big problem because it means that debug builds often don't work on hardware. What can we do to improve this? Two things that could help are: * Create optimized builds that are complied with DEBUG enabled. That way we still catch assertions and we still ensure that all the gecko debug code compiles and runs. We're just not catching compiler issues. Though those are rare and since this isn't builds that are shipping to end users that's probably not a big problem. * We can create a smaller set of tests to run than the full test that we are running for normal optimized builds. ***Firefox + js-API-shims*** We didn't end up talking about testing these builds that much. There is some hope that we can get developer tools for B2G desktop builds into a good enough shape soon enough that spending time on creating test suites for these builds won't be needed. What do people think? If this sounds good we need to get bugs filed for anything that's lacking bugs and find assignees. Getting this stuff in order should be top priority after leo blockers as lack of good testing is slowing us down *a lot*. / Jonas PS. I'm heading out on vacation without internet connection until Sep 9th. But hopefully lots of conversation can happen without me. _______________________________________________ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
