On 03/26/2013 07:10 AM, Ed Morley wrote: > (Please reply to dev.tree-management) > > Until now, the requirements for a new platform/test-suite to be shown in > the default TBPL view have been scattered across many newsgroup > discussions, bugs & IRC conversations, which understandably leads to > surprise when developers working on bringing a new job type into our > buildbot automation are told that it does not yet meet them. > > In order to make the existing criteria more discoverable, I have > documented them at: > https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy
I'm maintaining a somewhat irregular tier-1 job type, the spidermonkey root analysis builds. It makes an interesting example case, since it meets most but not all of your criteria presently, and I think I can weasel out of a few more. #1 yes #2 yes #3 no. The build only runs on mozilla-inbound and try. It could be added to mozilla-central, though it wouldn't change much (because no spidermonkey development lands straight on m-c afaik.) It should be added to the baseline compiler repo. I would argue that it doesn't really matter whether it's enabled on other trees, since it only runs when something under js/src is changed. I don't know if you want to update the wiki page to exempt things that are restricted to a subtree, or if there's an implicit "...but use common sense" override. #4 yes #5 yes but I didn't add anything specific for it in trychooser. I think I'm "good enough" here because it's a build job, not a test job, and the try syntax isn't easily expressible in the UI. (For the record, the build is scheduled whenever js/src is touched and you request a linux64-debug build as part of your request.) I can think of something better, but it seems like it would require more work than it's worth. #6 yes, but could the specifics be documented? Preferably with a mechanism, or at least advice, for handling numerical results. (eg a count of bad somethings, which you don't want to regress.) Given that we *just* fixed an output problem in JS tests, and we had no idea it was a problem, I think having an explicit description of the allowed formats would be useful. #7 yes. Though I worry that explicitly laying all this out could eliminate some needed wiggle room. For example, if a job really had a 5% failure rate, and it stayed that way for 3 months, I don't think you'd be happy. #8 yes #9 yes #10 no. I'll make one. #11 not really supported by mach, but it seems like this is somewhat test-specific anyway. My job is a build with specific configure options followed by a test run with an environment variable set. So it's not incompatible with mach, but I don't think it would do very much good to add it to mach, at least until we get to the point where we require each letter that appears on tbpl to be runnable directly from mach: ./mach tbpl 'SM(r)' ----- It sounds like you're already working on this with tbpl2, but I've thought for a while that we need some just a little below tier 1. Something like a persistent jobname pattern that shows certain jobs based on your group membership, or my preference: just a way in the tbpl UI to indicate failures that you don't need to backout for. There's no reason "visible" == "punishable by backout" if the UI makes it clear which failures are tolerated. Put them in a separate column on the right hand side or something. I like this better than hiding jobs entirely because it makes it harder to ignore flaky tests forever. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
