PS: I'm also happy if this class of tests only emails the responsible person, and not the pusher.
Robby On Mon, Aug 8, 2011 at 10:59 AM, Robby Findler <ro...@eecs.northwestern.edu> wrote: > I like the two-times-in-a-row thought. > > FWIW, please try to avoid race conditions of the second kind. > > I think the drracket test suites are special because they fail > not-so-often and I don't actually know how to fix them. If either of > those weren't true then I'd say they should just not run in drdr. (So > the race-condition/using the same file thing fails this test.) > > Robby > > On Mon, Aug 8, 2011 at 10:56 AM, Vincent St-Amour <stamo...@ccs.neu.edu> > wrote: >> >> I love DrDr, but there's a small thing that annoys me about it. >> >> Some tests are prone to intermittent failures. For example, some >> benchmarks need to create a file, and several benchmarks share the >> same file, which leads to race conditions. Similarly, some DrRacket >> tests sometimes fail for focus reasons. >> >> So, whenever someone pushes, they may get failures from these tests, >> then have go look at the actual errors, and try to figure out if they >> actually broke something or not. >> >> (Or, they ignore these failures, which is bad.) >> >> Here are two potential solutions. Let's assume that I just pushed >> something, and a test started failing. >> >> - Have DrDr send me email for every push about the broken test for as >> long as it fails. If I get email more than once, it's likely that I >> actually broke something. If I only get email once, the problem went >> away on its own, and was likely an intermittent failure. >> >> - Have the possiblity to flag some tests as intermittent (something >> like `drdr:random'), and only report failures for these tests if >> they fail twice in a row. This would reduce the amount of noise, >> since I expect most of these tests to pass most of the time. Actual >> breakage would still be detected, since it's unlikely that such >> failures would go away on their own. Detection would happen one push >> late, but that shouldn't be too much of an issue. >> >> Or, maybe only notify the pusher after two failures in a row, but >> notify the responsible person right away. >> >> Any thoughts? >> >> Vincent >> _________________________________________________ >> For list-related administrative tasks: >> http://lists.racket-lang.org/listinfo/dev >> > _________________________________________________ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev