I love DrDr, but there's a small thing that annoys me about it. Some tests are prone to intermittent failures. For example, some benchmarks need to create a file, and several benchmarks share the same file, which leads to race conditions. Similarly, some DrRacket tests sometimes fail for focus reasons.
So, whenever someone pushes, they may get failures from these tests, then have go look at the actual errors, and try to figure out if they actually broke something or not. (Or, they ignore these failures, which is bad.) Here are two potential solutions. Let's assume that I just pushed something, and a test started failing. - Have DrDr send me email for every push about the broken test for as long as it fails. If I get email more than once, it's likely that I actually broke something. If I only get email once, the problem went away on its own, and was likely an intermittent failure. - Have the possiblity to flag some tests as intermittent (something like `drdr:random'), and only report failures for these tests if they fail twice in a row. This would reduce the amount of noise, since I expect most of these tests to pass most of the time. Actual breakage would still be detected, since it's unlikely that such failures would go away on their own. Detection would happen one push late, but that shouldn't be too much of an issue. Or, maybe only notify the pusher after two failures in a row, but notify the responsible person right away. Any thoughts? Vincent _________________________________________________ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev