On Mon, Jun 26, 2017 at 7:38 PM, Daniel Krenn <kr...@aon.at> wrote: > On 2017-06-26 13:26, Erik Bray wrote: >> I submitted a PR [1] to the sage patchbot to add a feature I would >> like to have, that re-runs failed tests one or more times. If >> previously failing tests pass on a repeat attempt then the test is >> considered "passing". However, I added a new return status called >> "TestsPassedOnRetry" to distinguish this condition, so that it might >> still stand out if it's happening constantly, which may still indicate >> a real problem. >> >> This will help a lot though with transient failures that occur >> sometimes, especially if tests timeout (e.g. due to the system being >> under some other load). The addition of the new return status would >> require an upgrade to the patchbot server if accepted, which is why >> I'm pointing this out. > > Would it be possible to only retry if there is a failure due to timeout? > > If a doctest has a random-like behavior, then it should not be > considered a pass on a retry...
It would be doable but not as easily. It would require reading and parsing the actual log of the test run which I don't think the patchbot does currently (this would be better if the sage test runner could also save test results in a more machine-friendly file, assuming it doesn't already?) That said, there are other transient failure conditions that don't simply result in a test timeout as reported by the test runner. For example, the pexpect interfaces tend to be particularly fragile under testing in ways that aren't really a problem in real use, just due to overhead associated with I/O and process creation/destruction under Cygwin. I've made some efforts towards improving that and actually don't see such failure very often anymore but it can happen from time to time. Regardless, I think it's helpful to see if a test passes on its own if it didn't pass in the context of the full test suite. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
