The CI sometimes fails due to heisenbugs, which of course we should fix but mostly someone just kicks the CI off again and the test passes and the code is integrated. So these heisenbugs slow down development when the CI fails, and also we are not gaining any systematic statistics on where the heisenbugs come from.
So for a while I've thought it would be useful for the CI to automatically rerun a failed test to bypass the heisenbugs and at the same time log them to a central location for anaylsis. So I think such a feature could also suit this situation. After you correct the code and resume execution to the end, the test is automatically rerun and turns green, or fails again bringing you straight back to the debugger. Actually that last itself would be a useful feature for a tight cycle dev/testing a Test. cheers -ben On Sat, Nov 11, 2017 at 3:45 AM, Richard Sargent < richard.sarg...@gemtalksystems.com> wrote: > I hear you and I understand your pain. > > However, if you corrected the problem, not by a code change, but by > playing in the object's inspector or otherwise causing its internal state > to change, and then resumed from the debugger, would you still claim the > method was successful and should be coloured green? > > The only thing we can claim is that there were or were not further errors > in the test. Colour it red if there were, but you cannot honestly colour it > green. The code doesn't actually work. > > > On Fri, Nov 10, 2017 at 11:29 AM, Tim Mackinnon <tim@testit.works> wrote: > >> My specific usecase is from a pragmatic TDD perspective - failing test, >> in the debugger you fix the test and press proceed - expecting green. >> Getting red, and then immediately running again to get red takes away from >> our story of love coding and loving your debugger - and even Cassie me to >> mistrust the tools. >> >> I get the idea that you can jiffy in the debugger and cause a false pass >> - but I feel you are penalised for doing it right at the moment. >> >> Of course these tests will get run again, but I like the idea that if I >> did it right, it should recognise this, not incur an extra click and moment >> of doubt. >> >> A button to rerun the whole lot, or automatically rerun, or just run it >> would work for me. >> >> Tim >> >> Sent from my iPhone >> >> On 10 Nov 2017, at 17:56, Richard Sargent <richard.sargent@gemtalksystem >> s.com> wrote: >> >> That would be fine. >> The point is that, without running the test in its entirety, from start >> to finish, without interruption, error, or failure, one cannot claim >> success. >> >> On Fri, Nov 10, 2017 at 9:34 AM, Sean P. DeNigris <s...@clipperadams.com> >> wrote: >> >>> Richard Sargent wrote >>> > The only reliable conclusion one can make from such an interrupted run >>> is >>> > whether it failed again. So, it would be possible to determine that the >>> > test should be coloured red, but it is impossible to *reliably* claim >>> that >>> > the test should be coloured green. >>> >>> What if we ran the test again as if from the browser/runner before >>> setting >>> the icon? >>> >>> >>> >>> ----- >>> Cheers, >>> Sean >>> -- >>> Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html >>> >>> >> >
