I'm worried about it for two reasons: 1) It sets a precedent The test suite forms a "contract" between us and our users. They expect it to work. If we're prepared to ship software with failing tests, then what is the point of the tests?
2) I was recently contacted by the Homebrew people, and we had a conversation about tests. Apparently, they rely on "make distcheck", and when the tests fail, the build fails. They will not be the only downstream packagers that rely on this functionality. On 29 May 2013 09:42, Dirkjan Ochtman <[email protected]> wrote: > On Tue, May 28, 2013 at 11:55 PM, Noah Slater <[email protected]> wrote: > > Also, is there any place I can look that would alert me to these failures > > before I do the whole release procedure? > > Travis has been failing 140-attachment-comp.t intermittently for a while > now. > > On Wed, May 29, 2013 at 1:36 AM, Noah Slater <[email protected]> wrote: > > I'd sooner disable the failing test and re-roll rc2 than ship a failing > > test suite. > > Seems like a waste of time to me, in this case. The goal of the test > suite is to make sure our software works, not that the test suite is > infallible. I do think it would be particularly great if someone dug > into this test to see if it could be made more reliable for the next > release. > > Cheers, > > Dirkjan > -- NS
