On Thu, 22 Dec 2016, Florian Schlichting wrote: > On Wed, Dec 21, 2016 at 11:04:34PM +0100, Santiago Vila wrote: > > So, if establishing a threshold is the only way to achieve that for > > example Bug #843038 in "elki" is upgraded to serious again, so be it, > > but as I said, it would be a pity if we invest our time trying to > > estimate probabilities instead of actually ensuring that packages > > build all the time as a shared goal with reproducible builds. > > Santiago, I think you're doing important work drawing attention to a > class of bugs that has long gone unnoticed. > > However, you seem to suggest that simply disabling occasionally failing > tests will improve the quality of Debian.
Note: We are not always speaking about occasionally failing tests. Sometimes the tests fail in more than 90% of cases when built on single-CPU machines. We can't and shouldn't make multi-core to be part of the build-essential definition. Neither implicitly or explicitly. > I'm not so sure about that. > Those tests might provide important coverage for high-level functions > our users actually care about, so I'd want those tests to get fixed > instead of disabled. Sure. I would like those tests to be fixed as well, but you seem to suggest that every time a test fails, the test must be fixed or otherwise the quality of the package will suffer. That's not always the case. Very often, the test is wrongly designed. Funny example: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838828 The test above does not make the package better, it makes the package worse, because it fails gratuitously. What definitely will not improve the quality of Debian is to downgrade the bug and do absolutely nothing, because then everybody loses, the test will keep failing and the package will keep FTBFS for everybody who tries. Not so funny example: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=841098#78 This is one of the reasons I think these bugs should be RC regardless of probability. > But timing issues are hard to debug or even > reproduce on developers' machines, upstreams may need convincing, while > RC-bugs mere weeks before the freeze create enormous time pressure. Well, I started filing these bugs several months ago, certainly not weeks before the freeze. In general, most people fix these bugs, accept that FTBFS bugs are serious, and treat them as such. > How can we ensure those tests aren't just disabled and forgotten? We really can't, because we are all volunteers and you can't ensure that everybody fix things in the way you would like them to be fixed. If you have personal interest in the issue, you can always subscribe to the bugs and help maintainers to find the "right" fix. My list of (open) bugs of this kind is here: https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=ftbfs-randomly;users=sanv...@debian.org However, let me also answer your question with another question: How can we ensure that packages build ok and without failure, which is what Release Policy says? Certainly, not by downgrading to wishlist, as I've seen some maintainers to do, or by saying "I don't care anymore". Thanks.
