On Sat, Sep 1, 2018 at 11:56 AM Nyall Dawson <nyall.daw...@gmail.com> wrote:
> Hi devs, > > Just raising the question of what we should do about the constant WFS > test failures we get on Travis. I'd estimate 1 in 3 builds fails > because of WFS related tests hanging. > > I'm very reluctant to disable all these tests, because: > > 1. WFS is super important. > 2. I think there may be a real issue here - I've got at least one > customer who is having WFS issues with 3.2 and master. BUT: on the > other hand Travis has always been flaky with any test which uses > threads, regardless of which area of code it's from. So it could just > be Travis playing up again, in which case we'd need to disable these > tests like we do most of the other thread-related tests... > > The current situation is basically unworkable. Sooooo.... ideas? > > Nyall > Sorry I do not have any solution, the only consideration that comes up to my mind is that if sum up all the time wasted by developers by unrelated/random Travis failures we would probably be very badly surprised (as I usually say Travis is basically a very slow binary entropy generator). Talking about the tests, WFS tests, and (even for different reasons) http-based integration tests (some auth tests, qgsfiledownloader, many server OWS tests) are apparently more fragile than others, mainly because they depend on (sometimes external) http services, but they have proven in the past to be very effective in spotting out regressions on very important feature likes WMS, WFS etc., some of them are also important because they ensure that QGIS client can talk to its server component. The very bad things about disabling tests are: - their development costed a lot of time and efforts and disabling them is not an incentive for the developers to write more tests [1] - the tests are obviously important to prevent regressions - before disabling a test we should make 100% sure that we are not overlooking a real bug (or what are the tests written for in the first place?) So, my recommendation - not very useful in the short term I'm afraid - would be to start exploring other more reliable options for our CI, even if they are more expensive. For the time being, as a temporary solution, there are no other options than disabling though. Maybe a good idea would be to allocate some money specifically for the tests (a dedicated grant, a dedicated hackfest?) just an idea... [1] of course sometimes it would just mean to write "better" and more robust tests, but I suspect that this is not generally the case. -- Alessandro Pasotti w3: www.itopen.it
_______________________________________________ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
