[QGIS-Developer] Flaky tests

2017-11-09 Thread Even Rouault
Hi, I've had a hard time today with a pull request that changed only some python test code, but failures occuring in totally unrelated tests. I had to restart Travis-CI 5 times before it was green. Had similar issue with another PR The various failures I got : 3 times: 359 - PyQgsFil

Re: [QGIS-Developer] Flaky tests

2017-11-09 Thread Tom Chadwin
I don't know if it's related, but I've had real problems over the last two or three weeks with Travis tests for a Python plugin. Tests which use an external resource (a remote WFS) tend to work in the morning (GMT), but fail in the afternoon. I'm also getting segfaults which I hardly ever get (and

Re: [QGIS-Developer] Flaky tests

2017-11-09 Thread Alessandro Pasotti
PyQgsLocator is failing 50% of the times for me. Recently I've been lucky with the others. On Thu, Nov 9, 2017 at 4:24 PM, Even Rouault wrote: > Hi, > > > > I've had a hard time today with a pull request that changed only some > python test code, but failures occuring in totally unrelated test

Re: [QGIS-Developer] Flaky tests

2017-11-09 Thread Matthias Kuhn
Hi Even, Yes, despite things settling again I experienced the same/similar issues. I can confirm the issues with the spatialite and the locator test. I am in favor of disabling (parts of) these, since they do more damage than good in their current state. This will be much appreciated To avoid th

[QGIS-Developer] Flaky tests and Travis

2019-03-11 Thread Nyall Dawson
Hi all, For a long time now we've been plagued by intermittently failing tests on Travis, which are making the whole QGIS development experience quite painful. I propose that we take an absolute hard line approach from now and disable all tests which are causing false positive failures. I've star

Re: [QGIS-Developer] Flaky tests and Travis

2019-03-11 Thread Denis Rouzaud
I am also in favor of disabling them but maybe keep them running and have them as expected failure? If ones write some code related to the corresponding tests , they might still provide some valuable info. Is this possible? If not I'd still be in favor of disabling them. Cheers Denis On Mon, 11

Re: [QGIS-Developer] Flaky tests and Travis

2019-03-11 Thread Nyall Dawson
On Tue, 12 Mar 2019 at 07:59, Denis Rouzaud wrote: > > I am also in favor of disabling them but maybe keep them running and have > them as expected failure? That sounds preferable -- but I'm not (personally) sure if it's possible. You know the CI setup better than I do, can you see a way to do t

Re: [QGIS-Developer] Flaky tests and Travis

2019-03-12 Thread Matthias Kuhn
On 3/12/19 12:28 AM, Nyall Dawson wrote: > On Tue, 12 Mar 2019 at 07:59, Denis Rouzaud wrote: >> I am also in favor of disabling them but maybe keep them running and have >> them as expected failure? > That sounds preferable -- but I'm not (personally) sure if it's > possible. You know the CI set

Re: [QGIS-Developer] Flaky tests and Travis

2019-03-12 Thread Alessandro Pasotti
I understand we haven't much of a choice: +1 for disabling. I wonder if there is a way to make it more evident for developers that they need to run disabled tests locally by hand before making a PR. It happened to me once or twice that I made a change that I knew was covered by a test case, let t

Re: [QGIS-Developer] Flaky tests and Travis

2019-03-12 Thread Denis Rouzaud
Another approach would be to have a tag in the PR such as enable_alltests and it would run on all the tests. I am not sure we have access to labels in Travis (we didn't have access to title a year ago), but that would be a nice hack. On Tue, 12 Mar 2019, 10:42 Alessandro Pasotti, wrote: > I unde