HI Dmitry, > It is not about hiding. If a problematic test affects other test is will > continue to > affect. The main point here is only about the test, which will be affected. > With unpredictable order, testA may break testB, testC, testD.
The issue is that with a fixed order testA may not break any tests (e.g. because it will become last). Also, tests may not only break but "fix" each other, i.e. testA may make a should-be-failing testB pass. > > But for predictive & fixed order test affected by a failure of testA will be > always testC, and B&D will not be considered flaky because of the > randomized nature of execution. It will help for building at least good > statistics in the TC Bot. I agree that random test order may sound too scary and that file-system dependent order is no good because it's like a fixed one, but varies across platforms. In my experience the best way to tackle execution order of tests is to vary direct and reverse fixed order (say, alphabetical). It gives enough difference to make sure order-dependent tests fail sometimes, and, most importantly, it gives a way to catch them (you can tell that a problem is related to order when a test only fails on direct/reverse runs). I see that surefire already has in a form of runOrder=hourly. I suggest that if we change to something, we change to that. Stan > -----Original Message----- > From: Dmitriy Pavlov <dpav...@apache.org> > Sent: Friday, February 8, 2019 10:19 PM > To: dev@ignite.apache.org > Subject: Re: [DISCUSSION] Fixed test order to reduce flakyness > > I mean changing parameter > http://maven.apache.org/surefire/maven-surefire-plugin/test- > mojo.html#runOrder > which > is fileSystem by default. > > It is not about hiding. If a problematic test affects other test is will > continue to > affect. The main point here is only about the test, which will be affected. > With unpredictable order, testA may break testB, testC, testD. > > But for predictive & fixed order test affected by a failure of testA will be > always testC, and B&D will not be considered flaky because of the > randomized nature of execution. It will help for building at least good > statistics in the TC Bot. > > ср, 6 февр. 2019 г. в 17:37, Eduard Shangareev > <eduard.shangar...@gmail.com > >: > > > Dmitriy, > > > > Please, clarify the idea. > > What do we want to achieve by this? Making tests more stable by hiding > > some issues in our tests/codebase? > > > > > > On Tue, Feb 5, 2019 at 6:57 PM Павлухин Иван <vololo...@gmail.com> > wrote: > > > > > Dmitriy, > > > > > > Sounds like a good idea to me. Problems related to tests isolation > > > are very common in practice. And things can be complicated when > > > order varies. > > > > > > But by the way does the order of Ignite tests vary today? Junit 4 > > > javadocs claims something about "default deterministic order" [1]. > > > > > > [1] > > https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html > > > > > > вт, 5 февр. 2019 г. в 17:40, Dmitriy Pavlov <dpav...@apache.org>: > > > > > > > > Dear Ignite Developers, > > > > > > > > The original idea came from our recent habr.ru post related to > > > > Apache Ignite TeamCity Bot (for Russian native speakers, you can > > > > read an > > > original > > > > > https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976 > > > > ) > > > > > > > > It is a known phenomenon when tests have an influence on each other. > > The > > > > simplest case when Ignite Native persistence is used, and not > > > > properly cleared after a test run. This can make some test failed > afterward. > > > > > > > > So, what if we will set predictable, for example, alphabetical > > > > tests execution order > > > > (maven-surefire-plugin/runOrder/alphabetical). This may have the > > > > following effect: the set of tests failed because of being > > > > affected by the previous run will be constant, will be exactly the same > each run. > > > > > > > > At some point, when we stabilize flaky tests enough, we may select > > random > > > > order, but for now, this solution seems valid to me. > > > > > > > > What do you think? > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > >
