Dmitriy, Initially I thought that the topic is about order of test METHODS execution. You said about "fileSystem". I suppose file system order have no meaning for methods but rather for test classes. Do other possible values of the mentioned parameter affect order of methods execution?
For convenience I remind a way how to define method order I am aware of and wrote before [1]. [1] https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html пт, 8 февр. 2019 г. в 22:18, Dmitriy Pavlov <dpav...@apache.org>: > > I mean changing parameter > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#runOrder > which > is fileSystem by default. > > It is not about hiding. If a problematic test affects other test is will > continue to affect. The main point here is only about the test, which will > be affected. With unpredictable order, testA may break testB, testC, testD. > > But for predictive & fixed order test affected by a failure of testA will > be always testC, and B&D will not be considered flaky because of the > randomized nature of execution. It will help for building at least good > statistics in the TC Bot. > > ср, 6 февр. 2019 г. в 17:37, Eduard Shangareev <eduard.shangar...@gmail.com > >: > > > Dmitriy, > > > > Please, clarify the idea. > > What do we want to achieve by this? Making tests more stable by hiding some > > issues in our tests/codebase? > > > > > > On Tue, Feb 5, 2019 at 6:57 PM Павлухин Иван <vololo...@gmail.com> wrote: > > > > > Dmitriy, > > > > > > Sounds like a good idea to me. Problems related to tests isolation are > > > very common in practice. And things can be complicated when order > > > varies. > > > > > > But by the way does the order of Ignite tests vary today? Junit 4 > > > javadocs claims something about "default deterministic order" [1]. > > > > > > [1] > > https://junit.org/junit4/javadoc/latest/org/junit/FixMethodOrder.html > > > > > > вт, 5 февр. 2019 г. в 17:40, Dmitriy Pavlov <dpav...@apache.org>: > > > > > > > > Dear Ignite Developers, > > > > > > > > The original idea came from our recent habr.ru post related to Apache > > > > Ignite TeamCity Bot (for Russian native speakers, you can read an > > > original > > > > https://habr.com/ru/company/sberbank/blog/436070/#comment_19616976 ) > > > > > > > > It is a known phenomenon when tests have an influence on each other. > > The > > > > simplest case when Ignite Native persistence is used, and not properly > > > > cleared after a test run. This can make some test failed afterward. > > > > > > > > So, what if we will set predictable, for example, alphabetical tests > > > > execution order (maven-surefire-plugin/runOrder/alphabetical). This may > > > > have the following effect: the set of tests failed because of being > > > > affected by the previous run will be constant, will be exactly the same > > > > each run. > > > > > > > > At some point, when we stabilize flaky tests enough, we may select > > random > > > > order, but for now, this solution seems valid to me. > > > > > > > > What do you think? > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > -- Best regards, Ivan Pavlukhin
