> Efforts have been made in the past to clean things up, but the reality is that it is very hard to make tests reliable in an asynchronous framework and in general it is almost impossible to accommodate for all possible running environments.
I really want to highlight this takeaway, if we were so strict with other comparable Apache TLP (i.e. Kafka, Spark, Cassandra, Flink etc etc) then no release be ever made. While there is merit in discussing how bespoke the testing for Pekko is vs other "typical" ASF projects, if the expectation is that you can just run sbt test on a local laptop and the tests to reliably pass then thats not going to happen any time soon. As we speak I am adding in documentation in various places (i.e. https://cwiki.apache.org/confluence/display/PEKKO/Testing and https://github.com/apache/incubator-pekko/pull/469) for techniques on how to handle this (i.e. testQuick) and I will also document what Johannes said right now regarding the timing factor/timing tests specifically for Pekko core. On Thu, Jul 6, 2023 at 11:28 AM Johannes Rudolph <[email protected]> wrote: > The main test suite that is run nightly is run with this command: > > sbt \ > -Dpekko.cluster.assert=on \ > -Dpekko.log.timestamps=true \ > -Dpekko.test.timefactor=2 \ > -Dpekko.actor.testkit.typed.timefactor=2 \ > -Dpekko.test.tags.exclude=gh-exclude,timing \ > -Dpekko.test.multi-in-test=false \ > clean "+~ ${{ matrix.scalaVersion }} test" checkTestsHaveRun > > > https://github.com/apache/incubator-pekko/blob/88bf6329f193eedd45091f4f9a515943bd8ecb23/.github/workflows/nightly-builds.yml#L168-L175 > > Unfortunately, the amount of flaky tests is high, so the important bits are > > -Dpekko.test.timefactor=2 > -Dpekko.test.tags.exclude=gh-exclude,timing > > which makes timing in tests more lenient and also excludes some notorious > ones. > > Efforts have been made in the past to clean things up, but the reality > is that it is very hard to make tests reliable in an asynchronous > framework and in general it is almost impossible to accommodate for > all possible running environments. > > Johannes > > On Thu, Jul 6, 2023 at 10:30 AM Matthew de Detrich > <[email protected]> wrote: > > > > So in general testing software like Pekko is going to be problematic due > to > > it being a distributed/concurrent system i.e. there are determinism (i.e. > > flaky) test issues. One thing that I did however notice is that in the > > github actions CI we pass arguments to help alleviate these issues (i.e. > > > https://github.com/apache/incubator-pekko/blob/main/.github/workflows/nightly-builds.yml#L35-L42 > ). > > The way that ASF release process works where it compels committers to run > > tests locally has surfaced this, where as in the past the source of truth > > for tests was either in github actions CI or a in the case of Lightbend > > private machines/scripts that were specifically setup to test the > software > > before a release. > > > > A final thing to note is that when someone makes a PR against Pekko, > tests > > are only run on the module that has changed (this is achieved via > > https://github.com/sbt/sbt-pull-request-validator) and most of the > > flakiness occurs when you try to run all of the tests at once. For this, > > having a powerful machine helps. > > > > On Thu, Jul 6, 2023 at 10:20 AM Claude Warren, Jr > > <[email protected]> wrote: > > > > > My opinion is that if I check out the release code the tests should > pass, > > > or there should be a list of "flaky" tests that are known to have > problems > > > so I can at least verify that the failures are in them. > > > > > > On Thu, Jul 6, 2023 at 10:15 AM PJ Fanning <[email protected]> > wrote: > > > > > > > There are multiple modules. The tests for some modules are passing > but > > > > for other modules they are failing. > > > > > > > > With one example: > > > > > > > > [error] (remote-tests / Test / test) sbt.TestsFailedException: Tests > > > > unsuccessful > > > > > > > > remote-tests module is in the directory of the same name. > > > > > > > > You can use this command just to run the tests in that module > > > > > > > > sbt remote-tests/test > > > > > > > > Some of the tests can be sensitive to the performance of your > machine. > > > > > > > > If you continue to have trouble, maybe you could send me your full > > > > output. I don't think this public mailing list would be a good place > > > > for that large output but you can email me directly or message it > > > > using Slack. > > > > > > > > > > > > On Thu, 6 Jul 2023 at 09:06, Claude Warren, Jr > > > > <[email protected]> wrote: > > > > > > > > > > Hello, > > > > > > > > > > While testing RC3 I did the following: > > > > > > > > > > sbt test > > > > > > > > > > the result I got was: > > > > > [info] Total number of tests run: 628 > > > > > [info] Suites: completed 181, aborted 0 > > > > > [info] Tests: succeeded 628, failed 0, canceled 0, ignored 6, > pending 2 > > > > > [info] All tests passed. > > > > > [error] (remote-tests / Test / test) sbt.TestsFailedException: > Tests > > > > > unsuccessful > > > > > [error] (persistence / Test / test) sbt.TestsFailedException: Tests > > > > > unsuccessful > > > > > [error] (persistence-shared / Test / test) > sbt.TestsFailedException: > > > > Tests > > > > > unsuccessful > > > > > [error] (remote / Test / test) sbt.TestsFailedException: Tests > > > > unsuccessful > > > > > [error] (stream-tests / Test / test) sbt.TestsFailedException: > Tests > > > > > unsuccessful > > > > > [error] Total time: 7313 s (02:01:53), completed 5 Jul 2023, > 19:10:06 > > > > > > > > > > Why the success and yet the failures? > > > > > > > > > > Claude > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > -- > > > > Matthew de Detrich > > > > *Aiven Deutschland GmbH* > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > *m:* +491603708037 > > > > *w:* aiven.io *e:* [email protected] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Matthew de Detrich *Aiven Deutschland GmbH* Immanuelkirchstraße 26, 10405 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen *m:* +491603708037 *w:* aiven.io *e:* [email protected]
