So in general testing software like Pekko is going to be problematic due to it being a distributed/concurrent system i.e. there are determinism (i.e. flaky) test issues. One thing that I did however notice is that in the github actions CI we pass arguments to help alleviate these issues (i.e. https://github.com/apache/incubator-pekko/blob/main/.github/workflows/nightly-builds.yml#L35-L42). The way that ASF release process works where it compels committers to run tests locally has surfaced this, where as in the past the source of truth for tests was either in github actions CI or a in the case of Lightbend private machines/scripts that were specifically setup to test the software before a release.
A final thing to note is that when someone makes a PR against Pekko, tests are only run on the module that has changed (this is achieved via https://github.com/sbt/sbt-pull-request-validator) and most of the flakiness occurs when you try to run all of the tests at once. For this, having a powerful machine helps. On Thu, Jul 6, 2023 at 10:20 AM Claude Warren, Jr <[email protected]> wrote: > My opinion is that if I check out the release code the tests should pass, > or there should be a list of "flaky" tests that are known to have problems > so I can at least verify that the failures are in them. > > On Thu, Jul 6, 2023 at 10:15 AM PJ Fanning <[email protected]> wrote: > > > There are multiple modules. The tests for some modules are passing but > > for other modules they are failing. > > > > With one example: > > > > [error] (remote-tests / Test / test) sbt.TestsFailedException: Tests > > unsuccessful > > > > remote-tests module is in the directory of the same name. > > > > You can use this command just to run the tests in that module > > > > sbt remote-tests/test > > > > Some of the tests can be sensitive to the performance of your machine. > > > > If you continue to have trouble, maybe you could send me your full > > output. I don't think this public mailing list would be a good place > > for that large output but you can email me directly or message it > > using Slack. > > > > > > On Thu, 6 Jul 2023 at 09:06, Claude Warren, Jr > > <[email protected]> wrote: > > > > > > Hello, > > > > > > While testing RC3 I did the following: > > > > > > sbt test > > > > > > the result I got was: > > > [info] Total number of tests run: 628 > > > [info] Suites: completed 181, aborted 0 > > > [info] Tests: succeeded 628, failed 0, canceled 0, ignored 6, pending 2 > > > [info] All tests passed. > > > [error] (remote-tests / Test / test) sbt.TestsFailedException: Tests > > > unsuccessful > > > [error] (persistence / Test / test) sbt.TestsFailedException: Tests > > > unsuccessful > > > [error] (persistence-shared / Test / test) sbt.TestsFailedException: > > Tests > > > unsuccessful > > > [error] (remote / Test / test) sbt.TestsFailedException: Tests > > unsuccessful > > > [error] (stream-tests / Test / test) sbt.TestsFailedException: Tests > > > unsuccessful > > > [error] Total time: 7313 s (02:01:53), completed 5 Jul 2023, 19:10:06 > > > > > > Why the success and yet the failures? > > > > > > Claude > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > -- Matthew de Detrich *Aiven Deutschland GmbH* Immanuelkirchstraße 26, 10405 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen *m:* +491603708037 *w:* aiven.io *e:* [email protected]
