How are you running the tests? Just “ant test”? If so the output is all written to stdout so I usually just redirect it somewhere , e.g. “ant test > results 2>&1”….
That file should have the failures _and_ a “reproduce with” line for each failing test. One thing I didn’t mention is that there’s also a significant bit of randomization in the test harness. Different locales are chosen at random, different directory implementations, different timezones, etc… We had one issue that was a JVM issue that only showed up in the Turkish locale for instance that we’d never have found without the randomization. The “reproduce with” line has all that information echoed and will run the test again with all the same bits of randomization. It’s relatively rare for the test to fail reliably even if you use the “reproduce with” line because it’s, well, reproducible. When it is, you’ll see a JIRA raised something like “reproducible test failure” and/or someone will jump on it and fix it. Timing issues: Well, just that. Say a test creates a collection and _assumes_ (no, this isn’t a good practice) that it’ll finish in 5 seconds and it takes 6, then drives on. Oops. Other more subtle issues are just threading issues where some sequence of context switching happens to hit an unanticipated problem. etc. It’s not that we _never_ get reproducible tests, it’s that when we do someone fixes them. There are a _lot_ of tests in the full suite, so if timing-related tests fail 0.1% of the time… You can confirm this yourself pretty easily, just save the output and run the “reproduce with” line. Best, Erick > On Nov 8, 2019, at 8:26 PM, Raphael Bircher <[email protected]> wrote: > > Hi all > > On 2019/11/08 23:56:00, Raphael Bircher <[email protected]> wrote: > >> I was building solr and running the JUnit Tests now. > > The tests was running, but I don't find the testlogs ;-) > > I got two errors with a self builded solr from the head. I've also seen a > Ubuntu machine on Jenkins with two errors. I let now run the test for a > second time. > > [junit4] Execution time total: 1 hour 19 minutes 20 seconds > [junit4] Tests summary: 888 suites (6 ignored), 4543 tests, 2 errors, 212 > ignored (183 assumptions) > > Regards, Raphael > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
