Hi Erick, * On 2019/11/09 15:15:03, Erick Erickson <[email protected]> wrote: > How are you running the tests? Just “ant test”? If so the output is all > written to stdout so I usually just redirect it somewhere , e.g. “ant test > > results 2>&1”….
Yea, I run it with ant test. So the output is the log from the cmd, ok. Is there an other possibility to run the tests? > > That file should have the failures _and_ a “reproduce with” line for each > failing test. One thing I didn’t mention is that there’s also a significant > bit of randomization in the test harness. Different locales are chosen at > random, different directory implementations, different timezones, etc… We had > one issue that was a JVM issue that only showed up in the Turkish locale for > instance that we’d never have found without the randomization. The “reproduce > with” line has all that information echoed and will run the test again with > all the same bits of randomization. Ok, this sounds interesting > > It’s relatively rare for the test to fail reliably even if you use the > “reproduce with” line because it’s, well, reproducible. When it is, you’ll > see a JIRA raised something like “reproducible test failure” and/or someone > will jump on it and fix it. So just reproducible issues go into jira? In my experience, it makes sense in some case, to write an issue für a ireproducible bug. So you can collect all data on one place. Sometimes this helps to track the bugs down. > > Timing issues: Well, just that. Say a test creates a collection and _assumes_ > (no, this isn’t a good practice) that it’ll finish in 5 seconds and it takes > 6, then drives on. Oops. Other more subtle issues are just threading issues > where some sequence of context switching happens to hit an unanticipated > problem. etc. > > It’s not that we _never_ get reproducible tests, it’s that when we do someone > fixes them. There are a _lot_ of tests in the full suite, so if > timing-related tests fail 0.1% of the time… > > You can confirm this yourself pretty easily, just save the output and run the > “reproduce with” line. > Ok, I will go now into the source, and see, what some test does and trying to get a test without errors. Last time I have had 1 error. If it continues like this, I will have 0 next time ;-) Regards, Raphael > Best, > Erick > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
