Hi all, I like the idea of extending our test coverage and the fact people are putting effort on this topic.
I don't really mind having randomized or fixed CI checks as long as: 1. it doesn't break too often due to the environment; 2. it is capable to pinpoint regressions/problems on specific commits; 3. it doesn't take overly long to run; 4. failures are reproducible. I am a bit skeptical with respect to randomization mostly about point 1 but we can test it and see how it goes. Are other projects using this? How does it look so far? >From my perspective, I wouldn't opt for everything to be randomized but rather amend the fixed CI checks. Moreover, ideally I would like errors in randomized checks to not turn up as red builds but rather orange, with the semantics being let's log the issue but not block commits to master till the issue is resolved. Not sure if that's possible, just ideas. Also maybe we would like to keep randomization only on master (running it once per day) and not on every PR mostly to save some resources. Regarding tests on different timezones I find it a very good idea. Maybe, it would be better to do that on a per test basis rather than on the whole build but given that our unit tests run fast (~3-5min) I guess we can afford running everything on different zones. If we wanted to make timezone tests more focused we could possibly utilize JUnit5 extensions for running certain tests on multiple timezones (ideas for the future). Best, Stamatis On Sat, Oct 2, 2021 at 8:53 PM Vladimir Sitnikov < sitnikov.vladi...@gmail.com> wrote: > >I don’t think we should do randomized CI > > Then please incorporate: > a) Testing with "-XX:+UnlockExperimentalVMOptions -XX:hashCode=2" to ensure > the code does not rely on Object#hashCode uniqueness > b) different OSes > c) Different JDK vendors, versions and JIT implementations (e.g. OpenJ9, > Microsoft, Coretto) > d) interference of a, b, c, and Guava versions > > Vladimir >
