Having had to debug test failures for projects which previously did
(simple) randomized testing, I am also not a fan of randomization. If
there is a way to deterministically re-run a specific configuration,
this goes a long way.
I am more worried about debt in maintaining the complexity of a
high-cardinality test matrix. If we are randomly choosing one "diagonal"
(to steal Julian's term), we may miss a breakage from some code change
in the same way we might catch one.
I'm of the mindset that we cannot possibly test all variants. My opinion
is that we should be mindful of the incompatibilities which we have
observed with Avatica over the years and use that to define what is a
"meaningful" thing to test.
That said (going through Vlad's list), I don't think testing various
OS's is important. We should pick the JDK (or JDKs) which our devs/users
rely on today -- for me, that's Oracle or OpenJDK. I would also not want
to test against more than a few versions of Guava. Same goes for
timezones (picking two or three). For all of these, as long as we don't
get into hours-long precommit duration and we have the CPU in order to
run the tests, I'm happy with whatever.
On 10/2/21 6:03 PM, Stamatis Zampetakis wrote:
Hi all,
I like the idea of extending our test coverage and the fact people are
putting effort on this topic.
I don't really mind having randomized or fixed CI checks as long as:
1. it doesn't break too often due to the environment;
2. it is capable to pinpoint regressions/problems on specific commits;
3. it doesn't take overly long to run;
4. failures are reproducible.
I am a bit skeptical with respect to randomization mostly about point 1 but
we can test it and see how it goes.
Are other projects using this? How does it look so far?
From my perspective, I wouldn't opt for everything to be randomized but
rather amend the fixed CI checks.
Moreover, ideally I would like errors in randomized checks to not turn up
as red builds but rather orange, with the semantics being let's log the
issue but not block commits to master till the issue is resolved. Not sure
if that's possible, just ideas.
Also maybe we would like to keep randomization only on master (running it
once per day) and not on every PR mostly to save some resources.
Regarding tests on different timezones I find it a very good idea.
Maybe, it would be better to do that on a per test basis rather than on the
whole build but given that our unit tests run fast (~3-5min) I guess we can
afford running everything on different zones.
If we wanted to make timezone tests more focused we could possibly utilize
JUnit5 extensions for running certain tests on multiple timezones (ideas
for the future).
Best,
Stamatis
On Sat, Oct 2, 2021 at 8:53 PM Vladimir Sitnikov <
sitnikov.vladi...@gmail.com> wrote:
I don’t think we should do randomized CI
Then please incorporate:
a) Testing with "-XX:+UnlockExperimentalVMOptions -XX:hashCode=2" to ensure
the code does not rely on Object#hashCode uniqueness
b) different OSes
c) Different JDK vendors, versions and JIT implementations (e.g. OpenJ9,
Microsoft, Coretto)
d) interference of a, b, c, and Guava versions
Vladimir
