Some thoughts on Yetus: The only limit on what we can do is the load that we put on the ASF infrastructure, but that is a rather hard limit. The Yetus precommit check does not by default run the unit tests, but I hacked the personality to add the unit tests to it to preserve the old behaviour.
We have at least two dimensions that we can extend the testing to, one is the HBase version used, the other is the JDK version. The Hbase Yetus scripts and personality already has solutions to these issues, but I have cut some corners when adopting the script to Yetus, partly to mimic the old precommit behaviour and partly to simplify bringing up the tests. If we want to go in this direction, then we need to undo most of the simplifications that I've done while porting the HBase stuff to Phoenix: * Decouple running the IT suite from the standard Yetus precommit job to keep resource use in check * Restore the JDK multiversion and multi-Hadoop (in our case multi-HBase) functionality in the Yetus personality and scripts. * Use the intended Yetus mechanisms to run the IT suite and replace the current post-commit scripts with Yetus-based nightlies. I do not expect to have the time to work on the above until all of the 5.1 releases are done, and the related @dayjob tasks wind down, but would gladly give help / pointers to anyone willing to tackle these issues. On Mon, Oct 26, 2020 at 8:17 PM Viraj Jasani <vjas...@apache.org> wrote: > Based on a recent incident, I thought of bumping this thread. > > I agree with Chinmay on using JDK7 for our 4.x builds. Now with > Yetus support (after last discussion on this thread), maybe we > can consider supporting MULTIJDK environment (JDK7+JDK8) too > with parallel execution of precommit checks stage? > > > On 2020/05/11 22:41:51, Chinmay Kulkarni <chinmayskulka...@gmail.com> > wrote: > > Sorry to slightly digress, but based on these discussions, shouldn't we > be > > setting JAVA_HOME to point to JDK 7 in all our builds? Especially based > on > > Andrew's comment above: > > " > > > > > > *Java 7 is not forwards compatible with Java 8, especially with respect > > toJRE internals. What that means is if you compile something with Java > > 8,even if you are not using Java 8 language features, you won't be able > > torun it on a Java 7 runtime.*" > > > > Our 4.x build > > < > https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-4.x-matrix/configure > > > > is configured with java 1.8: > > [image: 4.x.png] > > In our master build > > < > https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-master-matrix/configure > >, > > we don't even specify JAVA_HOME, so essentially it picks up the latest > > version. Precommit gets JAVA_HOME set from $PHOENIX/dev/jenkensEnv.sh. > > > > There have been issues where master builds failed because they used JDK > 11 ( > > > https://builds.apache.org/job/Phoenix-master-matrix/HBASE_PROFILE=2.1/71/consoleFull > ), > > search for "JAVA VERSION" in the log file. The same build does not fail > if > > run with JDK 8. > > > > Also, doesn't this mean we are already compiling with Java 8 (at least in > > the 4.x branches) or am I missing something? > > > > < > https://builds.apache.org/view/M-R/view/Phoenix/job/Phoenix-4.x-matrix/configure > > > > > > On Wed, Apr 22, 2020 at 8:11 PM Ankit Singhal <ankitsingha...@gmail.com> > > wrote: > > > > > Just linking a discussion we had one and a half years back on the same > [1], > > > considering nothing has changed since then because Java 7 was EOLed in > July > > > 2015 > > > and Phoenix 5.0 was also out, it majorly comes to the point of the > > > inconvenience of working > > > on old code style and extra efforts required to create patches for each > > > branch. > > > > > > Let's say if we decide to upgrade to Java8(Option 3), don't we require > the > > > following changes? > > > > > > * Major version upgrade from 4.x-HBase-1.x to 5.x-HBase-1.x:- As > upgrading > > > 4.x branches > > > to Java8 breaks the compatibility with HBase and Java runtime, we need > to > > > ensure that > > > we adhere to dependency compatibility between the minor releases as it > is > > > expected that > > > Phoenix minor upgrade should be just a server jar drop and a restart > (even > > > though > > > we don't explicit covers Java runtime in our backward compatibility > [2] as > > > HBase does[3] > > > but people are used to it now) > > > > > > * Release Notes and convenience libraries:- Though we would say that > it is > > > compatible with > > > HBase version 1.x but require a Java8, so we need to be explicit with > our > > > convenience libraries > > > as well , like append "jdk8" to a name or something similar > > > (phoenix-server-HBase-1.x-jdk8.jar). > > > And also provide clarity on the version upgrade > > > > > > * Avoiding issues during runtime:- Though JAVA8 and JAVA7 are said to > be > > > binary compatible > > > and are expected to be fine in Java8 runtime but it has been called out > > > there are rare instances > > > which can cause issues due to some incompatibilities between JRE and > > > JDK[4]. (As Andrew also > > > called out and might have observed the same) > > > > > > > > > Though I agreed with Istvan that creating multiple patches and dealing > with > > > change in code style every time > > > we switch branches has put additional load on the contributor but > IMHO, we > > > should wait for > > > HBase-1.x to upgrade Java before we do it to avoid the some of the > issues > > > listed above related to Option 3. > > > > > > Option2 would be preferred at the time when we decide on whether we > want to > > > diverge from feature > > > parity in our major releases and we do only limited fixes for 4.x > branch. > > > So basically I'm also in favor of Option 1 (continuing Java 7 for > HBase-1.x > > > and code style as much possible for 5.x). > > > > > > [1] > > > > > > > https://lists.apache.org/thread.html/970db0b5cb0560c49654e450aafb8fb759596384cefe4f02808e80cc%40%3Cdev.phoenix.apache.org%3E > > > [2]http://phoenix.apache.org/upgrading.html > > > [3] https://hbase.apache.org/book.html#hbase.versioning > > > [4] > > > > > > > https://www.oracle.com/technetwork/java/javase/8-compatibility-guide-2156366.html#A999387 > > > > > > Regards, > > > Ankit Singhal > > > > > > > > > -- > > Chinmay Kulkarni > > > -- *István Tóth* | Staff Software Engineer st...@cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> <https://www.cloudera.com/> ------------------------------
