Dear ZooKeeper team,
Andor just reminded me of this JDK 11 vs. 8 discussion, for which we did not reach a resolution. Do we want to make a move for the 3.7.0 release? The original proposal, by Christopher, can be found here: https://mail-archives.apache.org/mod_mbox/zookeeper-dev/202010.mbox/%3ccal5zq9z2tnfawjz7etxpf91qmrovuwv+khnxgpsd_msekfp...@mail.gmail.com%3e I am explicitly replying to Christopher's nice and useful summary, which you will find directly below. An archived copy can be found here: https://mail-archives.apache.org/mod_mbox/zookeeper-dev/202010.mbox/%3cCAL5zq9ao9-4S9bmzxaYJM8n8=aolxidz-0zohunvnayqei0...@mail.gmail.com%3e It contains three proposals, to which we can add the trivial one: 1. Compile/test with JDK 11, support JDK 8 "passively"; 2. Compile/test with JDK 11, test client only with JDK 8 (requires test suite adaptations); 3. Move to JDK 11 fully; 4. Postpone any change to 3.8 (or 4.0). Flavio wrote: > [Christopher] mentioned a PMC vote, and I don't think this should be a > closed vote, independent of how the conversation goes. Do we have a better idea of what we want? Or should we organize a vote? Cheers, -D --8<--------------- original message ------------->8--- Christopher <ctubb...@apache.org> writes: > I'm happy that this discussion has been so lively! I just want to > emphasize a few things: > > I really do understand the desire to continue to support Java 8... I > get it. But all the conversations around this seem based on what > people are doing *today*. But, ZK 3.7 is *tomorrow's* version... a > *future* release... so it should be based more on reasonable > expectations for users in the future, and less based on what is > happening today. I suspect *most* people today are still using 3.4 > anyway (it was just so stable for so long...), but that shouldn't mean > the developers should hold back development on 3.5 and 3.6, any more > than today's users of 3.5/3.6 should hold back 3.7. > > Some of the opinions expressed in this discussion seem to propose a > scenario where users are going to be updating to "bleeding edge" > versions of ZooKeeper, but are going to insist on using Java 8. > Personally, I find this to be implausible. In my experience, people > either upgrade everything as soon as they are able to, or they upgrade > each thing individually, only when they are forced to. The first group > will be happy to move to Java 11 and ZK 3.7. The second group will > probably avoid 3.7 anyway, and are fine sticking with 3.6, but if they > had to update to 3.7, they'd also be fine updating to Java 11 if they > had to in order to use 3.7. I can't imagine the scenario where people > are eagerly choosing to upgrade to ZK 3.7, but miserly insisting on > using Java 8. Perhaps that scenario exists, but it's hard for me to > imagine. Even so, my proposal would still support even that group of > people. > > I think there are now effectively three proposals being discussed in > this thread: > > 1. (Christopher's original proposal) passively support Java 8 at > runtime by making JDK 11 the minimum requirement to build and test. > This scenario involves continuing to fix bugs, as they are discovered > and reported, that affect JDK 8, but passively, rather than > proactively. This proposal does *not* drop Java 8 support, but merely > de-emphasizes it in development of what will be 3.7 in the future, and > drops the requirement to do dedicated testing with Java 8. I think > this is low risk, because it is very unlikely that the ZK devs would > introduce a bug that would affect only Java 8 and the compiler > wouldn't catch it... because the cross-compilation features of newer > JDKs are really good. > > 2. (Enrico's alternate proposal) this variation of my proposal would > involve continuing to proactively support Java 8 by creating a > dedicated testing suite to test client code on Java 8. I think this is > a good option, but since it involves a significantly higher amount of > work than option 1, I think the cost-benefit analysis would show this > to be not worth the effort. Also, if it were implemented, it would > need to be done carefully to avoid requiring developers to have > concurrently installed both Java 8 and Java 11 in order to perform a > build, because requiring Java 8 at build time while developing would > be worse than we have today. > > 3. (Andor's preference) move to JDK 11 fully. This would provide no > support, passive or active, for Java 8 in ZK 3.7. To be honest, this > is my personal favorite, and is the simplest to implement and > communicate clearly to end users in release notes. The only reason I > proposed a passive support of Java 8 instead of this is because I was > trying to seek a compromise from the start. But, I think by far, this > is the best option for the next *future* release of ZK. If you wanted > to make the change even more visible to users, the version could even > be bumped to 4.0. > > If this were to come to a VOTE by the PMC, in order to make a final > decision, I would recommend they vote on option 3, and then if that > fails, vote on option 1, and if that fails, keep things the way they > are (because option 2 is more work). > > Christopher > > P.S. as for Hadoop on Java 11... I've been running Hadoop 3 on JDK 11 > and it works just fine there (as long as you add the missing runtime > jar for javax.activation:javax.activation-api:1.2.0 to its class path, > but that was fixed in Hadoop 3.3). > > On Wed, Oct 21, 2020 at 5:42 PM Tamas Penzes > <tam...@cloudera.com.invalid> wrote: >> >> Hi All, >> >> I've just talked with a Hadoop/HDFS developer who told me what I guessed. >> With Hadoop3 they have just dropped JDK7 support, dropping JDK8 would mean >> a release of Hadoop4. >> Since HADOOP-15338 is finished, they test with JDK8 and JDK11 both. As of >> today most of the Hadoop users are still on Hadoop2, he doesn't expect >> Hadoop4 soon. >> As many Apache components depend on Hadoop and ZooKeeper they won't hurry >> to JDK11 until they have to (they will probably go one-by-one very slowly), >> which means if Hadoop stays on JDK8, they would use the last ZK version >> which works on JDK8. >> Do we want a ZK 3.4 again? >> >> Regards, Tamaas >> >> >> On Wed, Oct 21, 2020 at 11:23 PM Tamas Penzes <tam...@cloudera.com> wrote: >> >> > Hi All, >> > >> > Just to add my two cents. >> > >> > Upgrading to JDK11 looks inevitable sooner or later and I would definitely >> > not wait until 2030 or 2026 when the extended support of JDK8 ends. >> > But on the other side I have to agree with Enrico and Patrick that far too >> > many users are tied to JDK8 yet (not because they want to use JDK8, but >> > because they have to), some of them are components of the Hadoop ecosystem, >> > which would be a loss to tie them to 3.6.x for years. >> > Do we know the state of Hadoop? It builds with JDK8 at the moment, but do >> > we know what are their plans to go to JDK11? >> > When they move, we should move too, but I don't think it would be wise to >> > do it earlier. >> > >> > Christopher's option looks like the golden path, but it needs some >> > investment on the testing side as Enrico pointed it out. >> > Could we agree on it as a compromise? >> > >> > Regards, Tamaas >> > >> > On Wed, Oct 21, 2020 at 7:11 PM Brent <brentwritesc...@gmail.com> wrote: >> > >> >> I think I was reacting to Enrico's earlier comment of: >> >> >> >> " ZooKeeper client is used by tons of users and unfortunately many >> >> projects >> >> are still on JDK8, if we move ZooKeeper to JDK11 the risk is to block >> >> users >> >> from the adoption, that is that we will see the world to stay on 3.6.x and >> >> we will have again a long lived release line, like 3.4." >> >> >> >> It's a matter of whether or not a long-lived release line is >> >> desirable/undesirable. If everyone is OK keeping 3.6.x up-to-date >> >> security/patch-wise (if not feature-wise) for the next N years, then >> >> that's >> >> a potentially valid approach. I interpreted that comment as "a long-lived >> >> release line is undesirable", but no one explicitly said that, I just read >> >> it that way. >> >> >> >> ~Brent >> >> >> >> On Wed, Oct 21, 2020 at 9:49 AM Patrick Hunt <ph...@apache.org> wrote: >> >> >> >> > On Wed, Oct 21, 2020 at 9:03 AM Andor Molnar <an...@apache.org> wrote: >> >> > >> >> > > As far as I know Hbase, Solr and Kafka are already Java 11 ready. >> >> > > >> >> > > IMHO contributors of those projects should also put efforts into >> >> moving >> >> > > forward. >> >> > > >> >> > > We’re not saying that you _have_ to move to Java 11. >> >> > > Staying on Java 8? No problem, 3.6 is for you. >> >> > > Want the fancy new features of 3.7? Work on it on your side too. >> >> > > >> >> > > >> >> > ppl want things like security fixes. I believe the highlighted downside >> >> is >> >> > that we would need to continue to maintain 3.6.x rather than allowing >> >> > users, and ourselves, to focus on trunk for production -"64 percent" >> >> would >> >> > be blocked. >> >> > >> >> > Patrick >> >> > >> >> > >> >> > > Andor >> >> > > >> >> > > >> >> > > >> >> > > > On 2020. Oct 21., at 17:52, Enrico Olivelli <eolive...@gmail.com> >> >> > wrote: >> >> > > > >> >> > > > Il giorno mer 21 ott 2020 alle ore 17:49 Andor Molnar < >> >> > an...@apache.org> >> >> > > ha >> >> > > > scritto: >> >> > > > >> >> > > >> Correct me if I'm wrong, but Oracle gets paid for extended support. >> >> > > >> Java 8 support until 2030 is not free of charge. >> >> > > >> >> >> > > >> "ZK may end up with a lot of users potentially locking themselves >> >> to >> >> > > >> 3.6.x for a while as Enrico mentioned." >> >> > > >> >> >> > > >> That's true. What's the downside of that which we should invest in >> >> to >> >> > > >> avoid? >> >> > > >> >> >> > > > >> >> > > > I see ZooKeeper used in many many projects, all of the >> >> > > > HBase/Pulsar/Kafka/Solr ecosystem... >> >> > > > they will have to move to JDK11 in order to move to the new ZK >> >> version >> >> > > > so probably they will stay on ZK 3.6 >> >> > > > >> >> > > > Probably with Java 17 LTS released the cards will change on the >> >> table >> >> > > > >> >> > > > Enrico >> >> > > > >> >> > > > >> >> > > > >> >> > > >> >> >> > > >> Andor >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> On Wed, 2020-10-21 at 08:03 -0700, Brent wrote: >> >> > > >>> As a slightly different consideration, if you look at the >> >> Long-Term >> >> > > >>> Support >> >> > > >>> (LTS) roadmaps for Java, currently Java 8 is set to have full >> >> support >> >> > > >>> until >> >> > > >>> 2030 from Oracle and at least 2026 from OpenJDK & Corretto: >> >> > > >>> >> >> > > >>> >> >> > https://www.oracle.com/java/technologies/java-se-support-roadmap.html >> >> > > >>> https://en.wikipedia.org/wiki/Java_version_history >> >> > > >>> >> >> > > >>> My guess is that a number of companies are still heavily invested >> >> at >> >> > > >>> the >> >> > > >>> Java 8 level (I know a few) and with that kind of time horizon, >> >> they >> >> > > >>> have >> >> > > >>> no real motivation to upgrade for quite a while. If the recent >> >> > > >>> Python 2 >> >> > > >>> deprecation is anything to go by, they won't do it until they have >> >> > > >>> to. >> >> > > >>> >> >> > > >>> Not saying Java 8 isn't *very* old (2014 release it seems like?) >> >> and >> >> > > >>> I'm >> >> > > >>> not invested heavily either way, but this might suggest that ZK >> >> may >> >> > > >>> end up >> >> > > >>> with a lot of users potentially locking themselves to 3.6.x for a >> >> > > >>> while as >> >> > > >>> Enrico mentioned. >> >> > > >>> >> >> > > >>> (Not a major contributor, but wanted to chime in since I just had >> >> > > >>> this >> >> > > >>> conversation with a bunch of people professionally recently) >> >> > > >>> >> >> > > >>> Brent >> >> > > >>> >> >> > > >>> On Wed, Oct 21, 2020 at 2:07 AM Andor Molnar <an...@apache.org> >> >> > > >>> wrote: >> >> > > >>> >> >> > > >>>> Thanks for the summary. >> >> > > >>>> >> >> > > >>>> I still vote for option 1). Move 3.7.0 to JDK 11 fully. Other >> >> > > >>>> projects >> >> > > >>>> will upgrade once they’re JDK11 compliant, otherwise they will >> >> stay >> >> > > >>>> on 3.5 >> >> > > >>>> or 3.6. Both version are quite recent in ZooKeeper-terms, we >> >> > > >>>> already >> >> > > >>>> planned big changes for 3.7.0 and JDK 11 could be one of them. >> >> > > >>>> >> >> > > >>>> Don’t put extra burden on the ZK community to help others staying >> >> > > >>>> on >> >> > > >>>> ancient Java versions. >> >> > > >>>> >> >> > > >>>> Andor >> >> > > >>>> >> >> > > >>>> >> >> > > >>>> >> >> > > >>>>> On 2020. Oct 21., at 10:57, Enrico Olivelli < >> >> eolive...@gmail.com> >> >> > > >>>>> wrote: >> >> > > >>>>> >> >> > > >>>>> Let me recap >> >> > > >>>>> - Christopher is proposing to move to JDK11 >> >> > > >>>>> - ZooKeeper client and server are bundled and coded and tested >> >> > > >>>>> together >> >> > > >>>> in >> >> > > >>>>> zookeeper-server >> >> > > >>>>> - Enrico is concerned about the need of testing ZooKeeper client >> >> > > >>>>> on JDK8 >> >> > > >>>>> (not a problem to move the server to JDK11) >> >> > > >>>>> >> >> > > >>>>> ZooKeeper client is used by tons of users and unfortunately many >> >> > > >>>>> projects >> >> > > >>>>> are still on JDK8, if we move ZooKeeper to JDK11 the risk is to >> >> > > >>>>> block >> >> > > >>>> users >> >> > > >>>>> from the adoption, >> >> > > >>>>> that is that we will see the world to stay on 3.6.x and we will >> >> > > >>>>> have >> >> > > >>>> again >> >> > > >>>>> a long lived release line, like 3.4. >> >> > > >>>>> >> >> > > >>>>> Testing the client on JDK8 would be possible if we create some >> >> > > >>>>> kind of >> >> > > >>>>> additional module with system tests, then we can start the >> >> server >> >> > > >>>>> on >> >> > > >>>> docker >> >> > > >>>>> on JDK11+ and start a client on JDK8 >> >> > > >>>>> with Maven toolchain it should possible to run surefire tests >> >> > > >>>>> using a >> >> > > >>>>> separate JVM. >> >> > > >>>>> >> >> > > >>>>> So in my vision 2 options: >> >> > > >>>>> 1) fully JDK11 - drop JDK8 at all >> >> > > >>>>> 2) build with JDK11 - server only on JDK11 - add system tests >> >> > > >>>>> with docker >> >> > > >>>>> and toolchains that ensure the ZooKeeper client (and all >> >> > > >>>>> dependencies) >> >> > > >>>>> still work on JDK8 >> >> > > >>>>> >> >> > > >>>>> From my point of view about the ZooKeeper ecosystem option 2) >> >> > > >>>>> will be far >> >> > > >>>>> better, but we need resources to work on a new test suite. >> >> > > >>>>> >> >> > > >>>>> Enrico >> >> > > >>>>> >> >> > > >>>>> >> >> > > >>>>> Il giorno mer 21 ott 2020 alle ore 10:43 Andor Molnar < >> >> > > >>>>> an...@apache.org> >> >> > > >>>> ha >> >> > > >>>>> scritto: >> >> > > >>>>> >> >> > > >>>>>> Tamas, Enrico, >> >> > > >>>>>> >> >> > > >>>>>> Sorry I don’t follow. Why do we have to test the client with >> >> > > >>>>>> JDK 8 in >> >> > > >>>>>> version 3.7.0? >> >> > > >>>>>> >> >> > > >>>>>> Andor >> >> > > >>>>>> >> >> > > >>>>>> >> >> > > >>>>>> >> >> > > >>>>>> >> >> > > >>>>>>> On 2020. Oct 20., at 22:29, Tamas Penzes < >> >> > > >>>>>>> tam...@cloudera.com.INVALID> >> >> > > >>>>>> wrote: >> >> > > >>>>>>> Hi Enrico, >> >> > > >>>>>>> >> >> > > >>>>>>> Separating ZooKeeper client and server is a huge work, but we >> >> > > >>>>>>> might not >> >> > > >>>>>>> need it. >> >> > > >>>>>>> As you mentioned we have to test ZK client with Java 8, what >> >> > > >>>>>>> about >> >> > > >>>>>>> separating only the test cases which we need to run with >> >> > > >>>>>>> Java8 too? >> >> > > >>>>>>> In Curator we have the ZK compatibility tests where we run a >> >> > > >>>>>>> limited >> >> > > >>>>>> amount >> >> > > >>>>>>> of Curator's jUnit tests with a different ZK version. >> >> > > >>>>>>> We might be able to do the same here, tag tests which are >> >> > > >>>>>>> testing ZK >> >> > > >>>>>> client >> >> > > >>>>>>> and run them separately with Java 8. The only limitation is >> >> > > >>>>>>> that these >> >> > > >>>>>>> tests must stay JDK8 compatible. >> >> > > >>>>>>> But from the tags we will see which ones are those. >> >> > > >>>>>>> >> >> > > >>>>>>> What do you think? >> >> > > >>>>>>> >> >> > > >>>>>>> Regards, Tamaas >> >> > > >>>>>>> >> >> > > >>>>>>> On Sat, Oct 17, 2020 at 7:45 AM Enrico Olivelli < >> >> > > >>>>>>> eolive...@gmail.com> >> >> > > >>>>>> wrote: >> >> > > >>>>>>>> Christopher >> >> > > >>>>>>>> I appreciate your idea and I also moved lots of my projects >> >> > > >>>>>>>> to work >> >> > > >>>> the >> >> > > >>>>>> way >> >> > > >>>>>>>> you are suggesting. >> >> > > >>>>>>>> >> >> > > >>>>>>>> >> >> > > >>>>>>>> We must run tests using real jdk8 to test the Zookeeper >> >> > > >>>>>>>> client. We >> >> > > >>>> must >> >> > > >>>>>>>> ensure that Zookeeper works well, especially while dealing >> >> > > >>>>>>>> with >> >> > > >>>> security >> >> > > >>>>>>>> stuff. >> >> > > >>>>>>>> Currently the client is in the same module of the server >> >> > > >>>>>>>> and it will >> >> > > >>>>>> take a >> >> > > >>>>>>>> good (huge) amount of work to separate them >> >> > > >>>>>>>> >> >> > > >>>>>>>> >> >> > > >>>>>>>> Enrico >> >> > > >>>>>>>> >> >> > > >>>>>>>> Il Ven 16 Ott 2020, 23:25 Christopher <ctubb...@apache.org> >> >> > > >>>>>>>> ha >> >> > > >>>> scritto: >> >> > > >>>>>>>>> Hi ZK Devs, >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> With recent advancements in Java (since Java 9), it is >> >> > > >>>>>>>>> now generally >> >> > > >>>>>>>>> no longer necessary to require that software be developed >> >> > > >>>>>>>>> on an older >> >> > > >>>>>>>>> JDK in order to have confidence that it will run on the >> >> > > >>>>>>>>> older version >> >> > > >>>>>>>>> of Java. This is because, as of Java 9, all JDK releases >> >> > > >>>>>>>>> have better >> >> > > >>>>>>>>> support for cross-compilation to older Java versions. >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> What this means is that developers can confidently make >> >> > > >>>>>>>>> the build >> >> > > >>>>>>>>> requirements for a project higher than the Java version >> >> > > >>>>>>>>> that will >> >> > > >>>>>>>>> actually be supported at runtime. >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> In fact, ZooKeeper already supports the necessary flags >> >> > > >>>>>>>>> in its Maven >> >> > > >>>>>>>>> build configuration to ensure that it uses JDK 8 >> >> > > >>>>>>>>> compliance when >> >> > > >>>>>>>>> building on a newer JDK (I added this way back in >> >> > > >>>>>>>>> ZOOKEEPER-3739 / >> >> > > >>>>>>>>> https://github.com/apache/zookeeper/pull/1269) >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> So, I propose that we make JDK 11 the new minimum version >> >> > > >>>>>>>>> to *build* >> >> > > >>>>>>>>> ZooKeeper with. This would not change the runtime >> >> > > >>>>>>>>> requirement, which >> >> > > >>>>>>>>> would remain at JDK 8. >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> The only necessary change to make this happen would be to >> >> > > >>>>>>>>> add the >> >> > > >>>>>>>>> minimum Java version to the maven-enforcer-plugin (like >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> >> >> > > >>>> >> >> > > >> >> >> > > >> >> > >> >> > https://github.com/apache/accumulo/blob/438f0efd34ef9d200bc8c7ecdd11d5dedb146519/pom.xml#L1162-L1164 >> >> > > >>>>>>>>> ) >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> This would allow ZooKeeper to to streamline its >> >> > > >>>>>>>>> development process a >> >> > > >>>>>>>>> little bit by reducing the amount of CI testing that is >> >> > > >>>>>>>>> done as part >> >> > > >>>>>>>>> of the build. In other words, we can drop the CI builds >> >> > > >>>>>>>>> for JDK 8, >> >> > > >>>>>>>>> which saves on build resources and time. The return on >> >> > > >>>>>>>>> investment is >> >> > > >>>>>>>>> so low for the JDK 8 builds anyway, because of the >> >> > > >>>>>>>>> improved >> >> > > >>>>>>>>> cross-compilation in newer JDKs. So, there's not much >> >> > > >>>>>>>>> value in >> >> > > >>>>>>>>> building on JDK 8 anyway. >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> Of course, I am only recommending this for *new* release >> >> > > >>>>>>>>> lines, >> >> > > >>>>>>>>> starting with ZooKeeper 3.7.0/master branch, because I >> >> > > >>>>>>>>> would not want >> >> > > >>>>>>>>> to change expectations for users who will build their own >> >> > > >>>>>>>>> 3.5 and 3.6 >> >> > > >>>>>>>>> versions as they continue to have patch versions >> >> > > >>>>>>>>> released. >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> What do you think? >> >> > > >>>>>>>>> >> >> > > >>>>>>>>> Kind Regards, >> >> > > >>>>>>>>> Christopher >> >> > > >>>>>>>>> >> >> > > >> >> >> > > >> >> >> > > >> >> > > >> >> > >> >> >> >
