Hi all, If I might chime in as a zookeeper user (in multiple products) and a follower of the project I think the drop of Java8 support (official and/or unofficial) could be a big mistake.
From my own company point of view we already support Java11 in all our applications so we are not directly impacted (and we have upgrade path for older versions to provide to our customers). My worries resides in the (high) probability of a userbase fragmentation: in the recent past Zookeeper development picked up speed thanks to a bunch of new committers and PMCs after a period of mostly maintenance focused works, but the number of active committers and PMCs is still very low for a project like this. I foresee the risk of spreading thin the resources of the project if we force the userbase to stick to an older version and, in turn, we are forced to backport many issue to the 3.6 branch. Alessandro Luccaroni Platform Manager @ Diennea - MagNews Tel.: (+39) 0546 066100 Int. 924 - Mob.: (+39) 393 7273519 Viale G.Marconi 30/14 - 48018 Faenza (RA) - Italy -----Messaggio originale----- Da: Christopher <ctubb...@apache.org> Inviato: giovedì 22 ottobre 2020 05:21 A: dev@zookeeper.apache.org Oggetto: Re: [DISCUSS][PROPOSAL] Require JDK 11 to build for 3.7 I'm happy that this discussion has been so lively! I just want to emphasize a few things: I really do understand the desire to continue to support Java 8... I get it. But all the conversations around this seem based on what people are doing *today*. But, ZK 3.7 is *tomorrow's* version... a *future* release... so it should be based more on reasonable expectations for users in the future, and less based on what is happening today. I suspect *most* people today are still using 3.4 anyway (it was just so stable for so long...), but that shouldn't mean the developers should hold back development on 3.5 and 3.6, any more than today's users of 3.5/3.6 should hold back 3.7. Some of the opinions expressed in this discussion seem to propose a scenario where users are going to be updating to "bleeding edge" versions of ZooKeeper, but are going to insist on using Java 8. Personally, I find this to be implausible. In my experience, people either upgrade everything as soon as they are able to, or they upgrade each thing individually, only when they are forced to. The first group will be happy to move to Java 11 and ZK 3.7. The second group will probably avoid 3.7 anyway, and are fine sticking with 3.6, but if they had to update to 3.7, they'd also be fine updating to Java 11 if they had to in order to use 3.7. I can't imagine the scenario where people are eagerly choosing to upgrade to ZK 3.7, but miserly insisting on using Java 8. Perhaps that scenario exists, but it's hard for me to imagine. Even so, my proposal would still support even that group of people. I think there are now effectively three proposals being discussed in this thread: 1. (Christopher's original proposal) passively support Java 8 at runtime by making JDK 11 the minimum requirement to build and test. This scenario involves continuing to fix bugs, as they are discovered and reported, that affect JDK 8, but passively, rather than proactively. This proposal does *not* drop Java 8 support, but merely de-emphasizes it in development of what will be 3.7 in the future, and drops the requirement to do dedicated testing with Java 8. I think this is low risk, because it is very unlikely that the ZK devs would introduce a bug that would affect only Java 8 and the compiler wouldn't catch it... because the cross-compilation features of newer JDKs are really good. 2. (Enrico's alternate proposal) this variation of my proposal would involve continuing to proactively support Java 8 by creating a dedicated testing suite to test client code on Java 8. I think this is a good option, but since it involves a significantly higher amount of work than option 1, I think the cost-benefit analysis would show this to be not worth the effort. Also, if it were implemented, it would need to be done carefully to avoid requiring developers to have concurrently installed both Java 8 and Java 11 in order to perform a build, because requiring Java 8 at build time while developing would be worse than we have today. 3. (Andor's preference) move to JDK 11 fully. This would provide no support, passive or active, for Java 8 in ZK 3.7. To be honest, this is my personal favorite, and is the simplest to implement and communicate clearly to end users in release notes. The only reason I proposed a passive support of Java 8 instead of this is because I was trying to seek a compromise from the start. But, I think by far, this is the best option for the next *future* release of ZK. If you wanted to make the change even more visible to users, the version could even be bumped to 4.0. If this were to come to a VOTE by the PMC, in order to make a final decision, I would recommend they vote on option 3, and then if that fails, vote on option 1, and if that fails, keep things the way they are (because option 2 is more work). Christopher P.S. as for Hadoop on Java 11... I've been running Hadoop 3 on JDK 11 and it works just fine there (as long as you add the missing runtime jar for javax.activation:javax.activation-api:1.2.0 to its class path, but that was fixed in Hadoop 3.3). On Wed, Oct 21, 2020 at 5:42 PM Tamas Penzes <tam...@cloudera.com.invalid> wrote: > > Hi All, > > I've just talked with a Hadoop/HDFS developer who told me what I guessed. > With Hadoop3 they have just dropped JDK7 support, dropping JDK8 would > mean a release of Hadoop4. > Since HADOOP-15338 is finished, they test with JDK8 and JDK11 both. As > of today most of the Hadoop users are still on Hadoop2, he doesn't > expect > Hadoop4 soon. > As many Apache components depend on Hadoop and ZooKeeper they won't > hurry to JDK11 until they have to (they will probably go one-by-one > very slowly), which means if Hadoop stays on JDK8, they would use the > last ZK version which works on JDK8. > Do we want a ZK 3.4 again? > > Regards, Tamaas > > > On Wed, Oct 21, 2020 at 11:23 PM Tamas Penzes <tam...@cloudera.com> wrote: > > > Hi All, > > > > Just to add my two cents. > > > > Upgrading to JDK11 looks inevitable sooner or later and I would > > definitely not wait until 2030 or 2026 when the extended support of JDK8 > > ends. > > But on the other side I have to agree with Enrico and Patrick that > > far too many users are tied to JDK8 yet (not because they want to > > use JDK8, but because they have to), some of them are components of > > the Hadoop ecosystem, which would be a loss to tie them to 3.6.x for years. > > Do we know the state of Hadoop? It builds with JDK8 at the moment, > > but do we know what are their plans to go to JDK11? > > When they move, we should move too, but I don't think it would be > > wise to do it earlier. > > > > Christopher's option looks like the golden path, but it needs some > > investment on the testing side as Enrico pointed it out. > > Could we agree on it as a compromise? > > > > Regards, Tamaas > > > > On Wed, Oct 21, 2020 at 7:11 PM Brent <brentwritesc...@gmail.com> wrote: > > > >> I think I was reacting to Enrico's earlier comment of: > >> > >> " ZooKeeper client is used by tons of users and unfortunately many > >> projects are still on JDK8, if we move ZooKeeper to JDK11 the risk > >> is to block users from the adoption, that is that we will see the > >> world to stay on 3.6.x and we will have again a long lived release > >> line, like 3.4." > >> > >> It's a matter of whether or not a long-lived release line is > >> desirable/undesirable. If everyone is OK keeping 3.6.x up-to-date > >> security/patch-wise (if not feature-wise) for the next N years, > >> then that's a potentially valid approach. I interpreted that > >> comment as "a long-lived release line is undesirable", but no one > >> explicitly said that, I just read it that way. > >> > >> ~Brent > >> > >> On Wed, Oct 21, 2020 at 9:49 AM Patrick Hunt <ph...@apache.org> wrote: > >> > >> > On Wed, Oct 21, 2020 at 9:03 AM Andor Molnar <an...@apache.org> wrote: > >> > > >> > > As far as I know Hbase, Solr and Kafka are already Java 11 ready. > >> > > > >> > > IMHO contributors of those projects should also put efforts > >> > > into > >> moving > >> > > forward. > >> > > > >> > > We’re not saying that you _have_ to move to Java 11. > >> > > Staying on Java 8? No problem, 3.6 is for you. > >> > > Want the fancy new features of 3.7? Work on it on your side too. > >> > > > >> > > > >> > ppl want things like security fixes. I believe the highlighted > >> > downside > >> is > >> > that we would need to continue to maintain 3.6.x rather than > >> > allowing users, and ourselves, to focus on trunk for production -"64 > >> > percent" > >> would > >> > be blocked. > >> > > >> > Patrick > >> > > >> > > >> > > Andor > >> > > > >> > > > >> > > > >> > > > On 2020. Oct 21., at 17:52, Enrico Olivelli > >> > > > <eolive...@gmail.com> > >> > wrote: > >> > > > > >> > > > Il giorno mer 21 ott 2020 alle ore 17:49 Andor Molnar < > >> > an...@apache.org> > >> > > ha > >> > > > scritto: > >> > > > > >> > > >> Correct me if I'm wrong, but Oracle gets paid for extended support. > >> > > >> Java 8 support until 2030 is not free of charge. > >> > > >> > >> > > >> "ZK may end up with a lot of users potentially locking > >> > > >> themselves > >> to > >> > > >> 3.6.x for a while as Enrico mentioned." > >> > > >> > >> > > >> That's true. What's the downside of that which we should > >> > > >> invest in > >> to > >> > > >> avoid? > >> > > >> > >> > > > > >> > > > I see ZooKeeper used in many many projects, all of the > >> > > > HBase/Pulsar/Kafka/Solr ecosystem... > >> > > > they will have to move to JDK11 in order to move to the new > >> > > > ZK > >> version > >> > > > so probably they will stay on ZK 3.6 > >> > > > > >> > > > Probably with Java 17 LTS released the cards will change on > >> > > > the > >> table > >> > > > > >> > > > Enrico > >> > > > > >> > > > > >> > > > > >> > > >> > >> > > >> Andor > >> > > >> > >> > > >> > >> > > >> > >> > > >> On Wed, 2020-10-21 at 08:03 -0700, Brent wrote: > >> > > >>> As a slightly different consideration, if you look at the > >> Long-Term > >> > > >>> Support > >> > > >>> (LTS) roadmaps for Java, currently Java 8 is set to have > >> > > >>> full > >> support > >> > > >>> until > >> > > >>> 2030 from Oracle and at least 2026 from OpenJDK & Corretto: > >> > > >>> > >> > > >>> > >> > https://www.oracle.com/java/technologies/java-se-support-roadmap. > >> > html > >> > > >>> https://en.wikipedia.org/wiki/Java_version_history > >> > > >>> > >> > > >>> My guess is that a number of companies are still heavily > >> > > >>> invested > >> at > >> > > >>> the > >> > > >>> Java 8 level (I know a few) and with that kind of time > >> > > >>> horizon, > >> they > >> > > >>> have > >> > > >>> no real motivation to upgrade for quite a while. If the > >> > > >>> recent Python 2 deprecation is anything to go by, they > >> > > >>> won't do it until they have to. > >> > > >>> > >> > > >>> Not saying Java 8 isn't *very* old (2014 release it seems > >> > > >>> like?) > >> and > >> > > >>> I'm > >> > > >>> not invested heavily either way, but this might suggest > >> > > >>> that ZK > >> may > >> > > >>> end up > >> > > >>> with a lot of users potentially locking themselves to 3.6.x > >> > > >>> for a while as Enrico mentioned. > >> > > >>> > >> > > >>> (Not a major contributor, but wanted to chime in since I > >> > > >>> just had this conversation with a bunch of people > >> > > >>> professionally recently) > >> > > >>> > >> > > >>> Brent > >> > > >>> > >> > > >>> On Wed, Oct 21, 2020 at 2:07 AM Andor Molnar > >> > > >>> <an...@apache.org> > >> > > >>> wrote: > >> > > >>> > >> > > >>>> Thanks for the summary. > >> > > >>>> > >> > > >>>> I still vote for option 1). Move 3.7.0 to JDK 11 fully. > >> > > >>>> Other projects will upgrade once they’re JDK11 compliant, > >> > > >>>> otherwise they will > >> stay > >> > > >>>> on 3.5 > >> > > >>>> or 3.6. Both version are quite recent in ZooKeeper-terms, > >> > > >>>> we already planned big changes for 3.7.0 and JDK 11 could > >> > > >>>> be one of them. > >> > > >>>> > >> > > >>>> Don’t put extra burden on the ZK community to help others > >> > > >>>> staying on ancient Java versions. > >> > > >>>> > >> > > >>>> Andor > >> > > >>>> > >> > > >>>> > >> > > >>>> > >> > > >>>>> On 2020. Oct 21., at 10:57, Enrico Olivelli < > >> eolive...@gmail.com> > >> > > >>>>> wrote: > >> > > >>>>> > >> > > >>>>> Let me recap > >> > > >>>>> - Christopher is proposing to move to JDK11 > >> > > >>>>> - ZooKeeper client and server are bundled and coded and > >> > > >>>>> tested together > >> > > >>>> in > >> > > >>>>> zookeeper-server > >> > > >>>>> - Enrico is concerned about the need of testing ZooKeeper > >> > > >>>>> client on JDK8 (not a problem to move the server to > >> > > >>>>> JDK11) > >> > > >>>>> > >> > > >>>>> ZooKeeper client is used by tons of users and > >> > > >>>>> unfortunately many projects are still on JDK8, if we move > >> > > >>>>> ZooKeeper to JDK11 the risk is to block > >> > > >>>> users > >> > > >>>>> from the adoption, > >> > > >>>>> that is that we will see the world to stay on 3.6.x and > >> > > >>>>> we will have > >> > > >>>> again > >> > > >>>>> a long lived release line, like 3.4. > >> > > >>>>> > >> > > >>>>> Testing the client on JDK8 would be possible if we create > >> > > >>>>> some kind of additional module with system tests, then we > >> > > >>>>> can start the > >> server > >> > > >>>>> on > >> > > >>>> docker > >> > > >>>>> on JDK11+ and start a client on JDK8 with Maven toolchain > >> > > >>>>> it should possible to run surefire tests using a separate > >> > > >>>>> JVM. > >> > > >>>>> > >> > > >>>>> So in my vision 2 options: > >> > > >>>>> 1) fully JDK11 - drop JDK8 at all > >> > > >>>>> 2) build with JDK11 - server only on JDK11 - add system > >> > > >>>>> tests with docker and toolchains that ensure the > >> > > >>>>> ZooKeeper client (and all > >> > > >>>>> dependencies) > >> > > >>>>> still work on JDK8 > >> > > >>>>> > >> > > >>>>> From my point of view about the ZooKeeper ecosystem > >> > > >>>>> option 2) will be far better, but we need resources to > >> > > >>>>> work on a new test suite. > >> > > >>>>> > >> > > >>>>> Enrico > >> > > >>>>> > >> > > >>>>> > >> > > >>>>> Il giorno mer 21 ott 2020 alle ore 10:43 Andor Molnar < > >> > > >>>>> an...@apache.org> > >> > > >>>> ha > >> > > >>>>> scritto: > >> > > >>>>> > >> > > >>>>>> Tamas, Enrico, > >> > > >>>>>> > >> > > >>>>>> Sorry I don’t follow. Why do we have to test the client > >> > > >>>>>> with JDK 8 in version 3.7.0? > >> > > >>>>>> > >> > > >>>>>> Andor > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>> > >> > > >>>>>>> On 2020. Oct 20., at 22:29, Tamas Penzes < > >> > > >>>>>>> tam...@cloudera.com.INVALID> > >> > > >>>>>> wrote: > >> > > >>>>>>> Hi Enrico, > >> > > >>>>>>> > >> > > >>>>>>> Separating ZooKeeper client and server is a huge work, > >> > > >>>>>>> but we might not need it. > >> > > >>>>>>> As you mentioned we have to test ZK client with Java 8, > >> > > >>>>>>> what about separating only the test cases which we need > >> > > >>>>>>> to run with > >> > > >>>>>>> Java8 too? > >> > > >>>>>>> In Curator we have the ZK compatibility tests where we > >> > > >>>>>>> run a limited > >> > > >>>>>> amount > >> > > >>>>>>> of Curator's jUnit tests with a different ZK version. > >> > > >>>>>>> We might be able to do the same here, tag tests which > >> > > >>>>>>> are testing ZK > >> > > >>>>>> client > >> > > >>>>>>> and run them separately with Java 8. The only > >> > > >>>>>>> limitation is that these tests must stay JDK8 > >> > > >>>>>>> compatible. > >> > > >>>>>>> But from the tags we will see which ones are those. > >> > > >>>>>>> > >> > > >>>>>>> What do you think? > >> > > >>>>>>> > >> > > >>>>>>> Regards, Tamaas > >> > > >>>>>>> > >> > > >>>>>>> On Sat, Oct 17, 2020 at 7:45 AM Enrico Olivelli < > >> > > >>>>>>> eolive...@gmail.com> > >> > > >>>>>> wrote: > >> > > >>>>>>>> Christopher > >> > > >>>>>>>> I appreciate your idea and I also moved lots of my > >> > > >>>>>>>> projects to work > >> > > >>>> the > >> > > >>>>>> way > >> > > >>>>>>>> you are suggesting. > >> > > >>>>>>>> > >> > > >>>>>>>> > >> > > >>>>>>>> We must run tests using real jdk8 to test the > >> > > >>>>>>>> Zookeeper client. We > >> > > >>>> must > >> > > >>>>>>>> ensure that Zookeeper works well, especially while > >> > > >>>>>>>> dealing with > >> > > >>>> security > >> > > >>>>>>>> stuff. > >> > > >>>>>>>> Currently the client is in the same module of the > >> > > >>>>>>>> server and it will > >> > > >>>>>> take a > >> > > >>>>>>>> good (huge) amount of work to separate them > >> > > >>>>>>>> > >> > > >>>>>>>> > >> > > >>>>>>>> Enrico > >> > > >>>>>>>> > >> > > >>>>>>>> Il Ven 16 Ott 2020, 23:25 Christopher > >> > > >>>>>>>> <ctubb...@apache.org> ha > >> > > >>>> scritto: > >> > > >>>>>>>>> Hi ZK Devs, > >> > > >>>>>>>>> > >> > > >>>>>>>>> With recent advancements in Java (since Java 9), it > >> > > >>>>>>>>> is now generally no longer necessary to require that > >> > > >>>>>>>>> software be developed on an older JDK in order to > >> > > >>>>>>>>> have confidence that it will run on the older version > >> > > >>>>>>>>> of Java. This is because, as of Java 9, all JDK > >> > > >>>>>>>>> releases have better support for cross-compilation to > >> > > >>>>>>>>> older Java versions. > >> > > >>>>>>>>> > >> > > >>>>>>>>> What this means is that developers can confidently > >> > > >>>>>>>>> make the build requirements for a project higher than > >> > > >>>>>>>>> the Java version that will actually be supported at > >> > > >>>>>>>>> runtime. > >> > > >>>>>>>>> > >> > > >>>>>>>>> In fact, ZooKeeper already supports the necessary > >> > > >>>>>>>>> flags in its Maven build configuration to ensure that > >> > > >>>>>>>>> it uses JDK 8 compliance when building on a newer JDK > >> > > >>>>>>>>> (I added this way back in > >> > > >>>>>>>>> ZOOKEEPER-3739 / > >> > > >>>>>>>>> https://github.com/apache/zookeeper/pull/1269) > >> > > >>>>>>>>> > >> > > >>>>>>>>> So, I propose that we make JDK 11 the new minimum > >> > > >>>>>>>>> version to *build* ZooKeeper with. This would not > >> > > >>>>>>>>> change the runtime requirement, which would remain at > >> > > >>>>>>>>> JDK 8. > >> > > >>>>>>>>> > >> > > >>>>>>>>> The only necessary change to make this happen would > >> > > >>>>>>>>> be to add the minimum Java version to the > >> > > >>>>>>>>> maven-enforcer-plugin (like > >> > > >>>>>>>>> > >> > > >>>>>>>>> > >> > > >>>> > >> > > >> > >> > > > >> > > >> https://github.com/apache/accumulo/blob/438f0efd34ef9d200bc8c7ecdd1 > >> 1d5dedb146519/pom.xml#L1162-L1164 > >> > > >>>>>>>>> ) > >> > > >>>>>>>>> > >> > > >>>>>>>>> This would allow ZooKeeper to to streamline its > >> > > >>>>>>>>> development process a little bit by reducing the > >> > > >>>>>>>>> amount of CI testing that is done as part of the > >> > > >>>>>>>>> build. In other words, we can drop the CI builds for > >> > > >>>>>>>>> JDK 8, which saves on build resources and time. The > >> > > >>>>>>>>> return on investment is so low for the JDK 8 builds > >> > > >>>>>>>>> anyway, because of the improved cross-compilation in > >> > > >>>>>>>>> newer JDKs. So, there's not much value in building on > >> > > >>>>>>>>> JDK 8 anyway. > >> > > >>>>>>>>> > >> > > >>>>>>>>> Of course, I am only recommending this for *new* > >> > > >>>>>>>>> release lines, starting with ZooKeeper 3.7.0/master > >> > > >>>>>>>>> branch, because I would not want to change > >> > > >>>>>>>>> expectations for users who will build their own > >> > > >>>>>>>>> 3.5 and 3.6 > >> > > >>>>>>>>> versions as they continue to have patch versions > >> > > >>>>>>>>> released. > >> > > >>>>>>>>> > >> > > >>>>>>>>> What do you think? > >> > > >>>>>>>>> > >> > > >>>>>>>>> Kind Regards, > >> > > >>>>>>>>> Christopher > >> > > >>>>>>>>> > >> > > >> > >> > > >> > >> > > > >> > > > >> > > >> > > ________________________________ CONFIDENTIALITY & PRIVACY NOTICE This e-mail (including any attachments) is strictly confidential and may also contain privileged information. If you are not the intended recipient you are not authorised to read, print, save, process or disclose this message. If you have received this message by mistake, please inform the sender immediately and destroy this e-mail, its attachments and any copies. Any use, distribution, reproduction or disclosure by any person other than the intended recipient is strictly prohibited and the person responsible may incur in penalties. The use of this e-mail is only for professional purposes; there is no guarantee that the correspondence towards this e-mail will be read only by the recipient, because, under certain circumstances, there may be a need to access this email by third subjects belonging to the Company.
