There are three points that stand out for me in this thread: - How do we determine how such a change affects our user base? - How much effort do the different options induce with respect to maintenance? - What's the right timeline for changes and how do we communicate them so that our users have enough time to prepare?
Someone mentioned a PMC vote, and I don't think this should be a closed vote, independent of how the conversation goes. -Flavio > On 22 Oct 2020, at 08:39, Alessandro Luccaroni - Diennea > <alessandro.luccar...@diennea.com.INVALID> wrote: > > Hi all, > If I might chime in as a zookeeper user (in multiple products) and a follower > of the project I think the drop of Java8 support (official and/or unofficial) > could be a big mistake. > > From my own company point of view we already support Java11 in all our > applications so we are not directly impacted (and we have upgrade path for > older versions to provide to our customers). > My worries resides in the (high) probability of a userbase fragmentation: in > the recent past Zookeeper development picked up speed thanks to a bunch of > new committers and PMCs after a period of mostly maintenance focused works, > but the number of active committers and PMCs is still very low for a project > like this. > > I foresee the risk of spreading thin the resources of the project if we force > the userbase to stick to an older version and, in turn, we are forced to > backport many issue to the 3.6 branch. > > Alessandro Luccaroni > Platform Manager @ Diennea - MagNews > Tel.: (+39) 0546 066100 Int. 924 - Mob.: (+39) 393 7273519 > Viale G.Marconi 30/14 - 48018 Faenza (RA) - Italy > > -----Messaggio originale----- > Da: Christopher <ctubb...@apache.org> > Inviato: giovedì 22 ottobre 2020 05:21 > A: dev@zookeeper.apache.org > Oggetto: Re: [DISCUSS][PROPOSAL] Require JDK 11 to build for 3.7 > > I'm happy that this discussion has been so lively! I just want to emphasize a > few things: > > I really do understand the desire to continue to support Java 8... I get it. > But all the conversations around this seem based on what people are doing > *today*. But, ZK 3.7 is *tomorrow's* version... a > *future* release... so it should be based more on reasonable expectations for > users in the future, and less based on what is happening today. I suspect > *most* people today are still using 3.4 anyway (it was just so stable for so > long...), but that shouldn't mean the developers should hold back development > on 3.5 and 3.6, any more than today's users of 3.5/3.6 should hold back 3.7. > > Some of the opinions expressed in this discussion seem to propose a scenario > where users are going to be updating to "bleeding edge" > versions of ZooKeeper, but are going to insist on using Java 8. > Personally, I find this to be implausible. In my experience, people either > upgrade everything as soon as they are able to, or they upgrade each thing > individually, only when they are forced to. The first group will be happy to > move to Java 11 and ZK 3.7. The second group will probably avoid 3.7 anyway, > and are fine sticking with 3.6, but if they had to update to 3.7, they'd also > be fine updating to Java 11 if they had to in order to use 3.7. I can't > imagine the scenario where people are eagerly choosing to upgrade to ZK 3.7, > but miserly insisting on using Java 8. Perhaps that scenario exists, but it's > hard for me to imagine. Even so, my proposal would still support even that > group of people. > > I think there are now effectively three proposals being discussed in this > thread: > > 1. (Christopher's original proposal) passively support Java 8 at runtime by > making JDK 11 the minimum requirement to build and test. > This scenario involves continuing to fix bugs, as they are discovered and > reported, that affect JDK 8, but passively, rather than proactively. This > proposal does *not* drop Java 8 support, but merely de-emphasizes it in > development of what will be 3.7 in the future, and drops the requirement to > do dedicated testing with Java 8. I think this is low risk, because it is > very unlikely that the ZK devs would introduce a bug that would affect only > Java 8 and the compiler wouldn't catch it... because the cross-compilation > features of newer JDKs are really good. > > 2. (Enrico's alternate proposal) this variation of my proposal would involve > continuing to proactively support Java 8 by creating a dedicated testing > suite to test client code on Java 8. I think this is a good option, but since > it involves a significantly higher amount of work than option 1, I think the > cost-benefit analysis would show this to be not worth the effort. Also, if it > were implemented, it would need to be done carefully to avoid requiring > developers to have concurrently installed both Java 8 and Java 11 in order to > perform a build, because requiring Java 8 at build time while developing > would be worse than we have today. > > 3. (Andor's preference) move to JDK 11 fully. This would provide no support, > passive or active, for Java 8 in ZK 3.7. To be honest, this is my personal > favorite, and is the simplest to implement and communicate clearly to end > users in release notes. The only reason I proposed a passive support of Java > 8 instead of this is because I was trying to seek a compromise from the > start. But, I think by far, this is the best option for the next *future* > release of ZK. If you wanted to make the change even more visible to users, > the version could even be bumped to 4.0. > > If this were to come to a VOTE by the PMC, in order to make a final decision, > I would recommend they vote on option 3, and then if that fails, vote on > option 1, and if that fails, keep things the way they are (because option 2 > is more work). > > Christopher > > P.S. as for Hadoop on Java 11... I've been running Hadoop 3 on JDK 11 and it > works just fine there (as long as you add the missing runtime jar for > javax.activation:javax.activation-api:1.2.0 to its class path, but that was > fixed in Hadoop 3.3). > > On Wed, Oct 21, 2020 at 5:42 PM Tamas Penzes <tam...@cloudera.com.invalid> > wrote: >> >> Hi All, >> >> I've just talked with a Hadoop/HDFS developer who told me what I guessed. >> With Hadoop3 they have just dropped JDK7 support, dropping JDK8 would >> mean a release of Hadoop4. >> Since HADOOP-15338 is finished, they test with JDK8 and JDK11 both. As >> of today most of the Hadoop users are still on Hadoop2, he doesn't >> expect >> Hadoop4 soon. >> As many Apache components depend on Hadoop and ZooKeeper they won't >> hurry to JDK11 until they have to (they will probably go one-by-one >> very slowly), which means if Hadoop stays on JDK8, they would use the >> last ZK version which works on JDK8. >> Do we want a ZK 3.4 again? >> >> Regards, Tamaas >> >> >> On Wed, Oct 21, 2020 at 11:23 PM Tamas Penzes <tam...@cloudera.com> wrote: >> >>> Hi All, >>> >>> Just to add my two cents. >>> >>> Upgrading to JDK11 looks inevitable sooner or later and I would >>> definitely not wait until 2030 or 2026 when the extended support of JDK8 >>> ends. >>> But on the other side I have to agree with Enrico and Patrick that >>> far too many users are tied to JDK8 yet (not because they want to >>> use JDK8, but because they have to), some of them are components of >>> the Hadoop ecosystem, which would be a loss to tie them to 3.6.x for years. >>> Do we know the state of Hadoop? It builds with JDK8 at the moment, >>> but do we know what are their plans to go to JDK11? >>> When they move, we should move too, but I don't think it would be >>> wise to do it earlier. >>> >>> Christopher's option looks like the golden path, but it needs some >>> investment on the testing side as Enrico pointed it out. >>> Could we agree on it as a compromise? >>> >>> Regards, Tamaas >>> >>> On Wed, Oct 21, 2020 at 7:11 PM Brent <brentwritesc...@gmail.com> wrote: >>> >>>> I think I was reacting to Enrico's earlier comment of: >>>> >>>> " ZooKeeper client is used by tons of users and unfortunately many >>>> projects are still on JDK8, if we move ZooKeeper to JDK11 the risk >>>> is to block users from the adoption, that is that we will see the >>>> world to stay on 3.6.x and we will have again a long lived release >>>> line, like 3.4." >>>> >>>> It's a matter of whether or not a long-lived release line is >>>> desirable/undesirable. If everyone is OK keeping 3.6.x up-to-date >>>> security/patch-wise (if not feature-wise) for the next N years, >>>> then that's a potentially valid approach. I interpreted that >>>> comment as "a long-lived release line is undesirable", but no one >>>> explicitly said that, I just read it that way. >>>> >>>> ~Brent >>>> >>>> On Wed, Oct 21, 2020 at 9:49 AM Patrick Hunt <ph...@apache.org> wrote: >>>> >>>>> On Wed, Oct 21, 2020 at 9:03 AM Andor Molnar <an...@apache.org> wrote: >>>>> >>>>>> As far as I know Hbase, Solr and Kafka are already Java 11 ready. >>>>>> >>>>>> IMHO contributors of those projects should also put efforts >>>>>> into >>>> moving >>>>>> forward. >>>>>> >>>>>> We’re not saying that you _have_ to move to Java 11. >>>>>> Staying on Java 8? No problem, 3.6 is for you. >>>>>> Want the fancy new features of 3.7? Work on it on your side too. >>>>>> >>>>>> >>>>> ppl want things like security fixes. I believe the highlighted >>>>> downside >>>> is >>>>> that we would need to continue to maintain 3.6.x rather than >>>>> allowing users, and ourselves, to focus on trunk for production -"64 >>>>> percent" >>>> would >>>>> be blocked. >>>>> >>>>> Patrick >>>>> >>>>> >>>>>> Andor >>>>>> >>>>>> >>>>>> >>>>>>> On 2020. Oct 21., at 17:52, Enrico Olivelli >>>>>>> <eolive...@gmail.com> >>>>> wrote: >>>>>>> >>>>>>> Il giorno mer 21 ott 2020 alle ore 17:49 Andor Molnar < >>>>> an...@apache.org> >>>>>> ha >>>>>>> scritto: >>>>>>> >>>>>>>> Correct me if I'm wrong, but Oracle gets paid for extended support. >>>>>>>> Java 8 support until 2030 is not free of charge. >>>>>>>> >>>>>>>> "ZK may end up with a lot of users potentially locking >>>>>>>> themselves >>>> to >>>>>>>> 3.6.x for a while as Enrico mentioned." >>>>>>>> >>>>>>>> That's true. What's the downside of that which we should >>>>>>>> invest in >>>> to >>>>>>>> avoid? >>>>>>>> >>>>>>> >>>>>>> I see ZooKeeper used in many many projects, all of the >>>>>>> HBase/Pulsar/Kafka/Solr ecosystem... >>>>>>> they will have to move to JDK11 in order to move to the new >>>>>>> ZK >>>> version >>>>>>> so probably they will stay on ZK 3.6 >>>>>>> >>>>>>> Probably with Java 17 LTS released the cards will change on >>>>>>> the >>>> table >>>>>>> >>>>>>> Enrico >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Andor >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 2020-10-21 at 08:03 -0700, Brent wrote: >>>>>>>>> As a slightly different consideration, if you look at the >>>> Long-Term >>>>>>>>> Support >>>>>>>>> (LTS) roadmaps for Java, currently Java 8 is set to have >>>>>>>>> full >>>> support >>>>>>>>> until >>>>>>>>> 2030 from Oracle and at least 2026 from OpenJDK & Corretto: >>>>>>>>> >>>>>>>>> >>>>> https://www.oracle.com/java/technologies/java-se-support-roadmap. >>>>> html >>>>>>>>> https://en.wikipedia.org/wiki/Java_version_history >>>>>>>>> >>>>>>>>> My guess is that a number of companies are still heavily >>>>>>>>> invested >>>> at >>>>>>>>> the >>>>>>>>> Java 8 level (I know a few) and with that kind of time >>>>>>>>> horizon, >>>> they >>>>>>>>> have >>>>>>>>> no real motivation to upgrade for quite a while. If the >>>>>>>>> recent Python 2 deprecation is anything to go by, they >>>>>>>>> won't do it until they have to. >>>>>>>>> >>>>>>>>> Not saying Java 8 isn't *very* old (2014 release it seems >>>>>>>>> like?) >>>> and >>>>>>>>> I'm >>>>>>>>> not invested heavily either way, but this might suggest >>>>>>>>> that ZK >>>> may >>>>>>>>> end up >>>>>>>>> with a lot of users potentially locking themselves to 3.6.x >>>>>>>>> for a while as Enrico mentioned. >>>>>>>>> >>>>>>>>> (Not a major contributor, but wanted to chime in since I >>>>>>>>> just had this conversation with a bunch of people >>>>>>>>> professionally recently) >>>>>>>>> >>>>>>>>> Brent >>>>>>>>> >>>>>>>>> On Wed, Oct 21, 2020 at 2:07 AM Andor Molnar >>>>>>>>> <an...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the summary. >>>>>>>>>> >>>>>>>>>> I still vote for option 1). Move 3.7.0 to JDK 11 fully. >>>>>>>>>> Other projects will upgrade once they’re JDK11 compliant, >>>>>>>>>> otherwise they will >>>> stay >>>>>>>>>> on 3.5 >>>>>>>>>> or 3.6. Both version are quite recent in ZooKeeper-terms, >>>>>>>>>> we already planned big changes for 3.7.0 and JDK 11 could >>>>>>>>>> be one of them. >>>>>>>>>> >>>>>>>>>> Don’t put extra burden on the ZK community to help others >>>>>>>>>> staying on ancient Java versions. >>>>>>>>>> >>>>>>>>>> Andor >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 2020. Oct 21., at 10:57, Enrico Olivelli < >>>> eolive...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Let me recap >>>>>>>>>>> - Christopher is proposing to move to JDK11 >>>>>>>>>>> - ZooKeeper client and server are bundled and coded and >>>>>>>>>>> tested together >>>>>>>>>> in >>>>>>>>>>> zookeeper-server >>>>>>>>>>> - Enrico is concerned about the need of testing ZooKeeper >>>>>>>>>>> client on JDK8 (not a problem to move the server to >>>>>>>>>>> JDK11) >>>>>>>>>>> >>>>>>>>>>> ZooKeeper client is used by tons of users and >>>>>>>>>>> unfortunately many projects are still on JDK8, if we move >>>>>>>>>>> ZooKeeper to JDK11 the risk is to block >>>>>>>>>> users >>>>>>>>>>> from the adoption, >>>>>>>>>>> that is that we will see the world to stay on 3.6.x and >>>>>>>>>>> we will have >>>>>>>>>> again >>>>>>>>>>> a long lived release line, like 3.4. >>>>>>>>>>> >>>>>>>>>>> Testing the client on JDK8 would be possible if we create >>>>>>>>>>> some kind of additional module with system tests, then we >>>>>>>>>>> can start the >>>> server >>>>>>>>>>> on >>>>>>>>>> docker >>>>>>>>>>> on JDK11+ and start a client on JDK8 with Maven toolchain >>>>>>>>>>> it should possible to run surefire tests using a separate >>>>>>>>>>> JVM. >>>>>>>>>>> >>>>>>>>>>> So in my vision 2 options: >>>>>>>>>>> 1) fully JDK11 - drop JDK8 at all >>>>>>>>>>> 2) build with JDK11 - server only on JDK11 - add system >>>>>>>>>>> tests with docker and toolchains that ensure the >>>>>>>>>>> ZooKeeper client (and all >>>>>>>>>>> dependencies) >>>>>>>>>>> still work on JDK8 >>>>>>>>>>> >>>>>>>>>>> From my point of view about the ZooKeeper ecosystem >>>>>>>>>>> option 2) will be far better, but we need resources to >>>>>>>>>>> work on a new test suite. >>>>>>>>>>> >>>>>>>>>>> Enrico >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Il giorno mer 21 ott 2020 alle ore 10:43 Andor Molnar < >>>>>>>>>>> an...@apache.org> >>>>>>>>>> ha >>>>>>>>>>> scritto: >>>>>>>>>>> >>>>>>>>>>>> Tamas, Enrico, >>>>>>>>>>>> >>>>>>>>>>>> Sorry I don’t follow. Why do we have to test the client >>>>>>>>>>>> with JDK 8 in version 3.7.0? >>>>>>>>>>>> >>>>>>>>>>>> Andor >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 2020. Oct 20., at 22:29, Tamas Penzes < >>>>>>>>>>>>> tam...@cloudera.com.INVALID> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> Hi Enrico, >>>>>>>>>>>>> >>>>>>>>>>>>> Separating ZooKeeper client and server is a huge work, >>>>>>>>>>>>> but we might not need it. >>>>>>>>>>>>> As you mentioned we have to test ZK client with Java 8, >>>>>>>>>>>>> what about separating only the test cases which we need >>>>>>>>>>>>> to run with >>>>>>>>>>>>> Java8 too? >>>>>>>>>>>>> In Curator we have the ZK compatibility tests where we >>>>>>>>>>>>> run a limited >>>>>>>>>>>> amount >>>>>>>>>>>>> of Curator's jUnit tests with a different ZK version. >>>>>>>>>>>>> We might be able to do the same here, tag tests which >>>>>>>>>>>>> are testing ZK >>>>>>>>>>>> client >>>>>>>>>>>>> and run them separately with Java 8. The only >>>>>>>>>>>>> limitation is that these tests must stay JDK8 >>>>>>>>>>>>> compatible. >>>>>>>>>>>>> But from the tags we will see which ones are those. >>>>>>>>>>>>> >>>>>>>>>>>>> What do you think? >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, Tamaas >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Oct 17, 2020 at 7:45 AM Enrico Olivelli < >>>>>>>>>>>>> eolive...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Christopher >>>>>>>>>>>>>> I appreciate your idea and I also moved lots of my >>>>>>>>>>>>>> projects to work >>>>>>>>>> the >>>>>>>>>>>> way >>>>>>>>>>>>>> you are suggesting. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> We must run tests using real jdk8 to test the >>>>>>>>>>>>>> Zookeeper client. We >>>>>>>>>> must >>>>>>>>>>>>>> ensure that Zookeeper works well, especially while >>>>>>>>>>>>>> dealing with >>>>>>>>>> security >>>>>>>>>>>>>> stuff. >>>>>>>>>>>>>> Currently the client is in the same module of the >>>>>>>>>>>>>> server and it will >>>>>>>>>>>> take a >>>>>>>>>>>>>> good (huge) amount of work to separate them >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Enrico >>>>>>>>>>>>>> >>>>>>>>>>>>>> Il Ven 16 Ott 2020, 23:25 Christopher >>>>>>>>>>>>>> <ctubb...@apache.org> ha >>>>>>>>>> scritto: >>>>>>>>>>>>>>> Hi ZK Devs, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> With recent advancements in Java (since Java 9), it >>>>>>>>>>>>>>> is now generally no longer necessary to require that >>>>>>>>>>>>>>> software be developed on an older JDK in order to >>>>>>>>>>>>>>> have confidence that it will run on the older version >>>>>>>>>>>>>>> of Java. This is because, as of Java 9, all JDK >>>>>>>>>>>>>>> releases have better support for cross-compilation to >>>>>>>>>>>>>>> older Java versions. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What this means is that developers can confidently >>>>>>>>>>>>>>> make the build requirements for a project higher than >>>>>>>>>>>>>>> the Java version that will actually be supported at >>>>>>>>>>>>>>> runtime. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In fact, ZooKeeper already supports the necessary >>>>>>>>>>>>>>> flags in its Maven build configuration to ensure that >>>>>>>>>>>>>>> it uses JDK 8 compliance when building on a newer JDK >>>>>>>>>>>>>>> (I added this way back in >>>>>>>>>>>>>>> ZOOKEEPER-3739 / >>>>>>>>>>>>>>> https://github.com/apache/zookeeper/pull/1269) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So, I propose that we make JDK 11 the new minimum >>>>>>>>>>>>>>> version to *build* ZooKeeper with. This would not >>>>>>>>>>>>>>> change the runtime requirement, which would remain at >>>>>>>>>>>>>>> JDK 8. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The only necessary change to make this happen would >>>>>>>>>>>>>>> be to add the minimum Java version to the >>>>>>>>>>>>>>> maven-enforcer-plugin (like >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> https://github.com/apache/accumulo/blob/438f0efd34ef9d200bc8c7ecdd1 >>>> 1d5dedb146519/pom.xml#L1162-L1164 >>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This would allow ZooKeeper to to streamline its >>>>>>>>>>>>>>> development process a little bit by reducing the >>>>>>>>>>>>>>> amount of CI testing that is done as part of the >>>>>>>>>>>>>>> build. In other words, we can drop the CI builds for >>>>>>>>>>>>>>> JDK 8, which saves on build resources and time. The >>>>>>>>>>>>>>> return on investment is so low for the JDK 8 builds >>>>>>>>>>>>>>> anyway, because of the improved cross-compilation in >>>>>>>>>>>>>>> newer JDKs. So, there's not much value in building on >>>>>>>>>>>>>>> JDK 8 anyway. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Of course, I am only recommending this for *new* >>>>>>>>>>>>>>> release lines, starting with ZooKeeper 3.7.0/master >>>>>>>>>>>>>>> branch, because I would not want to change >>>>>>>>>>>>>>> expectations for users who will build their own >>>>>>>>>>>>>>> 3.5 and 3.6 >>>>>>>>>>>>>>> versions as they continue to have patch versions >>>>>>>>>>>>>>> released. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What do you think? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kind Regards, >>>>>>>>>>>>>>> Christopher >>>>>>>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> > > ________________________________ > > CONFIDENTIALITY & PRIVACY NOTICE > This e-mail (including any attachments) is strictly confidential and may also > contain privileged information. If you are not the intended recipient you are > not authorised to read, print, save, process or disclose this message. If you > have received this message by mistake, please inform the sender immediately > and destroy this e-mail, its attachments and any copies. Any use, > distribution, reproduction or disclosure by any person other than the > intended recipient is strictly prohibited and the person responsible may > incur in penalties. > The use of this e-mail is only for professional purposes; there is no > guarantee that the correspondence towards this e-mail will be read only by > the recipient, because, under certain circumstances, there may be a need to > access this email by third subjects belonging to the Company.
