Re: [DISCUSS] About the details of JDK-8 support
I forgot to mention - we also need to upgrade Google Guice to 0.4.0 for source-level JDK8 support in addition to asm and cglib. https://github.com/google/guice/wiki/Guice40 > Some highlights: > Java8 runtime compatibility for Guice core & all extensions. Best regards, - Tsuyoshi On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa wrote: > Hi commiters and users of Hadoop stack, > > I’ll share the current status of JDK-8 support here. We can take a > two-step approach to support JDK-8 - runtime-level support and > source-level support. > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g. > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As > long as I tested, it works well completely since JDK-8 doesn’t have > any incompatibility at binary level. We can say Hadoop has supported > JDK8 runtime already. Do you have any concern about this? I’ve not > tested with HBase yet. I need help of HBase community. I think only > problem about runtime is HADOOP-11364, default value of > colntainer-killer of YARN. After fixing the issue, we can declare the > support of JDK on Hadoop Wiki to make it clear for users. > https://wiki.apache.org/hadoop/HadoopJavaVersions > > About source-level, however, we have one big problem - upgrading > dependency of asm and cglib. We need to upgrade all libraries which > depends on asm to support new byte code introduced in JDK8[1]. The > dependencies which uses asm are jersey-server for compile and provide > scope, and cglib for test scope(I checked it with mvn dependency:tree > command). HADOOP-9613 is addressing the problem. > > One complex problem I’ve faced is Jersey depends on grizzly - to > upgrade jersey to 1.19, which supports JDK8, > we need to upgrade grizzly to 2.2.16 to use > jersey-test-framework-grizzly2. I’d like to discuss which version we > will target this change. Can we do this in branch-2? Should we take > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also > confirm whether HADOOP-11993 means to remove Jersey, which depends on > asm, or not. I think we can collaborate with Yetus community here. > > Also, another simple problem is that source code cannot be compiled > because javadoc format or variable identifier are illegal(e.g. > HADOOP-12457, HADOOP-11875). I think this can be solved > straightforwardly. > > Please share any concern I’ve missed. The opinions of users are also welcome > :-) > I'd like to go forward this step by step to make Hadoop user friendly. > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about > JDK-8. > > Best regards, > - Tsuyoshi > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Hello Thank you for sharing information about JDK support discussion. I'd like to share what I know now. In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of course there may be several diffs between HDP and upstream(trunk, branch-2), it is generally working well. We are running hadoop, mapreduce, yarn, hdfs, hive(HS2, command line). So I also think runtime level support of JDK8 is generally completed. And new features such as Hive LLAP or HDFS erasure coding also work well currently from the viewpoint of runtime environment. Regard Kai Sasaki Sent from my tiny typewriter. > On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa wrote: > > I forgot to mention - we also need to upgrade Google Guice to 0.4.0 > for source-level JDK8 support in addition to asm and cglib. > > https://github.com/google/guice/wiki/Guice40 > >> Some highlights: >> Java8 runtime compatibility for Guice core & all extensions. > > Best regards, > - Tsuyoshi > >> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa wrote: >> Hi commiters and users of Hadoop stack, >> >> I’ll share the current status of JDK-8 support here. We can take a >> two-step approach to support JDK-8 - runtime-level support and >> source-level support. >> >> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g. >> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As >> long as I tested, it works well completely since JDK-8 doesn’t have >> any incompatibility at binary level. We can say Hadoop has supported >> JDK8 runtime already. Do you have any concern about this? I’ve not >> tested with HBase yet. I need help of HBase community. I think only >> problem about runtime is HADOOP-11364, default value of >> colntainer-killer of YARN. After fixing the issue, we can declare the >> support of JDK on Hadoop Wiki to make it clear for users. >> https://wiki.apache.org/hadoop/HadoopJavaVersions >> >> About source-level, however, we have one big problem - upgrading >> dependency of asm and cglib. We need to upgrade all libraries which >> depends on asm to support new byte code introduced in JDK8[1]. The >> dependencies which uses asm are jersey-server for compile and provide >> scope, and cglib for test scope(I checked it with mvn dependency:tree >> command). HADOOP-9613 is addressing the problem. >> >> One complex problem I’ve faced is Jersey depends on grizzly - to >> upgrade jersey to 1.19, which supports JDK8, >> we need to upgrade grizzly to 2.2.16 to use >> jersey-test-framework-grizzly2. I’d like to discuss which version we >> will target this change. Can we do this in branch-2? Should we take >> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also >> confirm whether HADOOP-11993 means to remove Jersey, which depends on >> asm, or not. I think we can collaborate with Yetus community here. >> >> Also, another simple problem is that source code cannot be compiled >> because javadoc format or variable identifier are illegal(e.g. >> HADOOP-12457, HADOOP-11875). I think this can be solved >> straightforwardly. >> >> Please share any concern I’ve missed. The opinions of users are also welcome >> :-) >> I'd like to go forward this step by step to make Hadoop user friendly. >> >> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew >> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about >> JDK-8. >> >> Best regards, >> - Tsuyoshi >> >> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale >> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
> On 7 Oct 2015, at 07:29, Masatake Iwasaki wrote: > > Thanks for clear summary, Tsuyoshi. > > I read some related past discussions. > > https://wiki.apache.org/hadoop/MovingToJdk7and8 > http://search-hadoop.com/m/uOzYtGSiCs1acRnh > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1 > > Though there seems to be no consensus about when to drop java 7 support yet, > it would not be 2.8 for which the preparation is already started. > If the works for making source compatible with java 8 does not result in > dropping java 7 support, it would be nice and easy to backport to branch-2. > > > > we need to upgrade grizzly to 2.2.16 to use > > jersey-test-framework-grizzly2. I’d like to discuss which version we > > will target this change. Can we do this in branch-2? > > At lease, the newest grizzly, jersey and asm seems to support java 7 too > and HADOOP-11993 may work in branch-2. > Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change? I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase. In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing > > Masatake Iwasaki > > > On 10/6/15 09:35, Tsuyoshi Ozawa wrote: > > Hi commiters and users of Hadoop stack, > > > > I’ll share the current status of JDK-8 support here. We can take a > > two-step approach to support JDK-8 - runtime-level support and > > source-level support. > > > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g. > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As > > long as I tested, it works well completely since JDK-8 doesn’t have > > any incompatibility at binary level. We can say Hadoop has supported > > JDK8 runtime already. Do you have any concern about this? I’ve not > > tested with HBase yet. I need help of HBase community. I think only > > problem about runtime is HADOOP-11364, default value of > > colntainer-killer of YARN. After fixing the issue, we can declare the > > support of JDK on Hadoop Wiki to make it clear for users. > > https://wiki.apache.org/hadoop/HadoopJavaVersions > > > > About source-level, however, we have one big problem - upgrading > > dependency of asm and cglib. We need to upgrade all libraries which > > depends on asm to support new byte code introduced in JDK8[1]. The > > dependencies which uses asm are jersey-server for compile and provide > > scope, and cglib for test scope(I checked it with mvn dependency:tree > > command). HADOOP-9613 is addressing the problem. > > > > One complex problem I’ve faced is Jersey depends on grizzly - to > > upgrade jersey to 1.19, which supports JDK8, > > we need to upgrade grizzly to 2.2.16 to use > > jersey-test-framework-grizzly2. I’d like to discuss which version we > > will target this change. Can we do this in branch-2? Should we take > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also > > confirm whether HADOOP-11993 means to remove Jersey, which depends on > > asm, or not. I think we can collaborate with Yetus community here. > > > > Also, another simple problem is that source code cannot be compiled > > because javadoc format or variable identifier are illegal(e.g. > > HADOOP-12457, HADOOP-11875). I think this can be solved > > straightforwardly. > > > > Please share any concern I’ve missed. The opinions of users are also > > welcome :-) > > I'd like to go forward this step by step to make Hadoop user friendly. > > > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about > > JDK-8. > > > > Best regards, > > - Tsuyoshi > > > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker > >
Re: [DISCUSS] About the details of JDK-8 support
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning the full stack including HBase), so I agree that we're good there. I'm against dropping JDK7 support though in branch-2. Even bumping dependency versions scares me, since it often leads to downstream pain. Any comment about the compatibility of said bump? We need to have very high confidence if it's targeted for branch-2. Best, Andrew On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran wrote: > > > On 7 Oct 2015, at 07:29, Masatake Iwasaki > wrote: > > > > Thanks for clear summary, Tsuyoshi. > > > > I read some related past discussions. > > > > https://wiki.apache.org/hadoop/MovingToJdk7and8 > > http://search-hadoop.com/m/uOzYtGSiCs1acRnh > > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1 > > > > Though there seems to be no consensus about when to drop java 7 support > yet, > > it would not be 2.8 for which the preparation is already started. > > If the works for making source compatible with java 8 does not result in > > dropping java 7 support, it would be nice and easy to backport to > branch-2. > > > > > > > we need to upgrade grizzly to 2.2.16 to use > > > jersey-test-framework-grizzly2. I’d like to discuss which version we > > > will target this change. Can we do this in branch-2? > > > > At lease, the newest grizzly, jersey and asm seems to support java 7 too > > and HADOOP-11993 may work in branch-2. > > > > Certainly for trunk, I'm +1 for making the leap. For branch 2, how > backwards compatible/incompatible is the change? > > I think we'd have to test it downstream; I can use slider & spark as test > builds locally —YARN apps are the failure points. Someone else would have > to try HBase. > > In that world, we could think of having a short-lived branch-2-java-8 > branch, which cherry picks the grizzly changes from trunk, and which we can > then use for that downstream testing > > > > > Masatake Iwasaki > > > > > > On 10/6/15 09:35, Tsuyoshi Ozawa wrote: > > > Hi commiters and users of Hadoop stack, > > > > > > I’ll share the current status of JDK-8 support here. We can take a > > > two-step approach to support JDK-8 - runtime-level support and > > > source-level support. > > > > > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g. > > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As > > > long as I tested, it works well completely since JDK-8 doesn’t have > > > any incompatibility at binary level. We can say Hadoop has supported > > > JDK8 runtime already. Do you have any concern about this? I’ve not > > > tested with HBase yet. I need help of HBase community. I think only > > > problem about runtime is HADOOP-11364, default value of > > > colntainer-killer of YARN. After fixing the issue, we can declare the > > > support of JDK on Hadoop Wiki to make it clear for users. > > > https://wiki.apache.org/hadoop/HadoopJavaVersions > > > > > > About source-level, however, we have one big problem - upgrading > > > dependency of asm and cglib. We need to upgrade all libraries which > > > depends on asm to support new byte code introduced in JDK8[1]. The > > > dependencies which uses asm are jersey-server for compile and provide > > > scope, and cglib for test scope(I checked it with mvn dependency:tree > > > command). HADOOP-9613 is addressing the problem. > > > > > > One complex problem I’ve faced is Jersey depends on grizzly - to > > > upgrade jersey to 1.19, which supports JDK8, > > > we need to upgrade grizzly to 2.2.16 to use > > > jersey-test-framework-grizzly2. I’d like to discuss which version we > > > will target this change. Can we do this in branch-2? Should we take > > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also > > > confirm whether HADOOP-11993 means to remove Jersey, which depends on > > > asm, or not. I think we can collaborate with Yetus community here. > > > > > > Also, another simple problem is that source code cannot be compiled > > > because javadoc format or variable identifier are illegal(e.g. > > > HADOOP-12457, HADOOP-11875). I think this can be solved > > > straightforwardly. > > > > > > Please share any concern I’ve missed. The opinions of users are also > welcome :-) > > > I'd like to go forward this step by step to make Hadoop user friendly. > > > > > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew > > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about > > > JDK-8. > > > > > > Best regards, > > > - Tsuyoshi > > > > > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale > > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker > > > > > >
Re: [DISCUSS] About the details of JDK-8 support
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa wrote: > Do you have any concern about this? I’ve not > tested with HBase yet. > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for quite a while. Everything has been very stable for us. We're running and compiling with jdk8. We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs didn't do too well. I'd be +1 on dropping jdk7 support. However as downstream developer it would be very weird for that to happen on anything but a major release.
