Re: [DISCUSS] About the details of JDK-8 support

[Jean-Baptiste Note]

Hi,

As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732

I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.

JB

Re: [DISCUSS] About the details of JDK-8 support

[Kihwal Lee]

sun.security.krb5.KrbApReq was creating a static MD5 digest object and not 
synchronizing access.
This has been fixed in jdk8u60. 

http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89

One of the visible symptom is RPC reader thread getting 
ArrayIndexOutOfBoundsException from 
sun.security.provider.DigestBase.engineUpdate. More concerning case is a reader 
operating on a wrong digest.

Kihwal



From: Jean-Baptiste Note <jbn...@gmail.com>
To: common-...@hadoop.apache.org 
Cc: "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>; 
"hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; dev 
<dev@hbase.apache.org>; "yarn-...@hadoop.apache.org" 
<yarn-...@hadoop.apache.org> 
Sent: Tuesday, October 13, 2015 5:00 AM
Subject: Re: [DISCUSS] About the details of JDK-8 support


Hi,

As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732

I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.

JB

Re: [DISCUSS] About the details of JDK-8 support

[Andrew Wang]

>
> > On 7 Oct 2015, at 17:23, Andrew Wang  wrote:
> >
> > We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> > the full stack including HBase), so I agree that we're good there.
> >
>
>
> with Kerberos on?
>
> Yea, I haven't been that involved with our internal JDK validation
efforts, but I know there have been an assortment of JDK8 bugs related to
Kerberos. Our latest docs currently recommend 1.8.0_40 or above:

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp_unique_1

Re: [DISCUSS] About the details of JDK-8 support

[Elliott Clark]

On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa  wrote:

> Do you have any concern about this? I’ve not
> tested with HBase yet.
>

We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.

We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.

I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.

Re: [DISCUSS] About the details of JDK-8 support

[Masatake Iwasaki]

Thanks for clear summary, Tsuyoshi.

I read some related past discussions.

  https://wiki.apache.org/hadoop/MovingToJdk7and8
  http://search-hadoop.com/m/uOzYtGSiCs1acRnh
  http://search-hadoop.com/m/uOzYthdWJqpGdSZ1

Though there seems to be no consensus about when to drop java 7 support yet,
it would not be 2.8 for which the preparation is already started.
If the works for making source compatible with java 8 does not result in
dropping java 7 support, it would be nice and easy to backport to branch-2.


> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2?

At lease, the newest grizzly, jersey and asm seems to support java 7 too
and HADOOP-11993 may work in branch-2.


Masatake Iwasaki


On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8  e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
>  we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also 
welcome :-)

> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker

Re: [DISCUSS] About the details of JDK-8 support

[Steve Loughran]

> On 7 Oct 2015, at 07:29, Masatake Iwasaki  wrote:
> 
> Thanks for clear summary, Tsuyoshi.
> 
> I read some related past discussions.
> 
>  https://wiki.apache.org/hadoop/MovingToJdk7and8
>  http://search-hadoop.com/m/uOzYtGSiCs1acRnh
>  http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> 
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
> 
> 
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
> 
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
> 

Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards 
compatible/incompatible is the change? 

I think we'd have to test it downstream; I can use slider & spark as test 
builds locally —YARN apps are the failure points. Someone else would have to 
try HBase.

In that world, we could think of having a short-lived branch-2-java-8 branch, 
which cherry picks the grizzly changes from trunk, and which we can then use 
for that downstream testing

> 
> Masatake Iwasaki
> 
> 
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8  e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> >  we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also 
> > welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> 
> 

Re: [DISCUSS] About the details of JDK-8 support

[Andrew Wang]

We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.

I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.

Best,
Andrew

On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran 
wrote:

>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki 
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> >  https://wiki.apache.org/hadoop/MovingToJdk7and8
> >  http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> >  http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8  e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > >  we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>

Re: [DISCUSS] About the details of JDK-8 support

[Steve Loughran]

> On 7 Oct 2015, at 17:23, Andrew Wang  wrote:
> 
> We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> the full stack including HBase), so I agree that we're good there.
> 


with Kerberos on?

> I'm against dropping JDK7 support though in branch-2.

+1. We're only talking about dependencies here

> Even bumping
> dependency versions scares me, since it often leads to downstream pain. Any
> comment about the compatibility of said bump? We need to have very high
> confidence if it's targeted for branch-2.
> 

Which is why we need to test things downstream with the bumped jersey/grizzly 
libs before making any commitment to branch-2.

Re: [DISCUSS] About the details of JDK-8 support

[SasakiKai]

Hello

Thank you for sharing information about JDK support discussion. I'd like to 
share what I know now.

In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of 
course there may be several diffs between HDP and upstream(trunk, branch-2), it 
is generally working well. We are running hadoop, mapreduce, yarn, hdfs, 
hive(HS2, command line). So I also think runtime level support of JDK8 is 
generally completed. 
And new features such as Hive LLAP or HDFS erasure coding also work well 
currently from the viewpoint of runtime environment.

Regard
Kai Sasaki

Sent from my tiny typewriter.

> On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa  wrote:
> 
> I forgot to mention - we also need to upgrade Google Guice to 0.4.0
> for source-level JDK8 support in addition to asm and cglib.
> 
> https://github.com/google/guice/wiki/Guice40
> 
>> Some highlights:
>> Java8 runtime compatibility for Guice core & all extensions.
> 
> Best regards,
> - Tsuyoshi
> 
>> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa  wrote:
>> Hi commiters and users of Hadoop stack,
>> 
>> I’ll share the current status of JDK-8 support here. We can take a
>> two-step approach to support JDK-8 - runtime-level support and
>> source-level support.
>> 
>> About runtime-level support, I’ve tested Hadoop stack with JDK-8  e.g.
>> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
>> long as I tested, it works well completely since JDK-8 doesn’t have
>> any incompatibility at binary level. We can say Hadoop has supported
>> JDK8 runtime already. Do you have any concern about this? I’ve not
>> tested with HBase yet. I need help of HBase community. I think only
>> problem about runtime is HADOOP-11364, default value of
>> colntainer-killer of YARN. After fixing the issue, we can declare the
>> support of JDK on Hadoop Wiki to make it clear for users.
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>> 
>> About source-level, however, we have one big problem - upgrading
>> dependency of asm and cglib. We need to upgrade all libraries which
>> depends on asm to support new byte code introduced in JDK8[1]. The
>> dependencies which uses asm are jersey-server for compile and provide
>> scope, and cglib for test scope(I checked it with mvn dependency:tree
>> command). HADOOP-9613 is addressing the problem.
>> 
>> One complex problem I’ve faced is Jersey depends on grizzly - to
>> upgrade jersey to 1.19, which supports JDK8,
>> we need to upgrade grizzly to 2.2.16 to use
>> jersey-test-framework-grizzly2. I’d like to discuss which version we
>> will target this change. Can we do this in branch-2? Should we take
>> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
>> confirm whether HADOOP-11993 means to remove Jersey, which depends on
>> asm, or not. I think we can collaborate with Yetus community here.
>> 
>> Also, another simple problem is that source code cannot be compiled
>> because javadoc format or variable identifier are illegal(e.g.
>> HADOOP-12457, HADOOP-11875). I think this can be solved
>> straightforwardly.
>> 
>> Please share any concern I’ve missed. The opinions of users are also welcome 
>> :-)
>> I'd like to go forward this step by step to make Hadoop user friendly.
>> 
>> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
>> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
>> JDK-8.
>> 
>> Best regards,
>> - Tsuyoshi
>> 
>> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
>> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker

Re: [DISCUSS] About the details of JDK-8 support

[Tsuyoshi Ozawa]

I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.

https://github.com/google/guice/wiki/Guice40

> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.

Best regards,
- Tsuyoshi

On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa  wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8  e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
>  we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome 
> :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker