Hey all,
This one started as an innocuous thread of enabling JDK7 on trunk and now it
seems like (haven't still finished reading the entire thing, and I started a
while ago) it has become a full blown proposal on 2.x, 3.x and 4.x releases.
Some of us haven't been tracking this (at least me and
Andrew, correct me if I'm misunderstanding, but the incompatible change
that would require a major version bump is dropping support for JDK6.
On Mon, Jun 23, 2014 at 1:53 PM, sanjay Radia
wrote:
>
> On Jun 21, 2014, at 8:01 AM, Andrew Wang wrote:
>
> > This is why I'd like to keep my original
On Jun 21, 2014, at 8:01 AM, Andrew Wang wrote:
> This is why I'd like to keep my original proposal on the table: keep going
> with branch-2 in the near term, while working towards a JDK8-based Hadoop 3
> by April next year. It doesn't need to be a big bang release either. I'd be
> delighted if
On Fri, Jun 20, 2014 at 10:02 PM, Arun C Murthy wrote:
> > Hadoop 3.x out the door later this year
>
> +1 that makes sense to me. Thanks for volunteering Steve - I'm glad to
> share the pain… ;-)
Hey Arun, you may have missed that Andrew volunteered for doing this as
well (the thread is long,
After further consideration, here is an alternate.
On Jun 21, 2014, at 11:14 AM, "Arun C. Murthy" wrote:
>
> JDK6 eol was Feb 2013 and, a year later, we are still have customers using it
> - which means we can't drop it yet.
>
> http://www.oracle.com/technetwork/java/eol-135779.html
>
> Give
Andrew,
> On Jun 21, 2014, at 8:01 AM, Andrew Wang wrote:
>
> Hi Steve, let me confirm that I understand your proposal correctly:
>
> - Release an intermediate Hadoop 3 a few months out, based on JDK7 and with
> bumped library versions
> - Release a Hadoop 4 mid next year, based on JDK8
>
> I
ozen and tell everyone "move to java 7+", everything
downstream gets updated binaries and a chance to move forwards.
There's another issue, which is one Alejandro highlit:
-- Forwarded message --
From: Alejandro Abdelnur
Date: 10 April 2014 10:30
Subject: Re: Plans o
Hi Steve, let me confirm that I understand your proposal correctly:
- Release an intermediate Hadoop 3 a few months out, based on JDK7 and with
bumped library versions
- Release a Hadoop 4 mid next year, based on JDK8
I question the utility of an intermediate Hadoop 3 like this. Assuming that
it
On Jun 20, 2014, at 9:51 PM, Steve Loughran wrote:
> On 20 June 2014 21:35, Steve Loughran wrote:
>
>>
>> This actually argues in favour of
>>
>> -renaming branch-2 branch-3 after a release
>> -making trunk hadoop-4
>>
>> -getting hadoop 3 released off the new branch-3 out in 2014, effectiv
On 20 June 2014 21:35, Steve Loughran wrote:
>
> This actually argues in favour of
>
> -renaming branch-2 branch-3 after a release
> -making trunk hadoop-4
>
> -getting hadoop 3 released off the new branch-3 out in 2014, effectively
> being an iteration of branch-2 with updated java , moves of (o
Having gone back through the entire thread I can see we've made progress
here, as the discussion has moved on from when to move to java 7 to when to
move to java 8... which, I've alway felt the appeal of from the
coding-side. Java 8 tomorrow is the most compelling reason to move to java
7 today.
On 20 June 2014 17:01, Andrew Wang wrote:
> Thanks everyone for the discussion so far. I talked with some of our other
> teams and thought about the issue some more.
>
> Regarding branch-2, we can't do much because of compatibility. Dropping
> support for a JDK is supposed to happen in a major re
gt;>
> > > >>> >> “I don't see any point to switching” is an interesting
> > perspective,
> > > >>> given
> > > >>> >> the well-known risks of running unsafe software.
>> >>
> > >>> >>
> > >>> >>
> > >>> >> You also said "we still test and support JDK6". I searched but
> have
> > not
> > >>> >> been able to find Cloudera critical security fi
;>> >>
> >>> >>
> >>> >>
> >>> >> Can you clarify, for example, Java 6 Update 51 for CVE-2013-2465? In
> >>> other
> >>> >> words, did you release to your customers any kind of public alert or
> >>> >>
2013-2465? In
>>> other
>>> >> words, did you release to your customers any kind of public alert or
>>> >> warning of this CVSS 10.0 event as part of your JDK6 support?
>>> >>
>>> >>
>>> >>
>>> >> http://w
gt;> >>
>> >> http://www.cvedetails.com/cve/CVE-2013-2465/
>> >>
>> >>
>> >>
>> >> If you are not releasing your own security fixes for JDK6 post-EOL would
>> >> it perhaps be safer to say Cloudera is hands-off; neither supports, nor
>> >> opposes the know
ra is hands-off; neither supports, nor
> >> opposes the known insecure and deprecated/unpatched JDK?
> >>
> >>
> >>
> >> I mentioned before in this thread the Oracle support timeline:
> >>
> >>
> >>
> >> - official public EOL (end o
d of life) was more than a year ago
>>
>> - premier support ended more than six months ago
>>
>> - extended support may get critical security fixes until the end of 2016
>>
>>
>>
>> Given this timeline, does Cloudera officially take responsibility for
ago
> >
> > - premier support ended more than six months ago
> >
> > - extended support may get critical security fixes until the end of 2016
> >
> >
> >
> > Given this timeline, does Cloudera officially take responsibility for
> > Hadoop custom
?
>
>
>
> Davi
>
>
>
>
>
>
>
> > -----Original Message-
>
> > From: Andrew Wang [mailto:andrew.w...@cloudera.com]
>
> > Sent: Wednesday, June 18, 2014 12:33 PM
>
> > To: common-dev@hadoop.apache.org
>
> > Subject: Re
Sent: Wednesday, June 18, 2014 12:33 PM
> To: common-dev@hadoop.apache.org
> Subject: Re: Plans of moving towards JDK7 in trunk
>
> Actually, a lot of our customers are still on JDK6, so if anything, its
> popularity
> hasn't significantly decreased. We still test and su
On 18 June 2014 12:32, Andrew Wang wrote:
> Actually, a lot of our customers are still on JDK6, so if anything, its
> popularity hasn't significantly decreased. We still test and support JDK6
> for CDH4 and CDH5. The claim that branch-2 is effectively JDK7 because no
> one supports JDK6 is untrue
Actually, a lot of our customers are still on JDK6, so if anything, its
popularity hasn't significantly decreased. We still test and support JDK6
for CDH4 and CDH5. The claim that branch-2 is effectively JDK7 because no
one supports JDK6 is untrue.
One issue with your proposal is that java 7+ libr
I also think we need to recognise that its been three months since that
last discussion, and Java 6 has not suddenly burst back into popularity
- nobody providing commercial support for Hadoop is offering branch-2
support on Java 6 AFAIK
- therefore, nobody is testing it at scale except
I think we should come up with a plan for when the next Hadoop release
will drop support for JDK6. We all know that day needs to come... the
only question is when. I agree that writing the JDK7-only code
doesn't seem very productive unless we have a plan for when it will be
released and usable.
Reviving this thread, I noticed there's been a patch and +1 on
HADOOP-10530, and I don't think we actually reached a conclusion.
I (and others) have expressed concerns about moving to JDK7 for trunk.
Summarizing a few points:
- We can't move to JDK7 in branch-2 because of compatibility
- branch-2
On 14 April 2014 17:46, Andrew Purtell wrote:
> How well is trunk tested? Does anyone deploy it with real applications
> running on top? When will the trunk codebase next be the basis for a
> production release? An impromptu diff of hadoop-common trunk against
> branch-2 as of today is 38,625 lin
I would say, to an extent. The current state of the jetty version is
*severe*. We're 3 major versions behind, and if my understanding is
correct, it was a long time ago they EOF'ed version 6.x.
Yes, upgrading jetty could break some customers. However, we need to view
it in balance. We're constantl
How well is trunk tested? Does anyone deploy it with real applications
running on top? When will the trunk codebase next be the basis for a
production release? An impromptu diff of hadoop-common trunk against
branch-2 as of today is 38,625 lines. Can they be said to be the same
animal? I ask becaus
I think the bottom line here is that as long as our stable release
uses JDK6, there is going to be a very, very strong disincentive to
put any code which can't run on JDK6 into trunk.
Like I said earlier, the traditional reason for putting something in
trunk but not the stable release is that it n
There's an outstanding question addressed to me: "Are there particular
features or new dependencies that you would like to contribute (or see
contributed) that require using the Java 1.7 APIs?" The question
misses the point: We'd figure out how to write something we wanted to
contribute to Hadoop
Hi,
+1 for Karthik's idea(non-binding).
IMO, we should keep the compatibility between JDK 6 and JDK 7 on both branch-1
and branch-2, because users can be using them. For future releases that we can
declare breaking compatibility(e.g. 3.0.0 release), we can use JDK 7
features if we
can get benefit
i disagree, mustn't break downstrea
Alejandro
(phone typing)
> On Apr 12, 2014, at 3:15, Steve Loughran wrote:
>
> 1. I wasn't thinking of sticking of jetty in in the web ui or webhdfs at
> all.
> 2. the later jetties change their packaing, so should be able to co-exist
> anyway.
>
> Jetty is
1. I wasn't thinking of sticking of jetty in in the web ui or webhdfs at
all.
2. the later jetties change their packaing, so should be able to co-exist
anyway.
Jetty is a fundamental cause of problems, especially on things like
webhdfs. We can't use the excuse of "mustn't break downstream app clas
newer jetties have non backwards compat APIs, we would break any user app
using jetty (consumed via hadoop classpath)
On Fri, Apr 11, 2014 at 2:16 PM, Steve Loughran wrote:
> that doesn't actually stop is from switching in our own code to alternate
> web servers, only that jetty can remain a p
that doesn't actually stop is from switching in our own code to alternate
web servers, only that jetty can remain a published artifact in the
hadoop/lib dir
On 11 April 2014 21:16, Alejandro Abdelnur wrote:
> because it is exposed as classpath dependency, changing it breaks backward
> compatib
because it is exposed as classpath dependency, changing it breaks backward
compatibility.
On Fri, Apr 11, 2014 at 1:02 PM, Steve Loughran wrote:
> Jetty's a big change, it's fairly intimately involved in bits of the code
>
> but: it's a source of grief, currently webhdfs is an example
> https://
Jetty's a big change, it's fairly intimately involved in bits of the code
but: it's a source of grief, currently webhdfs is an example
https://issues.apache.org/jira/browse/HDFS-6221
all YARN apps seem to get hosted by it too
On 11 April 2014 20:56, Robert Rati wrote:
> I don't mean to be den
I don't mean to be dense, but can you expand on why jetty 8 can't go
into branch2? What is the concern?
Rob
On 04/11/2014 10:55 AM, Alejandro Abdelnur wrote:
if you mean updating jetty on branch2, we cannot do that. it has to be done in
trunk.
thx
Alejandro
(phone typing)
On Apr 11, 2014
if you mean updating jetty on branch2, we cannot do that. it has to be done in
trunk.
thx
Alejandro
(phone typing)
> On Apr 11, 2014, at 4:46, Robert Rati wrote:
>
> Just an FYI, but I'm working on updating that jetty patch for the current
> 2.4.0 release. The one that is there won't clean
Just an FYI, but I'm working on updating that jetty patch for the
current 2.4.0 release. The one that is there won't cleanly apply
because so much has changed since it was posted. I'll post a new patch
when it's done.
Rob
On 04/11/2014 04:24 AM, Steve Loughran wrote:
On 10 April 2014 18:12
On 10 April 2014 18:12, Eli Collins wrote:
> Let's speak less abstractly, are there particular features or new
> dependencies that you would like to contribute (or see contributed) that
> require using the Java 1.7 APIs? Breaking compat in v2 or rolling a v3
> release are both non-trivial, not s
A bit of a different angle.
As the bottom of the stack Hadoop has to be conservative in adopting
things, but it should not preclude consumers of Hadoop (downstream projects
and Hadoop application developers) to have additional requirements such as
a higher JDK API than JDK6.
Hadoop 2.x should sti
On Thu, Apr 10, 2014 at 6:49 AM, Raymie Stata wrote:
> I think the problem to be solved here is to define a point in time
> when the average Hadoop contributor can start using Java7 dependencies
> in their code.
>
> The "use Java7 dependencies in trunk(/branch3)" plan, by itself, does
> not solve
On Thu, Apr 10, 2014 at 1:11 AM, Steve Loughran wrote:
> On 9 April 2014 23:52, Eli Collins wrote:
>
> >
> >
> > For the sake of this discussion we should separate the runtime from
> > the programming APIs. Users are already migrating to the java7 runtime
> > for most of the reasons listed below
I think the problem to be solved here is to define a point in time
when the average Hadoop contributor can start using Java7 dependencies
in their code.
The "use Java7 dependencies in trunk(/branch3)" plan, by itself, does
not solve this problem. The average Hadoop contributor wants to see
their
On 9 April 2014 23:52, Eli Collins wrote:
>
>
> For the sake of this discussion we should separate the runtime from
> the programming APIs. Users are already migrating to the java7 runtime
> for most of the reasons listed below (support, performance, bugs,
> etc), and the various distributions ce
+1 for keeping jdk 6 suppprt in branch-2 and start using jdk 7 in trunk.
I agree that this approach makes patch generation difficult for branch-2
and trunk.
Also the actual benefit and real issues after start using jdk7 will be
known only if atleast one of the release is out in trunk version.
Re
I think this thread isn't so much about whether java7, 8 etc features
are valuable, they are useful of course, and we'll want to adopt them,
it's a question of how we adopt them and in which releases.
For the sake of this discussion we should separate the runtime from
the programming APIs. Users a
A Java 8 runtime would also offer transparent performance improvements like
a reimplementation of ConcurrentSkipListMap, C2 support for AES cipher
acceleration with native CPU instructions, perf improvements for going from
String to byte[] or vice versa, and IIRC after 8u20 monitor lock elision
usi
> It might make sense to try to enumerate the benefits of switching to
> Java7 APIs and dependencies.
- Java7 introduced a huge number of language, byte-code, API, and
tooling enhancements! Just to name a few: try-with-resources, newer
and stronger encyrption methods, more scalable concurrency
It might make sense to try to enumerate the benefits of switching to Java7
APIs and dependencies. IMO, the ones listed so far on this thread don't
make a compelling enough case to drop Java6 in branch-2 on any time frame,
even if this means supporting Java6 through 2015. For example, the change
i
Is there broad consensus that, by end of 3Q2014 at the latest, "the
average" contributor to Hadoop should be free to use Java7 features?
And start pulling in libraries that have a Java7 dependency? And
start doing the "janitorial" work of taking advantage of the Java7
APIs? Or do we think that th
+1 to NOT breaking compatibility in branch-2.
I think it is reasonable to require JDK7 for trunk, if we limit use of
JDK7-only API to security fixes etc. If we make other optimizations (like
IO), it would be a pain to backport things to branch-2. I guess this all
depends on when we see ourselves s
On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi
wrote:
>> From: Eli Collins [mailto:e...@cloudera.com]
>> Sent: Monday, April 07, 2014 11:54 AM
>>
>>
>> IMO we should not drop support for Java 6 in a minor update of a stable
>> release (v2). I don't think the larger Hadoop user base would find
Davi,
If you look at the security issues, they mostly come down to the same
thing: the sandbox isn't secure. Instead of running applets or web
applications in a locked down environment, malicious code can get out and
access private data, manipulate the filesystem, get out on the network, etc.
As
+1 for maintaining Java 6 support in branch-2.
Hadoop continuing to support Java 6 is not an endorsement of Java 6. It's
an acknowledgement that many users of Hadoop 2 have Java 6 embedded in
their stack, and that upgrading is costly for some users and simply not an
option for others. If a simil
> From: Eli Collins [mailto:e...@cloudera.com]
> Sent: Monday, April 07, 2014 11:54 AM
>
>
> IMO we should not drop support for Java 6 in a minor update of a stable
> release (v2). I don't think the larger Hadoop user base would find it
> acceptable that upgrading to a minor update caused their
On Sat, Apr 5, 2014 at 12:54 PM, Raymie Stata wrote:
> To summarize the thread so far:
>
> a) Java7 is already a supported compile- and runtime environment for
> Hadoop branch2 and trunk
> b) Java6 must remain a supported compile- and runtime environment for
> Hadoop branch2
> c) (b) implies that
It looks to me that the majority of this thread welcomes JDK7. Just to
reiterate, there are two separate questions here:
1. When should hadoop-trunk can be only built on top of JDK7?
2. When should hadoop-branch-2 can be only built on top of JDK7?
The answers of the above questions directly imply
On 5 April 2014 20:54, Raymie Stata wrote:
> To summarize the thread so far:
>
> a) Java7 is already a supported compile- and runtime environment for
> Hadoop branch2 and trunk
> b) Java6 must remain a supported compile- and runtime environment for
> Hadoop branch2
> c) (b) implies that branch2 m
To summarize the thread so far:
a) Java7 is already a supported compile- and runtime environment for
Hadoop branch2 and trunk
b) Java6 must remain a supported compile- and runtime environment for
Hadoop branch2
c) (b) implies that branch2 must stick to Java6 APIs
I wonder if point (b) should be r
On 5 April 2014 11:53, Colin McCabe wrote:
> I've been using JDK7 for Hadoop development for a while now, and I
> know a lot of other folks have as well. Correct me if I'm wrong, but
> what we're talking about here is not "moving towards JDK7" but
> "breaking compatibility with JDK6."
>
+1
>
>
I've been using JDK7 for Hadoop development for a while now, and I
know a lot of other folks have as well. Correct me if I'm wrong, but
what we're talking about here is not "moving towards JDK7" but
"breaking compatibility with JDK6."
There are a lot of good reasons to ditch JDK6. It would let u
So, you want to compile hdfs/yarn/mapred clients (and hadoop-common and
hadoop-auth) with JDK6 and the rest with JDK7?
On Fri, Apr 4, 2014 at 3:15 PM, Haohui Mai wrote:
> I'm referring to the later case. Indeed migrating JDK7 for branch-2 is more
> difficult.
>
> I think one reasonable approach
bq. It might not be as clear cut...
Totally agree. I think the key is that we can do the work in an incremental
way. We can only introduce JDK7 dependency on the server side. In order to
do this we need to separate the client-side code to separate jars. I've
already proposed to create a hdfs-clien
Please don't forget the mac os build on JDK 7. :)
On Fri, Apr 4, 2014 at 3:15 PM, Haohui Mai wrote:
> I'm referring to the later case. Indeed migrating JDK7 for branch-2 is more
> difficult.
>
> I think one reasonable approach is to put the hdfs / yarn clients into
> separate jars. The client-s
I'm referring to the later case. Indeed migrating JDK7 for branch-2 is more
difficult.
I think one reasonable approach is to put the hdfs / yarn clients into
separate jars. The client-side jars can only use JDK6 APIs, so that
downstream projects running on top of JDK6 continue to work.
The HDFS/Y
Haohui,
Is the idea to compile/test with JDK7 and recommend it for runtime and stop
there? Or to start using JDK7 API stuff as well? If the later is the case,
then backporting stuff to branch-2 may break and patches may have to be
refactored for JDK6. Given that branch-2 got GA status not so long
70 matches
Mail list logo
