I also think we need to recognise that its been three months since that
last discussion, and Java 6 has not suddenly burst back into popularity
- nobody providing commercial support for Hadoop is offering branch-2
support on Java 6 AFAIK
- therefore, nobody is testing it at scale except privately, and they
aren't reporting bugs if they are
- if someone actually did file a bug on something on branch-2 which
didn't work on Java 6 but went away on Java7+, we'd probably close it as a
WORKSFORME
whether we acknowledge it or not, Hadoop 2.x is now really Java 7+.
We do all agree that hadoop 3 will not be java 6, so the only issue is
"when and how to make that transition".
That patch of mine just makes it possible to do today.
I have actually jumped to Java7 in the slider project, and actually being
using Java 8 and twill; the new language features there are significant and
would be great to use in Hadoop *at some point in the future*
For Java 7 though, based on that experience, the language changes are
convenient but not essential
- try-with-resources simply swallows close failures without the log
integration we have with IOUtils.closeStream(), so shoudn't be used in
hadoop core anyway.
- string based switching: convenient, but not critical
- type inference on template constructors. Modern IDEs handle the pain
anyway
The only feature I like is multi-catch and typed rethrow
catch(IOException | ExitException e) {
log.warn(e.toString();
throw e;
}
this would make "e" look like Exception, but when rethrown go back to its
original type.
This reduces duplicate work, and is the bit l actually value. Is it enough
to justify making code incompatible across branches? No.
So i'm going to propose this, and would like to start a vote on it soon
1. we parameterize java versions in the POMs on all branches, with
separate JDK versions and Java language
2. branch-2: java-6-language and JDK-6 minimum JDK
3. trunk: java-6-language and JDK-7 minimum JDK
This would guarantee that none of the java 7 language features went in, but
we could move trunk up to java 7+ only libraries (jersey, guava). Adopting
JDK7 features then becomes no more different from adopting java7+
libraries: those bits of code that have moved can't be backported.
-Steve
On 17 June 2014 22:08, Andrew Wang <andrew.w...@cloudera.com> wrote:
> Reviving this thread, I noticed there's been a patch and +1 on
> HADOOP-10530, and I don't think we actually reached a conclusion.
>
> I (and others) have expressed concerns about moving to JDK7 for trunk.
> Summarizing a few points:
>
> - We can't move to JDK7 in branch-2 because of compatibility
> - branch-2 is currently the only Hadoop release vehicle, there are no plans
> for a trunk-based Hadoop 3
> - Introducing JDK7-only APIs in trunk will increase divergence with
> branch-2 and make backports harder
> - Almost all developers care only about branch-2, since it is the only
> release vehicle
>
> With this in mind, I struggle to see any upsides to introducing JDK7-only
> APIs to trunk. Please let's not do anything on HADOOP-10530 or related
> until we agree on this.
>
> Thanks,
> Andrew
>
>
> On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran <ste...@hortonworks.com>
> wrote:
>
> > On 14 April 2014 17:46, Andrew Purtell <apurt...@apache.org> wrote:
> >
> > > How well is trunk tested? Does anyone deploy it with real applications
> > > running on top? When will the trunk codebase next be the basis for a
> > > production release? An impromptu diff of hadoop-common trunk against
> > > branch-2 as of today is 38,625 lines. Can they be said to be the same
> > > animal? I ask because any disincentive toward putting code in trunk is
> > > beside the point, if the only target worth pursuing today is branch-2
> > > unless one doesn't care if the code is released for production use.
> > > Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only matter
> for
> > > the vast majority of Hadoopers if talking about branch-2.
> > >
> > >
> > I think its partly a timescale issue; its also because the 1-2 transition
> > was so significant, especially at the YARN layer, that it's still taking
> > time to trickle through.
> >
> > If you do want code to ship this year, branch-2 is where you are going to
> > try and get it in -and like you say, that's where things get tried in the
> > field. At the same time, the constraints of stability are holding us back
> > -already-.
> >
> > I don't see why we should have such another major 1-2 transition in
> future;
> > the rate that Arun is pushing out 2.x releases its almost back to the
> 0.1x
> > timescale -though at that point most people were fending for themselves
> and
> > expectations of stability were less. We do want smaller version
> increments
> > in future, which branch-2 is -mostly- delivering.
> >
> > While Java 7 doesn't have some must-have features, Java 8 is a
> significant
> > improvement in the language, and we should be looking ahead to that,
> maybe
> > even doing some leading-edge work on the side, so the same discussion
> > doesn't come up in two years time when java 7 goes EOL.
> >
> >
> > -steve
> >
> > (personal opinions only, etc, )
> >
> >
> > >
> > > On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe <cmcc...@alumni.cmu.edu
> > > >wrote:
> > >
> > > > I think the bottom line here is that as long as our stable release
> > > > uses JDK6, there is going to be a very, very strong disincentive to
> > > > put any code which can't run on JDK6 into trunk.
> > > >
> > > > Like I said earlier, the traditional reason for putting something in
> > > > trunk but not the stable release is that it needs more testing. If a
> > > > stable release that drops support for JDK6 is more than a year away,
> > > > does it make sense to put anything in trunk like that? What might
> > > > need more than a year of testing? Certainly not changes to
> > > > LocalFileSystem to use the new APIs. I also don't think an upgrade
> to
> > > > various libraries qualifies.
> > > >
> > > > It might be best to shelve this for now, like we've done in the past,
> > > > until we're ready to talk about a stable release that requires JDK7+.
> > > > At least that's my feeling.
> > > >
> > > > If we're really desperate for the new file APIs JDK7 provides, we
> > > > could consider using loadable modules for it in branch-2. This is
> > > > similar to how we provide JNI versions of certain things on certain
> > > > platforms, without dropping support for the other platforms.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata <rst...@altiscale.com
> >
> > > > wrote:
> > > > > There's an outstanding question addressed to me: "Are there
> > particular
> > > > > features or new dependencies that you would like to contribute (or
> > see
> > > > > contributed) that require using the Java 1.7 APIs?" The question
> > > > > misses the point: We'd figure out how to write something we wanted
> to
> > > > > contribute to Hadoop against the APIs of Java4 if that's what it
> took
> > > > > to get them into a stable release. And at current course and
> speed,
> > > > > that's how ridiculous things could get.
> > > > >
> > > > > To summarize, it seems like there's a vague consensus that it might
> > be
> > > > > okay to eventually allow the use of Java7 in trunk, but there's no
> > > > > decision. And there's been no answer to the concern that even if
> > such
> > > > > dependencies were allowed in Java7, the only people using them
> would
> > > > > be people who uninterested in getting their patches into a stable
> > > > > release of Hadoop on any knowable timeframe, which doesn't bode
> well
> > > > > for the ability to stabilize that Java7 code when it comes time to
> > > > > attempt to.
> > > > >
> > > > > I don't have more to add, so I'll go back to lurking. It'll be
> > > > > interesting to see where we'll be standing a year from now.
> > > > >
> > > > > On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA
> > > > > <ozawa.tsuyo...@gmail.com> wrote:
> > > > >> Hi,
> > > > >>
> > > > >> +1 for Karthik's idea(non-binding).
> > > > >>
> > > > >> IMO, we should keep the compatibility between JDK 6 and JDK 7 on
> > both
> > > > branch-1
> > > > >> and branch-2, because users can be using them. For future releases
> > > that
> > > > we can
> > > > >> declare breaking compatibility(e.g. 3.0.0 release), we can use
> JDK 7
> > > > >> features if we
> > > > >> can get benefits. However, it can increase maintenance costs and
> > > > distributes the
> > > > >> efforts of contributions to maintain branches. Then, I think it is
> > > > >> reasonable approach
> > > > >> that we use limited and minimum JDK-7 APIs when we have reasons we
> > > need
> > > > to use
> > > > >> the features.
> > > > >> By the way, if we start to use JDK 7 APIs, we should declare the
> > basis
> > > > >> when to use
> > > > >> JDK 7 APIs on Wiki not to confuse contributors.
> > > > >>
> > > > >> Thanks,
> > > > >> - Tsuyoshi
> > > > >>
> > > > >> On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata <
> rst...@altiscale.com
> > >
> > > > wrote:
> > > > >>>> It might make sense to try to enumerate the benefits of
> switching
> > to
> > > > >>>> Java7 APIs and dependencies.
> > > > >>>
> > > > >>> - Java7 introduced a huge number of language, byte-code, API,
> and
> > > > >>> tooling enhancements! Just to name a few: try-with-resources,
> > newer
> > > > >>> and stronger encyrption methods, more scalable concurrency
> > > primitives.
> > > > >>> See http://www.slideshare.net/boulderjug/55-things-in-java-7
> > > > >>>
> > > > >>> - We can't update current dependencies, and we can't add cool
> new
> > > > ones.
> > > > >>>
> > > > >>> - Putting language/APIs aside, don't forget that a huge amount
> of
> > > > effort
> > > > >>> goes into qualifying for Java6 (at least, I hope the folks
> claiming
> > > to
> > > > >>> support Java6 are putting in such an effort :-). Wouldn't Hadoop
> > > > >>> users/customers be better served if qualification effort went
> into
> > > > >>> Java7/8 versus Java6/7?
> > > > >>>
> > > > >>> Getting to Java7 as a development env (and Java8 as a runtime
> env)
> > > > >>> seems like a no-brainer. Question is: How?
> > > > >>>
> > > > >>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza <
> > sandy.r...@cloudera.com
> > > >
> > > > wrote:
> > > > >>>> It might make sense to try to enumerate the benefits of
> switching
> > to
> > > > Java7
> > > > >>>> APIs and dependencies. IMO, the ones listed so far on this
> thread
> > > > don't
> > > > >>>> make a compelling enough case to drop Java6 in branch-2 on any
> > time
> > > > frame,
> > > > >>>> even if this means supporting Java6 through 2015. For example,
> > the
> > > > change
> > > > >>>> in RawLocalFileSystem semantics might be an incompatible change
> > for
> > > > >>>> branch-2 any way.
> > > > >>>>
> > > > >>>>
> > > > >>>> On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla <
> > > ka...@cloudera.com
> > > > >wrote:
> > > > >>>>
> > > > >>>>> +1 to NOT breaking compatibility in branch-2.
> > > > >>>>>
> > > > >>>>> I think it is reasonable to require JDK7 for trunk, if we limit
> > use
> > > > of
> > > > >>>>> JDK7-only API to security fixes etc. If we make other
> > optimizations
> > > > (like
> > > > >>>>> IO), it would be a pain to backport things to branch-2. I guess
> > > this
> > > > all
> > > > >>>>> depends on when we see ourselves shipping Hadoop-3. Any ideas
> on
> > > > that?
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins <e...@cloudera.com>
> > > > wrote:
> > > > >>>>>
> > > > >>>>> > On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi
> > > > >>>>> > <davi.ottenhei...@emc.com> wrote:
> > > > >>>>> > >> From: Eli Collins [mailto:e...@cloudera.com]
> > > > >>>>> > >> Sent: Monday, April 07, 2014 11:54 AM
> > > > >>>>> > >>
> > > > >>>>> > >>
> > > > >>>>> > >> IMO we should not drop support for Java 6 in a minor
> update
> > > of a
> > > > >>>>> stable
> > > > >>>>> > >> release (v2). I don't think the larger Hadoop user base
> > would
> > > > find it
> > > > >>>>> > >> acceptable that upgrading to a minor update caused their
> > > > systems to
> > > > >>>>> stop
> > > > >>>>> > >> working because they didn't upgrade Java. There are people
> > > still
> > > > >>>>> getting
> > > > >>>>> > >> support for Java 6. ...
> > > > >>>>> > >>
> > > > >>>>> > >> Thanks,
> > > > >>>>> > >> Eli
> > > > >>>>> > >
> > > > >>>>> > > Hi Eli,
> > > > >>>>> > >
> > > > >>>>> > > Technically you are correct those with extended support get
> > > > critical
> > > > >>>>> > security fixes for 6 until the end of 2016. I am curious
> > whether
> > > > many of
> > > > >>>>> > those are in the Hadoop user base. Do you know? My guess is
> the
> > > > vast
> > > > >>>>> > majority are within Oracle's official public end of life,
> which
> > > > was over
> > > > >>>>> 12
> > > > >>>>> > months ago. Even Premier support ended Dec 2013:
> > > > >>>>> > >
> > > > >>>>> > > http://www.oracle.com/technetwork/java/eol-135779.html
> > > > >>>>> > >
> > > > >>>>> > > The end of Java 6 support carries much risk. It has to be
> > > > considered in
> > > > >>>>> > terms of serious security vulnerabilities such as
> CVE-2013-2465
> > > > with CVSS
> > > > >>>>> > score 10.0.
> > > > >>>>> > >
> > > > >>>>> > > http://www.cvedetails.com/cve/CVE-2013-2465/
> > > > >>>>> > >
> > > > >>>>> > > Since you mentioned "caused systems to stop" as an example
> of
> > > > what
> > > > >>>>> would
> > > > >>>>> > be a concern to Hadoop users, please note the CVE-2013-2465
> > > > availability
> > > > >>>>> > impact:
> > > > >>>>> > >
> > > > >>>>> > > "Complete (There is a total shutdown of the affected
> > resource.
> > > > The
> > > > >>>>> > attacker can render the resource completely unavailable.)"
> > > > >>>>> > >
> > > > >>>>> > > This vulnerability was patched in Java 6 Update 51, but
> post
> > > end
> > > > of
> > > > >>>>> > life. Apple pushed out the update specifically because of
> this
> > > > >>>>> > vulnerability (http://support.apple.com/kb/HT5717) as did
> some
> > > > other
> > > > >>>>> > vendors privately, but for the majority of people using Java
> 6
> > > > means they
> > > > >>>>> > have a ticking time bomb.
> > > > >>>>> > >
> > > > >>>>> > > Allowing it to stay should be considered in terms of
> > accepting
> > > > the
> > > > >>>>> whole
> > > > >>>>> > risk posture.
> > > > >>>>> > >
> > > > >>>>> >
> > > > >>>>> > There are some who get extended support, but I suspect many
> > just
> > > > have
> > > > >>>>> > a if-it's-not-broke mentality when it comes to production
> > > > deployments.
> > > > >>>>> > The current code supports both java6 and java7 and so allows
> > > these
> > > > >>>>> > people to remain compatible, while enabling others to upgrade
> > to
> > > > the
> > > > >>>>> > java7 runtime. This seems like the right compromise for a
> > stable
> > > > >>>>> > release series. Again, absolutely makes sense for trunk (ie
> v3)
> > > to
> > > > >>>>> > require java7 or greater.
> > > > >>>>> >
> > > > >>>>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> - Tsuyoshi
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
