Reviving this thread, I noticed there's been a patch and +1 on HADOOP-10530, and I don't think we actually reached a conclusion.
I (and others) have expressed concerns about moving to JDK7 for trunk. Summarizing a few points: - We can't move to JDK7 in branch-2 because of compatibility - branch-2 is currently the only Hadoop release vehicle, there are no plans for a trunk-based Hadoop 3 - Introducing JDK7-only APIs in trunk will increase divergence with branch-2 and make backports harder - Almost all developers care only about branch-2, since it is the only release vehicle With this in mind, I struggle to see any upsides to introducing JDK7-only APIs to trunk. Please let's not do anything on HADOOP-10530 or related until we agree on this. Thanks, Andrew On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran <ste...@hortonworks.com> wrote: > On 14 April 2014 17:46, Andrew Purtell <apurt...@apache.org> wrote: > > > How well is trunk tested? Does anyone deploy it with real applications > > running on top? When will the trunk codebase next be the basis for a > > production release? An impromptu diff of hadoop-common trunk against > > branch-2 as of today is 38,625 lines. Can they be said to be the same > > animal? I ask because any disincentive toward putting code in trunk is > > beside the point, if the only target worth pursuing today is branch-2 > > unless one doesn't care if the code is released for production use. > > Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only matter for > > the vast majority of Hadoopers if talking about branch-2. > > > > > I think its partly a timescale issue; its also because the 1-2 transition > was so significant, especially at the YARN layer, that it's still taking > time to trickle through. > > If you do want code to ship this year, branch-2 is where you are going to > try and get it in -and like you say, that's where things get tried in the > field. At the same time, the constraints of stability are holding us back > -already-. > > I don't see why we should have such another major 1-2 transition in future; > the rate that Arun is pushing out 2.x releases its almost back to the 0.1x > timescale -though at that point most people were fending for themselves and > expectations of stability were less. We do want smaller version increments > in future, which branch-2 is -mostly- delivering. > > While Java 7 doesn't have some must-have features, Java 8 is a significant > improvement in the language, and we should be looking ahead to that, maybe > even doing some leading-edge work on the side, so the same discussion > doesn't come up in two years time when java 7 goes EOL. > > > -steve > > (personal opinions only, etc, ) > > > > > > On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe <cmcc...@alumni.cmu.edu > > >wrote: > > > > > I think the bottom line here is that as long as our stable release > > > uses JDK6, there is going to be a very, very strong disincentive to > > > put any code which can't run on JDK6 into trunk. > > > > > > Like I said earlier, the traditional reason for putting something in > > > trunk but not the stable release is that it needs more testing. If a > > > stable release that drops support for JDK6 is more than a year away, > > > does it make sense to put anything in trunk like that? What might > > > need more than a year of testing? Certainly not changes to > > > LocalFileSystem to use the new APIs. I also don't think an upgrade to > > > various libraries qualifies. > > > > > > It might be best to shelve this for now, like we've done in the past, > > > until we're ready to talk about a stable release that requires JDK7+. > > > At least that's my feeling. > > > > > > If we're really desperate for the new file APIs JDK7 provides, we > > > could consider using loadable modules for it in branch-2. This is > > > similar to how we provide JNI versions of certain things on certain > > > platforms, without dropping support for the other platforms. > > > > > > best, > > > Colin > > > > > > On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata <rst...@altiscale.com> > > > wrote: > > > > There's an outstanding question addressed to me: "Are there > particular > > > > features or new dependencies that you would like to contribute (or > see > > > > contributed) that require using the Java 1.7 APIs?" The question > > > > misses the point: We'd figure out how to write something we wanted to > > > > contribute to Hadoop against the APIs of Java4 if that's what it took > > > > to get them into a stable release. And at current course and speed, > > > > that's how ridiculous things could get. > > > > > > > > To summarize, it seems like there's a vague consensus that it might > be > > > > okay to eventually allow the use of Java7 in trunk, but there's no > > > > decision. And there's been no answer to the concern that even if > such > > > > dependencies were allowed in Java7, the only people using them would > > > > be people who uninterested in getting their patches into a stable > > > > release of Hadoop on any knowable timeframe, which doesn't bode well > > > > for the ability to stabilize that Java7 code when it comes time to > > > > attempt to. > > > > > > > > I don't have more to add, so I'll go back to lurking. It'll be > > > > interesting to see where we'll be standing a year from now. > > > > > > > > On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA > > > > <ozawa.tsuyo...@gmail.com> wrote: > > > >> Hi, > > > >> > > > >> +1 for Karthik's idea(non-binding). > > > >> > > > >> IMO, we should keep the compatibility between JDK 6 and JDK 7 on > both > > > branch-1 > > > >> and branch-2, because users can be using them. For future releases > > that > > > we can > > > >> declare breaking compatibility(e.g. 3.0.0 release), we can use JDK 7 > > > >> features if we > > > >> can get benefits. However, it can increase maintenance costs and > > > distributes the > > > >> efforts of contributions to maintain branches. Then, I think it is > > > >> reasonable approach > > > >> that we use limited and minimum JDK-7 APIs when we have reasons we > > need > > > to use > > > >> the features. > > > >> By the way, if we start to use JDK 7 APIs, we should declare the > basis > > > >> when to use > > > >> JDK 7 APIs on Wiki not to confuse contributors. > > > >> > > > >> Thanks, > > > >> - Tsuyoshi > > > >> > > > >> On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata <rst...@altiscale.com > > > > > wrote: > > > >>>> It might make sense to try to enumerate the benefits of switching > to > > > >>>> Java7 APIs and dependencies. > > > >>> > > > >>> - Java7 introduced a huge number of language, byte-code, API, and > > > >>> tooling enhancements! Just to name a few: try-with-resources, > newer > > > >>> and stronger encyrption methods, more scalable concurrency > > primitives. > > > >>> See http://www.slideshare.net/boulderjug/55-things-in-java-7 > > > >>> > > > >>> - We can't update current dependencies, and we can't add cool new > > > ones. > > > >>> > > > >>> - Putting language/APIs aside, don't forget that a huge amount of > > > effort > > > >>> goes into qualifying for Java6 (at least, I hope the folks claiming > > to > > > >>> support Java6 are putting in such an effort :-). Wouldn't Hadoop > > > >>> users/customers be better served if qualification effort went into > > > >>> Java7/8 versus Java6/7? > > > >>> > > > >>> Getting to Java7 as a development env (and Java8 as a runtime env) > > > >>> seems like a no-brainer. Question is: How? > > > >>> > > > >>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza < > sandy.r...@cloudera.com > > > > > > wrote: > > > >>>> It might make sense to try to enumerate the benefits of switching > to > > > Java7 > > > >>>> APIs and dependencies. IMO, the ones listed so far on this thread > > > don't > > > >>>> make a compelling enough case to drop Java6 in branch-2 on any > time > > > frame, > > > >>>> even if this means supporting Java6 through 2015. For example, > the > > > change > > > >>>> in RawLocalFileSystem semantics might be an incompatible change > for > > > >>>> branch-2 any way. > > > >>>> > > > >>>> > > > >>>> On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla < > > ka...@cloudera.com > > > >wrote: > > > >>>> > > > >>>>> +1 to NOT breaking compatibility in branch-2. > > > >>>>> > > > >>>>> I think it is reasonable to require JDK7 for trunk, if we limit > use > > > of > > > >>>>> JDK7-only API to security fixes etc. If we make other > optimizations > > > (like > > > >>>>> IO), it would be a pain to backport things to branch-2. I guess > > this > > > all > > > >>>>> depends on when we see ourselves shipping Hadoop-3. Any ideas on > > > that? > > > >>>>> > > > >>>>> > > > >>>>> On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins <e...@cloudera.com> > > > wrote: > > > >>>>> > > > >>>>> > On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi > > > >>>>> > <davi.ottenhei...@emc.com> wrote: > > > >>>>> > >> From: Eli Collins [mailto:e...@cloudera.com] > > > >>>>> > >> Sent: Monday, April 07, 2014 11:54 AM > > > >>>>> > >> > > > >>>>> > >> > > > >>>>> > >> IMO we should not drop support for Java 6 in a minor update > > of a > > > >>>>> stable > > > >>>>> > >> release (v2). I don't think the larger Hadoop user base > would > > > find it > > > >>>>> > >> acceptable that upgrading to a minor update caused their > > > systems to > > > >>>>> stop > > > >>>>> > >> working because they didn't upgrade Java. There are people > > still > > > >>>>> getting > > > >>>>> > >> support for Java 6. ... > > > >>>>> > >> > > > >>>>> > >> Thanks, > > > >>>>> > >> Eli > > > >>>>> > > > > > >>>>> > > Hi Eli, > > > >>>>> > > > > > >>>>> > > Technically you are correct those with extended support get > > > critical > > > >>>>> > security fixes for 6 until the end of 2016. I am curious > whether > > > many of > > > >>>>> > those are in the Hadoop user base. Do you know? My guess is the > > > vast > > > >>>>> > majority are within Oracle's official public end of life, which > > > was over > > > >>>>> 12 > > > >>>>> > months ago. Even Premier support ended Dec 2013: > > > >>>>> > > > > > >>>>> > > http://www.oracle.com/technetwork/java/eol-135779.html > > > >>>>> > > > > > >>>>> > > The end of Java 6 support carries much risk. It has to be > > > considered in > > > >>>>> > terms of serious security vulnerabilities such as CVE-2013-2465 > > > with CVSS > > > >>>>> > score 10.0. > > > >>>>> > > > > > >>>>> > > http://www.cvedetails.com/cve/CVE-2013-2465/ > > > >>>>> > > > > > >>>>> > > Since you mentioned "caused systems to stop" as an example of > > > what > > > >>>>> would > > > >>>>> > be a concern to Hadoop users, please note the CVE-2013-2465 > > > availability > > > >>>>> > impact: > > > >>>>> > > > > > >>>>> > > "Complete (There is a total shutdown of the affected > resource. > > > The > > > >>>>> > attacker can render the resource completely unavailable.)" > > > >>>>> > > > > > >>>>> > > This vulnerability was patched in Java 6 Update 51, but post > > end > > > of > > > >>>>> > life. Apple pushed out the update specifically because of this > > > >>>>> > vulnerability (http://support.apple.com/kb/HT5717) as did some > > > other > > > >>>>> > vendors privately, but for the majority of people using Java 6 > > > means they > > > >>>>> > have a ticking time bomb. > > > >>>>> > > > > > >>>>> > > Allowing it to stay should be considered in terms of > accepting > > > the > > > >>>>> whole > > > >>>>> > risk posture. > > > >>>>> > > > > > >>>>> > > > > >>>>> > There are some who get extended support, but I suspect many > just > > > have > > > >>>>> > a if-it's-not-broke mentality when it comes to production > > > deployments. > > > >>>>> > The current code supports both java6 and java7 and so allows > > these > > > >>>>> > people to remain compatible, while enabling others to upgrade > to > > > the > > > >>>>> > java7 runtime. This seems like the right compromise for a > stable > > > >>>>> > release series. Again, absolutely makes sense for trunk (ie v3) > > to > > > >>>>> > require java7 or greater. > > > >>>>> > > > > >>>>> > > > >> > > > >> > > > >> > > > >> -- > > > >> - Tsuyoshi > > > > > > > > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > > (via Tom White) > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
