I think we should come up with a plan for when the next Hadoop release will drop support for JDK6. We all know that day needs to come... the only question is when. I agree that writing the JDK7-only code doesn't seem very productive unless we have a plan for when it will be released and usable.
best, Colin On Tue, Jun 17, 2014 at 10:08 PM, Andrew Wang <andrew.w...@cloudera.com> wrote: > Reviving this thread, I noticed there's been a patch and +1 on > HADOOP-10530, and I don't think we actually reached a conclusion. > > I (and others) have expressed concerns about moving to JDK7 for trunk. > Summarizing a few points: > > - We can't move to JDK7 in branch-2 because of compatibility > - branch-2 is currently the only Hadoop release vehicle, there are no plans > for a trunk-based Hadoop 3 > - Introducing JDK7-only APIs in trunk will increase divergence with > branch-2 and make backports harder > - Almost all developers care only about branch-2, since it is the only > release vehicle > > With this in mind, I struggle to see any upsides to introducing JDK7-only > APIs to trunk. Please let's not do anything on HADOOP-10530 or related > until we agree on this. > > Thanks, > Andrew > > > On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran <ste...@hortonworks.com> > wrote: > >> On 14 April 2014 17:46, Andrew Purtell <apurt...@apache.org> wrote: >> >> > How well is trunk tested? Does anyone deploy it with real applications >> > running on top? When will the trunk codebase next be the basis for a >> > production release? An impromptu diff of hadoop-common trunk against >> > branch-2 as of today is 38,625 lines. Can they be said to be the same >> > animal? I ask because any disincentive toward putting code in trunk is >> > beside the point, if the only target worth pursuing today is branch-2 >> > unless one doesn't care if the code is released for production use. >> > Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only matter for >> > the vast majority of Hadoopers if talking about branch-2. >> > >> > >> I think its partly a timescale issue; its also because the 1-2 transition >> was so significant, especially at the YARN layer, that it's still taking >> time to trickle through. >> >> If you do want code to ship this year, branch-2 is where you are going to >> try and get it in -and like you say, that's where things get tried in the >> field. At the same time, the constraints of stability are holding us back >> -already-. >> >> I don't see why we should have such another major 1-2 transition in future; >> the rate that Arun is pushing out 2.x releases its almost back to the 0.1x >> timescale -though at that point most people were fending for themselves and >> expectations of stability were less. We do want smaller version increments >> in future, which branch-2 is -mostly- delivering. >> >> While Java 7 doesn't have some must-have features, Java 8 is a significant >> improvement in the language, and we should be looking ahead to that, maybe >> even doing some leading-edge work on the side, so the same discussion >> doesn't come up in two years time when java 7 goes EOL. >> >> >> -steve >> >> (personal opinions only, etc, ) >> >> >> > >> > On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe <cmcc...@alumni.cmu.edu >> > >wrote: >> > >> > > I think the bottom line here is that as long as our stable release >> > > uses JDK6, there is going to be a very, very strong disincentive to >> > > put any code which can't run on JDK6 into trunk. >> > > >> > > Like I said earlier, the traditional reason for putting something in >> > > trunk but not the stable release is that it needs more testing. If a >> > > stable release that drops support for JDK6 is more than a year away, >> > > does it make sense to put anything in trunk like that? What might >> > > need more than a year of testing? Certainly not changes to >> > > LocalFileSystem to use the new APIs. I also don't think an upgrade to >> > > various libraries qualifies. >> > > >> > > It might be best to shelve this for now, like we've done in the past, >> > > until we're ready to talk about a stable release that requires JDK7+. >> > > At least that's my feeling. >> > > >> > > If we're really desperate for the new file APIs JDK7 provides, we >> > > could consider using loadable modules for it in branch-2. This is >> > > similar to how we provide JNI versions of certain things on certain >> > > platforms, without dropping support for the other platforms. >> > > >> > > best, >> > > Colin >> > > >> > > On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata <rst...@altiscale.com> >> > > wrote: >> > > > There's an outstanding question addressed to me: "Are there >> particular >> > > > features or new dependencies that you would like to contribute (or >> see >> > > > contributed) that require using the Java 1.7 APIs?" The question >> > > > misses the point: We'd figure out how to write something we wanted to >> > > > contribute to Hadoop against the APIs of Java4 if that's what it took >> > > > to get them into a stable release. And at current course and speed, >> > > > that's how ridiculous things could get. >> > > > >> > > > To summarize, it seems like there's a vague consensus that it might >> be >> > > > okay to eventually allow the use of Java7 in trunk, but there's no >> > > > decision. And there's been no answer to the concern that even if >> such >> > > > dependencies were allowed in Java7, the only people using them would >> > > > be people who uninterested in getting their patches into a stable >> > > > release of Hadoop on any knowable timeframe, which doesn't bode well >> > > > for the ability to stabilize that Java7 code when it comes time to >> > > > attempt to. >> > > > >> > > > I don't have more to add, so I'll go back to lurking. It'll be >> > > > interesting to see where we'll be standing a year from now. >> > > > >> > > > On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA >> > > > <ozawa.tsuyo...@gmail.com> wrote: >> > > >> Hi, >> > > >> >> > > >> +1 for Karthik's idea(non-binding). >> > > >> >> > > >> IMO, we should keep the compatibility between JDK 6 and JDK 7 on >> both >> > > branch-1 >> > > >> and branch-2, because users can be using them. For future releases >> > that >> > > we can >> > > >> declare breaking compatibility(e.g. 3.0.0 release), we can use JDK 7 >> > > >> features if we >> > > >> can get benefits. However, it can increase maintenance costs and >> > > distributes the >> > > >> efforts of contributions to maintain branches. Then, I think it is >> > > >> reasonable approach >> > > >> that we use limited and minimum JDK-7 APIs when we have reasons we >> > need >> > > to use >> > > >> the features. >> > > >> By the way, if we start to use JDK 7 APIs, we should declare the >> basis >> > > >> when to use >> > > >> JDK 7 APIs on Wiki not to confuse contributors. >> > > >> >> > > >> Thanks, >> > > >> - Tsuyoshi >> > > >> >> > > >> On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata <rst...@altiscale.com >> > >> > > wrote: >> > > >>>> It might make sense to try to enumerate the benefits of switching >> to >> > > >>>> Java7 APIs and dependencies. >> > > >>> >> > > >>> - Java7 introduced a huge number of language, byte-code, API, and >> > > >>> tooling enhancements! Just to name a few: try-with-resources, >> newer >> > > >>> and stronger encyrption methods, more scalable concurrency >> > primitives. >> > > >>> See http://www.slideshare.net/boulderjug/55-things-in-java-7 >> > > >>> >> > > >>> - We can't update current dependencies, and we can't add cool new >> > > ones. >> > > >>> >> > > >>> - Putting language/APIs aside, don't forget that a huge amount of >> > > effort >> > > >>> goes into qualifying for Java6 (at least, I hope the folks claiming >> > to >> > > >>> support Java6 are putting in such an effort :-). Wouldn't Hadoop >> > > >>> users/customers be better served if qualification effort went into >> > > >>> Java7/8 versus Java6/7? >> > > >>> >> > > >>> Getting to Java7 as a development env (and Java8 as a runtime env) >> > > >>> seems like a no-brainer. Question is: How? >> > > >>> >> > > >>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza < >> sandy.r...@cloudera.com >> > > >> > > wrote: >> > > >>>> It might make sense to try to enumerate the benefits of switching >> to >> > > Java7 >> > > >>>> APIs and dependencies. IMO, the ones listed so far on this thread >> > > don't >> > > >>>> make a compelling enough case to drop Java6 in branch-2 on any >> time >> > > frame, >> > > >>>> even if this means supporting Java6 through 2015. For example, >> the >> > > change >> > > >>>> in RawLocalFileSystem semantics might be an incompatible change >> for >> > > >>>> branch-2 any way. >> > > >>>> >> > > >>>> >> > > >>>> On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla < >> > ka...@cloudera.com >> > > >wrote: >> > > >>>> >> > > >>>>> +1 to NOT breaking compatibility in branch-2. >> > > >>>>> >> > > >>>>> I think it is reasonable to require JDK7 for trunk, if we limit >> use >> > > of >> > > >>>>> JDK7-only API to security fixes etc. If we make other >> optimizations >> > > (like >> > > >>>>> IO), it would be a pain to backport things to branch-2. I guess >> > this >> > > all >> > > >>>>> depends on when we see ourselves shipping Hadoop-3. Any ideas on >> > > that? >> > > >>>>> >> > > >>>>> >> > > >>>>> On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins <e...@cloudera.com> >> > > wrote: >> > > >>>>> >> > > >>>>> > On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi >> > > >>>>> > <davi.ottenhei...@emc.com> wrote: >> > > >>>>> > >> From: Eli Collins [mailto:e...@cloudera.com] >> > > >>>>> > >> Sent: Monday, April 07, 2014 11:54 AM >> > > >>>>> > >> >> > > >>>>> > >> >> > > >>>>> > >> IMO we should not drop support for Java 6 in a minor update >> > of a >> > > >>>>> stable >> > > >>>>> > >> release (v2). I don't think the larger Hadoop user base >> would >> > > find it >> > > >>>>> > >> acceptable that upgrading to a minor update caused their >> > > systems to >> > > >>>>> stop >> > > >>>>> > >> working because they didn't upgrade Java. There are people >> > still >> > > >>>>> getting >> > > >>>>> > >> support for Java 6. ... >> > > >>>>> > >> >> > > >>>>> > >> Thanks, >> > > >>>>> > >> Eli >> > > >>>>> > > >> > > >>>>> > > Hi Eli, >> > > >>>>> > > >> > > >>>>> > > Technically you are correct those with extended support get >> > > critical >> > > >>>>> > security fixes for 6 until the end of 2016. I am curious >> whether >> > > many of >> > > >>>>> > those are in the Hadoop user base. Do you know? My guess is the >> > > vast >> > > >>>>> > majority are within Oracle's official public end of life, which >> > > was over >> > > >>>>> 12 >> > > >>>>> > months ago. Even Premier support ended Dec 2013: >> > > >>>>> > > >> > > >>>>> > > http://www.oracle.com/technetwork/java/eol-135779.html >> > > >>>>> > > >> > > >>>>> > > The end of Java 6 support carries much risk. It has to be >> > > considered in >> > > >>>>> > terms of serious security vulnerabilities such as CVE-2013-2465 >> > > with CVSS >> > > >>>>> > score 10.0. >> > > >>>>> > > >> > > >>>>> > > http://www.cvedetails.com/cve/CVE-2013-2465/ >> > > >>>>> > > >> > > >>>>> > > Since you mentioned "caused systems to stop" as an example of >> > > what >> > > >>>>> would >> > > >>>>> > be a concern to Hadoop users, please note the CVE-2013-2465 >> > > availability >> > > >>>>> > impact: >> > > >>>>> > > >> > > >>>>> > > "Complete (There is a total shutdown of the affected >> resource. >> > > The >> > > >>>>> > attacker can render the resource completely unavailable.)" >> > > >>>>> > > >> > > >>>>> > > This vulnerability was patched in Java 6 Update 51, but post >> > end >> > > of >> > > >>>>> > life. Apple pushed out the update specifically because of this >> > > >>>>> > vulnerability (http://support.apple.com/kb/HT5717) as did some >> > > other >> > > >>>>> > vendors privately, but for the majority of people using Java 6 >> > > means they >> > > >>>>> > have a ticking time bomb. >> > > >>>>> > > >> > > >>>>> > > Allowing it to stay should be considered in terms of >> accepting >> > > the >> > > >>>>> whole >> > > >>>>> > risk posture. >> > > >>>>> > > >> > > >>>>> > >> > > >>>>> > There are some who get extended support, but I suspect many >> just >> > > have >> > > >>>>> > a if-it's-not-broke mentality when it comes to production >> > > deployments. >> > > >>>>> > The current code supports both java6 and java7 and so allows >> > these >> > > >>>>> > people to remain compatible, while enabling others to upgrade >> to >> > > the >> > > >>>>> > java7 runtime. This seems like the right compromise for a >> stable >> > > >>>>> > release series. Again, absolutely makes sense for trunk (ie v3) >> > to >> > > >>>>> > require java7 or greater. >> > > >>>>> > >> > > >>>>> >> > > >> >> > > >> >> > > >> >> > > >> -- >> > > >> - Tsuyoshi >> > > >> > >> > >> > >> > -- >> > Best regards, >> > >> > - Andy >> > >> > Problems worthy of attack prove their worth by hitting back. - Piet Hein >> > (via Tom White) >> > >> >> -- >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity to >> which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >>
