Hi all, responding to multiple messages here, Arun, thanks for the clarification regarding MR classpaths. It sounds like the story there is improved and still improving.
However, I think we still suffer from this at least on the HDFS side. We have a single JAR for all of HDFS, and our clients need to have all the fun deps like Guava on the classpath. I'm told Spark sticks a newer Guava at the front of the classpath and the HDFS client still works okay, but this is more happy coincidence than anything else. While we're leaking deps, we're in a scary situation. API compat to me means that an app should be able to run on a new minor version of Hadoop and not have anything break. MAPREDUCE-4421 sounds like it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs and have nothing break. If we muck with the classpath, my understanding is that this could break. Owen, bumping the minimum JDK version in a minor release like this should be a one-time exception as Tucu stated. A number of people have pointed out how painful a forced JDK upgrade is for end users, and it's not something we should be springing on them in a minor release unless we're *very* confident like in this case. Chris, thanks for bringing up the ecosystem. For CDH5, we standardized on JDK7 across the CDH stack, so I think that's an indication that most ecosystem projects are ready to make the jump. Is that sufficient in your mind? For the record, I'm also +1 on the Tucu plan. Is it too late to do this for 2.5? I'll offer to help out with some of the mechanics. Thanks, Andrew On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth <cnaur...@hortonworks.com> wrote: > I understood the plan for avoiding JDK7-specific features in our code, and > your suggestion to add an extra Jenkins job is a great way to guard against > that. The thing I haven't seen discussed yet is how downstream projects > will continue to consume our built artifacts. If a downstream project > upgrades to pick up a bug fix, and the jar switches to 1.7 class files, but > their project is still building with 1.6, then it would be a nasty > surprise. > > These are the options I see: > > 1. Make sure all other projects upgrade first. This doesn't sound > feasible, unless all other ecosystem projects have moved to JDK7 already. > If not, then waiting on a single long pole project would hold up our > migration indefinitely. > > 2. We switch to JDK7, but run javac with -target 1.6 until the whole > ecosystem upgrades. I find this undesirable, because in a certain sense, > it still leaves a bit of 1.6 lingering in the project. (I'll assume that > end-of-life for JDK6 also means end-of-life for the 1.6 bytecode format.) > > 3. Just declare a clean break on some version (your earlier email said 2.5) > and start publishing artifacts built with JDK7 and no -target option. > Overall, this is my preferred option. However, as a side effect, this > sets us up for longer-term maintenance and patch releases off of the 2.4 > branch if a downstream project that's still on 1.6 needs to pick up a > critical bug fix. > > Of course, this is all a moot point if all the downstream ecosystem > projects have already made the switch to JDK7. I don't know the status of > that off the top of my head. Maybe someone else out there knows? If not, > then I expect I can free up enough in a few weeks to volunteer for tracking > down that information. > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > > On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur <t...@cloudera.com> > wrote: > > > Chris, > > > > Compiling with jdk7 and doing javac -target 1.6 is not sufficient, you > are > > still using jdk7 libraries and you could use new APIs, thus breaking jdk6 > > both at compile and runtime. > > > > you need to compile with jdk6 to ensure you are not running into that > > scenario. that is why i was suggesting the nightly jdk6 build/test > jenkins > > job. > > > > > > On Wed, Jun 25, 2014 at 2:04 PM, Chris Nauroth <cnaur...@hortonworks.com > > > > wrote: > > > > > I'm also +1 for getting us to JDK7 within the 2.x line after reading > the > > > proposals and catching up on the discussion in this thread. > > > > > > Has anyone yet considered how to coordinate this change with downstream > > > projects? Would we request downstream projects to upgrade to JDK7 > first > > > before we make the move? Would we switch to JDK7, but run javac > -target > > > 1.6 to maintain compatibility for downstream projects during an interim > > > period? > > > > > > Chris Nauroth > > > Hortonworks > > > http://hortonworks.com/ > > > > > > > > > > > > On Wed, Jun 25, 2014 at 9:48 AM, Owen O'Malley <omal...@apache.org> > > wrote: > > > > > > > On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur < > t...@cloudera.com > > > > > > > wrote: > > > > > > > > > After reading this thread and thinking a bit about it, I think it > > > should > > > > be > > > > > OK such move up to JDK7 in Hadoop > > > > > > > > > > > > I agree with Alejandro. Changing minimum JDKs is not an incompatible > > > change > > > > and is fine in the 2 branch. (Although I think it is would *not* be > > > > appropriate for a patch release.) Of course we need to do it with > > > > forethought and testing, but moving off of JDK 6, which is EOL'ed is > a > > > good > > > > thing. Moving to Java 8 as a minimum seems much too aggressive and I > > > would > > > > push back on that. > > > > > > > > I'm also think that we need to let the dust settle on the Hadoop 2 > line > > > for > > > > a while before we talk about Hadoop 3. It seems that it has only been > > in > > > > the last 6 months that Hadoop 2 adoption has reached the main stream > > > users. > > > > Our user community needs time to digest the changes in Hadoop 2.x > > before > > > we > > > > fracture the community by starting to discuss Hadoop 3 releases. > > > > > > > > .. Owen > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > > > > > -- > > Alejandro > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
