for #1-3, the answer is likely No. Recently we upgrade to Spark 1.5.1, with CDH5.3, CDH5.4 and HDP2.2 and others.
We were using CDH5.3 client to talk to CDH5.4. We were doing this to see if we support many different hadoop cluster versions without changing the build. This was ok for yarn-cluster spark 1.3.1, but could not get spark 1.5.1 started. We upgrade the client to CDH5.4, then everything works. There are API changes between Apache 2.4 and 2.6, not sure you can mix match them. Chester On Fri, Nov 20, 2015 at 1:59 PM, Sandy Ryza <sandy.r...@cloudera.com> wrote: > To answer your fourth question from Cloudera's perspective, we would never > support a customer running Spark 2.0 on a Hadoop version < 2.6. > > -Sandy > > On Fri, Nov 20, 2015 at 1:39 PM, Reynold Xin <r...@databricks.com> wrote: > >> OK I'm not exactly asking for a vote here :) >> >> I don't think we should look at it from only maintenance point of view -- >> because in that case the answer is clearly supporting as few versions as >> possible (or just rm -rf spark source code and call it a day). It is a >> tradeoff between the number of users impacted and the maintenance burden. >> >> So a few questions for those more familiar with Hadoop: >> >> 1. Can Hadoop 2.6 client read Hadoop 2.4 / 2.3? >> >> 2. If the answer to 1 is yes, are there known, major issues with backward >> compatibility? >> >> 3. Can Hadoop 2.6+ YARN work on older versions of YARN clusters? >> >> 4. (for Hadoop vendors) When did/will support for Hadoop 2.4 and below >> stop? To what extent do you care about running Spark on older Hadoop >> clusters. >> >> >> >> On Fri, Nov 20, 2015 at 7:52 AM, Steve Loughran <ste...@hortonworks.com> >> wrote: >> >>> >>> On 20 Nov 2015, at 14:28, ches...@alpinenow.com wrote: >>> >>> Assuming we have 1.6 and 1.7 releases, then spark 2.0 is about 9 months >>> away. >>> >>> customer will need to upgrade the new Hadoop clusters to Apache 2.6 or >>> later to leverage new spark 2.0 in one year. I think this possible as >>> latest release on cdh5.x, HDP 2.x are both on Apache 2.6.0 already. >>> Company will have enough time to upgrade cluster. >>> >>> +1 for me as well >>> >>> Chester >>> >>> >>> now, if you are looking that far ahead, the other big issue is "when to >>> retire Java 7 support".? >>> >>> That's a tough decision for all projects. Hadoop 3.x will be Java 8 >>> only, but nobody has committed the patch to the trunk codebase to force a >>> java 8 build; + most of *todays* hadoop clusters are Java 7. But as you >>> can't even download a Java 7 JDK for the desktop from oracle any more >>> today, 2016 is a time to look at the language support and decide what is >>> the baseline version >>> >>> Commentary from Twitter here -as they point out, it's not just the >>> server farm that matters, it's all the apps that talk to it >>> >>> >>> >>> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201503.mbox/%3ccab7mwte+kefcxsr6n46-ztcs19ed7cwc9vobtr1jqewdkye...@mail.gmail.com%3E >>> >>> -Steve >>> >> >> >
