Re: Dropping support for earlier Hadoop versions in Spark 2.0?

On Fri, Nov 20, 2015 at 10:39 PM, Reynold Xin wrote: > I don't think we should look at it from only maintenance point of view -- > because in that case the answer is clearly supporting as few versions as > possible (or just rm -rf spark source code and call it a day). It is a

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

> On 20 Nov 2015, at 21:39, Reynold Xin wrote: > > OK I'm not exactly asking for a vote here :) > > I don't think we should look at it from only maintenance point of view -- > because in that case the answer is clearly supporting as few versions as > possible (or just rm

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

On 19 Nov 2015, at 22:14, Reynold Xin > wrote: I proposed dropping support for Hadoop 1.x in the Spark 2.0 email, and I think everybody is for that. https://issues.apache.org/jira/browse/SPARK-11807 Sean suggested also dropping support for

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

On 20 Nov 2015, at 14:28, ches...@alpinenow.com wrote: Assuming we have 1.6 and 1.7 releases, then spark 2.0 is about 9 months away. customer will need to upgrade the new Hadoop clusters to Apache 2.6 or later to leverage new spark 2.0 in one year. I think this

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

To answer your fourth question from Cloudera's perspective, we would never support a customer running Spark 2.0 on a Hadoop version < 2.6. -Sandy On Fri, Nov 20, 2015 at 1:39 PM, Reynold Xin wrote: > OK I'm not exactly asking for a vote here :) > > I don't think we should

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

OK I'm not exactly asking for a vote here :) I don't think we should look at it from only maintenance point of view -- because in that case the answer is clearly supporting as few versions as possible (or just rm -rf spark source code and call it a day). It is a tradeoff between the number of

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

+1. Hadoop 2.6 would be a good choice with many features added (like supporting long running service, label based scheduling). Currently there's lot of reflection codes to support multiple version of Yarn, so upgrading to a newer version will really ease the pain :). Thanks Saisai On Fri, Nov

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

for #1-3, the answer is likely No. Recently we upgrade to Spark 1.5.1, with CDH5.3, CDH5.4 and HDP2.2 and others. We were using CDH5.3 client to talk to CDH5.4. We were doing this to see if we support many different hadoop cluster versions without changing the build. This was ok for

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

..@databricks.com> > 日 期：2015年11月20日 06:14:44 > 收件人：dev@spark.apache.org<dev@spark.apache.org>; Sean Owen<sro...@gmail.com>; > Thomas Graves<tgra...@apache.org> > 主 题：Dropping support for earlier Hadoop versions in Spark 2.0? > > > I proposed dropping supp

回复:Dropping support for earlier Hadoop versions in Spark 2.0?

I agreed +1--发件人：Reynold Xin<r...@databricks.com>日　期：2015年11月20日 06:14:44收件人：dev@spark.apache.org<dev@spark.apache.org>; Sean Owen<sro...@gmail.com>; Thomas Graves<tgra...@apache.org>主　题：Dropping sup

Dropping support for earlier Hadoop versions in Spark 2.0?

I proposed dropping support for Hadoop 1.x in the Spark 2.0 email, and I think everybody is for that. https://issues.apache.org/jira/browse/SPARK-11807 Sean suggested also dropping support for Hadoop 2.2, 2.3, and 2.4. That is to say, keep only Hadoop 2.6 and greater. What are the community's

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

+1 On 19 November 2015 at 14:14, Reynold Xin wrote: > I proposed dropping support for Hadoop 1.x in the Spark 2.0 email, and I > think everybody is for that. > > https://issues.apache.org/jira/browse/SPARK-11807 > > Sean suggested also dropping support for Hadoop 2.2, 2.3,

Re: Dropping support for earlier Hadoop versions in Spark 2.0?

+1 Regards JB On 11/19/2015 11:14 PM, Reynold Xin wrote: I proposed dropping support for Hadoop 1.x in the Spark 2.0 email, and I think everybody is for that. https://issues.apache.org/jira/browse/SPARK-11807 Sean suggested also dropping support for Hadoop 2.2, 2.3, and 2.4. That is to say,