Re: Hadoop 3 support

> On 16 Oct 2018, at 22:06, t4 wrote: > > has anyone got spark jars working with hadoop3.1 that they can share? i am > looking to be able to use the latest hadoop-aws fixes from v3.1 we do, but we do with * a patched hive JAR * bulding spark with

Re: Hadoop 3 support

See the discussion at https://github.com/apache/spark/pull/21588 2018년 10월 17일 (수) 오전 5:06, t4 님이 작성: > has anyone got spark jars working with hadoop3.1 that they can share? i am > looking to be able to use the latest hadoop-aws fixes from v3.1 > > > > -- > Sent from:

Re: Hadoop 3 support

has anyone got spark jars working with hadoop3.1 that they can share? i am looking to be able to use the latest hadoop-aws fixes from v3.1 -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To

Re: Hadoop 3 support

pril 3, 2018 1:33 PM Subject: Re: Hadoop 3 support To: Apache Spark Dev <dev@spark.apache.org> On 3 Apr 2018, at 01:30, Saisai Shao <sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> wrote: Yes, the main blocking issue is the hive version used in Spark (1.2.1.spark) doesn't

Re: Hadoop 3 support

On 3 Apr 2018, at 01:30, Saisai Shao > wrote: Yes, the main blocking issue is the hive version used in Spark (1.2.1.spark) doesn't support run on Hadoop 3. Hive will check the Hadoop version in the runtime [1]. Besides this I think some

Re: Hadoop 3 support

Yes, the main blocking issue is the hive version used in Spark (1.2.1.spark) doesn't support run on Hadoop 3. Hive will check the Hadoop version in the runtime [1]. Besides this I think some pom changes should be enough to support Hadoop 3. If we want to use Hadoop 3 shaded client jar, then the

Re: Hadoop 3 support

I haven't looked at it in detail... Somebody's been trying to do that in https://github.com/apache/spark/pull/20659, but that's kind of a huge change. The parts where I'd be concerned are: - using Hive's original hive-exec package brings in a bunch of shaded dependencies, which may break Spark

Re: Hadoop 3 support

Is it difficult to upgrade Hive execution version to the latest version? The metastore used to be an issue but now that part had been separated from the execution part. On Mon, Apr 2, 2018 at 1:57 PM, Marcelo Vanzin wrote: > Saisai filed SPARK-23534, but the main blocking

Re: Hadoop 3 support

Saisai filed SPARK-23534, but the main blocking issue is really SPARK-18673. On Mon, Apr 2, 2018 at 1:00 PM, Reynold Xin wrote: > Does anybody know what needs to be done in order for Spark to support Hadoop > 3? > -- Marcelo

Re: Hadoop 3 support

That's just a nice to have improvement right? I'm more curious what is the minimal amount of work required to support 3.0, without all the bells and whistles. (Of course we can also do the bells and whistles, but those would come after we can actually get 3.0 running). On Mon, Apr 2, 2018 at

Re: Hadoop 3 support

Specifically to run spark with hadoop 3 docker support, I have filed a few jira's tracked under [1]. Regards, Mridul [1] https://issues.apache.org/jira/browse/SPARK-23717 On Mon, Apr 2, 2018 at 1:00 PM, Reynold Xin wrote: > Does anybody know what needs to be done in order

Hadoop 3 support

Does anybody know what needs to be done in order for Spark to support Hadoop 3?