> On 16 Oct 2018, at 22:06, t4 wrote:
>
> has anyone got spark jars working with hadoop3.1 that they can share? i am
> looking to be able to use the latest hadoop-aws fixes from v3.1
we do, but we do with
* a patched hive JAR
* bulding spark with -Phive,yarn,hadoop-3.1,hadoop-cloud,kinesis
See the discussion at https://github.com/apache/spark/pull/21588
2018년 10월 17일 (수) 오전 5:06, t4 님이 작성:
> has anyone got spark jars working with hadoop3.1 that they can share? i am
> looking to be able to use the latest hadoop-aws fixes from v3.1
>
>
>
> --
> Sent from: http://apache-spark-develop
has anyone got spark jars working with hadoop3.1 that they can share? i am
looking to be able to use the latest hadoop-aws fixes from v3.1
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
-
To unsubscri
: Hadoop 3 support
To: Apache Spark Dev
On 3 Apr 2018, at 01:30, Saisai Shao
mailto:sai.sai.s...@gmail.com>> wrote:
Yes, the main blocking issue is the hive version used in Spark (1.2.1.spark)
doesn't support run on Hadoop 3. Hive will check the Hadoop version in the
runtime [1]. B
On 3 Apr 2018, at 01:30, Saisai Shao
mailto:sai.sai.s...@gmail.com>> wrote:
Yes, the main blocking issue is the hive version used in Spark (1.2.1.spark)
doesn't support run on Hadoop 3. Hive will check the Hadoop version in the
runtime [1]. Besides this I think some pom changes should be enou
On 3 Apr 2018, at 01:30, Saisai Shao
mailto:sai.sai.s...@gmail.com>> wrote:
Yes, the main blocking issue is the hive version used in Spark (1.2.1.spark)
doesn't support run on Hadoop 3. Hive will check the Hadoop version in the
runtime [1]. Besides this I think some pom changes should be enou
Yes, the main blocking issue is the hive version used in Spark
(1.2.1.spark) doesn't support run on Hadoop 3. Hive will check the Hadoop
version in the runtime [1]. Besides this I think some pom changes should be
enough to support Hadoop 3.
If we want to use Hadoop 3 shaded client jar, then the po
I haven't looked at it in detail...
Somebody's been trying to do that in
https://github.com/apache/spark/pull/20659, but that's kind of a huge
change.
The parts where I'd be concerned are:
- using Hive's original hive-exec package brings in a bunch of shaded
dependencies, which may break Spark in
Is it difficult to upgrade Hive execution version to the latest version?
The metastore used to be an issue but now that part had been separated from
the execution part.
On Mon, Apr 2, 2018 at 1:57 PM, Marcelo Vanzin wrote:
> Saisai filed SPARK-23534, but the main blocking issue is really
> SPAR
Saisai filed SPARK-23534, but the main blocking issue is really SPARK-18673.
On Mon, Apr 2, 2018 at 1:00 PM, Reynold Xin wrote:
> Does anybody know what needs to be done in order for Spark to support Hadoop
> 3?
>
--
Marcelo
--
That's just a nice to have improvement right? I'm more curious what is the
minimal amount of work required to support 3.0, without all the bells and
whistles. (Of course we can also do the bells and whistles, but those would
come after we can actually get 3.0 running).
On Mon, Apr 2, 2018 at 1:50
Specifically to run spark with hadoop 3 docker support, I have filed a
few jira's tracked under [1].
Regards,
Mridul
[1] https://issues.apache.org/jira/browse/SPARK-23717
On Mon, Apr 2, 2018 at 1:00 PM, Reynold Xin wrote:
> Does anybody know what needs to be done in order for Spark to support
Does anybody know what needs to be done in order for Spark to support
Hadoop 3?
13 matches
Mail list logo
