> On 16 Oct 2018, at 22:06, t4 <reubensaw...@hotmail.com> wrote:
>
> has anyone got spark jars working with hadoop3.1 that they can share? i am
> looking to be able to use the latest hadoop-aws fixes from v3.1
we do, but we do with
* a patched hive JAR
* bulding spark with -Phive,yarn,hadoop-3.1,hadoop-cloud,kinesis profiles to
pull in the object store stuff *while leaving out the things which cause
conflict*
* some extra stuff to wire up the 0-rename-committer
w.r.t hadoop aws, the hadoop-2.9 artifacts have the shaded aws JAR; 50 MB of
.class to avoid jackson dependency pain, and an early version of S3Guard. For
the new commit stuff you will need to go to hadoop 3.1
-steve
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
