Re: Spark 2.4.4 with Hadoop 3.2.0

[bsikander]

It could be that CDH6 has the integration but somehow I am getting the
following very frequently while building Spark2.4.4 with Hadoop 3.2.0 and
running spark tests.

caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major
version number: 3.2.0
at
org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
at
org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
at
org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:368)





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark 2.4.4 with Hadoop 3.2.0

[nihed mbarek]

Hi,
Spark 2.x is already part of Cloudera CDH6 who is based on Hadoop 3.x so
they support officially Spark2+Hadoop3
So for sure, there is tests and development done from this side. In other
part,  I don't know the status for Hadoop 3.2.

Regards,

On Tue, Nov 26, 2019 at 1:46 AM Alfredo Marquez 
wrote:

> Thank you Ismael! That's what I was looking for. I can take this to our
> platform team.
>
> Alfredo
>
> On Mon, Nov 25, 2019, 3:32 PM Ismaël Mejía  wrote:
>
>> Not officially. Apache Spark only announced support for Hadoop 3.x
>> starting with the upcoming Spark 3.
>> There is a preview release of Spark 3 with support for Hadoop 3.2 that
>> you can try now:
>>
>> https://archive.apache.org/dist/spark/spark-3.0.0-preview/spark-3.0.0-preview-bin-hadoop3.2.tgz
>>
>> Enjoy!
>>
>>
>>
>> On Tue, Nov 19, 2019 at 3:44 PM Alfredo Marquez <
>> alfredo.g.marq...@gmail.com> wrote:
>>
>>> I also would like know the answer to this question.
>>>
>>> Thanks,
>>>
>>> Alfredo
>>>
>>> On Tue, Nov 19, 2019, 8:24 AM bsikander  wrote:
>>>
 Hi,
 Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
 I tried to search the mailing list but couldn't find anything relevant.





 --
 Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

 -
 To unsubscribe e-mail: user-unsubscr...@spark.apache.org



-- 

M'BAREK Med Nihed,
Fedora Ambassador, TUNISIA, Northern Africa
http://www.nihed.com

Re: Spark 2.4.4 with Hadoop 3.2.0

[Alfredo Marquez]

Thank you Ismael! That's what I was looking for. I can take this to our
platform team.

Alfredo

On Mon, Nov 25, 2019, 3:32 PM Ismaël Mejía  wrote:

> Not officially. Apache Spark only announced support for Hadoop 3.x
> starting with the upcoming Spark 3.
> There is a preview release of Spark 3 with support for Hadoop 3.2 that you
> can try now:
>
> https://archive.apache.org/dist/spark/spark-3.0.0-preview/spark-3.0.0-preview-bin-hadoop3.2.tgz
>
> Enjoy!
>
>
>
> On Tue, Nov 19, 2019 at 3:44 PM Alfredo Marquez <
> alfredo.g.marq...@gmail.com> wrote:
>
>> I also would like know the answer to this question.
>>
>> Thanks,
>>
>> Alfredo
>>
>> On Tue, Nov 19, 2019, 8:24 AM bsikander  wrote:
>>
>>> Hi,
>>> Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
>>> I tried to search the mailing list but couldn't find anything relevant.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>>>
>>> -
>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>>
>>>

Re: Spark 2.4.4 with Hadoop 3.2.0

[Ismaël Mejía]

Not officially. Apache Spark only announced support for Hadoop 3.x starting
with the upcoming Spark 3.
There is a preview release of Spark 3 with support for Hadoop 3.2 that you
can try now:
https://archive.apache.org/dist/spark/spark-3.0.0-preview/spark-3.0.0-preview-bin-hadoop3.2.tgz

Enjoy!



On Tue, Nov 19, 2019 at 3:44 PM Alfredo Marquez 
wrote:

> I also would like know the answer to this question.
>
> Thanks,
>
> Alfredo
>
> On Tue, Nov 19, 2019, 8:24 AM bsikander  wrote:
>
>> Hi,
>> Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
>> I tried to search the mailing list but couldn't find anything relevant.
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>

Re: Spark 2.4.4 with Hadoop 3.2.0

[Alfredo Marquez]

I also would like know the answer to this question.

Thanks,

Alfredo

On Tue, Nov 19, 2019, 8:24 AM bsikander  wrote:

> Hi,
> Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
> I tried to search the mailing list but couldn't find anything relevant.
>
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>