Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
> The main thing is that this change is changing test coverage based on the
Hadoop version
> The Hive 2.1 suite you're disabling is also pretty important to keep
working, since it tests behavior that changed from Spark's built in version of
Hive, and most probably is similar in newer versions.
> We should be looking at what it means to support Hadoop 3, and answer
that before we go hacking and disabling things just to get tests to pass.
I believe that's what we currently cover with Hadoop 3 profile ... maybe we
should fix it back when other decisions are made or newer changes are landed.
In my point of view, we don't hack things here. It just explicitly makes the
tests passed as we cover with Hadoop 3 profile. The point of this PR and
changes are to make Spark to the working state ...
> The error message you saw seems familiar to you but I'm pretty sure it
would be very cryptic to someone who's not familiar with this problem. (Why is
Hive complaining about a Hadoop version if I'm running Spark?)
Yea, I agree with fixing the error message, sure. I was thinking of doing
this separately since this PR targets to make the tests passed and make Spark
into working state with Hadoop 3 at least (of course except that the external
(one liner) fix into Hive fork).
For me, I don't see the reasons to block fix everything else for Hadoop 3
within Spark side by one liner fix within the Hive fork. We should really make
a progress in parallel if we can ...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
