Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99588/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #99588 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99588/testReport)**
for PR 21588 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #99588 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99588/testReport)**
for PR 21588 at commit
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21588
If supporting Hadoop 3 means updating to Hive 2, more or less (not dropping
old Hive metastore support), then yes that seems pretty important. I didn't
hear objections to updating Hive, right? and
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
To all, so how about we start the fix @wangyum tried before? If we are
generally agreed upon the direction itself, upgrading Hive to 2.3 (or 3), I
would like to encourage him to continue
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
The test failure itself doesn't look caused by this change. The tests will
fail anyway with a different error message.
If the goal is really just to check if the tests pass or not, you
Github user LiehuoChen commented on the issue:
https://github.com/apache/spark/pull/21588
Hi HyukjinKwon,
Thanks for all the works to try to make the Jenkin test pass.
I patched this PR to spark 2.4, and anything works fine but failed in
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
Yes, that was what I was thinking at worst case. For clarification,
@wangyum made a try and all tests were passed at least -
https://github.com/apache/spark/pull/20659. Given this try, I think
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21588
Can you clarify what you mean by drop builtin metastore support? Are you
just saying users must always provide jars to use it or something more?
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
@dongjoon-hyun and @wangyum, please fix my comment if I am wrong at any
point - I believe you guys took a look for this part more then I did.
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
> Does this upgrade Hive for execution or also for metastore? Spark
supports virtually all Hive metastore versions out there, and a lot of
deployments do run different versions of Spark against
Github user tooptoop4 commented on the issue:
https://github.com/apache/spark/pull/21588
ship it!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
> Hive 2.3 works with Hadoop 2.x (Hive 3.x works with Hadoop 3.x).
This is essentially what we need for Hadoop 3 support
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/21588
Does this upgrade Hive for execution or also for metastore? Spark supports
virtually all Hive metastore versions out there, and a lot of deployments do
run different versions of Spark against the same
Github user functicons commented on the issue:
https://github.com/apache/spark/pull/21588
Do we really want to switch to Hive 2.3? From this page
https://hive.apache.org/downloads.html, Hive 2.3 works with Hadoop 2.x (Hive
3.x works with Hadoop 3.x).
---
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21588
Sounds like we should try this then
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21588
So, let's say we decide to only support Hive 2.3.x+, as a precursor to
this. We could already eliminate a lot of the Hive tests, right? that might be
useful in its own right as they take time and
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
Yup, it supports Hadoop 3, and other fixes what @wangyum mentioned.
---
-
To unsubscribe, e-mail:
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/21588
does Apache Hive 2.3.2 have all the fixes we need?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
@rxin and @gatorsmile, WDYT?
I already had to argue about Hadoop 3 support here and there (for instance
see [SPARK-18112|https://issues.apache.org/jira/browse/SPARK-18112] and
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/21588
Thanks @HyukjinKwon
Upgrade Hive to 2.3.2 can fix
[SPARK-12014](https://issues.apache.org/jira/browse/SPARK-12014),
[SPARK-18673](https://issues.apache.org/jira/browse/SPARK-18673),
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
ping @wangyum, if you're willing to make a progress about this, please
provide some input here and/or in the JIRA.
---
-
To
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
Yes it solves anything. We could consider to upgrade to Hive 3 but I am
unsure on this since any try (as far as I know) wasn't made yet. But for Hive
2.3.2, @wangyum made a try here
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21588
I know this is probably just reviving an old thread elsewhere, but, we
don't know how to update our 1.2.1 Hive fork anyway, it seems? if so, and the
fork is undesirable, seems like time to drop it.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
Since Hadoop 3 support is also being discussed for Spark 3
(http://apache-spark-developers-list.1001551.n3.nabble.com/time-for-Apache-Spark-3-0-td23755.html).
We should at least:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
This is currently blocked by
[SPARK-20202](https://issues.apache.org/jira/browse/SPARK-20202) per
https://github.com/apache/spark/pull/21588#issuecomment-399292229.
Please provide some
Github user tooptoop4 commented on the issue:
https://github.com/apache/spark/pull/21588
@jerryshao @vanzin can this be merged to master?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96685/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #96685 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96685/testReport)**
for PR 21588 at commit
Github user elgalu commented on the issue:
https://github.com/apache/spark/pull/21588
ð works!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #96685 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96685/testReport)**
for PR 21588 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user tooptoop4 commented on the issue:
https://github.com/apache/spark/pull/21588
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96664/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #96664 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96664/testReport)**
for PR 21588 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #96664 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96664/testReport)**
for PR 21588 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92573/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #92573 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92573/testReport)**
for PR 21588 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #92573 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92573/testReport)**
for PR 21588 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/644/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
Yea, that's all true. I admit what you and @jerryshao did makes sense in a
way. If we failed to replace the Hive fork to 2.3.x and keep the current fork,
I got it that's the last resort that
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/21588
There's a technical issue: trivial change to the case statement
and a ASF process one: the only ASF project which can release hive
artifacts is the hive team; it's that way due to ASF
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
> is it basically true that Hadoop 3 will work with only minor patches to
the Hive fork in Spark?
Up to my knowledge so far, yes, it basically works. At least, the
regression tests we
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21588
Also weighing in here to ask: is it basically true that Hadoop 3 will work
with only minor patches to the Hive fork in Spark? then that seems worth the
hacking. Is the blocker that we can't get a
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
Yup, will fix the hive fork thing and be back.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/21588
@HyukjinKwon , I'm in favor of @vanzin 's comment, we should fix things
first and then back to this one.
---
-
To
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
Will try to fix it then.
We can just enable it back. If we want to support those Hive versions in
Hadoop 3, we could simply enable them back with some fixes at that time. Adding
the
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21588
> The tests were passed in this PR builder
Against your private build of the Hive stuff.
Again, fix that and this will become a lot easier to discuss.
I'm also against
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
The tests were passed in this PR builder. The only hack I used is that I
landed a one liner fix to an artifact to use it in this PR, which is already in
Hive, and is proposed in Hive's fork
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21588
I already explained my view of why I don't think this should get in, in its
current form.
Passing tests in someone's private environment, for me, is not a worthy
goal.
You say the
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
I at least checked if this passed with that fix to fork manually. It fixes
everything else that can be fixed in Spark. I wonder why this should be blocked
to be honest yet. It can't be ran via
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21588
Let me phrase the question a different way: your title says "Make jenkins
tests passed" [sic]. If you check this in, and we enable a jenkins job for
hadoop 3, will it pass?
I'm 100% sure
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
> The main thing is that this change is changing test coverage based on the
Hadoop version
> The Hive 2.1 suite you're disabling is also pretty important to keep
working, since it
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21588
The main thing is that this change is changing test coverage based on the
Hadoop version. So that means that we're effectively changing supported
versions of Hive here, and we should do all the
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
I saw:
```
java.lang.IllegalArgumentException: Unrecognized Hadoop major version
number: 3.1.0
```
Other error messages are fixed by the current change. Yup, we
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
I saw:
```
Otherwise, we are not even able to start Spark shell. Currently, Hadoop 3
profile in Apache Spark doesn't work at all. It will face an error message such
as:
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21588
> but the error message was pretty readable to me
What is the error message you see? I didn't see any changes in the Spark
code that handles that (`IsolatedClientLoader.hiveVersion`), so
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
Yup, it will still fail but it fixes everything else to make it working
with Hadoop 3 within Spark. I think the current change is minimised as the
current status as is and I meant to target
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/21588
I'm talking about the `VersionsSuite` stuff. I think it needs to be a more
conscious decision about what happens.
If, when build with Hadoop 3, Spark will not support older versions of
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
@vanzin, which tests do you mean? Two types of tests are skipped. One is by
external Hive's limit which we can't control and the other one (two tests)
looks by a JDK bug which I think we
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/21588
If jenkins is happy, this is good.
* Be interesting to see what happens in a build with the
hadoop-cloud-storage module, if it adds new dependencies
* regarding commons-config, know
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92134/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #92134 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92134/testReport)**
for PR 21588 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92135/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #92135 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92135/testReport)**
for PR 21588 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21588
**[Test build #92135 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92135/testReport)**
for PR 21588 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/355/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4251/
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
+@steveloughran
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21588
@srowen, @vanzin, @jerryshao and @gatorsmile, I believe this is ready for a
look.
---
-
To unsubscribe, e-mail:
83 matches
Mail list logo
