-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24520/
-----------------------------------------------------------
(Updated Aug. 12, 2014, 12:21 a.m.)
Review request for hive and Brock Noland.
Changes
-------
Rebase + review comments.
Bugs: HIVE-7665
https://issues.apache.org/jira/browse/HIVE-7665
Repository: hive-git
Description
-------
Creates TestSparkCliDriver, which runs spark in local mode, using
spark.master=local[4].
It is a ton of dependency conflicts trying to run a spark local cluster within
hive test environment, its fragile and these may need to change with future
version of spark.
1. Netty needs to be greater than 3.4 to get compatibility with spark's akka,
but less than 4.0 (when the namespace changed). So the easiest way to achieve
this is to upgrade zookeeper (which pulled in netty) and this pulls in netty
3.7.0.FINAL. For reference, spark-core uses netty 3.6.6.FINAL.
2. Jetty needs to be 8.14, kryo needs to be 2.21. For now I have put the
spark test in a separate environment, but we can investigate upgrading these
versions across hive in later JIRA's. But it might not be possible, as for
example spark kryo is a downgrade from hive kryo.
Almost no need to mention, it does not work with hadoop-1, so I did not try.
I configured the TestSparkCliDriver framework so it can re-use any test within
clientpositive directory, but has a separate q.out directory (as explain
results will be different for spark).
Diffs (updated)
-----
data/conf/spark/hive-site.xml PRE-CREATION
itests/pom.xml f6d1d83
itests/qtest-spark/pom.xml PRE-CREATION
itests/qtest/testconfiguration.properties 2d3f331
pom.xml 90733df
ql/src/test/queries/clientpositive/spark_test.q PRE-CREATION
ql/src/test/results/clientpositive/spark/spark_test.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark_test.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/24520/diff/
Testing
-------
Ran -Dtest=TestSparkCliDriver. Added one very basic test that exercises spark
as an example.
Thanks,
Szehon Ho
