[
https://issues.apache.org/jira/browse/HIVE-7665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-7665:
----------------------------
Attachment: HIVE-7665.patch
This patch creates TestSparkCliDriver, which runs spark in local mode, using
spark.master=local[4].
It is a ton of dependency conflicts trying to run a spark local cluster within
hive test environment, its fragile and these may need to change with future
version of spark.
1. Netty needs to be greater than 3.4 to get compatibility with spark's akka,
but less than 4.0 (when the namespace changed). So the easiest way to achieve
this is to upgrade zookeeper (which pulled in netty) and this pulls in netty
3.7.0.FINAL. For reference, spark-core uses netty 3.6.6.FINAL.
2. Jetty needs to be 8.14, kryo needs to be 2.21. For now I have put the
spark test in a separate environment, but we can investigate upgrading these
versions across hive in later JIRA's. But it might not be possible, as for
example spark kryo is a downgrade from hive kryo.
Almost no need to mention, it does not work with hadoop-1, so I did not try.
I configured the TestSparkCliDriver framework so it can re-use any test within
clientpositive directory, but has a separate q.out directory (as explain
results will be different for spark).
> Create TestSparkCliDriver to run test in spark local mode
> ---------------------------------------------------------
>
> Key: HIVE-7665
> URL: https://issues.apache.org/jira/browse/HIVE-7665
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Testing Infrastructure
> Reporter: Szehon Ho
> Assignee: Szehon Ho
> Attachments: HIVE-7665.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
