[
https://issues.apache.org/jira/browse/HIVE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297147#comment-14297147
]
Brock Noland edited comment on HIVE-9211 at 1/29/15 5:14 PM:
-------------------------------------------------------------
Hi [~chengxiang li],
When we [moved over to a none SNAPSHOT version of
Spark|https://github.com/apache/hive/commit/dab416b2c492d22ab76fa2782f434d165c1144ab]
I used a tarball which does not include the hadoop jars in the spark assembly.
This can been seen by extracting the spark assembly in [our
tarball|http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.2.0-bin-hadoop2-without-hive.tgz].
As such you'll see I put {{$\{test.hive.hadoop.classpath\}}} in the classpath
to replace the missing hadoop jars from the spark assembly.
As such, I have the following questions:
# which class are you not finding that is required?
# when you say you need the latest branch-1.2, do you mean a released version
of spark? We can have a snapshot on the spark branch but not on trunk.
was (Author: brocknoland):
Hi [~chengxiang li],
When we [moved over to a none SNAPSHOT version of
Spark|https://github.com/apache/hive/commit/dab416b2c492d22ab76fa2782f434d165c1144ab]
I used a tarball which does not include the hadoop jars in the spark assembly.
This can been seen by extracting the spark assembly in [our
tarball|http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-${spark.version}-bin-hadoop2-without-hive.tgz].
As such you'll see I put {{$\{test.hive.hadoop.classpath\}}} in the classpath
to replace the missing hadoop jars from the spark assembly.
As such, I have the following questions:
# which class are you not finding that is required?
# when you say you need the latest branch-1.2, do you mean a released version
of spark? We can have a snapshot on the spark branch but not on trunk.
> Research on build mini HoS cluster on YARN for unit test[Spark Branch]
> ----------------------------------------------------------------------
>
> Key: HIVE-9211
> URL: https://issues.apache.org/jira/browse/HIVE-9211
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Chengxiang Li
> Assignee: Chengxiang Li
> Labels: Spark-M5
> Attachments: HIVE-9211.1-spark.patch, HIVE-9211.2-spark.patch,
> HIVE-9211.2-spark.patch, HIVE-9211.3-spark.patch, HIVE-9211.4-spark.patch
>
>
> HoS on YARN is a common use case in product environment, we'd better enable
> unit test for this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
