[
https://issues.apache.org/jira/browse/KYLIN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2421:
--------------------------------
Description:
Let Jenkins to test Spark cubing daily. Will change the engine type for
"ci_inner_join_cube" to "4" (spark).
After this change, you need do a couple changes manually in your HDP 2.4
sandbox to finish the Integration Test:
1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like
/usr/local/spark
cd /usr/local
tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz
ln -s spark-1.6.3-bin-hadoop2.6 spark
2. manually upload the spark-assembly jar to HDFS as
/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading
the jar to HDFS):
hadoop fs -mkdir /kylin/spark/
hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar
/kylin/spark/
3. manually create local temp folder for hbase client (if it doesn't exist):
mkdir -p /hadoop/hbase/local/jars
chmod 777 /hadoop/hbase/local/jars
4. Add SPARK_HOME property when run mvn:
mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P
sandbox
was:
Let Jenkins to test Spark cubing daily. Will change the engine type for
"ci_inner_join_cube" to "4" (spark).
After this change, you need do a couple changes manually in your HDP 2.4
sandbox to finish the Integration Test:
1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like
/usr/local/spark
2. manually upload the spark-assembly jar to HDFS as
/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar
3. manually create local temp folder for hbase client (if it doesn't exist):
mkdir -p /hadoop/hbase/local/jars
chmod 777 /hadoop/hbase/local/jars
4. Add SPARK_HOME property when run mvn:
mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P
sandbox
> Add spark engine to Integration Test
> ------------------------------------
>
> Key: KYLIN-2421
> URL: https://issues.apache.org/jira/browse/KYLIN-2421
> Project: Kylin
> Issue Type: Sub-task
> Components: Job Engine
> Affects Versions: v2.0.0
> Reporter: Shaofeng SHI
> Assignee: Shaofeng SHI
> Fix For: v2.0.0
>
>
> Let Jenkins to test Spark cubing daily. Will change the engine type for
> "ci_inner_join_cube" to "4" (spark).
> After this change, you need do a couple changes manually in your HDP 2.4
> sandbox to finish the Integration Test:
> 1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like
> /usr/local/spark
> cd /usr/local
> tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz
> ln -s spark-1.6.3-bin-hadoop2.6 spark
> 2. manually upload the spark-assembly jar to HDFS as
> /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading
> the jar to HDFS):
> hadoop fs -mkdir /kylin/spark/
> hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar
> /kylin/spark/
> 3. manually create local temp folder for hbase client (if it doesn't exist):
> mkdir -p /hadoop/hbase/local/jars
> chmod 777 /hadoop/hbase/local/jars
> 4. Add SPARK_HOME property when run mvn:
> mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P
> sandbox
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
