start hive metastore service OK
hadoop io compression codec is lzo, configure is core-site.xml
io.compression.codecs
codeurl: http://d3kbcqa49mib13.cloudfront.net/spark-1.5.0.tgz
build scripts:
build/mvn -Phadoop-2.3 -Dhadoop.version=2.3.0-cdh5.1.0 -Pyarn -Phive
-Dhive-version=0.12.0-cdh5.1.0 -Phive-thriftserver -Pspark-ganglia-lgpl
-DskipTests clean package
or
./make-distribution.sh --name 2.3.0 --tgz
Please look at more about hadoop logs, such as yarn logs -applicationId
xxx
attach more logs to this topic
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21779p24350.html
Sent from the Apache Spark User List
approach1:
submit spark job add bolow:
--conf
spark.driver.extraClassPath=/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
--conf
spark.executor.extraClassPath=/home/cluster/apps/hbase/lib/hbase-protocol-0.98.1-cdh5.1.0.jar
such as:
spark vesion:1.4.1
java version:1.7
hadoop version:
Hadoop 2.3.0-cdh5.1.0
submit spark job to yarn cluster that read hbase data,after job running, it
comes below error :
15/08/17 19:28:33 ERROR yarn.ApplicationMaster: User class threw exception:
org.apache.hadoop.hbase.DoNotRetryIOException:
*java:*
java version 1.7.0_60
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
*scala*
Scala code runner version 2.10.5 -- Copyright 2002-2013, LAMP/EPFL
*hadoop cluster:*
with 51 Servers that hadoop-2.3-cdh-5.1.0 version ,and
