Hi,
My spark was compiled with yarn profile, I can run spark on yarn without
problem.
For the spark job history server problem, I checked
spark-assembly-1.3.0-hadoop2.4.0.jar and found that the package
org.apache.spark.deploy.yarn.history is missing. I don't know why....
BR,
Patcharee
On 18. mars 2015 11:43, Akhil Das wrote:
You are not having yarn package in the classpath. You need to build
your spark it with yarn. You can read these docs.
<http://spark.apache.org/docs/1.3.0/running-on-yarn.html>
Thanks
Best Regards
On Wed, Mar 18, 2015 at 4:07 PM, patcharee <patcharee.thong...@uni.no
<mailto:patcharee.thong...@uni.no>> wrote:
I turned it on. But it failed to start. In the log,
Spark assembly has been built with Hive, including Datanucleus
jars on classpath
Spark Command: /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -cp
:/root/spark-1.3.0-bin-hadoop2.4/sbin/../conf:/root/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.3.0-hadoop2.4.0.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/root/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar:/etc/hadoop/conf
-XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m
-Xmx512m org.apache.spark.deploy.history.HistoryServer
========================================
15/03/18 10:23:46 WARN NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java
classes where applicable
Exception in thread "main" java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at
org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:183)
at
org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Patcharee
On 18. mars 2015 11:35, Akhil Das wrote:
You can simply turn it on using:
|./sbin/start-history-server.sh|
Read more here
<http://spark.apache.org/docs/1.3.0/monitoring.html>.
Thanks
Best Regards
On Wed, Mar 18, 2015 at 4:00 PM, patcharee
<patcharee.thong...@uni.no <mailto:patcharee.thong...@uni.no>> wrote:
Hi,
I am using spark 1.3. I would like to use Spark Job History
Server. I added the following line into conf/spark-defaults.conf
spark.yarn.services
org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.history.provider
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
spark.yarn.historyServer.address
sandbox.hortonworks.com:19888
<http://sandbox.hortonworks.com:19888>
But got Exception in thread "main"
java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
What class is really needed? How to fix it?
Br,
Patcharee
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
<mailto:user-unsubscr...@spark.apache.org>
For additional commands, e-mail: user-h...@spark.apache.org
<mailto:user-h...@spark.apache.org>
