On 22 Oct 2015, at 02:47, Ajay Chander <itsche...@gmail.com<mailto:itsche...@gmail.com>> wrote:
Thanks for your time. I have followed your inputs and downloaded "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a pie test everything seems to be working fine, except that the spark-history -server running on this node1 has gone down. It was complaining about missing class: 15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for [TERM, HUP, INT] 15/10/21 16:41:28 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead. 15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root 15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root 15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) Exception in thread "main" java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.history.YarnHistoryProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:173) at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231) at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala) OK, that's happening because the version of spark we ship comes with integration with the yarn history provider, and that's not on your classpath. find the history server config and either unset spark.history.provider or set it back to the filesystem one: spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider The 1.5.1-based binaries we'll push out will be set up for this and for the spark apps to be able to publish to the timeline server. note that irrespective of how histories are saved, a 1.3 or 1.4 history server cannot view a 1.5 one. So you will need to run a 1.5.x history server to view the most recent histories. But you can run a 1.4 history server talking to the timeline service alongside a 1.5 history server working through HDFS. see: SPARK-1537 for the details of what's happening