I don't know whether this would fix the problem. In v0.9, you need `yarn-standalone` instead of `yarn-cluster`.
See https://github.com/apache/spark/commit/328c73d037c17440c2a91a6c88b4258fbefa0c08 On Tue, May 13, 2014 at 11:36 PM, Xiangrui Meng <men...@gmail.com> wrote: > Does v0.9 support yarn-cluster mode? I checked SparkContext.scala in > v0.9.1 and didn't see special handling of `yarn-cluster`. -Xiangrui > > On Mon, May 12, 2014 at 11:14 AM, DB Tsai <dbt...@stanford.edu> wrote: >> We're deploying Spark in yarn-cluster mode (Spark 0.9), and we add jar >> dependencies in command line with "--addJars" option. However, those >> external jars are only available in the driver (application running in >> hadoop), and not available in the executors (workers). >> >> After doing some research, we realize that we've to push those jars to >> executors in driver via sc.AddJar(fileName). Although in the driver's log >> (see the following), the jar is successfully added in the http server in the >> driver, and I confirm that it's downloadable from any machine in the >> network, I still get `java.lang.NoClassDefFoundError` in the executors. >> >> 14/05/09 14:51:41 INFO spark.SparkContext: Added JAR >> analyticshadoop-eba5cdce1.jar at >> http://10.0.0.56:42522/jars/analyticshadoop-eba5cdce1.jar with timestamp >> 1399672301568 >> >> Then I check the log in the executors, and I don't find anything `Fetching >> <file> with timestamp <timestamp>`, which implies something is wrong; the >> executors are not downloading the external jars. >> >> Any suggestion what we can look at? >> >> After digging into how spark distributes external jars, I wonder the >> scalability of this approach. What if there are thousands of nodes >> downloading the jar from single http server in the driver? Why don't we push >> the jars into HDFS distributed cache by default instead of distributing them >> via http server? >> >> Thanks. >> >> Sincerely, >> >> DB Tsai >> ------------------------------------------------------- >> My Blog: https://www.dbtsai.com >> LinkedIn: https://www.linkedin.com/in/dbtsai
