Also, Spark on Mesos supports cluster mode: http://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode
Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Sun, Aug 9, 2015 at 4:30 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > Did you try this way? > > /usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050 > --conf spark.mesos.executor.docker.image=docker.repo/spark:latest --class > org.apache.spark.examples.SparkPi *--jars hdfs://hdfs1/tmp/spark-* > *examples-1.4.1-hadoop2.6.0-**cdh5.4.4.jar* 100 > > Thanks > Best Regards > > On Fri, Aug 7, 2015 at 5:51 AM, Alan Braithwaite <a...@cloudflare.com> > wrote: > >> Hi All, >> >> We're trying to run spark with mesos and docker in client mode (since >> mesos doesn't support cluster mode) and load the application Jar from >> HDFS. The following is the command we're running: >> >> /usr/local/spark/bin/spark-submit --master mesos://mesos.master:5050 >> --conf spark.mesos.executor.docker.image=docker.repo/spark:latest --class >> org.apache.spark.examples.SparkPi >> hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar 100 >> >> We're getting the following warning before an exception from that command: >> >> Warning: Skip remote jar >> hdfs://hdfs1/tmp/spark-examples-1.4.1-hadoop2.6.0-cdh5.4.4.jar. >> java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi >> >> Before I debug further, is this even supported? I started reading the >> code and it wasn't clear that it's possible to load a remote jar in client >> mode at all. I did see a related issue in [2] but it didn't quite clarify >> everything I was looking for. >> >> Thanks, >> - Alan >> >> [1] https://spark.apache.org/docs/latest/submitting-applications.html >> >> [2] >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-not-working-when-application-jar-is-in-hdfs-td21840.html >> > >
