Ah I see, I think it's because you've launched the Mesos slave in a docker container, and when you launch also the executor in a container it's not able to mount in the sandbox to the other container since the slave is in a chroot.
Can you try mounting in a volume from the host when you launch the slave for your slave's workdir? docker run -v /tmp/mesos/slave:/tmp/mesos/slave mesos_image mesos-slave --work_dir=/tmp/mesos/slave ........ Tim On Thu, Mar 3, 2016 at 4:42 AM, Ashish Soni <asoni.le...@gmail.com> wrote: > Hi Tim , > > > I think I know the problem but i do not have a solution , *The Mesos > Slave supposed to download the Jars from the URI specified and placed in > $MESOS_SANDBOX location but it is not downloading not sure why* .. see > below logs > > My command looks like below > > docker run -it --rm -m 2g -e SPARK_MASTER="mesos://10.0.2.15:7077" -e > SPARK_IMAGE="spark_driver:latest" spark_driver:latest ./bin/spark-submit > --deploy-mode cluster --class org.apache.spark.examples.SparkPi > http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar > > [root@Mindstorm spark-1.6.0]# docker logs d22d8e897b79 > *Warning: Local jar > /mnt/mesos/sandbox/spark-examples-1.6.0-hadoop2.6.0.jar does not exist, > skipping.* > java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:278) > at org.apache.spark.util.Utils$.classForName(Utils.scala:174) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > When i do docker inspect i see below command gets issued > > "Cmd": [ > "-c", > "./bin/spark-submit --name org.apache.spark.examples.SparkPi > --master mesos://10.0.2.15:5050 --driver-cores 1.0 --driver-memory 1024M > --class org.apache.spark.examples.SparkPi > $*MESOS_SANDBOX*/spark-examples-1.6.0-hadoop2.6.0.jar > " > > > > On Thu, Mar 3, 2016 at 12:09 AM, Tim Chen <t...@mesosphere.io> wrote: > >> You shouldn't need to specify --jars at all since you only have one jar. >> >> The error is pretty odd as it suggests it's trying to load >> /opt/spark/Example but that doesn't really seem to be anywhere in your >> image or command. >> >> Can you paste your stdout from the driver task launched by the cluster >> dispatcher, that shows you the spark-submit command it eventually ran? >> >> >> Tim >> >> >> >> On Wed, Mar 2, 2016 at 5:42 PM, Ashish Soni <asoni.le...@gmail.com> >> wrote: >> >>> See below and Attached the Dockerfile to build the spark image ( >>> between i just upgraded to 1.6 ) >>> >>> I am running below setup - >>> >>> Mesos Master - Docker Container >>> Mesos Slave 1 - Docker Container >>> Mesos Slave 2 - Docker Container >>> Marathon - Docker Container >>> Spark MESOS Dispatcher - Docker Container >>> >>> when i submit the Spark PI Example Job using below command >>> >>> *docker run -it --rm -m 2g -e SPARK_MASTER="mesos://10.0.2.15:7077 >>> <http://10.0.2.15:7077>" -e SPARK_IMAGE="spark_driver:**latest" >>> spark_driver:latest ./bin/spark-submit --deploy-mode cluster --name "PI >>> Example" --class org.apache.spark.examples.**SparkPi >>> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >>> <http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar> --jars >>> /opt/spark/lib/spark-examples-**1.6.0-hadoop2.6.0.jar --verbose* >>> >>> Below is the ERROR >>> Error: Cannot load main class from JAR file:/opt/spark/Example >>> Run with --help for usage help or --verbose for debug output >>> >>> >>> When i docker Inspect for the stopped / dead container i see below >>> output what is interesting to see is some one or executor replaced by >>> original command with below in highlighted and i do not see Executor is >>> downloading the JAR -- IS this a BUG i am hitting or not sure if that is >>> supposed to work this way and i am missing some configuration >>> >>> "Env": [ >>> "SPARK_IMAGE=spark_driver:latest", >>> "SPARK_SCALA_VERSION=2.10", >>> "SPARK_VERSION=1.6.0", >>> "SPARK_EXECUTOR_URI= >>> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-hadoop2.6.tgz";, >>> "MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos-0.25.0.so", >>> "SPARK_MASTER=mesos://10.0.2.15:7077", >>> >>> "SPARK_EXECUTOR_OPTS=-Dspark.executorEnv.MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/ >>> libmesos-0.25.0.so -Dspark.jars= >>> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >>> -Dspark.mesos.mesosExecutor.cores=0.1 -Dspark.driver.supervise=false - >>> Dspark.app.name=PI Example -Dspark.mesos.uris= >>> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >>> -Dspark.mesos.executor.docker.image=spark_driver:latest >>> -Dspark.submit.deployMode=cluster -Dspark.master=mesos://10.0.2.15:7077 >>> -Dspark.driver.extraClassPath=/opt/spark/custom/lib/* >>> -Dspark.executor.extraClassPath=/opt/spark/custom/lib/* >>> -Dspark.executor.uri= >>> http://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-hadoop2.6.tgz >>> -Dspark.mesos.executor.home=/opt/spark", >>> "MESOS_SANDBOX=/mnt/mesos/sandbox", >>> >>> "MESOS_CONTAINER_NAME=mesos-e47f8d4c-5ee1-4d01-ad07-0d9a03ced62d-S1.43c08f82-e508-4d57-8c0b-fa05bee77fd6", >>> >>> "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", >>> "HADOOP_VERSION=2.6", >>> "SPARK_HOME=/opt/spark" >>> ], >>> "Cmd": [ >>> "-c", >>> * "./bin/spark-submit --name PI Example --master >>> mesos://10.0.2.15:5050 <http://10.0.2.15:5050> --driver-cores 1.0 >>> --driver-memory 1024M --class org.apache.spark.examples.SparkPi >>> $MESOS_SANDBOX/spark-examples-1.6.0-hadoop2.6.0.jar --jars >>> /opt/spark/lib/spark-examples-1.6.0-hadoop2.6.0.jar --verbose"* >>> ], >>> "Image": "spark_driver:latest", >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Mar 2, 2016 at 5:49 PM, Charles Allen < >>> charles.al...@metamarkets.com> wrote: >>> >>>> @Tim yes, this is asking about 1.5 though >>>> >>>> On Wed, Mar 2, 2016 at 2:35 PM Tim Chen <t...@mesosphere.io> wrote: >>>> >>>>> Hi Charles, >>>>> >>>>> I thought that's fixed with your patch in latest master now right? >>>>> >>>>> Ashish, yes please give me your docker image name (if it's in the >>>>> public registry) and what you've tried and I can see what's wrong. I think >>>>> it's most likely just the configuration of where the Spark home folder is >>>>> in the image. >>>>> >>>>> Tim >>>>> >>>>> On Wed, Mar 2, 2016 at 2:28 PM, Charles Allen < >>>>> charles.al...@metamarkets.com> wrote: >>>>> >>>>>> Re: Spark on Mesos.... Warning regarding disk space: >>>>>> https://issues.apache.org/jira/browse/SPARK-12330 >>>>>> >>>>>> That's a spark flaw I encountered on a very regular basis on mesos. >>>>>> That and a few other annoyances are fixed in >>>>>> https://github.com/metamx/spark/tree/v1.5.2-mmx >>>>>> >>>>>> Here's another mild annoyance I've encountered: >>>>>> https://issues.apache.org/jira/browse/SPARK-11714 >>>>>> >>>>>> On Wed, Mar 2, 2016 at 1:31 PM Ashish Soni <asoni.le...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I have no luck and i would to ask the question to spark committers >>>>>>> will this be ever designed to run on mesos ? >>>>>>> >>>>>>> spark app as a docker container not working at all on mesos ,if any >>>>>>> one would like the code i can send it over to have a look. >>>>>>> >>>>>>> Ashish >>>>>>> >>>>>>> On Wed, Mar 2, 2016 at 12:23 PM, Sathish Kumaran Vairavelu < >>>>>>> vsathishkuma...@gmail.com> wrote: >>>>>>> >>>>>>>> Try passing jar using --jars option >>>>>>>> >>>>>>>> On Wed, Mar 2, 2016 at 10:17 AM Ashish Soni <asoni.le...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I made some progress but now i am stuck at this point , Please >>>>>>>>> help as looks like i am close to get it working >>>>>>>>> >>>>>>>>> I have everything running in docker container including mesos >>>>>>>>> slave and master >>>>>>>>> >>>>>>>>> When i try to submit the pi example i get below error >>>>>>>>> *Error: Cannot load main class from JAR file:/opt/spark/Example* >>>>>>>>> >>>>>>>>> Below is the command i use to submit as a docker container >>>>>>>>> >>>>>>>>> docker run -it --rm -e SPARK_MASTER="mesos://10.0.2.15:7077" -e >>>>>>>>> SPARK_IMAGE="spark_driver:latest" spark_driver:latest >>>>>>>>> ./bin/spark-submit >>>>>>>>> --deploy-mode cluster --name "PI Example" --class >>>>>>>>> org.apache.spark.examples.SparkPi --driver-memory 512m >>>>>>>>> --executor-memory >>>>>>>>> 512m --executor-cores 1 >>>>>>>>> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 1, 2016 at 2:59 PM, Timothy Chen <t...@mesosphere.io> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Can you go through the Mesos UI and look at the driver/executor >>>>>>>>>> log from steer file and see what the problem is? >>>>>>>>>> >>>>>>>>>> Tim >>>>>>>>>> >>>>>>>>>> On Mar 1, 2016, at 8:05 AM, Ashish Soni <asoni.le...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Not sure what is the issue but i am getting below error when i >>>>>>>>>> try to run spark PI example >>>>>>>>>> >>>>>>>>>> Blacklisting Mesos slave value: "5345asdasdasdkas234234asdasdasdasd" >>>>>>>>>> due to too many failures; is Spark installed on it? >>>>>>>>>> WARN TaskSchedulerImpl: Initial job has not accepted any >>>>>>>>>> resources; check your cluster UI to ensure that workers are >>>>>>>>>> registered and have sufficient resources >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Feb 29, 2016 at 1:39 PM, Sathish Kumaran Vairavelu < >>>>>>>>>> vsathishkuma...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> May be the Mesos executor couldn't find spark image or the >>>>>>>>>>> constraints are not satisfied. Check your Mesos UI if you see Spark >>>>>>>>>>> application in the Frameworks tab >>>>>>>>>>> >>>>>>>>>>> On Mon, Feb 29, 2016 at 12:23 PM Ashish Soni < >>>>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> What is the Best practice , I have everything running as docker >>>>>>>>>>>> container in single host ( mesos and marathon also as docker >>>>>>>>>>>> container ) >>>>>>>>>>>> and everything comes up fine but when i try to launch the spark >>>>>>>>>>>> shell i >>>>>>>>>>>> get below error >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> SQL context available as sqlContext. >>>>>>>>>>>> >>>>>>>>>>>> scala> val data = sc.parallelize(1 to 100) >>>>>>>>>>>> data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] >>>>>>>>>>>> at parallelize at <console>:27 >>>>>>>>>>>> >>>>>>>>>>>> scala> data.count >>>>>>>>>>>> [Stage 0:> >>>>>>>>>>>> (0 + 0) / 2]16/02/29 18:21:12 WARN TaskSchedulerImpl: Initial >>>>>>>>>>>> job has >>>>>>>>>>>> not accepted any resources; check your cluster UI to ensure that >>>>>>>>>>>> workers >>>>>>>>>>>> are registered and have sufficient resources >>>>>>>>>>>> 16/02/29 18:21:27 WARN TaskSchedulerImpl: Initial job has not >>>>>>>>>>>> accepted any resources; check your cluster UI to ensure that >>>>>>>>>>>> workers are >>>>>>>>>>>> registered and have sufficient resources >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Feb 29, 2016 at 12:04 PM, Tim Chen <t...@mesosphere.io> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> No you don't have to run Mesos in docker containers to run >>>>>>>>>>>>> Spark in docker containers. >>>>>>>>>>>>> >>>>>>>>>>>>> Once you have Mesos cluster running you can then specfiy the >>>>>>>>>>>>> Spark configurations in your Spark job (i.e: >>>>>>>>>>>>> spark.mesos.executor.docker.image=mesosphere/spark:1.6) >>>>>>>>>>>>> and Mesos will automatically launch docker containers for you. >>>>>>>>>>>>> >>>>>>>>>>>>> Tim >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Feb 29, 2016 at 7:36 AM, Ashish Soni < >>>>>>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yes i read that and not much details here. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is it true that we need to have spark installed on each mesos >>>>>>>>>>>>>> docker container ( master and slave ) ... >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Feb 26, 2016 at 2:14 PM, Tim Chen <t...@mesosphere.io> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html >>>>>>>>>>>>>>> should >>>>>>>>>>>>>>> be the best source, what problems were you running into? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tim >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Feb 26, 2016 at 11:06 AM, Yin Yang < >>>>>>>>>>>>>>> yy201...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Have you read this ? >>>>>>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Feb 26, 2016 at 11:03 AM, Ashish Soni < >>>>>>>>>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi All , >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is there any proper documentation as how to run spark on >>>>>>>>>>>>>>>>> mesos , I am trying from the last few days and not able to >>>>>>>>>>>>>>>>> make it work. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Please help >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> >> >
