See below and Attached the Dockerfile to build the spark image ( between i just upgraded to 1.6 )
I am running below setup - Mesos Master - Docker Container Mesos Slave 1 - Docker Container Mesos Slave 2 - Docker Container Marathon - Docker Container Spark MESOS Dispatcher - Docker Container when i submit the Spark PI Example Job using below command *docker run -it --rm -m 2g -e SPARK_MASTER="mesos://10.0.2.15:7077 <http://10.0.2.15:7077>" -e SPARK_IMAGE="spark_driver:**latest" spark_driver:latest ./bin/spark-submit --deploy-mode cluster --name "PI Example" --class org.apache.spark.examples.**SparkPi http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar <http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar> --jars /opt/spark/lib/spark-examples-**1.6.0-hadoop2.6.0.jar --verbose* Below is the ERROR Error: Cannot load main class from JAR file:/opt/spark/Example Run with --help for usage help or --verbose for debug output When i docker Inspect for the stopped / dead container i see below output what is interesting to see is some one or executor replaced by original command with below in highlighted and i do not see Executor is downloading the JAR -- IS this a BUG i am hitting or not sure if that is supposed to work this way and i am missing some configuration "Env": [ "SPARK_IMAGE=spark_driver:latest", "SPARK_SCALA_VERSION=2.10", "SPARK_VERSION=1.6.0", "SPARK_EXECUTOR_URI= http://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-hadoop2.6.tgz";, "MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos-0.25.0.so", "SPARK_MASTER=mesos://10.0.2.15:7077", "SPARK_EXECUTOR_OPTS=-Dspark.executorEnv.MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/ libmesos-0.25.0.so -Dspark.jars= http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar -Dspark.mesos.mesosExecutor.cores=0.1 -Dspark.driver.supervise=false - Dspark.app.name=PI Example -Dspark.mesos.uris= http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar -Dspark.mesos.executor.docker.image=spark_driver:latest -Dspark.submit.deployMode=cluster -Dspark.master=mesos://10.0.2.15:7077 -Dspark.driver.extraClassPath=/opt/spark/custom/lib/* -Dspark.executor.extraClassPath=/opt/spark/custom/lib/* -Dspark.executor.uri= http://d3kbcqa49mib13.cloudfront.net/spark-1.6.0-bin-hadoop2.6.tgz -Dspark.mesos.executor.home=/opt/spark", "MESOS_SANDBOX=/mnt/mesos/sandbox", "MESOS_CONTAINER_NAME=mesos-e47f8d4c-5ee1-4d01-ad07-0d9a03ced62d-S1.43c08f82-e508-4d57-8c0b-fa05bee77fd6", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "HADOOP_VERSION=2.6", "SPARK_HOME=/opt/spark" ], "Cmd": [ "-c", * "./bin/spark-submit --name PI Example --master mesos://10.0.2.15:5050 <http://10.0.2.15:5050> --driver-cores 1.0 --driver-memory 1024M --class org.apache.spark.examples.SparkPi $MESOS_SANDBOX/spark-examples-1.6.0-hadoop2.6.0.jar --jars /opt/spark/lib/spark-examples-1.6.0-hadoop2.6.0.jar --verbose"* ], "Image": "spark_driver:latest", On Wed, Mar 2, 2016 at 5:49 PM, Charles Allen <charles.al...@metamarkets.com > wrote: > @Tim yes, this is asking about 1.5 though > > On Wed, Mar 2, 2016 at 2:35 PM Tim Chen <t...@mesosphere.io> wrote: > >> Hi Charles, >> >> I thought that's fixed with your patch in latest master now right? >> >> Ashish, yes please give me your docker image name (if it's in the public >> registry) and what you've tried and I can see what's wrong. I think it's >> most likely just the configuration of where the Spark home folder is in the >> image. >> >> Tim >> >> On Wed, Mar 2, 2016 at 2:28 PM, Charles Allen < >> charles.al...@metamarkets.com> wrote: >> >>> Re: Spark on Mesos.... Warning regarding disk space: >>> https://issues.apache.org/jira/browse/SPARK-12330 >>> >>> That's a spark flaw I encountered on a very regular basis on mesos. That >>> and a few other annoyances are fixed in >>> https://github.com/metamx/spark/tree/v1.5.2-mmx >>> >>> Here's another mild annoyance I've encountered: >>> https://issues.apache.org/jira/browse/SPARK-11714 >>> >>> On Wed, Mar 2, 2016 at 1:31 PM Ashish Soni <asoni.le...@gmail.com> >>> wrote: >>> >>>> I have no luck and i would to ask the question to spark committers will >>>> this be ever designed to run on mesos ? >>>> >>>> spark app as a docker container not working at all on mesos ,if any >>>> one would like the code i can send it over to have a look. >>>> >>>> Ashish >>>> >>>> On Wed, Mar 2, 2016 at 12:23 PM, Sathish Kumaran Vairavelu < >>>> vsathishkuma...@gmail.com> wrote: >>>> >>>>> Try passing jar using --jars option >>>>> >>>>> On Wed, Mar 2, 2016 at 10:17 AM Ashish Soni <asoni.le...@gmail.com> >>>>> wrote: >>>>> >>>>>> I made some progress but now i am stuck at this point , Please help >>>>>> as looks like i am close to get it working >>>>>> >>>>>> I have everything running in docker container including mesos slave >>>>>> and master >>>>>> >>>>>> When i try to submit the pi example i get below error >>>>>> *Error: Cannot load main class from JAR file:/opt/spark/Example* >>>>>> >>>>>> Below is the command i use to submit as a docker container >>>>>> >>>>>> docker run -it --rm -e SPARK_MASTER="mesos://10.0.2.15:7077" -e >>>>>> SPARK_IMAGE="spark_driver:latest" spark_driver:latest ./bin/spark-submit >>>>>> --deploy-mode cluster --name "PI Example" --class >>>>>> org.apache.spark.examples.SparkPi --driver-memory 512m --executor-memory >>>>>> 512m --executor-cores 1 >>>>>> http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar >>>>>> >>>>>> >>>>>> On Tue, Mar 1, 2016 at 2:59 PM, Timothy Chen <t...@mesosphere.io> >>>>>> wrote: >>>>>> >>>>>>> Can you go through the Mesos UI and look at the driver/executor log >>>>>>> from steer file and see what the problem is? >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> On Mar 1, 2016, at 8:05 AM, Ashish Soni <asoni.le...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> Not sure what is the issue but i am getting below error when i try >>>>>>> to run spark PI example >>>>>>> >>>>>>> Blacklisting Mesos slave value: "5345asdasdasdkas234234asdasdasdasd" >>>>>>> due to too many failures; is Spark installed on it? >>>>>>> WARN TaskSchedulerImpl: Initial job has not accepted any resources; >>>>>>> check your cluster UI to ensure that workers are registered and have >>>>>>> sufficient resources >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 29, 2016 at 1:39 PM, Sathish Kumaran Vairavelu < >>>>>>> vsathishkuma...@gmail.com> wrote: >>>>>>> >>>>>>>> May be the Mesos executor couldn't find spark image or the >>>>>>>> constraints are not satisfied. Check your Mesos UI if you see Spark >>>>>>>> application in the Frameworks tab >>>>>>>> >>>>>>>> On Mon, Feb 29, 2016 at 12:23 PM Ashish Soni <asoni.le...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> What is the Best practice , I have everything running as docker >>>>>>>>> container in single host ( mesos and marathon also as docker >>>>>>>>> container ) >>>>>>>>> and everything comes up fine but when i try to launch the spark >>>>>>>>> shell i >>>>>>>>> get below error >>>>>>>>> >>>>>>>>> >>>>>>>>> SQL context available as sqlContext. >>>>>>>>> >>>>>>>>> scala> val data = sc.parallelize(1 to 100) >>>>>>>>> data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at >>>>>>>>> parallelize at <console>:27 >>>>>>>>> >>>>>>>>> scala> data.count >>>>>>>>> [Stage 0:> >>>>>>>>> (0 + 0) / 2]16/02/29 18:21:12 WARN TaskSchedulerImpl: Initial job >>>>>>>>> has not >>>>>>>>> accepted any resources; check your cluster UI to ensure that workers >>>>>>>>> are >>>>>>>>> registered and have sufficient resources >>>>>>>>> 16/02/29 18:21:27 WARN TaskSchedulerImpl: Initial job has not >>>>>>>>> accepted any resources; check your cluster UI to ensure that workers >>>>>>>>> are >>>>>>>>> registered and have sufficient resources >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 29, 2016 at 12:04 PM, Tim Chen <t...@mesosphere.io> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> No you don't have to run Mesos in docker containers to run Spark >>>>>>>>>> in docker containers. >>>>>>>>>> >>>>>>>>>> Once you have Mesos cluster running you can then specfiy the >>>>>>>>>> Spark configurations in your Spark job (i.e: >>>>>>>>>> spark.mesos.executor.docker.image=mesosphere/spark:1.6) >>>>>>>>>> and Mesos will automatically launch docker containers for you. >>>>>>>>>> >>>>>>>>>> Tim >>>>>>>>>> >>>>>>>>>> On Mon, Feb 29, 2016 at 7:36 AM, Ashish Soni < >>>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Yes i read that and not much details here. >>>>>>>>>>> >>>>>>>>>>> Is it true that we need to have spark installed on each mesos >>>>>>>>>>> docker container ( master and slave ) ... >>>>>>>>>>> >>>>>>>>>>> Ashish >>>>>>>>>>> >>>>>>>>>>> On Fri, Feb 26, 2016 at 2:14 PM, Tim Chen <t...@mesosphere.io> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html should >>>>>>>>>>>> be the best source, what problems were you running into? >>>>>>>>>>>> >>>>>>>>>>>> Tim >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Feb 26, 2016 at 11:06 AM, Yin Yang <yy201...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Have you read this ? >>>>>>>>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Feb 26, 2016 at 11:03 AM, Ashish Soni < >>>>>>>>>>>>> asoni.le...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi All , >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is there any proper documentation as how to run spark on >>>>>>>>>>>>>> mesos , I am trying from the last few days and not able to make >>>>>>>>>>>>>> it work. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please help >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ashish >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> >>
driver.tar.gz
Description: GNU Zip compressed data
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
