I made some progress but now i am stuck at this point , Please help as looks like i am close to get it working
I have everything running in docker container including mesos slave and master When i try to submit the pi example i get below error *Error: Cannot load main class from JAR file:/opt/spark/Example* Below is the command i use to submit as a docker container docker run -it --rm -e SPARK_MASTER="mesos://10.0.2.15:7077" -e SPARK_IMAGE="spark_driver:latest" spark_driver:latest ./bin/spark-submit --deploy-mode cluster --name "PI Example" --class org.apache.spark.examples.SparkPi --driver-memory 512m --executor-memory 512m --executor-cores 1 http://10.0.2.15/spark-examples-1.6.0-hadoop2.6.0.jar On Tue, Mar 1, 2016 at 2:59 PM, Timothy Chen <t...@mesosphere.io> wrote: > Can you go through the Mesos UI and look at the driver/executor log from > steer file and see what the problem is? > > Tim > > On Mar 1, 2016, at 8:05 AM, Ashish Soni <asoni.le...@gmail.com> wrote: > > Not sure what is the issue but i am getting below error when i try to run > spark PI example > > Blacklisting Mesos slave value: "5345asdasdasdkas234234asdasdasdasd" > due to too many failures; is Spark installed on it? > WARN TaskSchedulerImpl: Initial job has not accepted any resources; check > your cluster UI to ensure that workers are registered and have sufficient > resources > > > On Mon, Feb 29, 2016 at 1:39 PM, Sathish Kumaran Vairavelu < > vsathishkuma...@gmail.com> wrote: > >> May be the Mesos executor couldn't find spark image or the constraints >> are not satisfied. Check your Mesos UI if you see Spark application in the >> Frameworks tab >> >> On Mon, Feb 29, 2016 at 12:23 PM Ashish Soni <asoni.le...@gmail.com> >> wrote: >> >>> What is the Best practice , I have everything running as docker >>> container in single host ( mesos and marathon also as docker container ) >>> and everything comes up fine but when i try to launch the spark shell i >>> get below error >>> >>> >>> SQL context available as sqlContext. >>> >>> scala> val data = sc.parallelize(1 to 100) >>> data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at >>> parallelize at <console>:27 >>> >>> scala> data.count >>> [Stage 0:> (0 + >>> 0) / 2]16/02/29 18:21:12 WARN TaskSchedulerImpl: Initial job has not >>> accepted any resources; check your cluster UI to ensure that workers are >>> registered and have sufficient resources >>> 16/02/29 18:21:27 WARN TaskSchedulerImpl: Initial job has not accepted >>> any resources; check your cluster UI to ensure that workers are registered >>> and have sufficient resources >>> >>> >>> >>> On Mon, Feb 29, 2016 at 12:04 PM, Tim Chen <t...@mesosphere.io> wrote: >>> >>>> No you don't have to run Mesos in docker containers to run Spark in >>>> docker containers. >>>> >>>> Once you have Mesos cluster running you can then specfiy the Spark >>>> configurations in your Spark job (i.e: >>>> spark.mesos.executor.docker.image=mesosphere/spark:1.6) >>>> and Mesos will automatically launch docker containers for you. >>>> >>>> Tim >>>> >>>> On Mon, Feb 29, 2016 at 7:36 AM, Ashish Soni <asoni.le...@gmail.com> >>>> wrote: >>>> >>>>> Yes i read that and not much details here. >>>>> >>>>> Is it true that we need to have spark installed on each mesos docker >>>>> container ( master and slave ) ... >>>>> >>>>> Ashish >>>>> >>>>> On Fri, Feb 26, 2016 at 2:14 PM, Tim Chen <t...@mesosphere.io> wrote: >>>>> >>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html should be >>>>>> the best source, what problems were you running into? >>>>>> >>>>>> Tim >>>>>> >>>>>> On Fri, Feb 26, 2016 at 11:06 AM, Yin Yang <yy201...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Have you read this ? >>>>>>> https://spark.apache.org/docs/latest/running-on-mesos.html >>>>>>> >>>>>>> On Fri, Feb 26, 2016 at 11:03 AM, Ashish Soni <asoni.le...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi All , >>>>>>>> >>>>>>>> Is there any proper documentation as how to run spark on mesos , I >>>>>>>> am trying from the last few days and not able to make it work. >>>>>>>> >>>>>>>> Please help >>>>>>>> >>>>>>>> Ashish >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >
