Re: Spark 1.5 on Mesos

It did not helped , same error , Is this the issue i am running into https://issues.apache.org/jira/browse/SPARK-11638 *Warning: Local jar /mnt/mesos/sandbox/spark-examples-1.6.0-hadoop2.6.0.jar does not exist, skipping.* java.lang.ClassNotFoundException: org.apache.spark.examples.SparkPi On

Re: Spark 1.5 on Mesos

Ah I see, I think it's because you've launched the Mesos slave in a docker container, and when you launch also the executor in a container it's not able to mount in the sandbox to the other container since the slave is in a chroot. Can you try mounting in a volume from the host when you launch

Re: Spark 1.5 on Mesos

You shouldn't need to specify --jars at all since you only have one jar. The error is pretty odd as it suggests it's trying to load /opt/spark/Example but that doesn't really seem to be anywhere in your image or command. Can you paste your stdout from the driver task launched by the cluster

Re: Spark 1.5 on Mesos

See below and Attached the Dockerfile to build the spark image ( between i just upgraded to 1.6 ) I am running below setup - Mesos Master - Docker Container Mesos Slave 1 - Docker Container Mesos Slave 2 - Docker Container Marathon - Docker Container Spark MESOS Dispatcher - Docker

Re: Spark 1.5 on Mesos

Hi Charles, I thought that's fixed with your patch in latest master now right? Ashish, yes please give me your docker image name (if it's in the public registry) and what you've tried and I can see what's wrong. I think it's most likely just the configuration of where the Spark home folder is in

Re: Spark 1.5 on Mesos

Re: Spark on Mesos Warning regarding disk space: https://issues.apache.org/jira/browse/SPARK-12330 That's a spark flaw I encountered on a very regular basis on mesos. That and a few other annoyances are fixed in https://github.com/metamx/spark/tree/v1.5.2-mmx Here's another mild annoyance

Re: Spark 1.5 on Mesos

I have no luck and i would to ask the question to spark committers will this be ever designed to run on mesos ? spark app as a docker container not working at all on mesos ,if any one would like the code i can send it over to have a look. Ashish On Wed, Mar 2, 2016 at 12:23 PM, Sathish Kumaran

Re: Spark 1.5 on Mesos

Try passing jar using --jars option On Wed, Mar 2, 2016 at 10:17 AM Ashish Soni wrote: > I made some progress but now i am stuck at this point , Please help as > looks like i am close to get it working > > I have everything running in docker container including mesos slave

Re: Spark 1.5 on Mesos

I made some progress but now i am stuck at this point , Please help as looks like i am close to get it working I have everything running in docker container including mesos slave and master When i try to submit the pi example i get below error *Error: Cannot load main class from JAR

Re: Spark 1.5 on Mesos

Can you go through the Mesos UI and look at the driver/executor log from steer file and see what the problem is? Tim > On Mar 1, 2016, at 8:05 AM, Ashish Soni wrote: > > Not sure what is the issue but i am getting below error when i try to run > spark PI example > >

Re: Spark 1.5 on Mesos

Not sure what is the issue but i am getting below error when i try to run spark PI example Blacklisting Mesos slave value: "5345asdasdasdkas234234asdasdasdasd" due to too many failures; is Spark installed on it? WARN TaskSchedulerImpl: Initial job has not accepted any resources; check

Re: Spark 1.5 on Mesos

What is the Best practice , I have everything running as docker container in single host ( mesos and marathon also as docker container ) and everything comes up fine but when i try to launch the spark shell i get below error SQL context available as sqlContext. scala> val data =

Re: Spark 1.5 on Mesos

No you don't have to run Mesos in docker containers to run Spark in docker containers. Once you have Mesos cluster running you can then specfiy the Spark configurations in your Spark job (i.e: spark.mesos.executor.docker.image=mesosphere/spark:1.6) and Mesos will automatically launch docker

Re: Spark 1.5 on Mesos

Yes i read that and not much details here. Is it true that we need to have spark installed on each mesos docker container ( master and slave ) ... Ashish On Fri, Feb 26, 2016 at 2:14 PM, Tim Chen wrote: > https://spark.apache.org/docs/latest/running-on-mesos.html should be

Re: Spark 1.5 on Mesos

https://spark.apache.org/docs/latest/running-on-mesos.html should be the best source, what problems were you running into? Tim On Fri, Feb 26, 2016 at 11:06 AM, Yin Yang wrote: > Have you read this ? > https://spark.apache.org/docs/latest/running-on-mesos.html > > On Fri,

Spark 1.5 on Mesos

Hi All , Is there any proper documentation as how to run spark on mesos , I am trying from the last few days and not able to make it work. Please help Ashish

Re: Spark 1.5 on Mesos

Have you read this ? https://spark.apache.org/docs/latest/running-on-mesos.html On Fri, Feb 26, 2016 at 11:03 AM, Ashish Soni wrote: > Hi All , > > Is there any proper documentation as how to run spark on mesos , I am > trying from the last few days and not able to make