Re: Spark on Mesos 0.28 issue

hen wrote: >> Hi Yang, >> >> Can you share the master log/slave log? >> >> Tim >> >> >> On Apr 12, 2016, at 2:05 PM, Yang Lei wrote: >> >>> I have been able to run spark submission in docker container (HOST network) >>> t

Spark on Mesos 0.28 issue

I have been able to run spark submission in docker container (HOST network) through Marathon on mesos and target to Mesos cluster (zk address) for at least Spark 1.6, 1.5.2 over Mesos 0.26, 0.27. I do need to define SPARK_PUBLIC_DNS and SPARK_LOCAL_IP so that the spark driver can announce the

Re: Issue of running partitioned loading (RDD) in Spark External Datasource on Mesos

I finally isolated the issue to be related to the ActorSystem I reuse from SparkEnv.get.actorSystem. This ActorSystem will contain the configuration defined in my application jar's reference.conf in both local cluster case, and in the case I use it directly in an extension to BaseRelation's buildSc

Re: Issue of running partitioned loading (RDD) in Spark External Datasource on Mesos

ding some local config file that doesn't > exist on your executors? > > You might need to email the Spray list. > > >> On Fri, Apr 24, 2015 at 2:38 PM, Yang Lei wrote: >> forward to dev. >> >> On Mon, Apr 20, 2015 at 10:46 AM, Yang Lei wrote: &g

Re: Issue of running partitioned loading (RDD) in Spark External Datasource on Mesos

forward to dev. On Mon, Apr 20, 2015 at 10:46 AM, Yang Lei wrote: > I implemented two kinds of DataSource, one load data during buildScan, > the other returning my RDD class with partition information for future > loading. > > My RDD's compute gets actorSystem from Spar