Re: Dependencies with runing Spark Streaming on Mesos cluster using Python

I think there are two options for you: First you can set `--conf spark.mesos.executor.docker.image= adolphlwq/mesos-for-spark-exector-image:1.6.0.beta2` in your spark submit args, so mesos would launch the executor with your custom image. Or you can remove the `local:` prefix in the --jars flag,

Re: Dependencies with runing Spark Streaming on Mesos cluster using Python

Update: I revuild my mesos-exector-image ,I download *spark-streaming-kafka_2.10-1.6.0.jar* on *`/linker/jars`* I change my submit command: dcos spark run \ --submit-args='--jars > local:/linker/jars/spark-streaming-kafka_2.10-1.6.0.jar spark2cassandra.py > 10.140.0.14:2181

Dependencies with runing Spark Streaming on Mesos cluster using Python

Hi all: My spark runs on mesos.I write a spark streaming app using python, code on GitHub . The app has dependency "*org.apache.spark:spark-streaming-kafka_2.10:1.6.1* ". Spark on mesos has two important concepts: Spark Framework and Spark

Re: Spark Streaming on mesos

wrote: >>>> >>>> Hi, Nagaraj: >>>> Thanks for the response, but this does not solve my problem. >>>> I think executor memory should be proportional to number of cores, or >>>> number of core >>>> in each executor

Re: Spark Streaming on mesos

does not solve my problem. >> I think executor memory should be proportional to number of cores, or >> number of core >> in each executor should be the same. >> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar < >> nchandrashe...@innominds.com> wrote: >>

Re: Spark Streaming on mesos

gt;>> Tim >>> >>> On Nov 28, 2015, at 7:31 PM, Renjie Liu <liurenjie2...@gmail.com> wrote: >>> >>> Hi, Nagaraj: >>> Thanks for the response, but this does not solve my problem. >>> I think executor memory should be proportional

Re: Spark Streaming on mesos

at 1:48 AM Nagaraj Chandrashekar >> <nchandrashe...@innominds.com> wrote: >> Hi Renjie, >> >> I have not setup Spark Streaming on Mesos but there is something called >> reservations in Mesos. It supports both Static and Dynamic reservations. >>

Re: Spark Streaming on mesos

agaraj Chandrashekar < > nchandrashe...@innominds.com> wrote: > >> Hi Renjie, >> >> I have not setup Spark Streaming on Mesos but there is something called >> reservations in Mesos. It supports both Static and Dynamic reservations. >> Both types of

Re: Spark Streaming on mesos

> Hi Renjie, > > I have not setup Spark Streaming on Mesos but there is something called > reservations in Mesos. It supports both Static and Dynamic reservations. > Both types of reservations must have role defined. You may want to explore > these options. Excerpts from the Ap

Re: Spark Streaming on mesos

Hi Renjie, I have not setup Spark Streaming on Mesos but there is something called reservations in Mesos. It supports both Static and Dynamic reservations. Both types of reservations must have role defined. You may want to explore these options. Excerpts from the Apache Mesos documentation

Spark Streaming on mesos

Hi, all: I'm trying to run spark streaming on mesos and it seems that none of the scheduler is suitable for that. Fine grain scheduler will start an executor for each task so it will significantly increase the latency. While coarse grained mode can only set the max core numbers and executor memory

Re: dynamic allocation w/ spark streaming on mesos?

ticket https://issues.apache.org/jira/browse/SPARK-6287 >> with the right configuration, I have a simple example working with the >> spark-shell connected to a Mesos cluster. By working I mean the number of >> executors scales up/down based on workload. However, the sp

Re: dynamic allocation w/ spark streaming on mesos?

> spark-shell connected to a Mesos cluster. By working I mean the number of > executors scales up/down based on workload. However, the spark-shell is not > a streaming example. > > What is that status of dynamic resource allocation with Spark Streaming on > Mesos? Is it supported at

dynamic allocation w/ spark streaming on mesos?

Streaming on Mesos? Is it supported at all? Or supported but with some caveats to ensure no data loss? thanks, Duc

Re: dynamic allocation w/ spark streaming on mesos?

gt;>>>> I'm trying to get Spark Streaming to scale up/down its number of >>>>> executors within Mesos based on workload. It's not scaling down. I'm using >>>>> Spark 1.5.1 reading from Kafka using the direct (receiver-less) approach. >>>>> >>>>> Based on this ticket https://issues.apache.org/jira/browse/SPARK-6287 >>>>> with the right configuration, I have a simple example working with the >>>>> spark-shell connected to a Mesos cluster. By working I mean the number of >>>>> executors scales up/down based on workload. However, the spark-shell is >>>>> not >>>>> a streaming example. >>>>> >>>>> What is that status of dynamic resource allocation with Spark >>>>> Streaming on Mesos? Is it supported at all? Or supported but with some >>>>> caveats to ensure no data loss? >>>>> >>>>> thanks, >>>>> Duc >>>>> >>>> >>>> >>> >> >

Re: dynamic allocation w/ spark streaming on mesos?

scenario. There was a previous thread on this topic which >>>>>> discusses >>>>>> the issues that need to be resolved. >>>>>> >>>>>> Dean Wampler, Ph.D. >>>>>> Author: Programming Scala, 2nd Edition

Re: dynamic allocation w/ spark streaming on mesos?

a/browse/SPARK-6287 >>> with the right configuration, I have a simple example working with the >>> spark-shell connected to a Mesos cluster. By working I mean the number of >>> executors scales up/down based on workload. However, the spark-shell is not >>> a streaming example. >>> >>> What is that status of dynamic resource allocation with Spark Streaming >>> on Mesos? Is it supported at all? Or supported but with some caveats to >>> ensure no data loss? >>> >>> thanks, >>> Duc >>> >> >> >

Re: dynamic allocation w/ spark streaming on mesos?

@gmail.com> >>> wrote: >>> >>>> I'm trying to get Spark Streaming to scale up/down its number of >>>> executors within Mesos based on workload. It's not scaling down. I'm using >>>> Spark 1.5.1 reading from Kafka using the direct (receiver-le

Re: [Spark Streaming on Mesos (good practices)]

at 12:15 PM, Aram Mkrtchyan aram.mkrtchyan...@gmail.com wrote: which are the best practices to submit spark streaming application on mesos. I would like to know about scheduler mode. Is `coarse-grained` mode right solution? Thanks

[Spark Streaming on Mesos (good practices)]

which are the best practices to submit spark streaming application on mesos. I would like to know about scheduler mode. Is `coarse-grained` mode right solution? Thanks

Re: kafka spark streaming with mesos

, Bartek Radziszewski bar...@scalaric.com wrote: Hey, I’m trying to run kafka spark streaming using mesos with following example: *sc.stop* *import org.apache.spark.SparkConf* *import org.apache.spark.SparkContext._* *import kafka.serializer.StringDecoder* *import org.apache.spark.streaming

kafka spark streaming with mesos

Hey, I’m trying to run kafka spark streaming using mesos with following example: sc.stop import org.apache.spark.SparkConf import org.apache.spark.SparkContext._ import kafka.serializer.StringDecoder import org.apache.spark.streaming._ import org.apache.spark.streaming.kafka._ import

akka.actor.ActorNotFound In Spark Streaming on Mesos (using ssc.actorStream)

actor? Can the mesos 0.21.0 be the source of my problem? Have I misconfigured anything? Any ideas? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/akka-actor-ActorNotFound-In-Spark-Streaming-on-Mesos-using-ssc-actorStream-tp21014.html Sent from the Apache

Spark Streaming on Mesos, various questions

Hi, with the hints from Gerard I was able to get my locally working Spark code running on Mesos. Thanks! Basically, on my local dev machine, I use sbt assembly to create a fat jar (which is actually not so fat since I use ... % 'provided' in my sbt file for the Spark dependencies), upload it to

Re: Spark Streaming + Kafka + Mesos/Marathon strangeness

the processes are running in lxc containers they wont be sharing a filesystem (or machine for that matter) I cant use an s3n:// url for local dir can I? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-Mesos-Marathon-strangeness-tp3285p3373

Re: Spark Streaming + Kafka + Mesos/Marathon strangeness

://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-Mesos-Marathon-strangeness-tp3285p3356.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark Streaming + Kafka + Mesos/Marathon strangeness

this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-Mesos-Marathon-strangeness-tp3285p3373.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark Streaming + Kafka + Mesos/Marathon strangeness

a filesystem (or machine for that matter) I cant use an s3n:// url for local dir can I? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-Mesos-Marathon-strangeness-tp3285p3373.html Sent from the Apache Spark User List mailing list

Re: Spark Streaming + Kafka + Mesos/Marathon strangeness

. Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-Mesos-Marathon-strangeness-tp3285p3312.html Sent from the Apache Spark User List mailing list archive at Nabble.com.