Re: Dependencies with runing Spark Streaming on Mesos cluster using Python

I think there are two options for you: First you can set `--conf spark.mesos.executor.docker.image= adolphlwq/mesos-for-spark-exector-image:1.6.0.beta2` in your spark submit args, so mesos would launch the executor with your custom image. Or you can remove the `local:` prefix in the --jars flag,

Re: Dependencies with runing Spark Streaming on Mesos cluster using Python

Update: I revuild my mesos-exector-image ,I download *spark-streaming-kafka_2.10-1.6.0.jar* on *`/linker/jars`* I change my submit command: dcos spark run \ --submit-args='--jars > local:/linker/jars/spark-streaming-kafka_2.10-1.6.0.jar spark2cassandra.py > 10.140.0.14:2181 wlu_spark2cassand

Dependencies with runing Spark Streaming on Mesos cluster using Python

Hi all: My spark runs on mesos.I write a spark streaming app using python, code on GitHub . The app has dependency "*org.apache.spark:spark-streaming-kafka_2.10:1.6.1* ". Spark on mesos has two important concepts: Spark Framework and Spark exect

Re: Spark Streaming on mesos

cheduler is getting >>>> revamped now. >>>> >>>> Tim >>>> >>>> On Nov 28, 2015, at 7:31 PM, Renjie Liu >>>> wrote: >>>> >>>> Hi, Nagaraj: >>>> Thanks for the response, but this does not solve my

Re: Spark Streaming on mesos

Renjie Liu wrote: >>> >>> Hi, Nagaraj: >>> Thanks for the response, but this does not solve my problem. >>> I think executor memory should be proportional to number of cores, or >>> number of core >>> in each executor should be the same. >

Re: Spark Streaming on mesos

nal to number of cores, or >> number of core >> in each executor should be the same. >> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar < >> nchandrashe...@innominds.com> wrote: >> >>> Hi Renjie, >>> >>> I have not setup Spark Stre

Re: Spark Streaming on mesos

the >> slave like so: >> >> $ mesos-slave \ >> --master=: \ >> --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096" >> >> We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave. >> >> >> From: Renjie

Re: Spark Streaming on mesos

ar >> wrote: >> Hi Renjie, >> >> I have not setup Spark Streaming on Mesos but there is something called >> reservations in Mesos. It supports both Static and Dynamic reservations. >> Both types of reservations must have role defined. You may want to explore &

Re: Spark Streaming on mesos

> Hi Renjie, > > I have not setup Spark Streaming on Mesos but there is something called > reservations in Mesos. It supports both Static and Dynamic reservations. > Both types of reservations must have role defined. You may want to explore > these options. Excerpts from the Ap

Re: Spark Streaming on mesos

Hi Renjie, I have not setup Spark Streaming on Mesos but there is something called reservations in Mesos. It supports both Static and Dynamic reservations. Both types of reservations must have role defined. You may want to explore these options. Excerpts from the Apache Mesos documentation

Spark Streaming on mesos

Hi, all: I'm trying to run spark streaming on mesos and it seems that none of the scheduler is suitable for that. Fine grain scheduler will start an executor for each task so it will significantly increase the latency. While coarse grained mode can only set the max core numbers and executor m

Re: dynamic allocation w/ spark streaming on mesos?

t;> the issues that need to be resolved. >>>>>> >>>>>> Dean Wampler, Ph.D. >>>>>> Author: Programming Scala, 2nd Edition >>>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) >>>>>> Typesafe <http://typesafe.com> >>>>>> @deanwampler <http://twitter.com/deanwampler> >>>>>> http://polyglotprogramming.com >>>>>> >>>>>> On Wed, Nov 11, 2015 at 8:09 AM, PhuDuc Nguyen < >>>>>> duc.was.h...@gmail.com> wrote: >>>>>> >>>>>>> I'm trying to get Spark Streaming to scale up/down its number of >>>>>>> executors within Mesos based on workload. It's not scaling down. I'm >>>>>>> using >>>>>>> Spark 1.5.1 reading from Kafka using the direct (receiver-less) >>>>>>> approach. >>>>>>> >>>>>>> Based on this ticket >>>>>>> https://issues.apache.org/jira/browse/SPARK-6287 with the right >>>>>>> configuration, I have a simple example working with the spark-shell >>>>>>> connected to a Mesos cluster. By working I mean the number of executors >>>>>>> scales up/down based on workload. However, the spark-shell is not a >>>>>>> streaming example. >>>>>>> >>>>>>> What is that status of dynamic resource allocation with Spark >>>>>>> Streaming on Mesos? Is it supported at all? Or supported but with some >>>>>>> caveats to ensure no data loss? >>>>>>> >>>>>>> thanks, >>>>>>> Duc >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >

Re: dynamic allocation w/ spark streaming on mesos?

illy.com/product/0636920033073.do> (O'Reilly) >>>>> Typesafe <http://typesafe.com> >>>>> @deanwampler <http://twitter.com/deanwampler> >>>>> http://polyglotprogramming.com >>>>> >>>>> On Wed, Nov 11, 2015 at 8:09 AM, PhuDuc Nguyen >>>> > wrote: >>>>> >>>>>> I'm trying to get Spark Streaming to scale up/down its number of >>>>>> executors within Mesos based on workload. It's not scaling down. I'm >>>>>> using >>>>>> Spark 1.5.1 reading from Kafka using the direct (receiver-less) approach. >>>>>> >>>>>> Based on this ticket https://issues.apache.org/jira/browse/SPARK-6287 >>>>>> with the right configuration, I have a simple example working with the >>>>>> spark-shell connected to a Mesos cluster. By working I mean the number of >>>>>> executors scales up/down based on workload. However, the spark-shell is >>>>>> not >>>>>> a streaming example. >>>>>> >>>>>> What is that status of dynamic resource allocation with Spark >>>>>> Streaming on Mesos? Is it supported at all? Or supported but with some >>>>>> caveats to ensure no data loss? >>>>>> >>>>>> thanks, >>>>>> Duc >>>>>> >>>>> >>>>> >>>> >>> >> >

Re: dynamic allocation w/ spark streaming on mesos?

;http://twitter.com/deanwampler> >>>> http://polyglotprogramming.com >>>> >>>> On Wed, Nov 11, 2015 at 8:09 AM, PhuDuc Nguyen >>>> wrote: >>>> >>>>> I'm trying to get Spark Streaming to scale up/down its number of &g

Re: dynamic allocation w/ spark streaming on mesos?

> I'm trying to get Spark Streaming to scale up/down its number of >>>> executors within Mesos based on workload. It's not scaling down. I'm using >>>> Spark 1.5.1 reading from Kafka using the direct (receiver-less) approach. >>>> >&g

Re: dynamic allocation w/ spark streaming on mesos?

287 >>> with the right configuration, I have a simple example working with the >>> spark-shell connected to a Mesos cluster. By working I mean the number of >>> executors scales up/down based on workload. However, the spark-shell is not >>> a streaming example. >>> >>> What is that status of dynamic resource allocation with Spark Streaming >>> on Mesos? Is it supported at all? Or supported but with some caveats to >>> ensure no data loss? >>> >>> thanks, >>> Duc >>> >> >> >

Re: dynamic allocation w/ spark streaming on mesos?

; Based on this ticket https://issues.apache.org/jira/browse/SPARK-6287 >> with the right configuration, I have a simple example working with the >> spark-shell connected to a Mesos cluster. By working I mean the number of >> executors scales up/down based on workload. However, the

Re: dynamic allocation w/ spark streaming on mesos?

spark-shell connected to a Mesos cluster. By working I mean the number of > executors scales up/down based on workload. However, the spark-shell is not > a streaming example. > > What is that status of dynamic resource allocation with Spark Streaming on > Mesos? Is it supported at all? Or supported but with some caveats to ensure > no data loss? > > thanks, > Duc >

dynamic allocation w/ spark streaming on mesos?

tion with Spark Streaming on Mesos? Is it supported at all? Or supported but with some caveats to ensure no data loss? thanks, Duc

Re: [Spark Streaming on Mesos (good practices)]

Here is the answer to my question if somebody needs it Running Spark in Standalone mode or coarse-grained Mesos mode leads to better task launch times than the fine-grained Mesos mode. The resource is http://spark.apache.org/docs/latest/streaming-programming-guide.html On Mon, Aug 24, 2015 at

[Spark Streaming on Mesos (good practices)]

which are the best practices to submit spark streaming application on mesos. I would like to know about scheduler mode. Is `coarse-grained` mode right solution? Thanks

akka.actor.ActorNotFound In Spark Streaming on Mesos (using ssc.actorStream)

dn't manage to communicate with my actor How can I reach my actor? Can the mesos 0.21.0 be the source of my problem? Have I misconfigured anything? Any ideas? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/akka-actor-ActorNotFound-In-Spark-Streami

Spark Streaming on Mesos: How is the nr of coarse-grained executors calculated?

Hi, We're running several Spark Streaming-kafka-Cassandra jobs on Mesos. I'm currently working on tuning and validating scalability, and I'm looking for the way to configure the number of coarse grained task executors for a job. For example: I'm consuming 2 Kafka topics with 12 partitions and I'

Spark Streaming on Mesos, various questions

Hi, with the hints from Gerard I was able to get my locally working Spark code running on Mesos. Thanks! Basically, on my local dev machine, I use "sbt assembly" to create a fat jar (which is actually not so fat since I use "... % 'provided'" in my sbt file for the Spark dependencies), upload it