Hi, Latency isn't such a big issue as it sounds. Did you try it out and failed some performance metrics?
In short, the *Mesos* executor on a given slave is going to be long-running (consuming memory, but no CPUs). Each Spark task will be scheduled using Mesos CPU resources, but they don't suffer much latency. iulian On Mon, Nov 30, 2015 at 4:17 AM, Renjie Liu <liurenjie2...@gmail.com> wrote: > Hi, Tim: > Fine grain mode is not suitable for streaming applications since it need > to start up an executor each time. When will the revamp get release? In the > coming 1.6.0? > > On Sun, Nov 29, 2015 at 6:16 PM Timothy Chen <t...@mesosphere.io> wrote: > >> Hi Renjie, >> >> You can set number of cores per executor with spark executor cores in >> fine grain mode. >> >> If you want coarse grain mode to support that it will >> Be supported in the near term as he coarse grain scheduler is getting >> revamped now. >> >> Tim >> >> On Nov 28, 2015, at 7:31 PM, Renjie Liu <liurenjie2...@gmail.com> wrote: >> >> Hi, Nagaraj: >> Thanks for the response, but this does not solve my problem. >> I think executor memory should be proportional to number of cores, or >> number of core >> in each executor should be the same. >> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar < >> nchandrashe...@innominds.com> wrote: >> >>> Hi Renjie, >>> >>> I have not setup Spark Streaming on Mesos but there is something called >>> reservations in Mesos. It supports both Static and Dynamic reservations. >>> Both types of reservations must have role defined. You may want to explore >>> these options. Excerpts from the Apache Mesos documentation. >>> >>> Cheers >>> Nagaraj C >>> Reservation >>> >>> Mesos provides mechanisms to reserve resources in specific slaves. The >>> concept was first introduced with static reservation in 0.14.0 which >>> enabled operators to specify the reserved resources on slave startup. This >>> was extended with dynamic reservation in 0.23.0 which enabled operators >>> and authorized frameworks to dynamically reserve resources in the >>> cluster. >>> >>> No breaking changes were introduced with dynamic reservation, which >>> means the existing static reservation mechanism continues to be fully >>> supported. >>> >>> In both types of reservations, resources are reserved for a role. >>> Static Reservation (since 0.14.0) >>> >>> An operator can configure a slave with resources reserved for a role. >>> The reserved resources are specified via the --resources flag. For >>> example, suppose we have 12 CPUs and 6144 MB of RAM available on a slave >>> and that we want to reserve 8 CPUs and 4096 MB of RAM for the ads role. >>> We start the slave like so: >>> >>> $ mesos-slave \ >>> --master=<ip>:<port> \ >>> --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096" >>> >>> We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave. >>> >>> >>> From: Renjie Liu <liurenjie2...@gmail.com> >>> Date: Friday, November 27, 2015 at 9:57 PM >>> To: "user@spark.apache.org" <user@spark.apache.org> >>> Subject: Spark Streaming on mesos >>> >>> Hi, all: >>> I'm trying to run spark streaming on mesos and it seems that none of the >>> scheduler is suitable for that. Fine grain scheduler will start an executor >>> for each task so it will significantly increase the latency. While coarse >>> grained mode can only set the max core numbers and executor memory but >>> there's no way to set the number of cores for each executor. Has anyone >>> deployed spark streaming on mesos? And what's your settings? >>> -- >>> Liu, Renjie >>> Software Engineer, MVAD >>> >> -- >> Liu, Renjie >> Software Engineer, MVAD >> >> -- > Liu, Renjie > Software Engineer, MVAD > -- -- Iulian Dragos ------ Reactive Apps on the JVM www.typesafe.com
