On Mon, Nov 30, 2015 at 4:09 PM, Renjie Liu <liurenjie2...@gmail.com> wrote:
> Hi, Lulian: > Please, it's Iulian, not Lulian. > Are you sure that it'll be a long running process in fine-grained mode? I > think you have a misunderstanding about it. An executor will be launched > for some tasks, but not a long running process. When a group of tasks > finished, it will get shutdown. > Sorry I missed your answer. Yes, I'm pretty sure, and if you have SSH access to one of the slaves it's pretty easy to check. What makes you think otherwise? > > On Mon, Nov 30, 2015 at 6:25 PM Iulian Dragoș <iulian.dra...@typesafe.com> > wrote: > >> Hi, >> >> Latency isn't such a big issue as it sounds. Did you try it out and >> failed some performance metrics? >> >> In short, the *Mesos* executor on a given slave is going to be >> long-running (consuming memory, but no CPUs). Each Spark task will be >> scheduled using Mesos CPU resources, but they don't suffer much latency. >> >> iulian >> >> >> On Mon, Nov 30, 2015 at 4:17 AM, Renjie Liu <liurenjie2...@gmail.com> >> wrote: >> >>> Hi, Tim: >>> Fine grain mode is not suitable for streaming applications since it need >>> to start up an executor each time. When will the revamp get release? In the >>> coming 1.6.0? >>> >>> On Sun, Nov 29, 2015 at 6:16 PM Timothy Chen <t...@mesosphere.io> wrote: >>> >>>> Hi Renjie, >>>> >>>> You can set number of cores per executor with spark executor cores in >>>> fine grain mode. >>>> >>>> If you want coarse grain mode to support that it will >>>> Be supported in the near term as he coarse grain scheduler is getting >>>> revamped now. >>>> >>>> Tim >>>> >>>> On Nov 28, 2015, at 7:31 PM, Renjie Liu <liurenjie2...@gmail.com> >>>> wrote: >>>> >>>> Hi, Nagaraj: >>>> Thanks for the response, but this does not solve my problem. >>>> I think executor memory should be proportional to number of cores, or >>>> number of core >>>> in each executor should be the same. >>>> On Sat, Nov 28, 2015 at 1:48 AM Nagaraj Chandrashekar < >>>> nchandrashe...@innominds.com> wrote: >>>> >>>>> Hi Renjie, >>>>> >>>>> I have not setup Spark Streaming on Mesos but there is something >>>>> called reservations in Mesos. It supports both Static and Dynamic >>>>> reservations. Both types of reservations must have role defined. You may >>>>> want to explore these options. Excerpts from the Apache Mesos >>>>> documentation. >>>>> >>>>> Cheers >>>>> Nagaraj C >>>>> Reservation >>>>> >>>>> Mesos provides mechanisms to reserve resources in specific slaves. >>>>> The concept was first introduced with static reservation in 0.14.0 >>>>> which enabled operators to specify the reserved resources on slave >>>>> startup. >>>>> This was extended with dynamic reservation in 0.23.0 which enabled >>>>> operators and authorized frameworks to dynamically reserve resources >>>>> in the cluster. >>>>> >>>>> No breaking changes were introduced with dynamic reservation, which >>>>> means the existing static reservation mechanism continues to be fully >>>>> supported. >>>>> >>>>> In both types of reservations, resources are reserved for a role. >>>>> Static Reservation (since 0.14.0) >>>>> >>>>> An operator can configure a slave with resources reserved for a role. >>>>> The reserved resources are specified via the --resources flag. For >>>>> example, suppose we have 12 CPUs and 6144 MB of RAM available on a slave >>>>> and that we want to reserve 8 CPUs and 4096 MB of RAM for the ads role. >>>>> We start the slave like so: >>>>> >>>>> $ mesos-slave \ >>>>> --master=<ip>:<port> \ >>>>> --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096" >>>>> >>>>> We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave. >>>>> >>>>> >>>>> From: Renjie Liu <liurenjie2...@gmail.com> >>>>> Date: Friday, November 27, 2015 at 9:57 PM >>>>> To: "user@spark.apache.org" <user@spark.apache.org> >>>>> Subject: Spark Streaming on mesos >>>>> >>>>> Hi, all: >>>>> I'm trying to run spark streaming on mesos and it seems that none of >>>>> the scheduler is suitable for that. Fine grain scheduler will start an >>>>> executor for each task so it will significantly increase the latency. >>>>> While >>>>> coarse grained mode can only set the max core numbers and executor memory >>>>> but there's no way to set the number of cores for each executor. Has >>>>> anyone >>>>> deployed spark streaming on mesos? And what's your settings? >>>>> -- >>>>> Liu, Renjie >>>>> Software Engineer, MVAD >>>>> >>>> -- >>>> Liu, Renjie >>>> Software Engineer, MVAD >>>> >>>> -- >>> Liu, Renjie >>> Software Engineer, MVAD >>> >> >> >> >> -- >> >> -- >> Iulian Dragos >> >> ------ >> Reactive Apps on the JVM >> www.typesafe.com >> >> -- > Liu, Renjie > Software Engineer, MVAD > -- -- Iulian Dragos ------ Reactive Apps on the JVM www.typesafe.com