Re: Removing the Mesos fine-grained mode
As a recent fine-grained mode adopter I'm now confused after reading this and other resources from spark-summit, the docs, ... so can someone please advise me for our use-case? We'll have 1 or 2 streaming jobs and an will run scheduled batch jobs which should take resources away from the streaming jobs and give 'em back upon completion. Can someone point me at the docs or a guide to set this up? Thanks! - Jo Voordeckers On Thu, Nov 19, 2015 at 5:52 AM, Heller, Chris <chel...@akamai.com> wrote: > I was one that argued for fine-grain mode, and there is something I still > appreciate about how fine-grain mode operates in terms of the way one would > define a Mesos framework. That said, with dyn-allocation and Mesos support > for both resource reservation, oversubscription and revocation, I think the > direction is clear that the coarse mode is the proper way forward, and > having the two code paths is just noise. > > -Chris > > From: Iulian Dragoș <iulian.dra...@typesafe.com> > Date: Thursday, November 19, 2015 at 6:42 AM > To: "dev@spark.apache.org" <dev@spark.apache.org> > Subject: Removing the Mesos fine-grained mode > > Hi all, > > Mesos is the only cluster manager that has a fine-grained mode, but it's > more often than not problematic, and it's a maintenance burden. I'd like to > suggest removing it in the 2.0 release. > > A few reasons: > > - code/maintenance complexity. The two modes duplicate a lot of > functionality (and sometimes code) that leads to subtle differences or > bugs. See SPARK-10444 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SPARK-2D10444=CwMFaQ=96ZbZZcaMF4w0F4jpN6LZg=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc=4_2dJBDiLqTcfXfX1LZluOo1U6tRKR2wKGGzfwiKdVY=> > and > also this thread > <https://urldefense.proofpoint.com/v2/url?u=https-3A__mail-2Darchives.apache.org_mod-5Fmbox_spark-2Duser_201510.mbox_-253CCALxMP-2DA-2BaygNwSiyTM8ff20-2DMGWHykbhct94a2hwZTh1jWHp-5Fg-40mail.gmail.com-253E=CwMFaQ=96ZbZZcaMF4w0F4jpN6LZg=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc=SNFPzodGw7sgp3km9NKYM46gZHLguvxVNzCIeUlJzOw=> > and MESOS-3202 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_MESOS-2D3202=CwMFaQ=96ZbZZcaMF4w0F4jpN6LZg=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc=d-U4CohYsiZc0Zmj4KETn2dT_2ZFe5s3_IIbMm2tjJo=> > - it's not widely used (Reynold's previous thread > <https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Ddevelopers-2Dlist.1001551.n3.nabble.com_Please-2Dreply-2Dif-2Dyou-2Duse-2DMesos-2Dfine-2Dgrained-2Dmode-2Dtd14930.html=CwMFaQ=96ZbZZcaMF4w0F4jpN6LZg=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc=HGMiKyzxFDhpbomduKVIIRHWk9RDGDCk7tneJVQqTwo=> > got very few responses from people relying on it) > - similar functionality can be achieved with dynamic allocation + > coarse-grained mode > > I suggest that Spark 1.6 already issues a warning if it detects > fine-grained use, with removal in the 2.0 release. > > Thoughts? > > iulian > >
Re: Mesos cluster dispatcher doesn't respect most args from the submit req
On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș <iulian.dra...@typesafe.com> wrote: > I think it actually tries to send all properties as part of > `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed: > > > https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377 > > Aha that's interesting, I overlooked that line, I'll debug some more today because I know for sure that those options don't make it onto the commandline when I was running it in my debugger. > Can you please open a Jira ticket and describe also the symptoms? This > might be related, or the same issue: SPARK-11280 > <https://issues.apache.org/jira/browse/SPARK-11280> and also SPARK-11327 > <https://issues.apache.org/jira/browse/SPARK-11327> > SPARK-11327 <https://issues.apache.org/jira/browse/SPARK-11327> is exactly my problem, but I don't run docker. - Jo On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <jo.voordeck...@gmail.com> > wrote: > >> >> Hi all, >> >> I'm running the mesos cluster dispatcher, however when I submit jobs with >> things like jvm args, classpath order and UI port aren't added to the >> commandline executed by the mesos scheduler. In fact it only cares about >> the class, jar and num cores/mem. >> >> >> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424 >> >> I've made an attempt at adding a few of the args that I believe are >> useful to the MesosClusterScheduler class, which seems to solve my problem. >> >> Please have a look: >> >> https://github.com/apache/spark/pull/9752 >> >> Thanks >> >> - Jo Voordeckers >> >> > > > -- > > -- > Iulian Dragos > > -- > Reactive Apps on the JVM > www.typesafe.com > >
Re: Mesos cluster dispatcher doesn't respect most args from the submit req
Hi Tim, I've done more forensics on this bug, see my comment here: https://issues.apache.org/jira/browse/SPARK-11327?focusedCommentId=15009843=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15009843 - Jo Voordeckers On Tue, Nov 17, 2015 at 12:01 PM, Timothy Chen <tnac...@gmail.com> wrote: > Hi Jo, > > Thanks for the links, I would expected the properties to be in > scheduler properties but I need to double check. > > I'll be looking into these problems this week. > > Tim > > On Tue, Nov 17, 2015 at 10:28 AM, Jo Voordeckers > <jo.voordeck...@gmail.com> wrote: > > On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș < > iulian.dra...@typesafe.com> > > wrote: > >> > >> I think it actually tries to send all properties as part of > >> `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed: > >> > >> > >> > https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377 > >> > > > > Aha that's interesting, I overlooked that line, I'll debug some more > today > > because I know for sure that those options don't make it onto the > > commandline when I was running it in my debugger. > > > >> > >> Can you please open a Jira ticket and describe also the symptoms? This > >> might be related, or the same issue: SPARK-11280 and also SPARK-11327 > > > > > > SPARK-11327 is exactly my problem, but I don't run docker. > > > > - Jo > > > >> On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers < > jo.voordeck...@gmail.com> > >> wrote: > >>> > >>> > >>> Hi all, > >>> > >>> I'm running the mesos cluster dispatcher, however when I submit jobs > with > >>> things like jvm args, classpath order and UI port aren't added to the > >>> commandline executed by the mesos scheduler. In fact it only cares > about the > >>> class, jar and num cores/mem. > >>> > >>> > >>> > https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424 > >>> > >>> I've made an attempt at adding a few of the args that I believe are > >>> useful to the MesosClusterScheduler class, which seems to solve my > problem. > >>> > >>> Please have a look: > >>> > >>> https://github.com/apache/spark/pull/9752 > >>> > >>> Thanks > >>> > >>> - Jo Voordeckers > >>> > >> > >> > >> > >> -- > >> > >> -- > >> Iulian Dragos > >> > >> -- > >> Reactive Apps on the JVM > >> www.typesafe.com > >> > > >
Mesos cluster dispatcher doesn't respect most args from the submit req
Hi all, I'm running the mesos cluster dispatcher, however when I submit jobs with things like jvm args, classpath order and UI port aren't added to the commandline executed by the mesos scheduler. In fact it only cares about the class, jar and num cores/mem. https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424 I've made an attempt at adding a few of the args that I believe are useful to the MesosClusterScheduler class, which seems to solve my problem. Please have a look: https://github.com/apache/spark/pull/9752 Thanks - Jo Voordeckers
