As a recent fine-grained mode adopter I'm now confused after reading this and other resources from spark-summit, the docs, ... so can someone please advise me for our use-case?
We'll have 1 or 2 streaming jobs and an will run scheduled batch jobs which should take resources away from the streaming jobs and give 'em back upon completion. Can someone point me at the docs or a guide to set this up? Thanks! - Jo Voordeckers On Thu, Nov 19, 2015 at 5:52 AM, Heller, Chris <chel...@akamai.com> wrote: > I was one that argued for fine-grain mode, and there is something I still > appreciate about how fine-grain mode operates in terms of the way one would > define a Mesos framework. That said, with dyn-allocation and Mesos support > for both resource reservation, oversubscription and revocation, I think the > direction is clear that the coarse mode is the proper way forward, and > having the two code paths is just noise. > > -Chris > > From: Iulian Dragoș <iulian.dra...@typesafe.com> > Date: Thursday, November 19, 2015 at 6:42 AM > To: "dev@spark.apache.org" <dev@spark.apache.org> > Subject: Removing the Mesos fine-grained mode > > Hi all, > > Mesos is the only cluster manager that has a fine-grained mode, but it's > more often than not problematic, and it's a maintenance burden. I'd like to > suggest removing it in the 2.0 release. > > A few reasons: > > - code/maintenance complexity. The two modes duplicate a lot of > functionality (and sometimes code) that leads to subtle differences or > bugs. See SPARK-10444 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SPARK-2D10444&d=CwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY&m=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc&s=4_2dJBDiLqTcfXfX1LZluOo1U6tRKR2wKGGzfwiKdVY&e=> > and > also this thread > <https://urldefense.proofpoint.com/v2/url?u=https-3A__mail-2Darchives.apache.org_mod-5Fmbox_spark-2Duser_201510.mbox_-253CCALxMP-2DA-2BaygNwSiyTM8ff20-2DMGWHykbhct94a2hwZTh1jWHp-5Fg-40mail.gmail.com-253E&d=CwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY&m=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc&s=SNFPzodGw7sgp3km9NKYM46gZHLguvxVNzCIeUlJzOw&e=> > and MESOS-3202 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_MESOS-2D3202&d=CwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY&m=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc&s=d-U4CohYsiZc0Zmj4KETn2dT_2ZFe5s3_IIbMm2tjJo&e=> > - it's not widely used (Reynold's previous thread > <https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Ddevelopers-2Dlist.1001551.n3.nabble.com_Please-2Dreply-2Dif-2Dyou-2Duse-2DMesos-2Dfine-2Dgrained-2Dmode-2Dtd14930.html&d=CwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY&m=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc&s=HGMiKyzxFDhpbomduKVIIRHWk9RDGDCk7tneJVQqTwo&e=> > got very few responses from people relying on it) > - similar functionality can be achieved with dynamic allocation + > coarse-grained mode > > I suggest that Spark 1.6 already issues a warning if it detects > fine-grained use, with removal in the 2.0 release. > > Thoughts? > > iulian > >
