Hi! Are your customers using YARN? In that case, the default configuration will start a new YARN application per Flink job, no JVMs are shared between jobs. By default, even each slot has its own JVM.
Greetings, Stephan PS: I think the "spawning new JVMs" is what Till referred to when saying "spinning up a new cluster". Keep in mind that Flink is also a batch processor, and it handles sequences of short batch jobs (as issued for example by interactive shells) and it pre-allocates and manages a lot of memory for batch jobs. On Mon, Dec 5, 2016 at 3:48 PM, Manu Zhang <owenzhang1...@gmail.com> wrote: > The pro for the multi-tenant cluster mode is that you can share data >> between jobs and you don't have to spin up a new cluster for each job. > > > I don't think we have to spin up a new cluster for each job if every job > gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a > new job when free slots are available. How can we share data between jobs > and why ? > > > > On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann <trohrm...@apache.org> > wrote: > >> The pro for the multi-tenant cluster mode is that you can share data >> between jobs and you don't have to spin up a new cluster for each job. This >> might be helpful for scenarios where you want to run many short-lived and >> light-weight jobs. >> >> But the important part is that you don't have to use this method. You can >> also start a new Flink cluster per job which will then execute the job >> isolated from any other jobs (given that you don't submit other jobs to >> this cluster). >> >> Cheers, >> Till >> >> On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang <owenzhang1...@gmail.com> >> wrote: >> >>> Thanks Fabian and Till. >>> >>> We have customers who are interested in using Flink but very concerned >>> about that "multiple jobs share the same set of TMs". I've just joined the >>> community recently so I'm not sure whether there has been a discussion over >>> the "multi-tenant cluster mode" before. >>> >>> The cons are one job/user's failure may crash another, which is >>> unacceptable in a multi-tenant scenario. >>> What are the pros ? Do the pros overweigh the cons ? >>> >>> Manu >>> >>> On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <trohrm...@apache.org> >>> wrote: >>> >>>> Hi Manu, >>>> >>>> with Flip-6 we will be able to support stricter application isolation >>>> by starting for each job a dedicated JobManager which will execute its >>>> tasks on TM reserved solely for this job. But at the same time we will >>>> continue supporting the multi-tenant cluster mode where tasks belonging to >>>> multiple jobs share the same set of TMs and, thus, might share information >>>> between them. >>>> >>>> Cheers, >>>> Till >>>> >>>> On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhue...@gmail.com> >>>> wrote: >>>> >>>> Hi Manu, >>>> >>>> As far as I know, there are not plans to change the stand-alone >>>> deployment. >>>> FLIP-6 is focusing on deployments via resource providers (YARN, Mesos, >>>> etc.) which allow to start Flink processes per job. >>>> >>>> Till (in CC) is more familiar with the FLIP-6 effort and might be able >>>> to add more detail. >>>> >>>> Best, >>>> Fabian >>>> >>>> 2016-12-01 4:16 GMT+01:00 Manu Zhang <owenzhang1...@gmail.com>: >>>> >>>> Hi all, >>>> >>>> It seems tasks of different Flink applications can end up in the same >>>> JVM (TaskManager) in standalone mode. Isn't this fragile since errors in >>>> one application could crash another ? I checked FLIP-6 >>>> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077> >>>> but >>>> didn't found any mention of changing it in the future. >>>> >>>> Any thoughts or have I missed anything ? >>>> >>>> Thanks, >>>> Manu Zhang >>>> >>>> >>>> >>>> >> >
