Good to know that. Is it the "standalone setup v2.0" section ? The wiki page has no Google-Doc-like change histories. Any jiras opened for that ? Not sure that will be noticed given FLIP-6 is almost finished.
Thanks, Manu On Tue, Dec 6, 2016 at 11:55 PM Stephan Ewen <se...@apache.org> wrote: > Hi! > > We are currently changing the resource and process model quite a bit: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 > As part of that, I think it makes sense to introduce something like that. > > What you can do today is to set TaskManagers to use one slot only, and > then start multiple TaskManagers per machine. That makes sure that JVMs are > never shared across machines. > If you use the "start-cluster.sh" script from Flink, you can enter the > same hostname multiple times in the workers file, and it will start > multiple TaskManagers on a machine. > > Best, > Stephan > > > > On Tue, Dec 6, 2016 at 3:51 AM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > > Thanks Stephan, > > They don't use YARN now but I think they will consider it. Do you think > it would be beneficial to provide such an option as "separate-jvm" in > stand-alone mode for streaming processor and long running services ? Or do > you think it would introduce too much complexity ? > > Manu > > On Tue, Dec 6, 2016 at 1:04 AM Stephan Ewen <se...@apache.org> wrote: > > Hi! > > Are your customers using YARN? In that case, the default configuration > will start a new YARN application per Flink job, no JVMs are shared between > jobs. By default, even each slot has its own JVM. > > Greetings, > Stephan > > PS: I think the "spawning new JVMs" is what Till referred to when saying > "spinning up a new cluster". Keep in mind that Flink is also a batch > processor, and it handles sequences of short batch jobs (as issued for > example by interactive shells) and it pre-allocates and manages a lot of > memory for batch jobs. > > > > On Mon, Dec 5, 2016 at 3:48 PM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job. > > > I don't think we have to spin up a new cluster for each job if every job > gets its own JVMs. For examples, Storm will launch a new worker(JVM) for a > new job when free slots are available. How can we share data between jobs > and why ? > > > > On Mon, Dec 5, 2016 at 6:27 PM, Till Rohrmann <trohrm...@apache.org> > wrote: > > The pro for the multi-tenant cluster mode is that you can share data > between jobs and you don't have to spin up a new cluster for each job. This > might be helpful for scenarios where you want to run many short-lived and > light-weight jobs. > > But the important part is that you don't have to use this method. You can > also start a new Flink cluster per job which will then execute the job > isolated from any other jobs (given that you don't submit other jobs to > this cluster). > > Cheers, > Till > > On Sat, Dec 3, 2016 at 2:50 PM, Manu Zhang <owenzhang1...@gmail.com> > wrote: > > Thanks Fabian and Till. > > We have customers who are interested in using Flink but very concerned > about that "multiple jobs share the same set of TMs". I've just joined the > community recently so I'm not sure whether there has been a discussion over > the "multi-tenant cluster mode" before. > > The cons are one job/user's failure may crash another, which is > unacceptable in a multi-tenant scenario. > What are the pros ? Do the pros overweigh the cons ? > > Manu > > On Fri, Dec 2, 2016 at 7:06 PM Till Rohrmann <trohrm...@apache.org> wrote: > > Hi Manu, > > with Flip-6 we will be able to support stricter application isolation by > starting for each job a dedicated JobManager which will execute its tasks > on TM reserved solely for this job. But at the same time we will continue > supporting the multi-tenant cluster mode where tasks belonging to multiple > jobs share the same set of TMs and, thus, might share information between > them. > > Cheers, > Till > > On Fri, Dec 2, 2016 at 11:19 AM, Fabian Hueske <fhue...@gmail.com> wrote: > > Hi Manu, > > As far as I know, there are not plans to change the stand-alone deployment. > FLIP-6 is focusing on deployments via resource providers (YARN, Mesos, > etc.) which allow to start Flink processes per job. > > Till (in CC) is more familiar with the FLIP-6 effort and might be able to > add more detail. > > Best, > Fabian > > 2016-12-01 4:16 GMT+01:00 Manu Zhang <owenzhang1...@gmail.com>: > > Hi all, > > It seems tasks of different Flink applications can end up in the same JVM > (TaskManager) in standalone mode. Isn't this fragile since errors in one > application could crash another ? I checked FLIP-6 > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077> > but > didn't found any mention of changing it in the future. > > Any thoughts or have I missed anything ? > > Thanks, > Manu Zhang > > > > > > > >
