Hi Ufuk, I'm planning to build Flink HA cluster and I may need to autoscale taskmanager based on the the requirement. I feel it makes more sense for me to start each taskmanager & jobmanager individually using taskmanager.sh and jobmanager.sh and let these taskmanager's discover jobmanager's using Zookeeper. The reason being that if I go with updating the slaves, then I have to push these changes in all the machines in the cluster to come to a consistent state.. I will get more flexibility if I can add taskmanager's on-the-fly without any update in rest of the machines in the cluster. I'm going to use Docker & Amazon ECS. So, if there is a change in my artifact, I will build a new cluster but as such each node of the cluster will be immutable.
On Wed, Feb 17, 2016 at 11:54 PM, Ufuk Celebi <u...@apache.org> wrote: > No, the "slaves" file is still used to ssh into the machines and start > the task manager processes in the start-cluster.sh script. So you > still need password-less ssh into the machines if you want to use > that. The task managers discover the job manager via ZooKeeper though > (therefore you don't need to configure the jobmanager address in the > config). > > In theory, you can also skip the "slaves" file if you ssh manually > into the machines and start the task managers via the taskmanger.sh > script, but I don't think that this is what you are looking for. Or > are you? > > – Ufuk > > > On Wed, Feb 17, 2016 at 10:27 PM, Deepak Jha <dkjhan...@gmail.com> wrote: > > Hi Max and Stephan, > > Does this mean that I can start Flink HA cluster without keeping any > entry > > in "slaves" file ? I'm asking this because then I should not worry about > > copying public key for password-less ssh in Flink HA cluster.... > > > > On Wed, Feb 17, 2016 at 12:38 PM, Deepak Jha <dkjhan...@gmail.com> > wrote: > > > >> Sorry for the typo Stephan > >> > >> > >> On Wednesday, February 17, 2016, Deepak Jha <dkjhan...@gmail.com> > wrote: > >> > >>> Thanks Max and Steven for the response. > >>> > >>> On Wednesday, February 17, 2016, Stephan Ewen <se...@apache.org> > wrote: > >>> > >>>> Hi Deepak! > >>>> > >>>> The "slaves" file is only used by the SSH script to start a standalone > >>>> cluster. > >>>> > >>>> As Max said, TaskManagers register dynamically at the JobManager. > >>>> > >>>> Discovery works via: > >>>> - config in non-HA mode > >>>> - ZooKeeper in HA mode > >>>> > >>>> > >>>> > >>>> On Wed, Feb 17, 2016 at 10:11 AM, Maximilian Michels <m...@apache.org> > >>>> wrote: > >>>> > >>>> > Hi Deepak, > >>>> > > >>>> > The job manager doesn't have to know about task managers. They will > >>>> > simply register at the job manager using the provided configuration. > >>>> > In HA mode, they will lookup the currently leading job manager first > >>>> > and then connect to it. The job manager can then assign work. > >>>> > > >>>> > Cheers, > >>>> > Max > >>>> > > >>>> > On Tue, Feb 16, 2016 at 10:41 PM, Deepak Jha <dkjhan...@gmail.com> > >>>> wrote: > >>>> > > Hi All, > >>>> > > I have a question on scaling-up/scaling-down flink cluster. > >>>> > > As per the documentation, in order to scale-up the cluster, I can > >>>> add a > >>>> > new > >>>> > > taskmanager on the fly and jobmanager can assign work to it. > >>>> Assuming, I > >>>> > > have Flink HA , so in the event of master JobManager failure, how > is > >>>> this > >>>> > > taskmanager detail is going to get transferred ? I believe new > master > >>>> > will > >>>> > > just read the contents from slaves config file. Can anyone give > more > >>>> > > clarity on how this is done ? Or, Is it union of slaves and the > >>>> > > taskmanager's that are added on the fly ? > >>>> > > > >>>> > > -- > >>>> > > Thanks, > >>>> > > Deepak > >>>> > > >>>> > >>> > >>> > >>> -- > >>> Sent from Gmail Mobile > >>> > >> > >> > >> -- > >> Sent from Gmail Mobile > >> > > > > > > > > -- > > Thanks, > > Deepak Jha > -- Thanks, Deepak Jha
