Hi, If I can give my 2 cents.
One simple solution to your problem is using weave (https://www.weave.works/) a Docker network plugin. We’ve been working for more then year with dockerized (Flink+zookeeper+Yarn+spark+Kafka+hadoop+elasticsearch ) cluster using weave. Design your docker container so that you can set the cluster size on startup (number of task manager stand job managers should be a docker arg). Weave will act as a switch with dns server embedded. Your containers will only have to be configured with hosts names such as : flink.taskmanager-1.weave.local, link.taskmanager-2.weave.local, flink.jobmanager-1.weave.local, and so on … with flink Yarn it’s even simpler, but you have to dockerize a Yarn cluster. It works perfectly on bare metal machines and in the cloud (digital-ocean, aws,…). > Le 24 mars 2017 à 08:50, Chakravarthy varaga <chakravarth...@gmail.com> a > écrit : > > Hi, > > I request someone to help here. > > Best Regards > CVP > > On Thu, Mar 23, 2017 at 10:13 PM, Chakravarthy varaga > <chakravarth...@gmail.com <mailto:chakravarth...@gmail.com>> wrote: > I'm looking forward to hearing some updates on this... > > Any help here is highly appreciated !! > > On Thu, Mar 23, 2017 at 4:20 PM, Chakravarthy varaga > <chakravarth...@gmail.com <mailto:chakravarth...@gmail.com>> wrote: > Hi Team, > > We are doing a PoC to deploy Flink cluster on AWS. All runtime > components will be dockerized. > > I have few questions in relation to discover & security: > > 1. How does Job Manager discover task managers? Do they talk to over > TCP ? > > 2. If the runtime components TM, JM are containerized how are the IPs > resolved dynamically? Basically do I have to configure the JM with the > hostnames of the TMs. If so, if the TMs are on ephemeral IPs and on restart > of TM how does the job manager know the TM's (IP/Host). Before I go into DNS > and subnets, I'd like to understand how they disvoer & talk to each other ! > > 3. I went through some Flink materials on the web on security > precisely on kerebros. However how do I ensure that user level authentication > is applied on job management. For ex., only certain users are allowed to > start/stop jobs ? This question is in relation to if flink is deployed as > standalone-cluster > > Thanks & Regards > CVP > >
