Hi Edward, you're right that Flink's Kubernetes documentation has not been updated with respect to Flip-6. This will be one of the tasks during the Flink 1.5 release testing and is still pending.
A Flink cluster can be run in two modes: session mode vs per-job mode. The former starts a cluster to which you can submit multiple jobs. The cluster shares the same ResourceManager and a Dispatcher which is responsible for spawning JobMasters which execute a single job each. The latter starts a Flink cluster which is pre-initialized with a JobGraph and only runs this job. Here we also start a ResourceManager and a MiniDispatcher whose job it is to simply start a single JobMaster with the pre-initialized JobGraph. StandaloneSessionClusterEntrypoint is the entrypoint for the session mode. The JobClusterEntrypoint is the entrypoint for the per-job mode. Take a look at YarnJobClusterEntrypoint to see how the entrypoint retrieves the JobGraph from HDFS and then automatically starts executing it. There is no script which directly starts this entrypoint, but the YarnClusterDescriptor uses it when `deployJobCluster` is called. Depending on what you want to achieve: Either building generic K8 images to which you can submit any number of Flink jobs or having a special image which contains the single job you want to exeucte, you either have to call into the SessionClusterEntrypoint or the JobClusterEntrypoint. When starting a session cluster, then you can use bin/flink run to submit a job to this cluster. Let me know if you have other questions. Cheers, Till On Thu, Mar 15, 2018 at 7:53 PM, Edward Rojas <edward.roja...@gmail.com> wrote: > Hello, > > Currently I have a Flink 1.4 cluster running on kubernetes based on the > configuration describe on > https://ci.apache.org/projects/flink/flink-docs- > release-1.4/ops/deployment/kubernetes.html > with additional config for HA with Zookeeper. > > With this I have several Taskmanagers, a single Jobmanager and I create a > container for each job to perform the Job submission and manage Job updates > with savepoints. > > > I'm looking into what would be needed to migrate to the new architecture on > FLIP6 as we are planning to use Flink 1.5 once it's ready. > > If I understand correctly from > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 > and the current code on master: > > * Taskmanagers would continue the same, i.e they will execute the > taskmanager.sh start-foreground script, which with the flip6 mode > activated > will execute the new taskexecutor.TaskManagerRunner. > > * We will have now one Job Manager per Job which is really good; but I > don't > fully understand how this would be started. > > I notice that the jobmanager.sh with flip6 mode activated will execute > entrypoint.StandaloneSessionClusterEntrypoint but I don't see how we could > pass the job jar and parameters (?) > > So I think the other possibility to start the job would be via the /flink > run/ command with maybe an option to tell that we are creating a job with > job manager or would be this the default behaviour ? > > Or would be this the role of the JobMaster ? I didn't take a look to its > code but it's mentioned on the flip6 page. (however I don't see an > entrypoint from the scripts (?)) > > Could you help me to understand how this is expected to be done ? > > > * Also I'm not sure to understand whether it would be better to have a > ResourceManager per job or a single ResourceManager per cluster, as in the > page is stated that there is a ResourceManager for > Self-contained-single-job, but it seems to me that it needs to have the > information about all JobManagers and TaskManagers (?) > > > Thanks in advance for the help you could provide. > > I'm interested in using Flip6 on kubernetes when it will be ready, so I > could help with some testing if needed. > > -- > Edward > > > > -- > Sent from: http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/ >
