I'm not sure whether removing the current per-job mode semantics all together is a good idea. It has some nice properties, for example the JobGraph stays constant. With your proposal which I would coin the driver mode, the JobGraph would be regenerated in case of a failover. Depending on the user code logic, this could generate a different JobGraph.
Aren't we unnecessarily widening the scope of this FLIP here? Wouldn't it be possible to introduce the Executors without changing Flink's deployment options in the first step? I don't fully understand where this need/requirement comes from. Cheers, Till On Wed, Oct 2, 2019 at 10:58 AM Zili Chen <wander4...@gmail.com> wrote: > Thanks for your thoughts Kostas! > > I agree Executor to be a concept on clients now. And sincerely second the > description > > Now the Executor simply uses a client, e.g. a ClusterClient, to submit > the job (JobGraph) that it will create from the user program. > In that sense, the Executor is one level of abstraction above the > clients, as it adds more functionality and it uses the one offered by > the client. > > In fact, let's think of the statement an Executor simply uses a client to > submit the job. > I'd like to give a description of how job submission works in per-job mode > and it will > follow a similar view now which > > (1) achieve run client on cluster side @Stephan Ewen <se...@apache.org> > (2) support multi-parts per-job program so that we don't hack to fallback > to session cluster > in this case @Till Rohrmann <trohrm...@apache.org> > > Let's start with an example we submit a user program via CLI in per-job > mode. > > 1) CLI generates configuration for getting all information about > deployment. > 2) CLI deploys a job cluster *with user jars* and specially mark the jar > contains user program > 3) JobClusterEntrypoint takes care of the bootstrap of flink cluster and > executes user program, > respects all configuration passed from client > 4) user program now runs on cluster side, it starts executing main method, > get a environment with > information of the associated job cluster. since the cluster has already > started, it can submit the > job to that cluster as in session cluster. > 5) job cluster shutdown on user program exits *and* Dispatcher doesn't > maintain any jobs. > > Since we actually runs client on cluster side we can execute multi-parts > program because we submit > to local cluster one by one. And because we change the process from > > - start a per job cluster with job graph > > to > > + start a per job cluster with user program > > we runs client on cluster side, it avoids that we "extract" job graph from > user program which limits > on multi-parts program and doesn't respect user logic outside of Flink > related code. > > Take session scenario into consideration, overall we now have > > 1. ClusterDeployer and its factory which are SPI for platform developers > so that they can deploy a > job cluster with user program or session cluster. > > 2. Environment and Executor is unified. Environment helps describe user > program logic and internally > compile the job as well as submit job with Executor. Executor always make > use of a ClusterClient > to submit the job. Specifically, in per-job mode, Environment reads > configuration refined by job cluster > so that it knows how to generate a ClusterClient. > > 3. Platform developers gets ClusterClient as return value of deploy method > of ClusterDeployer or > retrieves from an existing public known session Cluster(by > ClusterRetriever or extend ClusterDeploy to > another general concept). > > 4. JobClient can be used by user program writer or platform developer for > manage job in different condition. > > There are many other refactor we can do to respect this architecture but > let's re-emphasize the key difference > > ** job cluster doesn't start with a job graph anymore but start with a > user program and it runs the program > on the same place as the cluster runs on. So that for the program, it is > nothing different to a session cluster. > It just an existing cluster. ** > > Best, > tison. >
