Here is the issue, and I will keep on updating it as I find more issues. https://issues.apache.org/jira/browse/FLINK-13954
This will also cover the refactoring of the Executors that we discussed in this thread, without any additional functionality (such as the job client). Kostas On Wed, Sep 4, 2019 at 11:46 AM Kostas Kloudas <kklou...@gmail.com> wrote: > > Great idea Tison! > > I will create the umbrella issue and post it here so that we are all > on the same page! > > Cheers, > Kostas > > On Wed, Sep 4, 2019 at 11:36 AM Zili Chen <wander4...@gmail.com> wrote: > > > > Hi Kostas & Aljoscha, > > > > I notice that there is a JIRA(FLINK-13946) which could be included > > in this refactor thread. Since we agree on most of directions in > > big picture, is it reasonable that we create an umbrella issue for > > refactor client APIs and also linked subtasks? It would be a better > > way that we join forces of our community. > > > > Best, > > tison. > > > > > > Zili Chen <wander4...@gmail.com> 于2019年8月31日周六 下午12:52写道： > >> > >> Great Kostas! Looking forward to your POC! > >> > >> Best, > >> tison. > >> > >> > >> Jeff Zhang <zjf...@gmail.com> 于2019年8月30日周五 下午11:07写道： > >>> > >>> Awesome, @Kostas Looking forward your POC. > >>> > >>> Kostas Kloudas <kklou...@gmail.com> 于2019年8月30日周五 下午8:33写道： > >>> > >>> > Hi all, > >>> > > >>> > I am just writing here to let you know that I am working on a POC that > >>> > tries to refactor the current state of job submission in Flink. > >>> > I want to stress out that it introduces NO CHANGES to the current > >>> > behaviour of Flink. It just re-arranges things and introduces the > >>> > notion of an Executor, which is the entity responsible for taking the > >>> > user-code and submitting it for execution. > >>> > > >>> > Given this, the discussion about the functionality that the JobClient > >>> > will expose to the user can go on independently and the same > >>> > holds for all the open questions so far. > >>> > > >>> > I hope I will have some more new to share soon. > >>> > > >>> > Thanks, > >>> > Kostas > >>> > > >>> > On Mon, Aug 26, 2019 at 4:20 AM Yang Wang <danrtsey...@gmail.com> wrote: > >>> > > > >>> > > Hi Zili, > >>> > > > >>> > > It make sense to me that a dedicated cluster is started for a per-job > >>> > > cluster and will not accept more jobs. > >>> > > Just have a question about the command line. > >>> > > > >>> > > Currently we could use the following commands to start different > >>> > clusters. > >>> > > *per-job cluster* > >>> > > ./bin/flink run -d -p 5 -ynm perjob-cluster1 -m yarn-cluster > >>> > > examples/streaming/WindowJoin.jar > >>> > > *session cluster* > >>> > > ./bin/flink run -p 5 -ynm session-cluster1 -m yarn-cluster > >>> > > examples/streaming/WindowJoin.jar > >>> > > > >>> > > What will it look like after client enhancement? > >>> > > > >>> > > > >>> > > Best, > >>> > > Yang > >>> > > > >>> > > Zili Chen <wander4...@gmail.com> 于2019年8月23日周五 下午10:46写道： > >>> > > > >>> > > > Hi Till, > >>> > > > > >>> > > > Thanks for your update. Nice to hear :-) > >>> > > > > >>> > > > Best, > >>> > > > tison. > >>> > > > > >>> > > > > >>> > > > Till Rohrmann <trohrm...@apache.org> 于2019年8月23日周五 下午10:39写道： > >>> > > > > >>> > > > > Hi Tison, > >>> > > > > > >>> > > > > just a quick comment concerning the class loading issues when > >>> > > > > using > >>> > the > >>> > > > per > >>> > > > > job mode. The community wants to change it so that the > >>> > > > > StandaloneJobClusterEntryPoint actually uses the user code class > >>> > loader > >>> > > > > with child first class loading [1]. Hence, I hope that this > >>> > > > > problem > >>> > will > >>> > > > be > >>> > > > > resolved soon. > >>> > > > > > >>> > > > > [1] https://issues.apache.org/jira/browse/FLINK-13840 > >>> > > > > > >>> > > > > Cheers, > >>> > > > > Till > >>> > > > > > >>> > > > > On Fri, Aug 23, 2019 at 2:47 PM Kostas Kloudas > >>> > > > > <kklou...@gmail.com> > >>> > > > wrote: > >>> > > > > > >>> > > > > > Hi all, > >>> > > > > > > >>> > > > > > On the topic of web submission, I agree with Till that it only > >>> > seems > >>> > > > > > to complicate things. > >>> > > > > > It is bad for security, job isolation (anybody can submit/cancel > >>> > jobs), > >>> > > > > > and its > >>> > > > > > implementation complicates some parts of the code. So, if it > >>> > > > > > were > >>> > to > >>> > > > > > redesign the > >>> > > > > > WebUI, maybe this part could be left out. In addition, I would > >>> > > > > > say > >>> > > > > > that the ability to cancel > >>> > > > > > jobs could also be left out. > >>> > > > > > > >>> > > > > > Also I would also be in favour of removing the "detached" mode, > >>> > > > > > for > >>> > > > > > the reasons mentioned > >>> > > > > > above (i.e. because now we will have a future representing the > >>> > result > >>> > > > > > on which the user > >>> > > > > > can choose to wait or not). > >>> > > > > > > >>> > > > > > Now for the separating job submission and cluster creation, I > >>> > > > > > am in > >>> > > > > > favour of keeping both. > >>> > > > > > Once again, the reasons are mentioned above by Stephan, Till, > >>> > Aljoscha > >>> > > > > > and also Zili seems > >>> > > > > > to agree. They mainly have to do with security, isolation and > >>> > > > > > ease > >>> > of > >>> > > > > > resource management > >>> > > > > > for the user as he knows that "when my job is done, everything > >>> > will be > >>> > > > > > cleared up". This is > >>> > > > > > also the experience you get when launching a process on your > >>> > > > > > local > >>> > OS. > >>> > > > > > > >>> > > > > > On excluding the per-job mode from returning a JobClient or > >>> > > > > > not, I > >>> > > > > > believe that eventually > >>> > > > > > it would be nice to allow users to get back a jobClient. The > >>> > reason is > >>> > > > > > that 1) I cannot > >>> > > > > > find any objective reason why the user-experience should > >>> > > > > > diverge, > >>> > and > >>> > > > > > 2) this will be the > >>> > > > > > way that the user will be able to interact with his running job. > >>> > > > > > Assuming that the necessary > >>> > > > > > ports are open for the REST API to work, then I think that the > >>> > > > > > JobClient can run against the > >>> > > > > > REST API without problems. If the needed ports are not open, > >>> > > > > > then > >>> > we > >>> > > > > > are safe to not return > >>> > > > > > a JobClient, as the user explicitly chose to close all points of > >>> > > > > > communication to his running job. > >>> > > > > > > >>> > > > > > On the topic of not hijacking the "env.execute()" in order to > >>> > > > > > get > >>> > the > >>> > > > > > Plan, I definitely agree but > >>> > > > > > for the proposal of having a "compile()" method in the env, I > >>> > > > > > would > >>> > > > > > like to have a better look at > >>> > > > > > the existing code. > >>> > > > > > > >>> > > > > > Cheers, > >>> > > > > > Kostas > >>> > > > > > > >>> > > > > > On Fri, Aug 23, 2019 at 5:52 AM Zili Chen <wander4...@gmail.com> > >>> > > > wrote: > >>> > > > > > > > >>> > > > > > > Hi Yang, > >>> > > > > > > > >>> > > > > > > It would be helpful if you check Stephan's last comment, > >>> > > > > > > which states that isolation is important. > >>> > > > > > > > >>> > > > > > > For per-job mode, we run a dedicated cluster(maybe it > >>> > > > > > > should have been a couple of JM and TMs during FLIP-6 > >>> > > > > > > design) for a specific job. Thus the process is prevented > >>> > > > > > > from other jobs. > >>> > > > > > > > >>> > > > > > > In our cases there was a time we suffered from multi > >>> > > > > > > jobs submitted by different users and they affected > >>> > > > > > > each other so that all ran into an error state. Also, > >>> > > > > > > run the client inside the cluster could save client > >>> > > > > > > resource at some points. > >>> > > > > > > > >>> > > > > > > However, we also face several issues as you mentioned, > >>> > > > > > > that in per-job mode it always uses parent classloader > >>> > > > > > > thus classloading issues occur. > >>> > > > > > > > >>> > > > > > > BTW, one can makes an analogy between session/per-job mode > >>> > > > > > > in Flink, and client/cluster mode in Spark. > >>> > > > > > > > >>> > > > > > > Best, > >>> > > > > > > tison. > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > Yang Wang <danrtsey...@gmail.com> 于2019年8月22日周四 上午11:25写道： > >>> > > > > > > > >>> > > > > > > > From the user's perspective, it is really confused about the > >>> > scope > >>> > > > of > >>> > > > > > > > per-job cluster. > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > If it means a flink cluster with single job, so that we > >>> > > > > > > > could > >>> > get > >>> > > > > > better > >>> > > > > > > > isolation. > >>> > > > > > > > > >>> > > > > > > > Now it does not matter how we deploy the cluster, directly > >>> > > > > > deploy(mode1) > >>> > > > > > > > > >>> > > > > > > > or start a flink cluster and then submit job through cluster > >>> > > > > > client(mode2). > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > Otherwise, if it just means directly deploy, how should we > >>> > name the > >>> > > > > > mode2, > >>> > > > > > > > > >>> > > > > > > > session with job or something else? > >>> > > > > > > > > >>> > > > > > > > We could also benefit from the mode2. Users could get the > >>> > > > > > > > same > >>> > > > > > isolation > >>> > > > > > > > with mode1. > >>> > > > > > > > > >>> > > > > > > > The user code and dependencies will be loaded by user class > >>> > loader > >>> > > > > > > > > >>> > > > > > > > to avoid class conflict with framework. > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > Anyway, both of the two submission modes are useful. > >>> > > > > > > > > >>> > > > > > > > We just need to clarify the concepts. > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > Best, > >>> > > > > > > > > >>> > > > > > > > Yang > >>> > > > > > > > > >>> > > > > > > > Zili Chen <wander4...@gmail.com> 于2019年8月20日周二 下午5:58写道： > >>> > > > > > > > > >>> > > > > > > > > Thanks for the clarification. > >>> > > > > > > > > > >>> > > > > > > > > The idea JobDeployer ever came into my mind when I was > >>> > muddled > >>> > > > with > >>> > > > > > > > > how to execute per-job mode and session mode with the same > >>> > user > >>> > > > > code > >>> > > > > > > > > and framework codepath. > >>> > > > > > > > > > >>> > > > > > > > > With the concept JobDeployer we back to the statement that > >>> > > > > > environment > >>> > > > > > > > > knows every configs of cluster deployment and job > >>> > submission. We > >>> > > > > > > > > configure or generate from configuration a specific > >>> > JobDeployer > >>> > > > in > >>> > > > > > > > > environment and then code align on > >>> > > > > > > > > > >>> > > > > > > > > *JobClient client = env.execute().get();* > >>> > > > > > > > > > >>> > > > > > > > > which in session mode returned by clusterClient.submitJob > >>> > and in > >>> > > > > > per-job > >>> > > > > > > > > mode returned by clusterDescriptor.deployJobCluster. > >>> > > > > > > > > > >>> > > > > > > > > Here comes a problem that currently we directly run > >>> > > > > ClusterEntrypoint > >>> > > > > > > > > with extracted job graph. Follow the JobDeployer way we'd > >>> > better > >>> > > > > > > > > align entry point of per-job deployment at JobDeployer. > >>> > Users run > >>> > > > > > > > > their main method or by a Cli(finally call main method) to > >>> > deploy > >>> > > > > the > >>> > > > > > > > > job cluster. > >>> > > > > > > > > > >>> > > > > > > > > Best, > >>> > > > > > > > > tison. > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > Stephan Ewen <se...@apache.org> 于2019年8月20日周二 下午4:40写道： > >>> > > > > > > > > > >>> > > > > > > > > > Till has made some good comments here. > >>> > > > > > > > > > > >>> > > > > > > > > > Two things to add: > >>> > > > > > > > > > > >>> > > > > > > > > > - The job mode is very nice in the way that it runs > >>> > > > > > > > > > the > >>> > > > client > >>> > > > > > inside > >>> > > > > > > > > the > >>> > > > > > > > > > cluster (in the same image/process that is the JM) and > >>> > > > > > > > > > thus > >>> > > > > unifies > >>> > > > > > > > both > >>> > > > > > > > > > applications and what the Spark world calls the "driver > >>> > mode". > >>> > > > > > > > > > > >>> > > > > > > > > > - Another thing I would add is that during the FLIP-6 > >>> > design, > >>> > > > > we > >>> > > > > > were > >>> > > > > > > > > > thinking about setups where Dispatcher and JobManager > >>> > > > > > > > > > are > >>> > > > > separate > >>> > > > > > > > > > processes. > >>> > > > > > > > > > A Yarn or Mesos Dispatcher of a session could run > >>> > > > > independently > >>> > > > > > > > (even > >>> > > > > > > > > > as privileged processes executing no code). > >>> > > > > > > > > > Then you the "per-job" mode could still be helpful: > >>> > when a > >>> > > > > job > >>> > > > > > is > >>> > > > > > > > > > submitted to the dispatcher, it launches the JM again > >>> > > > > > > > > > in a > >>> > > > > per-job > >>> > > > > > > > mode, > >>> > > > > > > > > so > >>> > > > > > > > > > that JM and TM processes are bound to teh job only. For > >>> > higher > >>> > > > > > security > >>> > > > > > > > > > setups, it is important that processes are not reused > >>> > across > >>> > > > > jobs. > >>> > > > > > > > > > > >>> > > > > > > > > > On Tue, Aug 20, 2019 at 10:27 AM Till Rohrmann < > >>> > > > > > trohrm...@apache.org> > >>> > > > > > > > > > wrote: > >>> > > > > > > > > > > >>> > > > > > > > > > > I would not be in favour of getting rid of the per-job > >>> > mode > >>> > > > > > since it > >>> > > > > > > > > > > simplifies the process of running Flink jobs > >>> > considerably. > >>> > > > > > Moreover, > >>> > > > > > > > it > >>> > > > > > > > > > is > >>> > > > > > > > > > > not only well suited for container deployments but > >>> > > > > > > > > > > also > >>> > for > >>> > > > > > > > deployments > >>> > > > > > > > > > > where you want to guarantee job isolation. For > >>> > > > > > > > > > > example, a > >>> > > > user > >>> > > > > > could > >>> > > > > > > > > use > >>> > > > > > > > > > > the per-job mode on Yarn to execute his job on a > >>> > > > > > > > > > > separate > >>> > > > > > cluster. > >>> > > > > > > > > > > > >>> > > > > > > > > > > I think that having two notions of cluster deployments > >>> > > > (session > >>> > > > > > vs. > >>> > > > > > > > > > per-job > >>> > > > > > > > > > > mode) does not necessarily contradict your ideas for > >>> > > > > > > > > > > the > >>> > > > client > >>> > > > > > api > >>> > > > > > > > > > > refactoring. For example one could have the following > >>> > > > > interfaces: > >>> > > > > > > > > > > > >>> > > > > > > > > > > - ClusterDeploymentDescriptor: encapsulates the logic > >>> > how to > >>> > > > > > deploy a > >>> > > > > > > > > > > cluster. > >>> > > > > > > > > > > - ClusterClient: allows to interact with a cluster > >>> > > > > > > > > > > - JobClient: allows to interact with a running job > >>> > > > > > > > > > > > >>> > > > > > > > > > > Now the ClusterDeploymentDescriptor could have two > >>> > methods: > >>> > > > > > > > > > > > >>> > > > > > > > > > > - ClusterClient deploySessionCluster() > >>> > > > > > > > > > > - JobClusterClient/JobClient > >>> > deployPerJobCluster(JobGraph) > >>> > > > > > > > > > > > >>> > > > > > > > > > > where JobClusterClient is either a supertype of > >>> > ClusterClient > >>> > > > > > which > >>> > > > > > > > > does > >>> > > > > > > > > > > not give you the functionality to submit jobs or > >>> > > > > > deployPerJobCluster > >>> > > > > > > > > > > returns directly a JobClient. > >>> > > > > > > > > > > > >>> > > > > > > > > > > When setting up the ExecutionEnvironment, one would > >>> > > > > > > > > > > then > >>> > not > >>> > > > > > provide > >>> > > > > > > > a > >>> > > > > > > > > > > ClusterClient to submit jobs but a JobDeployer which, > >>> > > > depending > >>> > > > > > on > >>> > > > > > > > the > >>> > > > > > > > > > > selected mode, either uses a ClusterClient (session > >>> > mode) to > >>> > > > > > submit > >>> > > > > > > > > jobs > >>> > > > > > > > > > or > >>> > > > > > > > > > > a ClusterDeploymentDescriptor to deploy per a job mode > >>> > > > cluster > >>> > > > > > with > >>> > > > > > > > the > >>> > > > > > > > > > job > >>> > > > > > > > > > > to execute. > >>> > > > > > > > > > > > >>> > > > > > > > > > > These are just some thoughts how one could make it > >>> > working > >>> > > > > > because I > >>> > > > > > > > > > > believe there is some value in using the per job mode > >>> > from > >>> > > > the > >>> > > > > > > > > > > ExecutionEnvironment. > >>> > > > > > > > > > > > >>> > > > > > > > > > > Concerning the web submission, this is indeed a bit > >>> > tricky. > >>> > > > > From > >>> > > > > > a > >>> > > > > > > > > > cluster > >>> > > > > > > > > > > management stand point, I would in favour of not > >>> > executing > >>> > > > user > >>> > > > > > code > >>> > > > > > > > on > >>> > > > > > > > > > the > >>> > > > > > > > > > > REST endpoint. Especially when considering security, > >>> > > > > > > > > > > it > >>> > would > >>> > > > > be > >>> > > > > > good > >>> > > > > > > > > to > >>> > > > > > > > > > > have a well defined cluster behaviour where it is > >>> > explicitly > >>> > > > > > stated > >>> > > > > > > > > where > >>> > > > > > > > > > > user code and, thus, potentially risky code is > >>> > > > > > > > > > > executed. > >>> > > > > Ideally > >>> > > > > > we > >>> > > > > > > > > limit > >>> > > > > > > > > > > it to the TaskExecutor and JobMaster. > >>> > > > > > > > > > > > >>> > > > > > > > > > > Cheers, > >>> > > > > > > > > > > Till > >>> > > > > > > > > > > > >>> > > > > > > > > > > On Tue, Aug 20, 2019 at 9:40 AM Flavio Pompermaier < > >>> > > > > > > > > pomperma...@okkam.it > >>> > > > > > > > > > > > >>> > > > > > > > > > > wrote: > >>> > > > > > > > > > > > >>> > > > > > > > > > > > In my opinion the client should not use any > >>> > environment to > >>> > > > > get > >>> > > > > > the > >>> > > > > > > > > Job > >>> > > > > > > > > > > > graph because the jar should reside ONLY on the > >>> > > > > > > > > > > > cluster > >>> > > > (and > >>> > > > > > not in > >>> > > > > > > > > the > >>> > > > > > > > > > > > client classpath otherwise there are always > >>> > inconsistencies > >>> > > > > > between > >>> > > > > > > > > > > client > >>> > > > > > > > > > > > and Flink Job manager's classpath). > >>> > > > > > > > > > > > In the YARN, Mesos and Kubernetes scenarios you have > >>> > the > >>> > > > jar > >>> > > > > > but > >>> > > > > > > > you > >>> > > > > > > > > > > could > >>> > > > > > > > > > > > start a cluster that has the jar on the Job Manager > >>> > > > > > > > > > > > as > >>> > well > >>> > > > > > (but > >>> > > > > > > > this > >>> > > > > > > > > > is > >>> > > > > > > > > > > > the only case where I think you can assume that the > >>> > client > >>> > > > > has > >>> > > > > > the > >>> > > > > > > > > jar > >>> > > > > > > > > > on > >>> > > > > > > > > > > > the classpath..in the REST job submission you don't > >>> > have > >>> > > > any > >>> > > > > > > > > > classpath). > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > Thus, always in my opinion, the JobGraph should be > >>> > > > generated > >>> > > > > > by the > >>> > > > > > > > > Job > >>> > > > > > > > > > > > Manager REST API. > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > On Tue, Aug 20, 2019 at 9:00 AM Zili Chen < > >>> > > > > > wander4...@gmail.com> > >>> > > > > > > > > > wrote: > >>> > > > > > > > > > > > > >>> > > > > > > > > > > >> I would like to involve Till & Stephan here to > >>> > > > > > > > > > > >> clarify > >>> > > > some > >>> > > > > > > > concept > >>> > > > > > > > > of > >>> > > > > > > > > > > >> per-job mode. > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> The term per-job is one of modes a cluster could > >>> > > > > > > > > > > >> run > >>> > on. > >>> > > > It > >>> > > > > is > >>> > > > > > > > > mainly > >>> > > > > > > > > > > >> aimed > >>> > > > > > > > > > > >> at spawn > >>> > > > > > > > > > > >> a dedicated cluster for a specific job while the > >>> > > > > > > > > > > >> job > >>> > could > >>> > > > > be > >>> > > > > > > > > packaged > >>> > > > > > > > > > > >> with > >>> > > > > > > > > > > >> Flink > >>> > > > > > > > > > > >> itself and thus the cluster initialized with job so > >>> > that > >>> > > > get > >>> > > > > > rid > >>> > > > > > > > of > >>> > > > > > > > > a > >>> > > > > > > > > > > >> separated > >>> > > > > > > > > > > >> submission step. > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> This is useful for container deployments where one > >>> > create > >>> > > > > his > >>> > > > > > > > image > >>> > > > > > > > > > with > >>> > > > > > > > > > > >> the job > >>> > > > > > > > > > > >> and then simply deploy the container. > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> However, it is out of client scope since a > >>> > > > > > client(ClusterClient > >>> > > > > > > > for > >>> > > > > > > > > > > >> example) is for > >>> > > > > > > > > > > >> communicate with an existing cluster and > >>> > > > > > > > > > > >> performance > >>> > > > > actions. > >>> > > > > > > > > > Currently, > >>> > > > > > > > > > > >> in > >>> > > > > > > > > > > >> per-job > >>> > > > > > > > > > > >> mode, we extract the job graph and bundle it into > >>> > cluster > >>> > > > > > > > deployment > >>> > > > > > > > > > and > >>> > > > > > > > > > > >> thus no > >>> > > > > > > > > > > >> concept of client get involved. It looks like > >>> > reasonable > >>> > > > to > >>> > > > > > > > exclude > >>> > > > > > > > > > the > >>> > > > > > > > > > > >> deployment > >>> > > > > > > > > > > >> of per-job cluster from client api and use > >>> > > > > > > > > > > >> dedicated > >>> > > > utility > >>> > > > > > > > > > > >> classes(deployers) for > >>> > > > > > > > > > > >> deployment. > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> Zili Chen <wander4...@gmail.com> 于2019年8月20日周二 > >>> > 下午12:37写道： > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > Hi Aljoscha, > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > Thanks for your reply and participance. The > >>> > > > > > > > > > > >> > Google > >>> > Doc > >>> > > > you > >>> > > > > > > > linked > >>> > > > > > > > > to > >>> > > > > > > > > > > >> > requires > >>> > > > > > > > > > > >> > permission and I think you could use a share link > >>> > > > instead. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > I agree with that we almost reach a consensus > >>> > > > > > > > > > > >> > that > >>> > > > > > JobClient is > >>> > > > > > > > > > > >> necessary > >>> > > > > > > > > > > >> > to > >>> > > > > > > > > > > >> > interacte with a running Job. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > Let me check your open questions one by one. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > 1. Separate cluster creation and job submission > >>> > > > > > > > > > > >> > for > >>> > > > > per-job > >>> > > > > > > > mode. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > As you mentioned here is where the opinions > >>> > diverge. In > >>> > > > my > >>> > > > > > > > > document > >>> > > > > > > > > > > >> there > >>> > > > > > > > > > > >> > is > >>> > > > > > > > > > > >> > an alternative[2] that proposes excluding per-job > >>> > > > > deployment > >>> > > > > > > > from > >>> > > > > > > > > > > client > >>> > > > > > > > > > > >> > api > >>> > > > > > > > > > > >> > scope and now I find it is more reasonable we do > >>> > > > > > > > > > > >> > the > >>> > > > > > exclusion. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > When in per-job mode, a dedicated JobCluster is > >>> > launched > >>> > > > > to > >>> > > > > > > > > execute > >>> > > > > > > > > > > the > >>> > > > > > > > > > > >> > specific job. It is like a Flink Application more > >>> > than a > >>> > > > > > > > > submission > >>> > > > > > > > > > > >> > of Flink Job. Client only takes care of job > >>> > submission > >>> > > > and > >>> > > > > > > > assume > >>> > > > > > > > > > > there > >>> > > > > > > > > > > >> is > >>> > > > > > > > > > > >> > an existing cluster. In this way we are able to > >>> > consider > >>> > > > > > per-job > >>> > > > > > > > > > > issues > >>> > > > > > > > > > > >> > individually and JobClusterEntrypoint would be > >>> > > > > > > > > > > >> > the > >>> > > > utility > >>> > > > > > class > >>> > > > > > > > > for > >>> > > > > > > > > > > >> > per-job > >>> > > > > > > > > > > >> > deployment. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > Nevertheless, user program works in both session > >>> > mode > >>> > > > and > >>> > > > > > > > per-job > >>> > > > > > > > > > mode > >>> > > > > > > > > > > >> > without > >>> > > > > > > > > > > >> > necessary to change code. JobClient in per-job > >>> > > > > > > > > > > >> > mode > >>> > is > >>> > > > > > returned > >>> > > > > > > > > from > >>> > > > > > > > > > > >> > env.execute as normal. However, it would be no > >>> > longer a > >>> > > > > > wrapper > >>> > > > > > > > of > >>> > > > > > > > > > > >> > RestClusterClient but a wrapper of > >>> > PerJobClusterClient > >>> > > > > which > >>> > > > > > > > > > > >> communicates > >>> > > > > > > > > > > >> > to Dispatcher locally. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > 2. How to deal with plan preview. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > With env.compile functions users can get > >>> > > > > > > > > > > >> > JobGraph or > >>> > > > > > FlinkPlan > >>> > > > > > > > and > >>> > > > > > > > > > > thus > >>> > > > > > > > > > > >> > they can preview the plan with programming. > >>> > Typically it > >>> > > > > > looks > >>> > > > > > > > > like > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > if (preview configured) { > >>> > > > > > > > > > > >> > FlinkPlan plan = env.compile(); > >>> > > > > > > > > > > >> > new JSONDumpGenerator(...).dump(plan); > >>> > > > > > > > > > > >> > } else { > >>> > > > > > > > > > > >> > env.execute(); > >>> > > > > > > > > > > >> > } > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > And `flink info` would be invalid any more. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > 3. How to deal with Jar Submission at the Web > >>> > Frontend. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > There is one more thread talked on this topic[1]. > >>> > Apart > >>> > > > > from > >>> > > > > > > > > > removing > >>> > > > > > > > > > > >> > the functions there are two alternatives. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > One is to introduce an interface has a method > >>> > returns > >>> > > > > > > > > > > JobGraph/FilnkPlan > >>> > > > > > > > > > > >> > and Jar Submission only support main-class > >>> > implements > >>> > > > this > >>> > > > > > > > > > interface. > >>> > > > > > > > > > > >> > And then extract the JobGraph/FlinkPlan just by > >>> > calling > >>> > > > > the > >>> > > > > > > > > method. > >>> > > > > > > > > > > >> > In this way, it is even possible to consider a > >>> > > > separation > >>> > > > > > of job > >>> > > > > > > > > > > >> creation > >>> > > > > > > > > > > >> > and job submission. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > The other is, as you mentioned, let execute() do > >>> > > > > > > > > > > >> > the > >>> > > > > actual > >>> > > > > > > > > > execution. > >>> > > > > > > > > > > >> > We won't execute the main method in the > >>> > > > > > > > > > > >> > WebFrontend > >>> > but > >>> > > > > > spawn a > >>> > > > > > > > > > > process > >>> > > > > > > > > > > >> > at WebMonitor side to execute. For return part we > >>> > could > >>> > > > > > generate > >>> > > > > > > > > the > >>> > > > > > > > > > > >> > JobID from WebMonitor and pass it to the > >>> > > > > > > > > > > >> > execution > >>> > > > > > environemnt. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > 4. How to deal with detached mode. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > I think detached mode is a temporary solution for > >>> > > > > > non-blocking > >>> > > > > > > > > > > >> submission. > >>> > > > > > > > > > > >> > In my document both submission and execution > >>> > > > > > > > > > > >> > return > >>> > a > >>> > > > > > > > > > > CompletableFuture > >>> > > > > > > > > > > >> and > >>> > > > > > > > > > > >> > users control whether or not wait for the > >>> > > > > > > > > > > >> > result. In > >>> > > > this > >>> > > > > > point > >>> > > > > > > > we > >>> > > > > > > > > > > don't > >>> > > > > > > > > > > >> > need a detached option but the functionality is > >>> > covered. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > 5. How does per-job mode interact with > >>> > > > > > > > > > > >> > interactive > >>> > > > > > programming. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > All of YARN, Mesos and Kubernetes scenarios > >>> > > > > > > > > > > >> > follow > >>> > the > >>> > > > > > pattern > >>> > > > > > > > > > launch > >>> > > > > > > > > > > a > >>> > > > > > > > > > > >> > JobCluster now. And I don't think there would be > >>> > > > > > inconsistency > >>> > > > > > > > > > between > >>> > > > > > > > > > > >> > different resource management. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > Best, > >>> > > > > > > > > > > >> > tison. > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > [1] > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://lists.apache.org/x/thread.html/6db869c53816f4e2917949a7c6992c2b90856d7d639d7f2e1cd13768@%3Cdev.flink.apache.org%3E > >>> > > > > > > > > > > >> > [2] > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://docs.google.com/document/d/1UWJE7eYWiMuZewBKS0YmdVO2LUTqXPd6-pbOCof9ddY/edit?disco=AAAADZaGGfs > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> > Aljoscha Krettek <aljos...@apache.org> > >>> > 于2019年8月16日周五 > >>> > > > > > 下午9:20写道： > >>> > > > > > > > > > > >> > > >>> > > > > > > > > > > >> >> Hi, > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> I read both Jeffs initial design document and > >>> > > > > > > > > > > >> >> the > >>> > newer > >>> > > > > > > > document > >>> > > > > > > > > by > >>> > > > > > > > > > > >> >> Tison. I also finally found the time to collect > >>> > > > > > > > > > > >> >> our > >>> > > > > > thoughts on > >>> > > > > > > > > the > >>> > > > > > > > > > > >> issue, > >>> > > > > > > > > > > >> >> I had quite some discussions with Kostas and > >>> > > > > > > > > > > >> >> this > >>> > is > >>> > > > the > >>> > > > > > > > result: > >>> > > > > > > > > > [1]. > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> I think overall we agree that this part of the > >>> > code is > >>> > > > in > >>> > > > > > dire > >>> > > > > > > > > need > >>> > > > > > > > > > > of > >>> > > > > > > > > > > >> >> some refactoring/improvements but I think there > >>> > > > > > > > > > > >> >> are > >>> > > > still > >>> > > > > > some > >>> > > > > > > > > open > >>> > > > > > > > > > > >> >> questions and some differences in opinion what > >>> > those > >>> > > > > > > > refactorings > >>> > > > > > > > > > > >> should > >>> > > > > > > > > > > >> >> look like. > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> I think the API-side is quite clear, i.e. we > >>> > > > > > > > > > > >> >> need > >>> > some > >>> > > > > > > > JobClient > >>> > > > > > > > > > API > >>> > > > > > > > > > > >> that > >>> > > > > > > > > > > >> >> allows interacting with a running Job. It could > >>> > > > > > > > > > > >> >> be > >>> > > > > > worthwhile > >>> > > > > > > > to > >>> > > > > > > > > > spin > >>> > > > > > > > > > > >> that > >>> > > > > > > > > > > >> >> off into a separate FLIP because we can probably > >>> > find > >>> > > > > > consensus > >>> > > > > > > > > on > >>> > > > > > > > > > > that > >>> > > > > > > > > > > >> >> part more easily. > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> For the rest, the main open questions from our > >>> > > > > > > > > > > >> >> doc > >>> > are > >>> > > > > > these: > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> - Do we want to separate cluster creation and > >>> > > > > > > > > > > >> >> job > >>> > > > > > submission > >>> > > > > > > > > for > >>> > > > > > > > > > > >> >> per-job mode? In the past, there were conscious > >>> > efforts > >>> > > > > to > >>> > > > > > > > *not* > >>> > > > > > > > > > > >> separate > >>> > > > > > > > > > > >> >> job submission from cluster creation for per-job > >>> > > > clusters > >>> > > > > > for > >>> > > > > > > > > > Mesos, > >>> > > > > > > > > > > >> YARN, > >>> > > > > > > > > > > >> >> Kubernets (see StandaloneJobClusterEntryPoint). > >>> > Tison > >>> > > > > > suggests > >>> > > > > > > > in > >>> > > > > > > > > > his > >>> > > > > > > > > > > >> >> design document to decouple this in order to > >>> > > > > > > > > > > >> >> unify > >>> > job > >>> > > > > > > > > submission. > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> - How to deal with plan preview, which needs > >>> > > > > > > > > > > >> >> to > >>> > > > hijack > >>> > > > > > > > > execute() > >>> > > > > > > > > > > and > >>> > > > > > > > > > > >> >> let the outside code catch an exception? > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> - How to deal with Jar Submission at the Web > >>> > > > Frontend, > >>> > > > > > which > >>> > > > > > > > > > needs > >>> > > > > > > > > > > to > >>> > > > > > > > > > > >> >> hijack execute() and let the outside code catch > >>> > > > > > > > > > > >> >> an > >>> > > > > > exception? > >>> > > > > > > > > > > >> >> CliFrontend.run() “hijacks” > >>> > > > > ExecutionEnvironment.execute() > >>> > > > > > to > >>> > > > > > > > > get a > >>> > > > > > > > > > > >> >> JobGraph and then execute that JobGraph > >>> > > > > > > > > > > >> >> manually. > >>> > We > >>> > > > > could > >>> > > > > > get > >>> > > > > > > > > > around > >>> > > > > > > > > > > >> that > >>> > > > > > > > > > > >> >> by letting execute() do the actual execution. > >>> > > > > > > > > > > >> >> One > >>> > > > caveat > >>> > > > > > for > >>> > > > > > > > this > >>> > > > > > > > > > is > >>> > > > > > > > > > > >> that > >>> > > > > > > > > > > >> >> now the main() method doesn’t return (or is > >>> > > > > > > > > > > >> >> forced > >>> > to > >>> > > > > > return by > >>> > > > > > > > > > > >> throwing an > >>> > > > > > > > > > > >> >> exception from execute()) which means that for > >>> > > > > > > > > > > >> >> Jar > >>> > > > > > Submission > >>> > > > > > > > > from > >>> > > > > > > > > > > the > >>> > > > > > > > > > > >> >> WebFrontend we have a long-running main() method > >>> > > > running > >>> > > > > > in the > >>> > > > > > > > > > > >> >> WebFrontend. This doesn’t sound very good. We > >>> > could get > >>> > > > > > around > >>> > > > > > > > > this > >>> > > > > > > > > > > by > >>> > > > > > > > > > > >> >> removing the plan preview feature and by > >>> > > > > > > > > > > >> >> removing > >>> > Jar > >>> > > > > > > > > > > >> Submission/Running. > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> - How to deal with detached mode? Right now, > >>> > > > > > > > > DetachedEnvironment > >>> > > > > > > > > > > will > >>> > > > > > > > > > > >> >> execute the job and return immediately. If users > >>> > > > control > >>> > > > > > when > >>> > > > > > > > > they > >>> > > > > > > > > > > >> want to > >>> > > > > > > > > > > >> >> return, by waiting on the job completion future, > >>> > how do > >>> > > > > we > >>> > > > > > deal > >>> > > > > > > > > > with > >>> > > > > > > > > > > >> this? > >>> > > > > > > > > > > >> >> Do we simply remove the distinction between > >>> > > > > > > > > detached/non-detached? > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> - How does per-job mode interact with > >>> > “interactive > >>> > > > > > > > programming” > >>> > > > > > > > > > > >> >> (FLIP-36). For YARN, each execute() call could > >>> > spawn a > >>> > > > > new > >>> > > > > > > > Flink > >>> > > > > > > > > > YARN > >>> > > > > > > > > > > >> >> cluster. What about Mesos and Kubernetes? > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> The first open question is where the opinions > >>> > diverge, > >>> > > > I > >>> > > > > > think. > >>> > > > > > > > > The > >>> > > > > > > > > > > >> rest > >>> > > > > > > > > > > >> >> are just open questions and interesting things > >>> > that we > >>> > > > > > need to > >>> > > > > > > > > > > >> consider. > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> Best, > >>> > > > > > > > > > > >> >> Aljoscha > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> [1] > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://docs.google.com/document/d/1E-8UjOLz4QPUTxetGWbU23OlsIH9VIdodpTsxwoQTs0/edit#heading=h.na7k0ad88tix > >>> > > > > > > > > > > >> >> < > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://docs.google.com/document/d/1E-8UjOLz4QPUTxetGWbU23OlsIH9VIdodpTsxwoQTs0/edit#heading=h.na7k0ad88tix > >>> > > > > > > > > > > >> >> > > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> > On 31. Jul 2019, at 15:23, Jeff Zhang < > >>> > > > > zjf...@gmail.com> > >>> > > > > > > > > wrote: > >>> > > > > > > > > > > >> >> > > >>> > > > > > > > > > > >> >> > Thanks tison for the effort. I left a few > >>> > comments. > >>> > > > > > > > > > > >> >> > > >>> > > > > > > > > > > >> >> > > >>> > > > > > > > > > > >> >> > Zili Chen <wander4...@gmail.com> 于2019年7月31日周三 > >>> > > > > 下午8:24写道： > >>> > > > > > > > > > > >> >> > > >>> > > > > > > > > > > >> >> >> Hi Flavio, > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> >> Thanks for your reply. > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> >> Either current impl and in the design, > >>> > ClusterClient > >>> > > > > > > > > > > >> >> >> never takes responsibility for generating > >>> > JobGraph. > >>> > > > > > > > > > > >> >> >> (what you see in current codebase is several > >>> > class > >>> > > > > > methods) > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> >> Instead, user describes his program in the > >>> > > > > > > > > > > >> >> >> main > >>> > > > method > >>> > > > > > > > > > > >> >> >> with ExecutionEnvironment apis and calls > >>> > > > env.compile() > >>> > > > > > > > > > > >> >> >> or env.optimize() to get FlinkPlan and > >>> > > > > > > > > > > >> >> >> JobGraph > >>> > > > > > > > respectively. > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> >> For listing main classes in a jar and choose > >>> > one for > >>> > > > > > > > > > > >> >> >> submission, you're now able to customize a > >>> > > > > > > > > > > >> >> >> CLI > >>> > to do > >>> > > > > it. > >>> > > > > > > > > > > >> >> >> Specifically, the path of jar is passed as > >>> > arguments > >>> > > > > and > >>> > > > > > > > > > > >> >> >> in the customized CLI you list main classes, > >>> > choose > >>> > > > > one > >>> > > > > > > > > > > >> >> >> to submit to the cluster. > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> >> Best, > >>> > > > > > > > > > > >> >> >> tison. > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> >> Flavio Pompermaier <pomperma...@okkam.it> > >>> > > > > 于2019年7月31日周三 > >>> > > > > > > > > > 下午8:12写道： > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> >>> Just one note on my side: it is not clear > >>> > > > > > > > > > > >> >> >>> to me > >>> > > > > > whether the > >>> > > > > > > > > > > client > >>> > > > > > > > > > > >> >> needs > >>> > > > > > > > > > > >> >> >> to > >>> > > > > > > > > > > >> >> >>> be able to generate a job graph or not. > >>> > > > > > > > > > > >> >> >>> In my opinion, the job jar must resides only > >>> > on the > >>> > > > > > > > > > > >> server/jobManager > >>> > > > > > > > > > > >> >> >> side > >>> > > > > > > > > > > >> >> >>> and the client requires a way to get the job > >>> > graph. > >>> > > > > > > > > > > >> >> >>> If you really want to access to the job > >>> > > > > > > > > > > >> >> >>> graph, > >>> > I'd > >>> > > > > add > >>> > > > > > a > >>> > > > > > > > > > > dedicated > >>> > > > > > > > > > > >> >> method > >>> > > > > > > > > > > >> >> >>> on the ClusterClient. like: > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >>> - getJobGraph(jarId, mainClass): JobGraph > >>> > > > > > > > > > > >> >> >>> - listMainClasses(jarId): List<String> > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >>> These would require some addition also on > >>> > > > > > > > > > > >> >> >>> the > >>> > job > >>> > > > > > manager > >>> > > > > > > > > > > endpoint > >>> > > > > > > > > > > >> as > >>> > > > > > > > > > > >> >> >>> well..what do you think? > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >>> On Wed, Jul 31, 2019 at 12:42 PM Zili Chen < > >>> > > > > > > > > > wander4...@gmail.com > >>> > > > > > > > > > > > > >>> > > > > > > > > > > >> >> wrote: > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >>>> Hi all, > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> Here is a document[1] on client api > >>> > enhancement > >>> > > > from > >>> > > > > > our > >>> > > > > > > > > > > >> perspective. > >>> > > > > > > > > > > >> >> >>>> We have investigated current > >>> > > > > > > > > > > >> >> >>>> implementations. > >>> > And > >>> > > > we > >>> > > > > > > > propose > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> 1. Unify the implementation of cluster > >>> > deployment > >>> > > > > and > >>> > > > > > job > >>> > > > > > > > > > > >> submission > >>> > > > > > > > > > > >> >> in > >>> > > > > > > > > > > >> >> >>>> Flink. > >>> > > > > > > > > > > >> >> >>>> 2. Provide programmatic interfaces to allow > >>> > > > flexible > >>> > > > > > job > >>> > > > > > > > and > >>> > > > > > > > > > > >> cluster > >>> > > > > > > > > > > >> >> >>>> management. > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> The first proposal is aimed at reducing > >>> > > > > > > > > > > >> >> >>>> code > >>> > paths > >>> > > > > of > >>> > > > > > > > > cluster > >>> > > > > > > > > > > >> >> >> deployment > >>> > > > > > > > > > > >> >> >>>> and > >>> > > > > > > > > > > >> >> >>>> job submission so that one can adopt Flink > >>> > > > > > > > > > > >> >> >>>> in > >>> > his > >>> > > > > > usage > >>> > > > > > > > > > easily. > >>> > > > > > > > > > > >> The > >>> > > > > > > > > > > >> >> >>> second > >>> > > > > > > > > > > >> >> >>>> proposal is aimed at providing rich > >>> > interfaces for > >>> > > > > > > > advanced > >>> > > > > > > > > > > users > >>> > > > > > > > > > > >> >> >>>> who want to make accurate control of these > >>> > stages. > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> Quick reference on open questions: > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> 1. Exclude job cluster deployment from > >>> > > > > > > > > > > >> >> >>>> client > >>> > side > >>> > > > > or > >>> > > > > > > > > redefine > >>> > > > > > > > > > > the > >>> > > > > > > > > > > >> >> >>> semantic > >>> > > > > > > > > > > >> >> >>>> of job cluster? Since it fits in a process > >>> > quite > >>> > > > > > different > >>> > > > > > > > > > from > >>> > > > > > > > > > > >> >> session > >>> > > > > > > > > > > >> >> >>>> cluster deployment and job submission. > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> 2. Maintain the codepaths handling class > >>> > > > > > > > > > > o.a.f.api.common.Program > >>> > > > > > > > > > > >> or > >>> > > > > > > > > > > >> >> >>>> implement customized program handling > >>> > > > > > > > > > > >> >> >>>> logic by > >>> > > > > > customized > >>> > > > > > > > > > > >> >> CliFrontend? > >>> > > > > > > > > > > >> >> >>>> See also this thread[2] and the > >>> > > > > > > > > > > >> >> >>>> document[1]. > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> 3. Expose ClusterClient as public api or > >>> > > > > > > > > > > >> >> >>>> just > >>> > > > expose > >>> > > > > > api > >>> > > > > > > > in > >>> > > > > > > > > > > >> >> >>>> ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>> and delegate them to ClusterClient? > >>> > > > > > > > > > > >> >> >>>> Further, > >>> > in > >>> > > > > > either way > >>> > > > > > > > > is > >>> > > > > > > > > > it > >>> > > > > > > > > > > >> >> worth > >>> > > > > > > > > > > >> >> >> to > >>> > > > > > > > > > > >> >> >>>> introduce a JobClient which is an > >>> > encapsulation of > >>> > > > > > > > > > ClusterClient > >>> > > > > > > > > > > >> that > >>> > > > > > > > > > > >> >> >>>> associated to specific job? > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> Best, > >>> > > > > > > > > > > >> >> >>>> tison. > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> [1] > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://docs.google.com/document/d/1UWJE7eYWiMuZewBKS0YmdVO2LUTqXPd6-pbOCof9ddY/edit?usp=sharing > >>> > > > > > > > > > > >> >> >>>> [2] > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://lists.apache.org/thread.html/7ffc9936a384b891dbcf0a481d26c6d13b2125607c200577780d1e18@%3Cdev.flink.apache.org%3E > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>> Jeff Zhang <zjf...@gmail.com> 于2019年7月24日周三 > >>> > > > > 上午9:19写道： > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>>>> Thanks Stephan, I will follow up this > >>> > > > > > > > > > > >> >> >>>>> issue > >>> > in > >>> > > > next > >>> > > > > > few > >>> > > > > > > > > > weeks, > >>> > > > > > > > > > > >> and > >>> > > > > > > > > > > >> >> >> will > >>> > > > > > > > > > > >> >> >>>>> refine the design doc. We could discuss > >>> > > > > > > > > > > >> >> >>>>> more > >>> > > > > details > >>> > > > > > > > after > >>> > > > > > > > > > 1.9 > >>> > > > > > > > > > > >> >> >> release. > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>>> Stephan Ewen <se...@apache.org> > >>> > 于2019年7月24日周三 > >>> > > > > > 上午12:58写道： > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>>>> Hi all! > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>>> This thread has stalled for a bit, which > >>> > > > > > > > > > > >> >> >>>>>> I > >>> > > > assume > >>> > > > > > ist > >>> > > > > > > > > mostly > >>> > > > > > > > > > > >> due to > >>> > > > > > > > > > > >> >> >>> the > >>> > > > > > > > > > > >> >> >>>>>> Flink 1.9 feature freeze and release > >>> > > > > > > > > > > >> >> >>>>>> testing > >>> > > > > effort. > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>>> I personally still recognize this issue > >>> > > > > > > > > > > >> >> >>>>>> as > >>> > one > >>> > > > > > important > >>> > > > > > > > > to > >>> > > > > > > > > > be > >>> > > > > > > > > > > >> >> >>> solved. > >>> > > > > > > > > > > >> >> >>>>> I'd > >>> > > > > > > > > > > >> >> >>>>>> be happy to help resume this discussion > >>> > > > > > > > > > > >> >> >>>>>> soon > >>> > > > > (after > >>> > > > > > the > >>> > > > > > > > > 1.9 > >>> > > > > > > > > > > >> >> >> release) > >>> > > > > > > > > > > >> >> >>>> and > >>> > > > > > > > > > > >> >> >>>>>> see if we can do some step towards this > >>> > > > > > > > > > > >> >> >>>>>> in > >>> > Flink > >>> > > > > > 1.10. > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>>> Best, > >>> > > > > > > > > > > >> >> >>>>>> Stephan > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>>> On Mon, Jun 24, 2019 at 10:41 AM Flavio > >>> > > > > Pompermaier > >>> > > > > > < > >>> > > > > > > > > > > >> >> >>>>> pomperma...@okkam.it> > >>> > > > > > > > > > > >> >> >>>>>> wrote: > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> That's exactly what I suggested a long > >>> > > > > > > > > > > >> >> >>>>>>> time > >>> > > > ago: > >>> > > > > > the > >>> > > > > > > > > Flink > >>> > > > > > > > > > > REST > >>> > > > > > > > > > > >> >> >>>> client > >>> > > > > > > > > > > >> >> >>>>>>> should not require any Flink dependency, > >>> > only > >>> > > > > http > >>> > > > > > > > > library > >>> > > > > > > > > > to > >>> > > > > > > > > > > >> >> >> call > >>> > > > > > > > > > > >> >> >>>> the > >>> > > > > > > > > > > >> >> >>>>>> REST > >>> > > > > > > > > > > >> >> >>>>>>> services to submit and monitor a job. > >>> > > > > > > > > > > >> >> >>>>>>> What I suggested also in [1] was to > >>> > > > > > > > > > > >> >> >>>>>>> have a > >>> > way > >>> > > > to > >>> > > > > > > > > > > automatically > >>> > > > > > > > > > > >> >> >>>> suggest > >>> > > > > > > > > > > >> >> >>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>> user (via a UI) the available main > >>> > > > > > > > > > > >> >> >>>>>>> classes > >>> > and > >>> > > > > > their > >>> > > > > > > > > > required > >>> > > > > > > > > > > >> >> >>>>>>> parameters[2]. > >>> > > > > > > > > > > >> >> >>>>>>> Another problem we have with Flink is > >>> > > > > > > > > > > >> >> >>>>>>> that > >>> > the > >>> > > > > Rest > >>> > > > > > > > > client > >>> > > > > > > > > > > and > >>> > > > > > > > > > > >> >> >> the > >>> > > > > > > > > > > >> >> >>>> CLI > >>> > > > > > > > > > > >> >> >>>>>> one > >>> > > > > > > > > > > >> >> >>>>>>> behaves differently and we use the CLI > >>> > client > >>> > > > > (via > >>> > > > > > ssh) > >>> > > > > > > > > > > because > >>> > > > > > > > > > > >> >> >> it > >>> > > > > > > > > > > >> >> >>>>> allows > >>> > > > > > > > > > > >> >> >>>>>>> to call some other method after > >>> > env.execute() > >>> > > > [3] > >>> > > > > > (we > >>> > > > > > > > > have > >>> > > > > > > > > > to > >>> > > > > > > > > > > >> >> >> call > >>> > > > > > > > > > > >> >> >>>>>> another > >>> > > > > > > > > > > >> >> >>>>>>> REST service to signal the end of the > >>> > > > > > > > > > > >> >> >>>>>>> job). > >>> > > > > > > > > > > >> >> >>>>>>> Int his regard, a dedicated interface, > >>> > like the > >>> > > > > > > > > JobListener > >>> > > > > > > > > > > >> >> >>> suggested > >>> > > > > > > > > > > >> >> >>>>> in > >>> > > > > > > > > > > >> >> >>>>>>> the previous emails, would be very > >>> > > > > > > > > > > >> >> >>>>>>> helpful > >>> > > > > (IMHO). > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> [1] > >>> > > > > > https://issues.apache.org/jira/browse/FLINK-10864 > >>> > > > > > > > > > > >> >> >>>>>>> [2] > >>> > > > > > https://issues.apache.org/jira/browse/FLINK-10862 > >>> > > > > > > > > > > >> >> >>>>>>> [3] > >>> > > > > > https://issues.apache.org/jira/browse/FLINK-10879 > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> Best, > >>> > > > > > > > > > > >> >> >>>>>>> Flavio > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> On Mon, Jun 24, 2019 at 9:54 AM Jeff > >>> > > > > > > > > > > >> >> >>>>>>> Zhang > >>> > < > >>> > > > > > > > > > zjf...@gmail.com > >>> > > > > > > > > > > > > >>> > > > > > > > > > > >> >> >>> wrote: > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> Hi, Tison, > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> Thanks for your comments. Overall I > >>> > > > > > > > > > > >> >> >>>>>>>> agree > >>> > with > >>> > > > > you > >>> > > > > > > > that > >>> > > > > > > > > it > >>> > > > > > > > > > > is > >>> > > > > > > > > > > >> >> >>>>> difficult > >>> > > > > > > > > > > >> >> >>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>> down stream project to integrate with > >>> > flink > >>> > > > and > >>> > > > > we > >>> > > > > > > > need > >>> > > > > > > > > to > >>> > > > > > > > > > > >> >> >>> refactor > >>> > > > > > > > > > > >> >> >>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>> current flink client api. > >>> > > > > > > > > > > >> >> >>>>>>>> And I agree that CliFrontend should > >>> > > > > > > > > > > >> >> >>>>>>>> only > >>> > > > parsing > >>> > > > > > > > command > >>> > > > > > > > > > > line > >>> > > > > > > > > > > >> >> >>>>> arguments > >>> > > > > > > > > > > >> >> >>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>> then pass them to ExecutionEnvironment. > >>> > It is > >>> > > > > > > > > > > >> >> >>>> ExecutionEnvironment's > >>> > > > > > > > > > > >> >> >>>>>>>> responsibility to compile job, create > >>> > cluster, > >>> > > > > and > >>> > > > > > > > > submit > >>> > > > > > > > > > > job. > >>> > > > > > > > > > > >> >> >>>>> Besides > >>> > > > > > > > > > > >> >> >>>>>>>> that, Currently flink has many > >>> > > > > > ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>> implementations, > >>> > > > > > > > > > > >> >> >>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>> flink will use the specific one based > >>> > > > > > > > > > > >> >> >>>>>>>> on > >>> > the > >>> > > > > > context. > >>> > > > > > > > > > IMHO, > >>> > > > > > > > > > > it > >>> > > > > > > > > > > >> >> >> is > >>> > > > > > > > > > > >> >> >>>> not > >>> > > > > > > > > > > >> >> >>>>>>>> necessary, ExecutionEnvironment should > >>> > > > > > > > > > > >> >> >>>>>>>> be > >>> > able > >>> > > > > to > >>> > > > > > do > >>> > > > > > > > the > >>> > > > > > > > > > > right > >>> > > > > > > > > > > >> >> >>>> thing > >>> > > > > > > > > > > >> >> >>>>>>> based > >>> > > > > > > > > > > >> >> >>>>>>>> on the FlinkConf it is received. Too > >>> > > > > > > > > > > >> >> >>>>>>>> many > >>> > > > > > > > > > > ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>> implementation is another burden for > >>> > > > downstream > >>> > > > > > > > project > >>> > > > > > > > > > > >> >> >>>> integration. > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> One thing I'd like to mention is > >>> > > > > > > > > > > >> >> >>>>>>>> flink's > >>> > scala > >>> > > > > > shell > >>> > > > > > > > and > >>> > > > > > > > > > sql > >>> > > > > > > > > > > >> >> >>>> client, > >>> > > > > > > > > > > >> >> >>>>>>>> although they are sub-modules of flink, > >>> > they > >>> > > > > > could be > >>> > > > > > > > > > > treated > >>> > > > > > > > > > > >> >> >> as > >>> > > > > > > > > > > >> >> >>>>>>> downstream > >>> > > > > > > > > > > >> >> >>>>>>>> project which use flink's client api. > >>> > > > Currently > >>> > > > > > you > >>> > > > > > > > will > >>> > > > > > > > > > > find > >>> > > > > > > > > > > >> >> >> it > >>> > > > > > > > > > > >> >> >>> is > >>> > > > > > > > > > > >> >> >>>>> not > >>> > > > > > > > > > > >> >> >>>>>>>> easy for them to integrate with flink, > >>> > they > >>> > > > > share > >>> > > > > > many > >>> > > > > > > > > > > >> >> >> duplicated > >>> > > > > > > > > > > >> >> >>>>> code. > >>> > > > > > > > > > > >> >> >>>>>>> It > >>> > > > > > > > > > > >> >> >>>>>>>> is another sign that we should refactor > >>> > flink > >>> > > > > > client > >>> > > > > > > > > api. > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> I believe it is a large and hard > >>> > > > > > > > > > > >> >> >>>>>>>> change, > >>> > and I > >>> > > > > am > >>> > > > > > > > afraid > >>> > > > > > > > > > we > >>> > > > > > > > > > > >> can > >>> > > > > > > > > > > >> >> >>> not > >>> > > > > > > > > > > >> >> >>>>>> keep > >>> > > > > > > > > > > >> >> >>>>>>>> compatibility since many of changes are > >>> > user > >>> > > > > > facing. > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> Zili Chen <wander4...@gmail.com> > >>> > > > 于2019年6月24日周一 > >>> > > > > > > > > 下午2:53写道： > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> Hi all, > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> After a closer look on our client > >>> > > > > > > > > > > >> >> >>>>>>>>> apis, > >>> > I can > >>> > > > > see > >>> > > > > > > > there > >>> > > > > > > > > > are > >>> > > > > > > > > > > >> >> >> two > >>> > > > > > > > > > > >> >> >>>>> major > >>> > > > > > > > > > > >> >> >>>>>>>>> issues to consistency and integration, > >>> > namely > >>> > > > > > > > different > >>> > > > > > > > > > > >> >> >>>> deployment > >>> > > > > > > > > > > >> >> >>>>> of > >>> > > > > > > > > > > >> >> >>>>>>>>> job cluster which couples job graph > >>> > creation > >>> > > > > and > >>> > > > > > > > > cluster > >>> > > > > > > > > > > >> >> >>>>> deployment, > >>> > > > > > > > > > > >> >> >>>>>>>>> and submission via CliFrontend > >>> > > > > > > > > > > >> >> >>>>>>>>> confusing > >>> > > > > control > >>> > > > > > flow > >>> > > > > > > > > of > >>> > > > > > > > > > > job > >>> > > > > > > > > > > >> >> >>>> graph > >>> > > > > > > > > > > >> >> >>>>>>>>> compilation and job submission. I'd > >>> > > > > > > > > > > >> >> >>>>>>>>> like > >>> > to > >>> > > > > > follow > >>> > > > > > > > the > >>> > > > > > > > > > > >> >> >> discuss > >>> > > > > > > > > > > >> >> >>>>> above, > >>> > > > > > > > > > > >> >> >>>>>>>>> mainly the process described by Jeff > >>> > > > > > > > > > > >> >> >>>>>>>>> and > >>> > > > > > Stephan, and > >>> > > > > > > > > > share > >>> > > > > > > > > > > >> >> >> my > >>> > > > > > > > > > > >> >> >>>>>>>>> ideas on these issues. > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> 1) CliFrontend confuses the control > >>> > > > > > > > > > > >> >> >>>>>>>>> flow > >>> > of > >>> > > > job > >>> > > > > > > > > > compilation > >>> > > > > > > > > > > >> >> >> and > >>> > > > > > > > > > > >> >> >>>>>>>> submission. > >>> > > > > > > > > > > >> >> >>>>>>>>> Following the process of job > >>> > > > > > > > > > > >> >> >>>>>>>>> submission > >>> > > > Stephan > >>> > > > > > and > >>> > > > > > > > > Jeff > >>> > > > > > > > > > > >> >> >>>> described, > >>> > > > > > > > > > > >> >> >>>>>>>>> execution environment knows all > >>> > > > > > > > > > > >> >> >>>>>>>>> configs > >>> > of > >>> > > > the > >>> > > > > > > > cluster > >>> > > > > > > > > > and > >>> > > > > > > > > > > >> >> >>>>>>> topos/settings > >>> > > > > > > > > > > >> >> >>>>>>>>> of the job. Ideally, in the main > >>> > > > > > > > > > > >> >> >>>>>>>>> method > >>> > of > >>> > > > user > >>> > > > > > > > > program, > >>> > > > > > > > > > it > >>> > > > > > > > > > > >> >> >>> calls > >>> > > > > > > > > > > >> >> >>>>>>>> #execute > >>> > > > > > > > > > > >> >> >>>>>>>>> (or named #submit) and Flink deploys > >>> > > > > > > > > > > >> >> >>>>>>>>> the > >>> > > > > cluster, > >>> > > > > > > > > compile > >>> > > > > > > > > > > the > >>> > > > > > > > > > > >> >> >>> job > >>> > > > > > > > > > > >> >> >>>>>> graph > >>> > > > > > > > > > > >> >> >>>>>>>>> and submit it to the cluster. However, > >>> > > > current > >>> > > > > > > > > > CliFrontend > >>> > > > > > > > > > > >> >> >> does > >>> > > > > > > > > > > >> >> >>>> all > >>> > > > > > > > > > > >> >> >>>>>>> these > >>> > > > > > > > > > > >> >> >>>>>>>>> things inside its #runProgram method, > >>> > which > >>> > > > > > > > introduces > >>> > > > > > > > > a > >>> > > > > > > > > > > lot > >>> > > > > > > > > > > >> >> >> of > >>> > > > > > > > > > > >> >> >>>>>>>> subclasses > >>> > > > > > > > > > > >> >> >>>>>>>>> of (stream) execution environment. > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> Actually, it sets up an exec env that > >>> > hijacks > >>> > > > > the > >>> > > > > > > > > > > >> >> >>>>>> #execute/executePlan > >>> > > > > > > > > > > >> >> >>>>>>>>> method, initializes the job graph and > >>> > abort > >>> > > > > > > > execution. > >>> > > > > > > > > > And > >>> > > > > > > > > > > >> >> >> then > >>> > > > > > > > > > > >> >> >>>>>>>>> control flow back to CliFrontend, it > >>> > deploys > >>> > > > > the > >>> > > > > > > > > > cluster(or > >>> > > > > > > > > > > >> >> >>>>> retrieve > >>> > > > > > > > > > > >> >> >>>>>>>>> the client) and submits the job graph. > >>> > This > >>> > > > is > >>> > > > > > quite > >>> > > > > > > > a > >>> > > > > > > > > > > >> >> >> specific > >>> > > > > > > > > > > >> >> >>>>>>> internal > >>> > > > > > > > > > > >> >> >>>>>>>>> process inside Flink and none of > >>> > consistency > >>> > > > to > >>> > > > > > > > > anything. > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> 2) Deployment of job cluster couples > >>> > > > > > > > > > > >> >> >>>>>>>>> job > >>> > > > graph > >>> > > > > > > > creation > >>> > > > > > > > > > and > >>> > > > > > > > > > > >> >> >>>> cluster > >>> > > > > > > > > > > >> >> >>>>>>>>> deployment. Abstractly, from user job > >>> > > > > > > > > > > >> >> >>>>>>>>> to > >>> > a > >>> > > > > > concrete > >>> > > > > > > > > > > >> >> >> submission, > >>> > > > > > > > > > > >> >> >>>> it > >>> > > > > > > > > > > >> >> >>>>>>>> requires > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> create JobGraph --\ > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> create ClusterClient --> submit > >>> > > > > > > > > > > >> >> >>>>>>>>> JobGraph > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> such a dependency. ClusterClient was > >>> > created > >>> > > > by > >>> > > > > > > > > deploying > >>> > > > > > > > > > > or > >>> > > > > > > > > > > >> >> >>>>>>> retrieving. > >>> > > > > > > > > > > >> >> >>>>>>>>> JobGraph submission requires a > >>> > > > > > > > > > > >> >> >>>>>>>>> compiled > >>> > > > > JobGraph > >>> > > > > > and > >>> > > > > > > > > > valid > >>> > > > > > > > > > > >> >> >>>>>>> ClusterClient, > >>> > > > > > > > > > > >> >> >>>>>>>>> but the creation of ClusterClient is > >>> > > > abstractly > >>> > > > > > > > > > independent > >>> > > > > > > > > > > >> >> >> of > >>> > > > > > > > > > > >> >> >>>> that > >>> > > > > > > > > > > >> >> >>>>>> of > >>> > > > > > > > > > > >> >> >>>>>>>>> JobGraph. However, in job cluster > >>> > > > > > > > > > > >> >> >>>>>>>>> mode, > >>> > we > >>> > > > > > deploy job > >>> > > > > > > > > > > cluster > >>> > > > > > > > > > > >> >> >>>> with > >>> > > > > > > > > > > >> >> >>>>> a > >>> > > > > > > > > > > >> >> >>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>> graph, which means we use another > >>> > process: > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> create JobGraph --> deploy cluster > >>> > > > > > > > > > > >> >> >>>>>>>>> with > >>> > the > >>> > > > > > JobGraph > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> Here is another inconsistency and > >>> > downstream > >>> > > > > > > > > > > projects/client > >>> > > > > > > > > > > >> >> >>> apis > >>> > > > > > > > > > > >> >> >>>>> are > >>> > > > > > > > > > > >> >> >>>>>>>>> forced to handle different cases with > >>> > rare > >>> > > > > > supports > >>> > > > > > > > > from > >>> > > > > > > > > > > >> >> >> Flink. > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> Since we likely reached a consensus on > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> 1. all configs gathered by Flink > >>> > > > configuration > >>> > > > > > and > >>> > > > > > > > > passed > >>> > > > > > > > > > > >> >> >>>>>>>>> 2. execution environment knows all > >>> > configs > >>> > > > and > >>> > > > > > > > handles > >>> > > > > > > > > > > >> >> >>>>> execution(both > >>> > > > > > > > > > > >> >> >>>>>>>>> deployment and submission) > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> to the issues above I propose > >>> > > > > > > > > > > >> >> >>>>>>>>> eliminating > >>> > > > > > > > > inconsistencies > >>> > > > > > > > > > > by > >>> > > > > > > > > > > >> >> >>>>>> following > >>> > > > > > > > > > > >> >> >>>>>>>>> approach: > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> 1) CliFrontend should exactly be a > >>> > > > > > > > > > > >> >> >>>>>>>>> front > >>> > end, > >>> > > > > at > >>> > > > > > > > least > >>> > > > > > > > > > for > >>> > > > > > > > > > > >> >> >>> "run" > >>> > > > > > > > > > > >> >> >>>>>>> command. > >>> > > > > > > > > > > >> >> >>>>>>>>> That means it just gathered and passed > >>> > all > >>> > > > > config > >>> > > > > > > > from > >>> > > > > > > > > > > >> >> >> command > >>> > > > > > > > > > > >> >> >>>> line > >>> > > > > > > > > > > >> >> >>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>> the main method of user program. > >>> > Execution > >>> > > > > > > > environment > >>> > > > > > > > > > > knows > >>> > > > > > > > > > > >> >> >>> all > >>> > > > > > > > > > > >> >> >>>>> the > >>> > > > > > > > > > > >> >> >>>>>>> info > >>> > > > > > > > > > > >> >> >>>>>>>>> and with an addition to utils for > >>> > > > > ClusterClient, > >>> > > > > > we > >>> > > > > > > > > > > >> >> >> gracefully > >>> > > > > > > > > > > >> >> >>>> get > >>> > > > > > > > > > > >> >> >>>>> a > >>> > > > > > > > > > > >> >> >>>>>>>>> ClusterClient by deploying or > >>> > retrieving. In > >>> > > > > this > >>> > > > > > > > way, > >>> > > > > > > > > we > >>> > > > > > > > > > > >> >> >> don't > >>> > > > > > > > > > > >> >> >>>>> need > >>> > > > > > > > > > > >> >> >>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>> hijack #execute/executePlan methods > >>> > > > > > > > > > > >> >> >>>>>>>>> and > >>> > can > >>> > > > > > remove > >>> > > > > > > > > > various > >>> > > > > > > > > > > >> >> >>>> hacking > >>> > > > > > > > > > > >> >> >>>>>>>>> subclasses of exec env, as well as > >>> > > > > > > > > > > >> >> >>>>>>>>> #run > >>> > > > methods > >>> > > > > > in > >>> > > > > > > > > > > >> >> >>>>> ClusterClient(for > >>> > > > > > > > > > > >> >> >>>>>> an > >>> > > > > > > > > > > >> >> >>>>>>>>> interface-ized ClusterClient). Now the > >>> > > > control > >>> > > > > > flow > >>> > > > > > > > > flows > >>> > > > > > > > > > > >> >> >> from > >>> > > > > > > > > > > >> >> >>>>>>>> CliFrontend > >>> > > > > > > > > > > >> >> >>>>>>>>> to the main method and never returns. > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> 2) Job cluster means a cluster for the > >>> > > > specific > >>> > > > > > job. > >>> > > > > > > > > From > >>> > > > > > > > > > > >> >> >>> another > >>> > > > > > > > > > > >> >> >>>>>>>>> perspective, it is an ephemeral > >>> > > > > > > > > > > >> >> >>>>>>>>> session. > >>> > We > >>> > > > may > >>> > > > > > > > > decouple > >>> > > > > > > > > > > the > >>> > > > > > > > > > > >> >> >>>>>> deployment > >>> > > > > > > > > > > >> >> >>>>>>>>> with a compiled job graph, but start a > >>> > > > session > >>> > > > > > with > >>> > > > > > > > > idle > >>> > > > > > > > > > > >> >> >>> timeout > >>> > > > > > > > > > > >> >> >>>>>>>>> and submit the job following. > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> These topics, before we go into more > >>> > details > >>> > > > on > >>> > > > > > > > design > >>> > > > > > > > > or > >>> > > > > > > > > > > >> >> >>>>>>> implementation, > >>> > > > > > > > > > > >> >> >>>>>>>>> are better to be aware and discussed > >>> > > > > > > > > > > >> >> >>>>>>>>> for > >>> > a > >>> > > > > > consensus. > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> Best, > >>> > > > > > > > > > > >> >> >>>>>>>>> tison. > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>> Zili Chen <wander4...@gmail.com> > >>> > > > 于2019年6月20日周四 > >>> > > > > > > > > 上午3:21写道： > >>> > > > > > > > > > > >> >> >>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> Hi Jeff, > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> Thanks for raising this thread and > >>> > > > > > > > > > > >> >> >>>>>>>>>> the > >>> > > > design > >>> > > > > > > > > document! > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> As @Thomas Weise mentioned above, > >>> > extending > >>> > > > > > config > >>> > > > > > > > to > >>> > > > > > > > > > > flink > >>> > > > > > > > > > > >> >> >>>>>>>>>> requires far more effort than it > >>> > > > > > > > > > > >> >> >>>>>>>>>> should > >>> > be. > >>> > > > > > Another > >>> > > > > > > > > > > example > >>> > > > > > > > > > > >> >> >>>>>>>>>> is we achieve detach mode by > >>> > > > > > > > > > > >> >> >>>>>>>>>> introduce > >>> > > > another > >>> > > > > > > > > execution > >>> > > > > > > > > > > >> >> >>>>>>>>>> environment which also hijack > >>> > > > > > > > > > > >> >> >>>>>>>>>> #execute > >>> > > > method. > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> I agree with your idea that user > >>> > > > > > > > > > > >> >> >>>>>>>>>> would > >>> > > > > > configure all > >>> > > > > > > > > > > things > >>> > > > > > > > > > > >> >> >>>>>>>>>> and flink "just" respect it. On this > >>> > topic I > >>> > > > > > think > >>> > > > > > > > the > >>> > > > > > > > > > > >> >> >> unusual > >>> > > > > > > > > > > >> >> >>>>>>>>>> control flow when CliFrontend handle > >>> > "run" > >>> > > > > > command > >>> > > > > > > > is > >>> > > > > > > > > > the > >>> > > > > > > > > > > >> >> >>>> problem. > >>> > > > > > > > > > > >> >> >>>>>>>>>> It handles several configs, mainly > >>> > > > > > > > > > > >> >> >>>>>>>>>> about > >>> > > > > cluster > >>> > > > > > > > > > settings, > >>> > > > > > > > > > > >> >> >> and > >>> > > > > > > > > > > >> >> >>>>>>>>>> thus main method of user program is > >>> > unaware > >>> > > > of > >>> > > > > > them. > >>> > > > > > > > > > Also > >>> > > > > > > > > > > it > >>> > > > > > > > > > > >> >> >>>>>> compiles > >>> > > > > > > > > > > >> >> >>>>>>>>>> app to job graph by run the main > >>> > > > > > > > > > > >> >> >>>>>>>>>> method > >>> > > > with a > >>> > > > > > > > > hijacked > >>> > > > > > > > > > > exec > >>> > > > > > > > > > > >> >> >>>> env, > >>> > > > > > > > > > > >> >> >>>>>>>>>> which constrain the main method > >>> > > > > > > > > > > >> >> >>>>>>>>>> further. > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> I'd like to write down a few of > >>> > > > > > > > > > > >> >> >>>>>>>>>> notes on > >>> > > > > > > > configs/args > >>> > > > > > > > > > pass > >>> > > > > > > > > > > >> >> >> and > >>> > > > > > > > > > > >> >> >>>>>>> respect, > >>> > > > > > > > > > > >> >> >>>>>>>>>> as well as decoupling job compilation > >>> > and > >>> > > > > > > > submission. > >>> > > > > > > > > > > Share > >>> > > > > > > > > > > >> >> >> on > >>> > > > > > > > > > > >> >> >>>>> this > >>> > > > > > > > > > > >> >> >>>>>>>>>> thread later. > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> Best, > >>> > > > > > > > > > > >> >> >>>>>>>>>> tison. > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> SHI Xiaogang <shixiaoga...@gmail.com> > >>> > > > > > 于2019年6月17日周一 > >>> > > > > > > > > > > >> >> >> 下午7:29写道： > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> Hi Jeff and Flavio, > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> Thanks Jeff a lot for proposing the > >>> > design > >>> > > > > > > > document. > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> We are also working on refactoring > >>> > > > > > ClusterClient to > >>> > > > > > > > > > allow > >>> > > > > > > > > > > >> >> >>>>> flexible > >>> > > > > > > > > > > >> >> >>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>> efficient job management in our > >>> > real-time > >>> > > > > > platform. > >>> > > > > > > > > > > >> >> >>>>>>>>>>> We would like to draft a document to > >>> > share > >>> > > > > our > >>> > > > > > > > ideas > >>> > > > > > > > > > with > >>> > > > > > > > > > > >> >> >>> you. > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> I think it's a good idea to have > >>> > something > >>> > > > > like > >>> > > > > > > > > Apache > >>> > > > > > > > > > > Livy > >>> > > > > > > > > > > >> >> >>> for > >>> > > > > > > > > > > >> >> >>>>>>> Flink, > >>> > > > > > > > > > > >> >> >>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>> the efforts discussed here will > >>> > > > > > > > > > > >> >> >>>>>>>>>>> take a > >>> > > > great > >>> > > > > > step > >>> > > > > > > > > > forward > >>> > > > > > > > > > > >> >> >> to > >>> > > > > > > > > > > >> >> >>>> it. > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> Regards, > >>> > > > > > > > > > > >> >> >>>>>>>>>>> Xiaogang > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> Flavio Pompermaier < > >>> > pomperma...@okkam.it> > >>> > > > > > > > > > 于2019年6月17日周一 > >>> > > > > > > > > > > >> >> >>>>> 下午7:13写道： > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> Is there any possibility to have > >>> > something > >>> > > > > > like > >>> > > > > > > > > Apache > >>> > > > > > > > > > > >> >> >> Livy > >>> > > > > > > > > > > >> >> >>>> [1] > >>> > > > > > > > > > > >> >> >>>>>>> also > >>> > > > > > > > > > > >> >> >>>>>>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> Flink in the future? > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> [1] https://livy.apache.org/ > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> On Tue, Jun 11, 2019 at 5:23 PM > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> Jeff > >>> > > > Zhang < > >>> > > > > > > > > > > >> >> >>> zjf...@gmail.com > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>>>>> wrote: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> Any API we expose should not > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> have > >>> > > > > > dependencies > >>> > > > > > > > > on > >>> > > > > > > > > > > >> >> >>> the > >>> > > > > > > > > > > >> >> >>>>>>> runtime > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> (flink-runtime) package or other > >>> > > > > > implementation > >>> > > > > > > > > > > >> >> >> details. > >>> > > > > > > > > > > >> >> >>> To > >>> > > > > > > > > > > >> >> >>>>> me, > >>> > > > > > > > > > > >> >> >>>>>>>> this > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> means > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> that the current ClusterClient > >>> > cannot be > >>> > > > > > exposed > >>> > > > > > > > to > >>> > > > > > > > > > > >> >> >> users > >>> > > > > > > > > > > >> >> >>>>>> because > >>> > > > > > > > > > > >> >> >>>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> uses > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> quite some classes from the > >>> > optimiser and > >>> > > > > > runtime > >>> > > > > > > > > > > >> >> >>> packages. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> We should change ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> from > >>> > class > >>> > > > > to > >>> > > > > > > > > > interface. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> ExecutionEnvironment only use the > >>> > > > interface > >>> > > > > > > > > > > >> >> >> ClusterClient > >>> > > > > > > > > > > >> >> >>>>> which > >>> > > > > > > > > > > >> >> >>>>>>>>>>> should be > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> in flink-clients while the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> concrete > >>> > > > > > > > implementation > >>> > > > > > > > > > > >> >> >> class > >>> > > > > > > > > > > >> >> >>>>> could > >>> > > > > > > > > > > >> >> >>>>>> be > >>> > > > > > > > > > > >> >> >>>>>>>> in > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> flink-runtime. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> What happens when a > >>> > failure/restart in > >>> > > > > the > >>> > > > > > > > > client > >>> > > > > > > > > > > >> >> >>>>> happens? > >>> > > > > > > > > > > >> >> >>>>>>>> There > >>> > > > > > > > > > > >> >> >>>>>>>>>>> need > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> to be a way of re-establishing the > >>> > > > > > connection to > >>> > > > > > > > > the > >>> > > > > > > > > > > >> >> >> job, > >>> > > > > > > > > > > >> >> >>>> set > >>> > > > > > > > > > > >> >> >>>>>> up > >>> > > > > > > > > > > >> >> >>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> listeners again, etc. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> Good point. First we need to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> define > >>> > what > >>> > > > > > does > >>> > > > > > > > > > > >> >> >>>>> failure/restart > >>> > > > > > > > > > > >> >> >>>>>> in > >>> > > > > > > > > > > >> >> >>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> client mean. IIUC, that usually > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> mean > >>> > > > > network > >>> > > > > > > > > failure > >>> > > > > > > > > > > >> >> >>> which > >>> > > > > > > > > > > >> >> >>>>> will > >>> > > > > > > > > > > >> >> >>>>>>>>>>> happen in > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> class RestClient. If my > >>> > understanding is > >>> > > > > > correct, > >>> > > > > > > > > > > >> >> >>>>> restart/retry > >>> > > > > > > > > > > >> >> >>>>>>>>>>> mechanism > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> should be done in RestClient. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> Aljoscha Krettek < > >>> > aljos...@apache.org> > >>> > > > > > > > > 于2019年6月11日周二 > >>> > > > > > > > > > > >> >> >>>>>> 下午11:10写道： > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> Some points to consider: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> * Any API we expose should not > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> have > >>> > > > > > dependencies > >>> > > > > > > > > on > >>> > > > > > > > > > > >> >> >> the > >>> > > > > > > > > > > >> >> >>>>>> runtime > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> (flink-runtime) package or other > >>> > > > > > implementation > >>> > > > > > > > > > > >> >> >>> details. > >>> > > > > > > > > > > >> >> >>>> To > >>> > > > > > > > > > > >> >> >>>>>> me, > >>> > > > > > > > > > > >> >> >>>>>>>>>>> this > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> means > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> that the current ClusterClient > >>> > cannot be > >>> > > > > > exposed > >>> > > > > > > > > to > >>> > > > > > > > > > > >> >> >>> users > >>> > > > > > > > > > > >> >> >>>>>>> because > >>> > > > > > > > > > > >> >> >>>>>>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> uses > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> quite some classes from the > >>> > optimiser > >>> > > > and > >>> > > > > > > > runtime > >>> > > > > > > > > > > >> >> >>>> packages. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> * What happens when a > >>> > failure/restart in > >>> > > > > the > >>> > > > > > > > > client > >>> > > > > > > > > > > >> >> >>>>> happens? > >>> > > > > > > > > > > >> >> >>>>>>>> There > >>> > > > > > > > > > > >> >> >>>>>>>>>>> need > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> be a way of re-establishing the > >>> > > > connection > >>> > > > > > to > >>> > > > > > > > the > >>> > > > > > > > > > > >> >> >> job, > >>> > > > > > > > > > > >> >> >>>> set > >>> > > > > > > > > > > >> >> >>>>> up > >>> > > > > > > > > > > >> >> >>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> listeners > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> again, etc. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> Aljoscha > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> On 29. May 2019, at 10:17, Jeff > >>> > Zhang < > >>> > > > > > > > > > > >> >> >>>> zjf...@gmail.com> > >>> > > > > > > > > > > >> >> >>>>>>>> wrote: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> Sorry folks, the design doc is > >>> > late as > >>> > > > > you > >>> > > > > > > > > > > >> >> >> expected. > >>> > > > > > > > > > > >> >> >>>>> Here's > >>> > > > > > > > > > > >> >> >>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> design > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> doc > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> I drafted, welcome any comments > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> and > >>> > > > > > feedback. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://docs.google.com/document/d/1VavBrYn8vJeZs-Mhu5VzKO6xrWCF40aY0nlQ_UVVTRg/edit?usp=sharing > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> Stephan Ewen <se...@apache.org> > >>> > > > > > 于2019年2月14日周四 > >>> > > > > > > > > > > >> >> >>>> 下午8:43写道： > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> Nice that this discussion is > >>> > > > happening. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> In the FLIP, we could also > >>> > revisit the > >>> > > > > > entire > >>> > > > > > > > > role > >>> > > > > > > > > > > >> >> >>> of > >>> > > > > > > > > > > >> >> >>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> environments > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> again. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> Initially, the idea was: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> - the environments take care of > >>> > the > >>> > > > > > specific > >>> > > > > > > > > > > >> >> >> setup > >>> > > > > > > > > > > >> >> >>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>> standalone > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> (no > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> setup needed), yarn, mesos, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> etc. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> - the session ones have control > >>> > over > >>> > > > the > >>> > > > > > > > > session. > >>> > > > > > > > > > > >> >> >>> The > >>> > > > > > > > > > > >> >> >>>>>>>>>>> environment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> holds > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> the session client. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> - running a job gives a > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> "control" > >>> > > > object > >>> > > > > > for > >>> > > > > > > > > that > >>> > > > > > > > > > > >> >> >>>> job. > >>> > > > > > > > > > > >> >> >>>>>> That > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> behavior > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> is > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> the same in all environments. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> The actual implementation > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> diverged > >>> > > > quite > >>> > > > > > a bit > >>> > > > > > > > > > > >> >> >> from > >>> > > > > > > > > > > >> >> >>>>> that. > >>> > > > > > > > > > > >> >> >>>>>>>> Happy > >>> > > > > > > > > > > >> >> >>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> see a > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> discussion about straitening > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> this > >>> > out > >>> > > > a > >>> > > > > > bit > >>> > > > > > > > > more. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> On Tue, Feb 12, 2019 at 4:58 AM > >>> > Jeff > >>> > > > > > Zhang < > >>> > > > > > > > > > > >> >> >>>>>>> zjf...@gmail.com> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> wrote: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> Hi folks, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> Sorry for late response, It > >>> > seems we > >>> > > > > > reach > >>> > > > > > > > > > > >> >> >>> consensus > >>> > > > > > > > > > > >> >> >>>> on > >>> > > > > > > > > > > >> >> >>>>>>>> this, I > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> will > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> create > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> FLIP for this with more > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> detailed > >>> > > > design > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> Thomas Weise <t...@apache.org> > >>> > > > > > 于2018年12月21日周五 > >>> > > > > > > > > > > >> >> >>>>> 上午11:43写道： > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Great to see this discussion > >>> > seeded! > >>> > > > > The > >>> > > > > > > > > > > >> >> >> problems > >>> > > > > > > > > > > >> >> >>>> you > >>> > > > > > > > > > > >> >> >>>>>> face > >>> > > > > > > > > > > >> >> >>>>>>>>>>> with > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Zeppelin integration are also > >>> > > > > affecting > >>> > > > > > > > other > >>> > > > > > > > > > > >> >> >>>>> downstream > >>> > > > > > > > > > > >> >> >>>>>>>>>>> projects, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> like > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Beam. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> We just enabled the savepoint > >>> > > > restore > >>> > > > > > option > >>> > > > > > > > > in > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> RemoteStreamEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> [1] > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> and that was more difficult > >>> > than it > >>> > > > > > should > >>> > > > > > > > be. > >>> > > > > > > > > > > >> >> >> The > >>> > > > > > > > > > > >> >> >>>>> main > >>> > > > > > > > > > > >> >> >>>>>>>> issue > >>> > > > > > > > > > > >> >> >>>>>>>>>>> is > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> that > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> environment and cluster > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> client > >>> > > > aren't > >>> > > > > > > > > decoupled. > >>> > > > > > > > > > > >> >> >>>>> Ideally > >>> > > > > > > > > > > >> >> >>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>> should > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> be > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> possible to just get the > >>> > matching > >>> > > > > > cluster > >>> > > > > > > > > client > >>> > > > > > > > > > > >> >> >>>> from > >>> > > > > > > > > > > >> >> >>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> environment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> then control the job through > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> it > >>> > > > > > (environment > >>> > > > > > > > > as > >>> > > > > > > > > > > >> >> >>>>> factory > >>> > > > > > > > > > > >> >> >>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> cluster > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> client). But note that the > >>> > > > environment > >>> > > > > > > > classes > >>> > > > > > > > > > > >> >> >> are > >>> > > > > > > > > > > >> >> >>>>> part > >>> > > > > > > > > > > >> >> >>>>>> of > >>> > > > > > > > > > > >> >> >>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> public > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> API, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> and it is not > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> straightforward to > >>> > > > make > >>> > > > > > larger > >>> > > > > > > > > > > >> >> >>> changes > >>> > > > > > > > > > > >> >> >>>>>>> without > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> breaking > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> backward compatibility. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> ClusterClient currently > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> exposes > >>> > > > > internal > >>> > > > > > > > > classes > >>> > > > > > > > > > > >> >> >>>> like > >>> > > > > > > > > > > >> >> >>>>>>>>>>> JobGraph and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> StreamGraph. But it should be > >>> > > > possible > >>> > > > > > to > >>> > > > > > > > wrap > >>> > > > > > > > > > > >> >> >>> this > >>> > > > > > > > > > > >> >> >>>>>> with a > >>> > > > > > > > > > > >> >> >>>>>>>> new > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> public > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> API > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> that brings the required job > >>> > control > >>> > > > > > > > > > > >> >> >> capabilities > >>> > > > > > > > > > > >> >> >>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>> downstream > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> projects. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Perhaps it is helpful to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> look at > >>> > > > some > >>> > > > > > of the > >>> > > > > > > > > > > >> >> >>>>> interfaces > >>> > > > > > > > > > > >> >> >>>>>> in > >>> > > > > > > > > > > >> >> >>>>>>>>>>> Beam > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> while > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> thinking about this: [2] for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> the > >>> > > > > > portable > >>> > > > > > > > job > >>> > > > > > > > > > > >> >> >> API > >>> > > > > > > > > > > >> >> >>>> and > >>> > > > > > > > > > > >> >> >>>>>> [3] > >>> > > > > > > > > > > >> >> >>>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> old > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> asynchronous job control from > >>> > the > >>> > > > Beam > >>> > > > > > Java > >>> > > > > > > > > SDK. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> The backward compatibility > >>> > > > discussion > >>> > > > > > [4] is > >>> > > > > > > > > > > >> >> >> also > >>> > > > > > > > > > > >> >> >>>>>> relevant > >>> > > > > > > > > > > >> >> >>>>>>>>>>> here. A > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> new > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> API > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> should shield downstream > >>> > projects > >>> > > > from > >>> > > > > > > > > internals > >>> > > > > > > > > > > >> >> >>> and > >>> > > > > > > > > > > >> >> >>>>>> allow > >>> > > > > > > > > > > >> >> >>>>>>>>>>> them to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> interoperate with multiple > >>> > future > >>> > > > > Flink > >>> > > > > > > > > versions > >>> > > > > > > > > > > >> >> >>> in > >>> > > > > > > > > > > >> >> >>>>> the > >>> > > > > > > > > > > >> >> >>>>>>> same > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> release > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> line > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> without forced upgrades. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Thanks, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Thomas > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> [1] > >>> > > > > > > > https://github.com/apache/flink/pull/7249 > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> [2] > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/beam_job_api.proto > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> [3] > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineResult.java > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> [4] > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > https://lists.apache.org/thread.html/064c75c5d10f0806095b14f6d76942598917a14429c1acbddd151fe2@%3Cdev.flink.apache.org%3E > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> On Thu, Dec 20, 2018 at 6:15 > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> PM > >>> > Jeff > >>> > > > > > Zhang < > >>> > > > > > > > > > > >> >> >>>>>>>> zjf...@gmail.com> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> wrote: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> I'm not so sure whether > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> the > >>> > user > >>> > > > > > should > >>> > > > > > > > be > >>> > > > > > > > > > > >> >> >>> able > >>> > > > > > > > > > > >> >> >>>> to > >>> > > > > > > > > > > >> >> >>>>>>>> define > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> where > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> runs (in your example Yarn). > >>> > This > >>> > > > is > >>> > > > > > > > actually > >>> > > > > > > > > > > >> >> >>>>>> independent > >>> > > > > > > > > > > >> >> >>>>>>>> of > >>> > > > > > > > > > > >> >> >>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> development and is something > >>> > which > >>> > > > is > >>> > > > > > > > decided > >>> > > > > > > > > > > >> >> >> at > >>> > > > > > > > > > > >> >> >>>>>>> deployment > >>> > > > > > > > > > > >> >> >>>>>>>>>>> time. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> User don't need to specify > >>> > > > execution > >>> > > > > > mode > >>> > > > > > > > > > > >> >> >>>>>>> programmatically. > >>> > > > > > > > > > > >> >> >>>>>>>>>>> They > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> can > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> also > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> pass the execution mode from > >>> > the > >>> > > > > > arguments > >>> > > > > > > > in > >>> > > > > > > > > > > >> >> >>> flink > >>> > > > > > > > > > > >> >> >>>>> run > >>> > > > > > > > > > > >> >> >>>>>>>>>>> command. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> e.g. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> bin/flink run -m > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> yarn-cluster > >>> > .... > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> bin/flink run -m local ... > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> bin/flink run -m host:port > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> ... > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Does this make sense to you > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> ? > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> To me it makes sense that > >>> > the > >>> > > > > > > > > > > >> >> >>>> ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>>>> is > >>> > > > > > > > > > > >> >> >>>>>>>> not > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> directly > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> initialized by the user and > >>> > instead > >>> > > > > > context > >>> > > > > > > > > > > >> >> >>>> sensitive > >>> > > > > > > > > > > >> >> >>>>>> how > >>> > > > > > > > > > > >> >> >>>>>>>> you > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> want > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> execute your job (Flink CLI > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> vs. > >>> > > > IDE, > >>> > > > > > for > >>> > > > > > > > > > > >> >> >>> example). > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Right, currently I notice > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Flink > >>> > > > would > >>> > > > > > > > create > >>> > > > > > > > > > > >> >> >>>>> different > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> ContextExecutionEnvironment > >>> > based > >>> > > > on > >>> > > > > > > > > different > >>> > > > > > > > > > > >> >> >>>>>> submission > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> scenarios > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> (Flink > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Cli vs IDE). To me this is > >>> > kind of > >>> > > > > hack > >>> > > > > > > > > > > >> >> >> approach, > >>> > > > > > > > > > > >> >> >>>> not > >>> > > > > > > > > > > >> >> >>>>>> so > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> straightforward. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> What I suggested above is > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> that > >>> > is > >>> > > > > that > >>> > > > > > > > flink > >>> > > > > > > > > > > >> >> >>> should > >>> > > > > > > > > > > >> >> >>>>>>> always > >>> > > > > > > > > > > >> >> >>>>>>>>>>> create > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> same > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> ExecutionEnvironment but > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> with > >>> > > > > different > >>> > > > > > > > > > > >> >> >>>>> configuration, > >>> > > > > > > > > > > >> >> >>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>> based > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> on > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> configuration it would > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> create > >>> > the > >>> > > > > > proper > >>> > > > > > > > > > > >> >> >>>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> different > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> behaviors. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Till Rohrmann < > >>> > > > trohrm...@apache.org> > >>> > > > > > > > > > > >> >> >>>> 于2018年12月20日周四 > >>> > > > > > > > > > > >> >> >>>>>>>>>>> 下午11:18写道： > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> You are probably right > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> that we > >>> > > > have > >>> > > > > > code > >>> > > > > > > > > > > >> >> >>>> duplication > >>> > > > > > > > > > > >> >> >>>>>>> when > >>> > > > > > > > > > > >> >> >>>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> comes > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> creation of the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> ClusterClient. > >>> > > > This > >>> > > > > > should > >>> > > > > > > > > be > >>> > > > > > > > > > > >> >> >>>>> reduced > >>> > > > > > > > > > > >> >> >>>>>> in > >>> > > > > > > > > > > >> >> >>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> future. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> I'm not so sure whether the > >>> > user > >>> > > > > > should be > >>> > > > > > > > > > > >> >> >> able > >>> > > > > > > > > > > >> >> >>> to > >>> > > > > > > > > > > >> >> >>>>>>> define > >>> > > > > > > > > > > >> >> >>>>>>>>>>> where > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> runs (in your example > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> Yarn). > >>> > This > >>> > > > is > >>> > > > > > > > > actually > >>> > > > > > > > > > > >> >> >>>>>>> independent > >>> > > > > > > > > > > >> >> >>>>>>>>>>> of the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> development and is > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> something > >>> > which > >>> > > > > is > >>> > > > > > > > > decided > >>> > > > > > > > > > > >> >> >> at > >>> > > > > > > > > > > >> >> >>>>>>>> deployment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> time. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> To > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> me > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> makes sense that the > >>> > > > > > ExecutionEnvironment > >>> > > > > > > > is > >>> > > > > > > > > > > >> >> >> not > >>> > > > > > > > > > > >> >> >>>>>>> directly > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> initialized > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> by > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> the user and instead > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> context > >>> > > > > > sensitive how > >>> > > > > > > > > you > >>> > > > > > > > > > > >> >> >>>> want > >>> > > > > > > > > > > >> >> >>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>> execute > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> your > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> (Flink CLI vs. IDE, for > >>> > example). > >>> > > > > > > > However, I > >>> > > > > > > > > > > >> >> >>> agree > >>> > > > > > > > > > > >> >> >>>>>> that > >>> > > > > > > > > > > >> >> >>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> ExecutionEnvironment should > >>> > give > >>> > > > you > >>> > > > > > > > access > >>> > > > > > > > > to > >>> > > > > > > > > > > >> >> >>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> job (maybe in the form of > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> the > >>> > > > > > JobGraph or > >>> > > > > > > > a > >>> > > > > > > > > > > >> >> >> job > >>> > > > > > > > > > > >> >> >>>>> plan). > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> Cheers, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> Till > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> On Thu, Dec 13, 2018 at > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> 4:36 > >>> > AM > >>> > > > Jeff > >>> > > > > > > > Zhang < > >>> > > > > > > > > > > >> >> >>>>>>>>>>> zjf...@gmail.com> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> wrote: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Hi Till, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> You > >>> > are > >>> > > > > > right > >>> > > > > > > > > that I > >>> > > > > > > > > > > >> >> >>>>> expect > >>> > > > > > > > > > > >> >> >>>>>>>> better > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> programmatic > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> job submission/control api > >>> > which > >>> > > > > > could be > >>> > > > > > > > > > > >> >> >> used > >>> > > > > > > > > > > >> >> >>> by > >>> > > > > > > > > > > >> >> >>>>>>>>>>> downstream > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> project. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> And > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> it would benefit for the > >>> > flink > >>> > > > > > ecosystem. > >>> > > > > > > > > > > >> >> >> When > >>> > > > > > > > > > > >> >> >>> I > >>> > > > > > > > > > > >> >> >>>>> look > >>> > > > > > > > > > > >> >> >>>>>>> at > >>> > > > > > > > > > > >> >> >>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> code > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> of > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> flink > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> scala-shell and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> sql-client (I > >>> > > > > believe > >>> > > > > > > > they > >>> > > > > > > > > > > >> >> >> are > >>> > > > > > > > > > > >> >> >>>> not > >>> > > > > > > > > > > >> >> >>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>> core of > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> flink, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> but > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> belong to the ecosystem of > >>> > > > flink), > >>> > > > > I > >>> > > > > > find > >>> > > > > > > > > > > >> >> >> many > >>> > > > > > > > > > > >> >> >>>>>>> duplicated > >>> > > > > > > > > > > >> >> >>>>>>>>>>> code > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> creating > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> ClusterClient from user > >>> > provided > >>> > > > > > > > > > > >> >> >> configuration > >>> > > > > > > > > > > >> >> >>>>>>>>>>> (configuration > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> format > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> may > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> be > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> different from scala-shell > >>> > and > >>> > > > > > > > sql-client) > >>> > > > > > > > > > > >> >> >> and > >>> > > > > > > > > > > >> >> >>>> then > >>> > > > > > > > > > > >> >> >>>>>> use > >>> > > > > > > > > > > >> >> >>>>>>>>>>> that > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> to manipulate jobs. I > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> don't > >>> > think > >>> > > > > > this is > >>> > > > > > > > > > > >> >> >>>>> convenient > >>> > > > > > > > > > > >> >> >>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> downstream > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> projects. What I expect is > >>> > that > >>> > > > > > > > downstream > >>> > > > > > > > > > > >> >> >>>> project > >>> > > > > > > > > > > >> >> >>>>>> only > >>> > > > > > > > > > > >> >> >>>>>>>>>>> needs > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> provide > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> necessary configuration > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> info > >>> > > > (maybe > >>> > > > > > > > > > > >> >> >> introducing > >>> > > > > > > > > > > >> >> >>>>> class > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> FlinkConf), > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> then > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> build ExecutionEnvironment > >>> > based > >>> > > > on > >>> > > > > > this > >>> > > > > > > > > > > >> >> >>>> FlinkConf, > >>> > > > > > > > > > > >> >> >>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment will > >>> > create > >>> > > > > the > >>> > > > > > > > proper > >>> > > > > > > > > > > >> >> >>>>>>>> ClusterClient. > >>> > > > > > > > > > > >> >> >>>>>>>>>>> It > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> not > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> only > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> benefit for the downstream > >>> > > > project > >>> > > > > > > > > > > >> >> >> development > >>> > > > > > > > > > > >> >> >>>> but > >>> > > > > > > > > > > >> >> >>>>>> also > >>> > > > > > > > > > > >> >> >>>>>>>> be > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> helpful > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> their integration test > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> with > >>> > > > flink. > >>> > > > > > Here's > >>> > > > > > > > > one > >>> > > > > > > > > > > >> >> >>>>> sample > >>> > > > > > > > > > > >> >> >>>>>>> code > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> snippet > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> that > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> I > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> expect. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> val conf = new > >>> > > > > > FlinkConf().mode("yarn") > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> val env = new > >>> > > > > > ExecutionEnvironment(conf) > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> val jobId = > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> env.submit(...) > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> val jobStatus = > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > env.getClusterClient().queryJobStatus(jobId) > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > env.getClusterClient().cancelJob(jobId) > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> What do you think ? > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Till Rohrmann < > >>> > > > > trohrm...@apache.org> > >>> > > > > > > > > > > >> >> >>>>> 于2018年12月11日周二 > >>> > > > > > > > > > > >> >> >>>>>>>>>>> 下午6:28写道： > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Hi Jeff, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> what you are proposing > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> is to > >>> > > > > > provide the > >>> > > > > > > > > > > >> >> >> user > >>> > > > > > > > > > > >> >> >>>> with > >>> > > > > > > > > > > >> >> >>>>>>>> better > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> programmatic > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> control. There was > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> actually > >>> > an > >>> > > > > > effort to > >>> > > > > > > > > > > >> >> >>> achieve > >>> > > > > > > > > > > >> >> >>>>>> this > >>> > > > > > > > > > > >> >> >>>>>>>> but > >>> > > > > > > > > > > >> >> >>>>>>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> has > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> never > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> been > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> completed [1]. However, > >>> > there > >>> > > > are > >>> > > > > > some > >>> > > > > > > > > > > >> >> >>>> improvement > >>> > > > > > > > > > > >> >> >>>>>> in > >>> > > > > > > > > > > >> >> >>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>> code > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> base > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> now. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Look for example at the > >>> > > > > > NewClusterClient > >>> > > > > > > > > > > >> >> >>>> interface > >>> > > > > > > > > > > >> >> >>>>>>> which > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> offers a > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> non-blocking job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> submission. > >>> > > > But I > >>> > > > > > agree > >>> > > > > > > > > > > >> >> >> that > >>> > > > > > > > > > > >> >> >>> we > >>> > > > > > > > > > > >> >> >>>>>> need > >>> > > > > > > > > > > >> >> >>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> improve > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Flink > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> in > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> this regard. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> I would not be in favour > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> if > >>> > > > > > exposing all > >>> > > > > > > > > > > >> >> >>>>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>> calls > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> via > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment > >>> > because it > >>> > > > > > would > >>> > > > > > > > > > > >> >> >> clutter > >>> > > > > > > > > > > >> >> >>>> the > >>> > > > > > > > > > > >> >> >>>>>>> class > >>> > > > > > > > > > > >> >> >>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> would > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> not > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> be > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> a > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> good separation of > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> concerns. > >>> > > > > > Instead one > >>> > > > > > > > > > > >> >> >> idea > >>> > > > > > > > > > > >> >> >>>>> could > >>> > > > > > > > > > > >> >> >>>>>> be > >>> > > > > > > > > > > >> >> >>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> retrieve > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> current ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> from > >>> > the > >>> > > > > > > > > > > >> >> >>>>> ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>> which > >>> > > > > > > > > > > >> >> >>>>>>>>>>> can > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> then > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> be > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> used > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> for cluster and job > >>> > control. But > >>> > > > > > before > >>> > > > > > > > we > >>> > > > > > > > > > > >> >> >>> start > >>> > > > > > > > > > > >> >> >>>>> an > >>> > > > > > > > > > > >> >> >>>>>>>> effort > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> here, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> we > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> need > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> agree and capture what > >>> > > > > > functionality we > >>> > > > > > > > > want > >>> > > > > > > > > > > >> >> >>> to > >>> > > > > > > > > > > >> >> >>>>>>> provide. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Initially, the idea was > >>> > that we > >>> > > > > > have the > >>> > > > > > > > > > > >> >> >>>>>>>> ClusterDescriptor > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> describing > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> how > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> to talk to cluster > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> manager > >>> > like > >>> > > > > > Yarn or > >>> > > > > > > > > > > >> >> >> Mesos. > >>> > > > > > > > > > > >> >> >>>> The > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> ClusterDescriptor > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> can > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> be > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> used for deploying Flink > >>> > > > clusters > >>> > > > > > (job > >>> > > > > > > > and > >>> > > > > > > > > > > >> >> >>>>> session) > >>> > > > > > > > > > > >> >> >>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>> gives > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> you a > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> ClusterClient. The > >>> > ClusterClient > >>> > > > > > > > controls > >>> > > > > > > > > > > >> >> >> the > >>> > > > > > > > > > > >> >> >>>>>> cluster > >>> > > > > > > > > > > >> >> >>>>>>>>>>> (e.g. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> submitting > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> jobs, listing all running > >>> > jobs). > >>> > > > > And > >>> > > > > > > > then > >>> > > > > > > > > > > >> >> >>> there > >>> > > > > > > > > > > >> >> >>>>> was > >>> > > > > > > > > > > >> >> >>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>> idea > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> introduce a > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> JobClient which you > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> obtain > >>> > from > >>> > > > > the > >>> > > > > > > > > > > >> >> >>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>> trigger > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> specific > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> operations (e.g. taking a > >>> > > > > savepoint, > >>> > > > > > > > > > > >> >> >>> cancelling > >>> > > > > > > > > > > >> >> >>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>> job). > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> [1] > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > https://issues.apache.org/jira/browse/FLINK-4272 > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Cheers, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Till > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> On Tue, Dec 11, 2018 at > >>> > 10:13 AM > >>> > > > > > Jeff > >>> > > > > > > > > Zhang > >>> > > > > > > > > > > >> >> >> < > >>> > > > > > > > > > > >> >> >>>>>>>>>>> zjf...@gmail.com > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> wrote: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Hi Folks, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> I am trying to integrate > >>> > flink > >>> > > > > into > >>> > > > > > > > > apache > >>> > > > > > > > > > > >> >> >>>>> zeppelin > >>> > > > > > > > > > > >> >> >>>>>>>>>>> which is > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> an > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> interactive > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> notebook. And I hit > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> several > >>> > > > > issues > >>> > > > > > that > >>> > > > > > > > > is > >>> > > > > > > > > > > >> >> >>>> caused > >>> > > > > > > > > > > >> >> >>>>>> by > >>> > > > > > > > > > > >> >> >>>>>>>>>>> flink > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> client > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> api. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> So > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> I'd like to proposal the > >>> > > > > following > >>> > > > > > > > > changes > >>> > > > > > > > > > > >> >> >>> for > >>> > > > > > > > > > > >> >> >>>>>> flink > >>> > > > > > > > > > > >> >> >>>>>>>>>>> client > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> api. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 1. Support nonblocking > >>> > > > execution. > >>> > > > > > > > > > > >> >> >> Currently, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment#execute > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> is a blocking method > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> which > >>> > > > would > >>> > > > > > do 2 > >>> > > > > > > > > > > >> >> >> things, > >>> > > > > > > > > > > >> >> >>>>> first > >>> > > > > > > > > > > >> >> >>>>>>>>>>> submit > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> then > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> wait for job until it is > >>> > > > > finished. > >>> > > > > > I'd > >>> > > > > > > > > like > >>> > > > > > > > > > > >> >> >>>>>>> introduce a > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> nonblocking > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> execution method like > >>> > > > > > > > > > > >> >> >>>> ExecutionEnvironment#submit > >>> > > > > > > > > > > >> >> >>>>>>> which > >>> > > > > > > > > > > >> >> >>>>>>>>>>> only > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> submit > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> then return jobId to > >>> > client. > >>> > > > And > >>> > > > > > allow > >>> > > > > > > > > user > >>> > > > > > > > > > > >> >> >>> to > >>> > > > > > > > > > > >> >> >>>>>> query > >>> > > > > > > > > > > >> >> >>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> status > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> via > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> jobId. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 2. Add cancel api in > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >> ExecutionEnvironment/StreamExecutionEnvironment, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> currently the only way > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> to > >>> > > > cancel > >>> > > > > > job is > >>> > > > > > > > > via > >>> > > > > > > > > > > >> >> >>> cli > >>> > > > > > > > > > > >> >> >>>>>>>>>>> (bin/flink), > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> this > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> is > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> not > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> convenient for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> downstream > >>> > > > project > >>> > > > > > to > >>> > > > > > > > use > >>> > > > > > > > > > > >> >> >> this > >>> > > > > > > > > > > >> >> >>>>>>> feature. > >>> > > > > > > > > > > >> >> >>>>>>>>>>> So I'd > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> like > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> add > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> cancel api in > >>> > > > > ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 3. Add savepoint api in > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>> > >>> > ExecutionEnvironment/StreamExecutionEnvironment. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> It > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> is similar as cancel > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> api, > >>> > we > >>> > > > > > should use > >>> > > > > > > > > > > >> >> >>>>>>>>>>> ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> as > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> unified > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> api for third party to > >>> > > > integrate > >>> > > > > > with > >>> > > > > > > > > > > >> >> >> flink. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 4. Add listener for job > >>> > > > execution > >>> > > > > > > > > > > >> >> >> lifecycle. > >>> > > > > > > > > > > >> >> >>>>>>> Something > >>> > > > > > > > > > > >> >> >>>>>>>>>>> like > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> following, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> so > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> that downstream project > >>> > can do > >>> > > > > > custom > >>> > > > > > > > > logic > >>> > > > > > > > > > > >> >> >>> in > >>> > > > > > > > > > > >> >> >>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>> lifecycle > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> of > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> job. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> e.g. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Zeppelin would capture > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> the > >>> > > > jobId > >>> > > > > > after > >>> > > > > > > > > job > >>> > > > > > > > > > > >> >> >> is > >>> > > > > > > > > > > >> >> >>>>>>> submitted > >>> > > > > > > > > > > >> >> >>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> then > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> use > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> this > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> jobId to cancel it later > >>> > when > >>> > > > > > > > necessary. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> public interface > >>> > JobListener { > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> void > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> onJobSubmitted(JobID > >>> > > > > jobId); > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> void > >>> > > > > > onJobExecuted(JobExecutionResult > >>> > > > > > > > > > > >> >> >>>>> jobResult); > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> void > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> onJobCanceled(JobID > >>> > > > jobId); > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> } > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 5. Enable session in > >>> > > > > > > > > ExecutionEnvironment. > >>> > > > > > > > > > > >> >> >>>>>> Currently > >>> > > > > > > > > > > >> >> >>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>> is > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> disabled, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> but > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> session is very > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> convenient > >>> > for > >>> > > > > > third > >>> > > > > > > > > party > >>> > > > > > > > > > > >> >> >> to > >>> > > > > > > > > > > >> >> >>>>>>>> submitting > >>> > > > > > > > > > > >> >> >>>>>>>>>>> jobs > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> continually. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> I hope flink can enable > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> it > >>> > > > again. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 6. Unify all flink > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> client > >>> > api > >>> > > > > into > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > ExecutionEnvironment/StreamExecutionEnvironment. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> This is a long term > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> issue > >>> > which > >>> > > > > > needs > >>> > > > > > > > > more > >>> > > > > > > > > > > >> >> >>>>> careful > >>> > > > > > > > > > > >> >> >>>>>>>>>>> thinking > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> design. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Currently some of > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> features > >>> > of > >>> > > > > > flink is > >>> > > > > > > > > > > >> >> >>> exposed > >>> > > > > > > > > > > >> >> >>>> in > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > ExecutionEnvironment/StreamExecutionEnvironment, > >>> > > > > > > > > > > >> >> >>>>>> but > >>> > > > > > > > > > > >> >> >>>>>>>>>>> some are > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> exposed > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> in > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> cli instead of api, like > >>> > the > >>> > > > > > cancel and > >>> > > > > > > > > > > >> >> >>>>> savepoint I > >>> > > > > > > > > > > >> >> >>>>>>>>>>> mentioned > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> above. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> I > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> think the root cause is > >>> > due to > >>> > > > > that > >>> > > > > > > > flink > >>> > > > > > > > > > > >> >> >>>> didn't > >>> > > > > > > > > > > >> >> >>>>>>> unify > >>> > > > > > > > > > > >> >> >>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> interaction > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> with > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> flink. Here I list 3 > >>> > scenarios > >>> > > > of > >>> > > > > > flink > >>> > > > > > > > > > > >> >> >>>> operation > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> - Local job execution. > >>> > Flink > >>> > > > > will > >>> > > > > > > > > create > >>> > > > > > > > > > > >> >> >>>>>>>>>>> LocalEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> then > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> use > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> this LocalEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> to > >>> > > > create > >>> > > > > > > > > > > >> >> >>> LocalExecutor > >>> > > > > > > > > > > >> >> >>>>> for > >>> > > > > > > > > > > >> >> >>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> execution. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> - Remote job execution. > >>> > Flink > >>> > > > > will > >>> > > > > > > > > create > >>> > > > > > > > > > > >> >> >>>>>>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> first > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> then > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> create > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> ContextEnvironment > >>> > > > based > >>> > > > > > on the > >>> > > > > > > > > > > >> >> >>>>>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> then > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> run > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> job. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> - Job cancelation. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Flink > >>> > will > >>> > > > > > create > >>> > > > > > > > > > > >> >> >>>>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>> first > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> then > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> cancel > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> this job via this > >>> > > > ClusterClient. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> As you can see in the > >>> > above 3 > >>> > > > > > > > scenarios. > >>> > > > > > > > > > > >> >> >>> Flink > >>> > > > > > > > > > > >> >> >>>>>> didn't > >>> > > > > > > > > > > >> >> >>>>>>>>>>> use the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> same > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> approach(code path) to > >>> > interact > >>> > > > > > with > >>> > > > > > > > > flink > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> What I propose is > >>> > following: > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Create the proper > >>> > > > > > > > > > > >> >> >>>>>> LocalEnvironment/RemoteEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>> (based > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> on > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> user > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> configuration) --> Use > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> this > >>> > > > > > Environment > >>> > > > > > > > > to > >>> > > > > > > > > > > >> >> >>>> create > >>> > > > > > > > > > > >> >> >>>>>>>> proper > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> ClusterClient > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> (LocalClusterClient or > >>> > > > > > > > RestClusterClient) > >>> > > > > > > > > > > >> >> >> to > >>> > > > > > > > > > > >> >> >>>>>>>> interactive > >>> > > > > > > > > > > >> >> >>>>>>>>>>> with > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Flink ( > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> job > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> execution or > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> cancelation) > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> This way we can unify > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> the > >>> > > > process > >>> > > > > > of > >>> > > > > > > > > local > >>> > > > > > > > > > > >> >> >>>>>> execution > >>> > > > > > > > > > > >> >> >>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> remote > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> execution. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> And it is much easier > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> for > >>> > third > >>> > > > > > party > >>> > > > > > > > to > >>> > > > > > > > > > > >> >> >>>>> integrate > >>> > > > > > > > > > > >> >> >>>>>>> with > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> flink, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> because > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment is > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> the > >>> > > > > unified > >>> > > > > > > > entry > >>> > > > > > > > > > > >> >> >>> point > >>> > > > > > > > > > > >> >> >>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>> flink. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> What > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> third > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> party > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> needs to do is just pass > >>> > > > > > configuration > >>> > > > > > > > to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> and > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> will > >>> > do > >>> > > > the > >>> > > > > > right > >>> > > > > > > > > > > >> >> >> thing > >>> > > > > > > > > > > >> >> >>>>> based > >>> > > > > > > > > > > >> >> >>>>>> on > >>> > > > > > > > > > > >> >> >>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> configuration. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Flink cli can also be > >>> > > > considered > >>> > > > > as > >>> > > > > > > > flink > >>> > > > > > > > > > > >> >> >> api > >>> > > > > > > > > > > >> >> >>>>>>> consumer. > >>> > > > > > > > > > > >> >> >>>>>>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> just > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> pass > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> the > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> configuration to > >>> > > > > > ExecutionEnvironment > >>> > > > > > > > and > >>> > > > > > > > > > > >> >> >> let > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> ExecutionEnvironment > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> create the proper > >>> > ClusterClient > >>> > > > > > instead > >>> > > > > > > > > of > >>> > > > > > > > > > > >> >> >>>>> letting > >>> > > > > > > > > > > >> >> >>>>>>> cli > >>> > > > > > > > > > > >> >> >>>>>>>> to > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> create > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> ClusterClient directly. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 6 would involve large > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> code > >>> > > > > > refactoring, > >>> > > > > > > > > so > >>> > > > > > > > > > > >> >> >> I > >>> > > > > > > > > > > >> >> >>>>> think > >>> > > > > > > > > > > >> >> >>>>>> we > >>> > > > > > > > > > > >> >> >>>>>>>> can > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> defer > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> it > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> for > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> future release, > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 1,2,3,4,5 > >>> > could > >>> > > > > be > >>> > > > > > done > >>> > > > > > > > > at > >>> > > > > > > > > > > >> >> >>>> once I > >>> > > > > > > > > > > >> >> >>>>>>>>>>> believe. > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> Let > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> me > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> know > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> your > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> comments and feedback, > >>> > thanks > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> -- > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Best Regards > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Jeff Zhang > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> -- > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Best Regards > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Jeff Zhang > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> -- > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Best Regards > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Jeff Zhang > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> -- > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> Best Regards > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> Jeff Zhang > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> -- > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> Best Regards > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>>> Jeff Zhang > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> -- > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> Best Regards > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> Jeff Zhang > >>> > > > > > > > > > > >> >> >>>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> -- > >>> > > > > > > > > > > >> >> >>>>>>>> Best Regards > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>>> Jeff Zhang > >>> > > > > > > > > > > >> >> >>>>>>>> > >>> > > > > > > > > > > >> >> >>>>>>> > >>> > > > > > > > > > > >> >> >>>>>> > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>>> -- > >>> > > > > > > > > > > >> >> >>>>> Best Regards > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>>> Jeff Zhang > >>> > > > > > > > > > > >> >> >>>>> > >>> > > > > > > > > > > >> >> >>>> > >>> > > > > > > > > > > >> >> >>> > >>> > > > > > > > > > > >> >> >> > >>> > > > > > > > > > > >> >> > > >>> > > > > > > > > > > >> >> > > >>> > > > > > > > > > > >> >> > -- > >>> > > > > > > > > > > >> >> > Best Regards > >>> > > > > > > > > > > >> >> > > >>> > > > > > > > > > > >> >> > Jeff Zhang > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > >>> > >>> > >>> -- > >>> Best Regards > >>> > >>> Jeff Zhang
