Hi Till, Thanks for your update. Nice to hear :-)
Best, tison. Till Rohrmann <trohrm...@apache.org> 于2019年8月23日周五 下午10:39写道： > Hi Tison, > > just a quick comment concerning the class loading issues when using the per > job mode. The community wants to change it so that the > StandaloneJobClusterEntryPoint actually uses the user code class loader > with child first class loading [1]. Hence, I hope that this problem will be > resolved soon. > > [1] https://issues.apache.org/jira/browse/FLINK-13840 > > Cheers, > Till > > On Fri, Aug 23, 2019 at 2:47 PM Kostas Kloudas <kklou...@gmail.com> wrote: > > > Hi all, > > > > On the topic of web submission, I agree with Till that it only seems > > to complicate things. > > It is bad for security, job isolation (anybody can submit/cancel jobs), > > and its > > implementation complicates some parts of the code. So, if it were to > > redesign the > > WebUI, maybe this part could be left out. In addition, I would say > > that the ability to cancel > > jobs could also be left out. > > > > Also I would also be in favour of removing the "detached" mode, for > > the reasons mentioned > > above (i.e. because now we will have a future representing the result > > on which the user > > can choose to wait or not). > > > > Now for the separating job submission and cluster creation, I am in > > favour of keeping both. > > Once again, the reasons are mentioned above by Stephan, Till, Aljoscha > > and also Zili seems > > to agree. They mainly have to do with security, isolation and ease of > > resource management > > for the user as he knows that "when my job is done, everything will be > > cleared up". This is > > also the experience you get when launching a process on your local OS. > > > > On excluding the per-job mode from returning a JobClient or not, I > > believe that eventually > > it would be nice to allow users to get back a jobClient. The reason is > > that 1) I cannot > > find any objective reason why the user-experience should diverge, and > > 2) this will be the > > way that the user will be able to interact with his running job. > > Assuming that the necessary > > ports are open for the REST API to work, then I think that the > > JobClient can run against the > > REST API without problems. If the needed ports are not open, then we > > are safe to not return > > a JobClient, as the user explicitly chose to close all points of > > communication to his running job. > > > > On the topic of not hijacking the "env.execute()" in order to get the > > Plan, I definitely agree but > > for the proposal of having a "compile()" method in the env, I would > > like to have a better look at > > the existing code. > > > > Cheers, > > Kostas > > > > On Fri, Aug 23, 2019 at 5:52 AM Zili Chen <wander4...@gmail.com> wrote: > > > > > > Hi Yang, > > > > > > It would be helpful if you check Stephan's last comment, > > > which states that isolation is important. > > > > > > For per-job mode, we run a dedicated cluster(maybe it > > > should have been a couple of JM and TMs during FLIP-6 > > > design) for a specific job. Thus the process is prevented > > > from other jobs. > > > > > > In our cases there was a time we suffered from multi > > > jobs submitted by different users and they affected > > > each other so that all ran into an error state. Also, > > > run the client inside the cluster could save client > > > resource at some points. > > > > > > However, we also face several issues as you mentioned, > > > that in per-job mode it always uses parent classloader > > > thus classloading issues occur. > > > > > > BTW, one can makes an analogy between session/per-job mode > > > in Flink, and client/cluster mode in Spark. > > > > > > Best, > > > tison. > > > > > > > > > Yang Wang <danrtsey...@gmail.com> 于2019年8月22日周四 上午11:25写道： > > > > > > > From the user's perspective, it is really confused about the scope of > > > > per-job cluster. > > > > > > > > > > > > If it means a flink cluster with single job, so that we could get > > better > > > > isolation. > > > > > > > > Now it does not matter how we deploy the cluster, directly > > deploy(mode1) > > > > > > > > or start a flink cluster and then submit job through cluster > > client(mode2). > > > > > > > > > > > > Otherwise, if it just means directly deploy, how should we name the > > mode2, > > > > > > > > session with job or something else? > > > > > > > > We could also benefit from the mode2. Users could get the same > > isolation > > > > with mode1. > > > > > > > > The user code and dependencies will be loaded by user class loader > > > > > > > > to avoid class conflict with framework. > > > > > > > > > > > > > > > > Anyway, both of the two submission modes are useful. > > > > > > > > We just need to clarify the concepts. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > Yang > > > > > > > > Zili Chen <wander4...@gmail.com> 于2019年8月20日周二 下午5:58写道： > > > > > > > > > Thanks for the clarification. > > > > > > > > > > The idea JobDeployer ever came into my mind when I was muddled with > > > > > how to execute per-job mode and session mode with the same user > code > > > > > and framework codepath. > > > > > > > > > > With the concept JobDeployer we back to the statement that > > environment > > > > > knows every configs of cluster deployment and job submission. We > > > > > configure or generate from configuration a specific JobDeployer in > > > > > environment and then code align on > > > > > > > > > > *JobClient client = env.execute().get();* > > > > > > > > > > which in session mode returned by clusterClient.submitJob and in > > per-job > > > > > mode returned by clusterDescriptor.deployJobCluster. > > > > > > > > > > Here comes a problem that currently we directly run > ClusterEntrypoint > > > > > with extracted job graph. Follow the JobDeployer way we'd better > > > > > align entry point of per-job deployment at JobDeployer. Users run > > > > > their main method or by a Cli(finally call main method) to deploy > the > > > > > job cluster. > > > > > > > > > > Best, > > > > > tison. > > > > > > > > > > > > > > > Stephan Ewen <se...@apache.org> 于2019年8月20日周二 下午4:40写道： > > > > > > > > > > > Till has made some good comments here. > > > > > > > > > > > > Two things to add: > > > > > > > > > > > > - The job mode is very nice in the way that it runs the client > > inside > > > > > the > > > > > > cluster (in the same image/process that is the JM) and thus > unifies > > > > both > > > > > > applications and what the Spark world calls the "driver mode". > > > > > > > > > > > > - Another thing I would add is that during the FLIP-6 design, > we > > were > > > > > > thinking about setups where Dispatcher and JobManager are > separate > > > > > > processes. > > > > > > A Yarn or Mesos Dispatcher of a session could run > independently > > > > (even > > > > > > as privileged processes executing no code). > > > > > > Then you the "per-job" mode could still be helpful: when a > job > > is > > > > > > submitted to the dispatcher, it launches the JM again in a > per-job > > > > mode, > > > > > so > > > > > > that JM and TM processes are bound to teh job only. For higher > > security > > > > > > setups, it is important that processes are not reused across > jobs. > > > > > > > > > > > > On Tue, Aug 20, 2019 at 10:27 AM Till Rohrmann < > > trohrm...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > I would not be in favour of getting rid of the per-job mode > > since it > > > > > > > simplifies the process of running Flink jobs considerably. > > Moreover, > > > > it > > > > > > is > > > > > > > not only well suited for container deployments but also for > > > > deployments > > > > > > > where you want to guarantee job isolation. For example, a user > > could > > > > > use > > > > > > > the per-job mode on Yarn to execute his job on a separate > > cluster. > > > > > > > > > > > > > > I think that having two notions of cluster deployments (session > > vs. > > > > > > per-job > > > > > > > mode) does not necessarily contradict your ideas for the client > > api > > > > > > > refactoring. For example one could have the following > interfaces: > > > > > > > > > > > > > > - ClusterDeploymentDescriptor: encapsulates the logic how to > > deploy a > > > > > > > cluster. > > > > > > > - ClusterClient: allows to interact with a cluster > > > > > > > - JobClient: allows to interact with a running job > > > > > > > > > > > > > > Now the ClusterDeploymentDescriptor could have two methods: > > > > > > > > > > > > > > - ClusterClient deploySessionCluster() > > > > > > > - JobClusterClient/JobClient deployPerJobCluster(JobGraph) > > > > > > > > > > > > > > where JobClusterClient is either a supertype of ClusterClient > > which > > > > > does > > > > > > > not give you the functionality to submit jobs or > > deployPerJobCluster > > > > > > > returns directly a JobClient. > > > > > > > > > > > > > > When setting up the ExecutionEnvironment, one would then not > > provide > > > > a > > > > > > > ClusterClient to submit jobs but a JobDeployer which, depending > > on > > > > the > > > > > > > selected mode, either uses a ClusterClient (session mode) to > > submit > > > > > jobs > > > > > > or > > > > > > > a ClusterDeploymentDescriptor to deploy per a job mode cluster > > with > > > > the > > > > > > job > > > > > > > to execute. > > > > > > > > > > > > > > These are just some thoughts how one could make it working > > because I > > > > > > > believe there is some value in using the per job mode from the > > > > > > > ExecutionEnvironment. > > > > > > > > > > > > > > Concerning the web submission, this is indeed a bit tricky. > From > > a > > > > > > cluster > > > > > > > management stand point, I would in favour of not executing user > > code > > > > on > > > > > > the > > > > > > > REST endpoint. Especially when considering security, it would > be > > good > > > > > to > > > > > > > have a well defined cluster behaviour where it is explicitly > > stated > > > > > where > > > > > > > user code and, thus, potentially risky code is executed. > Ideally > > we > > > > > limit > > > > > > > it to the TaskExecutor and JobMaster. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Tue, Aug 20, 2019 at 9:40 AM Flavio Pompermaier < > > > > > pomperma...@okkam.it > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > In my opinion the client should not use any environment to > get > > the > > > > > Job > > > > > > > > graph because the jar should reside ONLY on the cluster (and > > not in > > > > > the > > > > > > > > client classpath otherwise there are always inconsistencies > > between > > > > > > > client > > > > > > > > and Flink Job manager's classpath). > > > > > > > > In the YARN, Mesos and Kubernetes scenarios you have the jar > > but > > > > you > > > > > > > could > > > > > > > > start a cluster that has the jar on the Job Manager as well > > (but > > > > this > > > > > > is > > > > > > > > the only case where I think you can assume that the client > has > > the > > > > > jar > > > > > > on > > > > > > > > the classpath..in the REST job submission you don't have any > > > > > > classpath). > > > > > > > > > > > > > > > > Thus, always in my opinion, the JobGraph should be generated > > by the > > > > > Job > > > > > > > > Manager REST API. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 20, 2019 at 9:00 AM Zili Chen < > > wander4...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > >> I would like to involve Till & Stephan here to clarify some > > > > concept > > > > > of > > > > > > > >> per-job mode. > > > > > > > >> > > > > > > > >> The term per-job is one of modes a cluster could run on. It > is > > > > > mainly > > > > > > > >> aimed > > > > > > > >> at spawn > > > > > > > >> a dedicated cluster for a specific job while the job could > be > > > > > packaged > > > > > > > >> with > > > > > > > >> Flink > > > > > > > >> itself and thus the cluster initialized with job so that get > > rid > > > > of > > > > > a > > > > > > > >> separated > > > > > > > >> submission step. > > > > > > > >> > > > > > > > >> This is useful for container deployments where one create > his > > > > image > > > > > > with > > > > > > > >> the job > > > > > > > >> and then simply deploy the container. > > > > > > > >> > > > > > > > >> However, it is out of client scope since a > > client(ClusterClient > > > > for > > > > > > > >> example) is for > > > > > > > >> communicate with an existing cluster and performance > actions. > > > > > > Currently, > > > > > > > >> in > > > > > > > >> per-job > > > > > > > >> mode, we extract the job graph and bundle it into cluster > > > > deployment > > > > > > and > > > > > > > >> thus no > > > > > > > >> concept of client get involved. It looks like reasonable to > > > > exclude > > > > > > the > > > > > > > >> deployment > > > > > > > >> of per-job cluster from client api and use dedicated utility > > > > > > > >> classes(deployers) for > > > > > > > >> deployment. > > > > > > > >> > > > > > > > >> Zili Chen <wander4...@gmail.com> 于2019年8月20日周二 下午12:37写道： > > > > > > > >> > > > > > > > >> > Hi Aljoscha, > > > > > > > >> > > > > > > > > >> > Thanks for your reply and participance. The Google Doc you > > > > linked > > > > > to > > > > > > > >> > requires > > > > > > > >> > permission and I think you could use a share link instead. > > > > > > > >> > > > > > > > > >> > I agree with that we almost reach a consensus that > > JobClient is > > > > > > > >> necessary > > > > > > > >> > to > > > > > > > >> > interacte with a running Job. > > > > > > > >> > > > > > > > > >> > Let me check your open questions one by one. > > > > > > > >> > > > > > > > > >> > 1. Separate cluster creation and job submission for > per-job > > > > mode. > > > > > > > >> > > > > > > > > >> > As you mentioned here is where the opinions diverge. In my > > > > > document > > > > > > > >> there > > > > > > > >> > is > > > > > > > >> > an alternative[2] that proposes excluding per-job > deployment > > > > from > > > > > > > client > > > > > > > >> > api > > > > > > > >> > scope and now I find it is more reasonable we do the > > exclusion. > > > > > > > >> > > > > > > > > >> > When in per-job mode, a dedicated JobCluster is launched > to > > > > > execute > > > > > > > the > > > > > > > >> > specific job. It is like a Flink Application more than a > > > > > submission > > > > > > > >> > of Flink Job. Client only takes care of job submission and > > > > assume > > > > > > > there > > > > > > > >> is > > > > > > > >> > an existing cluster. In this way we are able to consider > > per-job > > > > > > > issues > > > > > > > >> > individually and JobClusterEntrypoint would be the utility > > class > > > > > for > > > > > > > >> > per-job > > > > > > > >> > deployment. > > > > > > > >> > > > > > > > > >> > Nevertheless, user program works in both session mode and > > > > per-job > > > > > > mode > > > > > > > >> > without > > > > > > > >> > necessary to change code. JobClient in per-job mode is > > returned > > > > > from > > > > > > > >> > env.execute as normal. However, it would be no longer a > > wrapper > > > > of > > > > > > > >> > RestClusterClient but a wrapper of PerJobClusterClient > which > > > > > > > >> communicates > > > > > > > >> > to Dispatcher locally. > > > > > > > >> > > > > > > > > >> > 2. How to deal with plan preview. > > > > > > > >> > > > > > > > > >> > With env.compile functions users can get JobGraph or > > FlinkPlan > > > > and > > > > > > > thus > > > > > > > >> > they can preview the plan with programming. Typically it > > looks > > > > > like > > > > > > > >> > > > > > > > > >> > if (preview configured) { > > > > > > > >> > FlinkPlan plan = env.compile(); > > > > > > > >> > new JSONDumpGenerator(...).dump(plan); > > > > > > > >> > } else { > > > > > > > >> > env.execute(); > > > > > > > >> > } > > > > > > > >> > > > > > > > > >> > And `flink info` would be invalid any more. > > > > > > > >> > > > > > > > > >> > 3. How to deal with Jar Submission at the Web Frontend. > > > > > > > >> > > > > > > > > >> > There is one more thread talked on this topic[1]. Apart > from > > > > > > removing > > > > > > > >> > the functions there are two alternatives. > > > > > > > >> > > > > > > > > >> > One is to introduce an interface has a method returns > > > > > > > JobGraph/FilnkPlan > > > > > > > >> > and Jar Submission only support main-class implements this > > > > > > interface. > > > > > > > >> > And then extract the JobGraph/FlinkPlan just by calling > the > > > > > method. > > > > > > > >> > In this way, it is even possible to consider a separation > > of job > > > > > > > >> creation > > > > > > > >> > and job submission. > > > > > > > >> > > > > > > > > >> > The other is, as you mentioned, let execute() do the > actual > > > > > > execution. > > > > > > > >> > We won't execute the main method in the WebFrontend but > > spawn a > > > > > > > process > > > > > > > >> > at WebMonitor side to execute. For return part we could > > generate > > > > > the > > > > > > > >> > JobID from WebMonitor and pass it to the execution > > environemnt. > > > > > > > >> > > > > > > > > >> > 4. How to deal with detached mode. > > > > > > > >> > > > > > > > > >> > I think detached mode is a temporary solution for > > non-blocking > > > > > > > >> submission. > > > > > > > >> > In my document both submission and execution return a > > > > > > > CompletableFuture > > > > > > > >> and > > > > > > > >> > users control whether or not wait for the result. In this > > point > > > > we > > > > > > > don't > > > > > > > >> > need a detached option but the functionality is covered. > > > > > > > >> > > > > > > > > >> > 5. How does per-job mode interact with interactive > > programming. > > > > > > > >> > > > > > > > > >> > All of YARN, Mesos and Kubernetes scenarios follow the > > pattern > > > > > > launch > > > > > > > a > > > > > > > >> > JobCluster now. And I don't think there would be > > inconsistency > > > > > > between > > > > > > > >> > different resource management. > > > > > > > >> > > > > > > > > >> > Best, > > > > > > > >> > tison. > > > > > > > >> > > > > > > > > >> > [1] > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/x/thread.html/6db869c53816f4e2917949a7c6992c2b90856d7d639d7f2e1cd13768@%3Cdev.flink.apache.org%3E > > > > > > > >> > [2] > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1UWJE7eYWiMuZewBKS0YmdVO2LUTqXPd6-pbOCof9ddY/edit?disco=AAAADZaGGfs > > > > > > > >> > > > > > > > > >> > Aljoscha Krettek <aljos...@apache.org> 于2019年8月16日周五 > > 下午9:20写道： > > > > > > > >> > > > > > > > > >> >> Hi, > > > > > > > >> >> > > > > > > > >> >> I read both Jeffs initial design document and the newer > > > > document > > > > > by > > > > > > > >> >> Tison. I also finally found the time to collect our > > thoughts on > > > > > the > > > > > > > >> issue, > > > > > > > >> >> I had quite some discussions with Kostas and this is the > > > > result: > > > > > > [1]. > > > > > > > >> >> > > > > > > > >> >> I think overall we agree that this part of the code is in > > dire > > > > > need > > > > > > > of > > > > > > > >> >> some refactoring/improvements but I think there are still > > some > > > > > open > > > > > > > >> >> questions and some differences in opinion what those > > > > refactorings > > > > > > > >> should > > > > > > > >> >> look like. > > > > > > > >> >> > > > > > > > >> >> I think the API-side is quite clear, i.e. we need some > > > > JobClient > > > > > > API > > > > > > > >> that > > > > > > > >> >> allows interacting with a running Job. It could be > > worthwhile > > > > to > > > > > > spin > > > > > > > >> that > > > > > > > >> >> off into a separate FLIP because we can probably find > > consensus > > > > > on > > > > > > > that > > > > > > > >> >> part more easily. > > > > > > > >> >> > > > > > > > >> >> For the rest, the main open questions from our doc are > > these: > > > > > > > >> >> > > > > > > > >> >> - Do we want to separate cluster creation and job > > submission > > > > > for > > > > > > > >> >> per-job mode? In the past, there were conscious efforts > to > > > > *not* > > > > > > > >> separate > > > > > > > >> >> job submission from cluster creation for per-job clusters > > for > > > > > > Mesos, > > > > > > > >> YARN, > > > > > > > >> >> Kubernets (see StandaloneJobClusterEntryPoint). Tison > > suggests > > > > in > > > > > > his > > > > > > > >> >> design document to decouple this in order to unify job > > > > > submission. > > > > > > > >> >> > > > > > > > >> >> - How to deal with plan preview, which needs to hijack > > > > > execute() > > > > > > > and > > > > > > > >> >> let the outside code catch an exception? > > > > > > > >> >> > > > > > > > >> >> - How to deal with Jar Submission at the Web Frontend, > > which > > > > > > needs > > > > > > > to > > > > > > > >> >> hijack execute() and let the outside code catch an > > exception? > > > > > > > >> >> CliFrontend.run() “hijacks” > ExecutionEnvironment.execute() > > to > > > > > get a > > > > > > > >> >> JobGraph and then execute that JobGraph manually. We > could > > get > > > > > > around > > > > > > > >> that > > > > > > > >> >> by letting execute() do the actual execution. One caveat > > for > > > > this > > > > > > is > > > > > > > >> that > > > > > > > >> >> now the main() method doesn’t return (or is forced to > > return by > > > > > > > >> throwing an > > > > > > > >> >> exception from execute()) which means that for Jar > > Submission > > > > > from > > > > > > > the > > > > > > > >> >> WebFrontend we have a long-running main() method running > > in the > > > > > > > >> >> WebFrontend. This doesn’t sound very good. We could get > > around > > > > > this > > > > > > > by > > > > > > > >> >> removing the plan preview feature and by removing Jar > > > > > > > >> Submission/Running. > > > > > > > >> >> > > > > > > > >> >> - How to deal with detached mode? Right now, > > > > > DetachedEnvironment > > > > > > > will > > > > > > > >> >> execute the job and return immediately. If users control > > when > > > > > they > > > > > > > >> want to > > > > > > > >> >> return, by waiting on the job completion future, how do > we > > deal > > > > > > with > > > > > > > >> this? > > > > > > > >> >> Do we simply remove the distinction between > > > > > detached/non-detached? > > > > > > > >> >> > > > > > > > >> >> - How does per-job mode interact with “interactive > > > > programming” > > > > > > > >> >> (FLIP-36). For YARN, each execute() call could spawn a > new > > > > Flink > > > > > > YARN > > > > > > > >> >> cluster. What about Mesos and Kubernetes? > > > > > > > >> >> > > > > > > > >> >> The first open question is where the opinions diverge, I > > think. > > > > > The > > > > > > > >> rest > > > > > > > >> >> are just open questions and interesting things that we > > need to > > > > > > > >> consider. > > > > > > > >> >> > > > > > > > >> >> Best, > > > > > > > >> >> Aljoscha > > > > > > > >> >> > > > > > > > >> >> [1] > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1E-8UjOLz4QPUTxetGWbU23OlsIH9VIdodpTsxwoQTs0/edit#heading=h.na7k0ad88tix > > > > > > > >> >> < > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1E-8UjOLz4QPUTxetGWbU23OlsIH9VIdodpTsxwoQTs0/edit#heading=h.na7k0ad88tix > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > On 31. Jul 2019, at 15:23, Jeff Zhang < > zjf...@gmail.com> > > > > > wrote: > > > > > > > >> >> > > > > > > > > >> >> > Thanks tison for the effort. I left a few comments. > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > Zili Chen <wander4...@gmail.com> 于2019年7月31日周三 > 下午8:24写道： > > > > > > > >> >> > > > > > > > > >> >> >> Hi Flavio, > > > > > > > >> >> >> > > > > > > > >> >> >> Thanks for your reply. > > > > > > > >> >> >> > > > > > > > >> >> >> Either current impl and in the design, ClusterClient > > > > > > > >> >> >> never takes responsibility for generating JobGraph. > > > > > > > >> >> >> (what you see in current codebase is several class > > methods) > > > > > > > >> >> >> > > > > > > > >> >> >> Instead, user describes his program in the main method > > > > > > > >> >> >> with ExecutionEnvironment apis and calls env.compile() > > > > > > > >> >> >> or env.optimize() to get FlinkPlan and JobGraph > > > > respectively. > > > > > > > >> >> >> > > > > > > > >> >> >> For listing main classes in a jar and choose one for > > > > > > > >> >> >> submission, you're now able to customize a CLI to do > it. > > > > > > > >> >> >> Specifically, the path of jar is passed as arguments > and > > > > > > > >> >> >> in the customized CLI you list main classes, choose > one > > > > > > > >> >> >> to submit to the cluster. > > > > > > > >> >> >> > > > > > > > >> >> >> Best, > > > > > > > >> >> >> tison. > > > > > > > >> >> >> > > > > > > > >> >> >> > > > > > > > >> >> >> Flavio Pompermaier <pomperma...@okkam.it> > 于2019年7月31日周三 > > > > > > 下午8:12写道： > > > > > > > >> >> >> > > > > > > > >> >> >>> Just one note on my side: it is not clear to me > > whether the > > > > > > > client > > > > > > > >> >> needs > > > > > > > >> >> >> to > > > > > > > >> >> >>> be able to generate a job graph or not. > > > > > > > >> >> >>> In my opinion, the job jar must resides only on the > > > > > > > >> server/jobManager > > > > > > > >> >> >> side > > > > > > > >> >> >>> and the client requires a way to get the job graph. > > > > > > > >> >> >>> If you really want to access to the job graph, I'd > add > > a > > > > > > > dedicated > > > > > > > >> >> method > > > > > > > >> >> >>> on the ClusterClient. like: > > > > > > > >> >> >>> > > > > > > > >> >> >>> - getJobGraph(jarId, mainClass): JobGraph > > > > > > > >> >> >>> - listMainClasses(jarId): List<String> > > > > > > > >> >> >>> > > > > > > > >> >> >>> These would require some addition also on the job > > manager > > > > > > > endpoint > > > > > > > >> as > > > > > > > >> >> >>> well..what do you think? > > > > > > > >> >> >>> > > > > > > > >> >> >>> On Wed, Jul 31, 2019 at 12:42 PM Zili Chen < > > > > > > wander4...@gmail.com > > > > > > > > > > > > > > > >> >> wrote: > > > > > > > >> >> >>> > > > > > > > >> >> >>>> Hi all, > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> Here is a document[1] on client api enhancement from > > our > > > > > > > >> perspective. > > > > > > > >> >> >>>> We have investigated current implementations. And we > > > > propose > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> 1. Unify the implementation of cluster deployment > and > > job > > > > > > > >> submission > > > > > > > >> >> in > > > > > > > >> >> >>>> Flink. > > > > > > > >> >> >>>> 2. Provide programmatic interfaces to allow flexible > > job > > > > and > > > > > > > >> cluster > > > > > > > >> >> >>>> management. > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> The first proposal is aimed at reducing code paths > of > > > > > cluster > > > > > > > >> >> >> deployment > > > > > > > >> >> >>>> and > > > > > > > >> >> >>>> job submission so that one can adopt Flink in his > > usage > > > > > > easily. > > > > > > > >> The > > > > > > > >> >> >>> second > > > > > > > >> >> >>>> proposal is aimed at providing rich interfaces for > > > > advanced > > > > > > > users > > > > > > > >> >> >>>> who want to make accurate control of these stages. > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> Quick reference on open questions: > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> 1. Exclude job cluster deployment from client side > or > > > > > redefine > > > > > > > the > > > > > > > >> >> >>> semantic > > > > > > > >> >> >>>> of job cluster? Since it fits in a process quite > > different > > > > > > from > > > > > > > >> >> session > > > > > > > >> >> >>>> cluster deployment and job submission. > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> 2. Maintain the codepaths handling class > > > > > > > o.a.f.api.common.Program > > > > > > > >> or > > > > > > > >> >> >>>> implement customized program handling logic by > > customized > > > > > > > >> >> CliFrontend? > > > > > > > >> >> >>>> See also this thread[2] and the document[1]. > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> 3. Expose ClusterClient as public api or just expose > > api > > > > in > > > > > > > >> >> >>>> ExecutionEnvironment > > > > > > > >> >> >>>> and delegate them to ClusterClient? Further, in > > either way > > > > > is > > > > > > it > > > > > > > >> >> worth > > > > > > > >> >> >> to > > > > > > > >> >> >>>> introduce a JobClient which is an encapsulation of > > > > > > ClusterClient > > > > > > > >> that > > > > > > > >> >> >>>> associated to specific job? > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> Best, > > > > > > > >> >> >>>> tison. > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> [1] > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> > > > > > > > >> >> >>> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1UWJE7eYWiMuZewBKS0YmdVO2LUTqXPd6-pbOCof9ddY/edit?usp=sharing > > > > > > > >> >> >>>> [2] > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> > > > > > > > >> >> >>> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/7ffc9936a384b891dbcf0a481d26c6d13b2125607c200577780d1e18@%3Cdev.flink.apache.org%3E > > > > > > > >> >> >>>> > > > > > > > >> >> >>>> Jeff Zhang <zjf...@gmail.com> 于2019年7月24日周三 > 上午9:19写道： > > > > > > > >> >> >>>> > > > > > > > >> >> >>>>> Thanks Stephan, I will follow up this issue in next > > few > > > > > > weeks, > > > > > > > >> and > > > > > > > >> >> >> will > > > > > > > >> >> >>>>> refine the design doc. We could discuss more > details > > > > after > > > > > > 1.9 > > > > > > > >> >> >> release. > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>>> Stephan Ewen <se...@apache.org> 于2019年7月24日周三 > > 上午12:58写道： > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>>>> Hi all! > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>>> This thread has stalled for a bit, which I assume > > ist > > > > > mostly > > > > > > > >> due to > > > > > > > >> >> >>> the > > > > > > > >> >> >>>>>> Flink 1.9 feature freeze and release testing > effort. > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>>> I personally still recognize this issue as one > > important > > > > > to > > > > > > be > > > > > > > >> >> >>> solved. > > > > > > > >> >> >>>>> I'd > > > > > > > >> >> >>>>>> be happy to help resume this discussion soon > (after > > the > > > > > 1.9 > > > > > > > >> >> >> release) > > > > > > > >> >> >>>> and > > > > > > > >> >> >>>>>> see if we can do some step towards this in Flink > > 1.10. > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>>> Best, > > > > > > > >> >> >>>>>> Stephan > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>>> On Mon, Jun 24, 2019 at 10:41 AM Flavio > Pompermaier > > < > > > > > > > >> >> >>>>> pomperma...@okkam.it> > > > > > > > >> >> >>>>>> wrote: > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>>>> That's exactly what I suggested a long time ago: > > the > > > > > Flink > > > > > > > REST > > > > > > > >> >> >>>> client > > > > > > > >> >> >>>>>>> should not require any Flink dependency, only > http > > > > > library > > > > > > to > > > > > > > >> >> >> call > > > > > > > >> >> >>>> the > > > > > > > >> >> >>>>>> REST > > > > > > > >> >> >>>>>>> services to submit and monitor a job. > > > > > > > >> >> >>>>>>> What I suggested also in [1] was to have a way to > > > > > > > automatically > > > > > > > >> >> >>>> suggest > > > > > > > >> >> >>>>>> the > > > > > > > >> >> >>>>>>> user (via a UI) the available main classes and > > their > > > > > > required > > > > > > > >> >> >>>>>>> parameters[2]. > > > > > > > >> >> >>>>>>> Another problem we have with Flink is that the > Rest > > > > > client > > > > > > > and > > > > > > > >> >> >> the > > > > > > > >> >> >>>> CLI > > > > > > > >> >> >>>>>> one > > > > > > > >> >> >>>>>>> behaves differently and we use the CLI client > (via > > ssh) > > > > > > > because > > > > > > > >> >> >> it > > > > > > > >> >> >>>>> allows > > > > > > > >> >> >>>>>>> to call some other method after env.execute() [3] > > (we > > > > > have > > > > > > to > > > > > > > >> >> >> call > > > > > > > >> >> >>>>>> another > > > > > > > >> >> >>>>>>> REST service to signal the end of the job). > > > > > > > >> >> >>>>>>> Int his regard, a dedicated interface, like the > > > > > JobListener > > > > > > > >> >> >>> suggested > > > > > > > >> >> >>>>> in > > > > > > > >> >> >>>>>>> the previous emails, would be very helpful > (IMHO). > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>>> [1] > > https://issues.apache.org/jira/browse/FLINK-10864 > > > > > > > >> >> >>>>>>> [2] > > https://issues.apache.org/jira/browse/FLINK-10862 > > > > > > > >> >> >>>>>>> [3] > > https://issues.apache.org/jira/browse/FLINK-10879 > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>>> Best, > > > > > > > >> >> >>>>>>> Flavio > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>>> On Mon, Jun 24, 2019 at 9:54 AM Jeff Zhang < > > > > > > zjf...@gmail.com > > > > > > > > > > > > > > > >> >> >>> wrote: > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>>>> Hi, Tison, > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>> Thanks for your comments. Overall I agree with > you > > > > that > > > > > it > > > > > > > is > > > > > > > >> >> >>>>> difficult > > > > > > > >> >> >>>>>>> for > > > > > > > >> >> >>>>>>>> down stream project to integrate with flink and > we > > > > need > > > > > to > > > > > > > >> >> >>> refactor > > > > > > > >> >> >>>>> the > > > > > > > >> >> >>>>>>>> current flink client api. > > > > > > > >> >> >>>>>>>> And I agree that CliFrontend should only parsing > > > > command > > > > > > > line > > > > > > > >> >> >>>>> arguments > > > > > > > >> >> >>>>>>> and > > > > > > > >> >> >>>>>>>> then pass them to ExecutionEnvironment. It is > > > > > > > >> >> >>>> ExecutionEnvironment's > > > > > > > >> >> >>>>>>>> responsibility to compile job, create cluster, > and > > > > > submit > > > > > > > job. > > > > > > > >> >> >>>>> Besides > > > > > > > >> >> >>>>>>>> that, Currently flink has many > > ExecutionEnvironment > > > > > > > >> >> >>>> implementations, > > > > > > > >> >> >>>>>> and > > > > > > > >> >> >>>>>>>> flink will use the specific one based on the > > context. > > > > > > IMHO, > > > > > > > it > > > > > > > >> >> >> is > > > > > > > >> >> >>>> not > > > > > > > >> >> >>>>>>>> necessary, ExecutionEnvironment should be able > to > > do > > > > the > > > > > > > right > > > > > > > >> >> >>>> thing > > > > > > > >> >> >>>>>>> based > > > > > > > >> >> >>>>>>>> on the FlinkConf it is received. Too many > > > > > > > ExecutionEnvironment > > > > > > > >> >> >>>>>>>> implementation is another burden for downstream > > > > project > > > > > > > >> >> >>>> integration. > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>> One thing I'd like to mention is flink's scala > > shell > > > > and > > > > > > sql > > > > > > > >> >> >>>> client, > > > > > > > >> >> >>>>>>>> although they are sub-modules of flink, they > > could be > > > > > > > treated > > > > > > > >> >> >> as > > > > > > > >> >> >>>>>>> downstream > > > > > > > >> >> >>>>>>>> project which use flink's client api. Currently > > you > > > > will > > > > > > > find > > > > > > > >> >> >> it > > > > > > > >> >> >>> is > > > > > > > >> >> >>>>> not > > > > > > > >> >> >>>>>>>> easy for them to integrate with flink, they > share > > many > > > > > > > >> >> >> duplicated > > > > > > > >> >> >>>>> code. > > > > > > > >> >> >>>>>>> It > > > > > > > >> >> >>>>>>>> is another sign that we should refactor flink > > client > > > > > api. > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>> I believe it is a large and hard change, and I > am > > > > afraid > > > > > > we > > > > > > > >> can > > > > > > > >> >> >>> not > > > > > > > >> >> >>>>>> keep > > > > > > > >> >> >>>>>>>> compatibility since many of changes are user > > facing. > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>> Zili Chen <wander4...@gmail.com> 于2019年6月24日周一 > > > > > 下午2:53写道： > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>>> Hi all, > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> After a closer look on our client apis, I can > see > > > > there > > > > > > are > > > > > > > >> >> >> two > > > > > > > >> >> >>>>> major > > > > > > > >> >> >>>>>>>>> issues to consistency and integration, namely > > > > different > > > > > > > >> >> >>>> deployment > > > > > > > >> >> >>>>> of > > > > > > > >> >> >>>>>>>>> job cluster which couples job graph creation > and > > > > > cluster > > > > > > > >> >> >>>>> deployment, > > > > > > > >> >> >>>>>>>>> and submission via CliFrontend confusing > control > > flow > > > > > of > > > > > > > job > > > > > > > >> >> >>>> graph > > > > > > > >> >> >>>>>>>>> compilation and job submission. I'd like to > > follow > > > > the > > > > > > > >> >> >> discuss > > > > > > > >> >> >>>>> above, > > > > > > > >> >> >>>>>>>>> mainly the process described by Jeff and > > Stephan, and > > > > > > share > > > > > > > >> >> >> my > > > > > > > >> >> >>>>>>>>> ideas on these issues. > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> 1) CliFrontend confuses the control flow of job > > > > > > compilation > > > > > > > >> >> >> and > > > > > > > >> >> >>>>>>>> submission. > > > > > > > >> >> >>>>>>>>> Following the process of job submission Stephan > > and > > > > > Jeff > > > > > > > >> >> >>>> described, > > > > > > > >> >> >>>>>>>>> execution environment knows all configs of the > > > > cluster > > > > > > and > > > > > > > >> >> >>>>>>> topos/settings > > > > > > > >> >> >>>>>>>>> of the job. Ideally, in the main method of user > > > > > program, > > > > > > it > > > > > > > >> >> >>> calls > > > > > > > >> >> >>>>>>>> #execute > > > > > > > >> >> >>>>>>>>> (or named #submit) and Flink deploys the > cluster, > > > > > compile > > > > > > > the > > > > > > > >> >> >>> job > > > > > > > >> >> >>>>>> graph > > > > > > > >> >> >>>>>>>>> and submit it to the cluster. However, current > > > > > > CliFrontend > > > > > > > >> >> >> does > > > > > > > >> >> >>>> all > > > > > > > >> >> >>>>>>> these > > > > > > > >> >> >>>>>>>>> things inside its #runProgram method, which > > > > introduces > > > > > a > > > > > > > lot > > > > > > > >> >> >> of > > > > > > > >> >> >>>>>>>> subclasses > > > > > > > >> >> >>>>>>>>> of (stream) execution environment. > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> Actually, it sets up an exec env that hijacks > the > > > > > > > >> >> >>>>>> #execute/executePlan > > > > > > > >> >> >>>>>>>>> method, initializes the job graph and abort > > > > execution. > > > > > > And > > > > > > > >> >> >> then > > > > > > > >> >> >>>>>>>>> control flow back to CliFrontend, it deploys > the > > > > > > cluster(or > > > > > > > >> >> >>>>> retrieve > > > > > > > >> >> >>>>>>>>> the client) and submits the job graph. This is > > quite > > > > a > > > > > > > >> >> >> specific > > > > > > > >> >> >>>>>>> internal > > > > > > > >> >> >>>>>>>>> process inside Flink and none of consistency to > > > > > anything. > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> 2) Deployment of job cluster couples job graph > > > > creation > > > > > > and > > > > > > > >> >> >>>> cluster > > > > > > > >> >> >>>>>>>>> deployment. Abstractly, from user job to a > > concrete > > > > > > > >> >> >> submission, > > > > > > > >> >> >>>> it > > > > > > > >> >> >>>>>>>> requires > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> create JobGraph --\ > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> create ClusterClient --> submit JobGraph > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> such a dependency. ClusterClient was created by > > > > > deploying > > > > > > > or > > > > > > > >> >> >>>>>>> retrieving. > > > > > > > >> >> >>>>>>>>> JobGraph submission requires a compiled > JobGraph > > and > > > > > > valid > > > > > > > >> >> >>>>>>> ClusterClient, > > > > > > > >> >> >>>>>>>>> but the creation of ClusterClient is abstractly > > > > > > independent > > > > > > > >> >> >> of > > > > > > > >> >> >>>> that > > > > > > > >> >> >>>>>> of > > > > > > > >> >> >>>>>>>>> JobGraph. However, in job cluster mode, we > > deploy job > > > > > > > cluster > > > > > > > >> >> >>>> with > > > > > > > >> >> >>>>> a > > > > > > > >> >> >>>>>>> job > > > > > > > >> >> >>>>>>>>> graph, which means we use another process: > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> create JobGraph --> deploy cluster with the > > JobGraph > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> Here is another inconsistency and downstream > > > > > > > projects/client > > > > > > > >> >> >>> apis > > > > > > > >> >> >>>>> are > > > > > > > >> >> >>>>>>>>> forced to handle different cases with rare > > supports > > > > > from > > > > > > > >> >> >> Flink. > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> Since we likely reached a consensus on > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> 1. all configs gathered by Flink configuration > > and > > > > > passed > > > > > > > >> >> >>>>>>>>> 2. execution environment knows all configs and > > > > handles > > > > > > > >> >> >>>>> execution(both > > > > > > > >> >> >>>>>>>>> deployment and submission) > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> to the issues above I propose eliminating > > > > > inconsistencies > > > > > > > by > > > > > > > >> >> >>>>>> following > > > > > > > >> >> >>>>>>>>> approach: > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> 1) CliFrontend should exactly be a front end, > at > > > > least > > > > > > for > > > > > > > >> >> >>> "run" > > > > > > > >> >> >>>>>>> command. > > > > > > > >> >> >>>>>>>>> That means it just gathered and passed all > config > > > > from > > > > > > > >> >> >> command > > > > > > > >> >> >>>> line > > > > > > > >> >> >>>>>> to > > > > > > > >> >> >>>>>>>>> the main method of user program. Execution > > > > environment > > > > > > > knows > > > > > > > >> >> >>> all > > > > > > > >> >> >>>>> the > > > > > > > >> >> >>>>>>> info > > > > > > > >> >> >>>>>>>>> and with an addition to utils for > ClusterClient, > > we > > > > > > > >> >> >> gracefully > > > > > > > >> >> >>>> get > > > > > > > >> >> >>>>> a > > > > > > > >> >> >>>>>>>>> ClusterClient by deploying or retrieving. In > this > > > > way, > > > > > we > > > > > > > >> >> >> don't > > > > > > > >> >> >>>>> need > > > > > > > >> >> >>>>>> to > > > > > > > >> >> >>>>>>>>> hijack #execute/executePlan methods and can > > remove > > > > > > various > > > > > > > >> >> >>>> hacking > > > > > > > >> >> >>>>>>>>> subclasses of exec env, as well as #run methods > > in > > > > > > > >> >> >>>>> ClusterClient(for > > > > > > > >> >> >>>>>> an > > > > > > > >> >> >>>>>>>>> interface-ized ClusterClient). Now the control > > flow > > > > > flows > > > > > > > >> >> >> from > > > > > > > >> >> >>>>>>>> CliFrontend > > > > > > > >> >> >>>>>>>>> to the main method and never returns. > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> 2) Job cluster means a cluster for the specific > > job. > > > > > From > > > > > > > >> >> >>> another > > > > > > > >> >> >>>>>>>>> perspective, it is an ephemeral session. We may > > > > > decouple > > > > > > > the > > > > > > > >> >> >>>>>> deployment > > > > > > > >> >> >>>>>>>>> with a compiled job graph, but start a session > > with > > > > > idle > > > > > > > >> >> >>> timeout > > > > > > > >> >> >>>>>>>>> and submit the job following. > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> These topics, before we go into more details on > > > > design > > > > > or > > > > > > > >> >> >>>>>>> implementation, > > > > > > > >> >> >>>>>>>>> are better to be aware and discussed for a > > consensus. > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> Best, > > > > > > > >> >> >>>>>>>>> tison. > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>> Zili Chen <wander4...@gmail.com> 于2019年6月20日周四 > > > > > 上午3:21写道： > > > > > > > >> >> >>>>>>>>> > > > > > > > >> >> >>>>>>>>>> Hi Jeff, > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>>>> Thanks for raising this thread and the design > > > > > document! > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>>>> As @Thomas Weise mentioned above, extending > > config > > > > to > > > > > > > flink > > > > > > > >> >> >>>>>>>>>> requires far more effort than it should be. > > Another > > > > > > > example > > > > > > > >> >> >>>>>>>>>> is we achieve detach mode by introduce another > > > > > execution > > > > > > > >> >> >>>>>>>>>> environment which also hijack #execute method. > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>>>> I agree with your idea that user would > > configure all > > > > > > > things > > > > > > > >> >> >>>>>>>>>> and flink "just" respect it. On this topic I > > think > > > > the > > > > > > > >> >> >> unusual > > > > > > > >> >> >>>>>>>>>> control flow when CliFrontend handle "run" > > command > > > > is > > > > > > the > > > > > > > >> >> >>>> problem. > > > > > > > >> >> >>>>>>>>>> It handles several configs, mainly about > cluster > > > > > > settings, > > > > > > > >> >> >> and > > > > > > > >> >> >>>>>>>>>> thus main method of user program is unaware of > > them. > > > > > > Also > > > > > > > it > > > > > > > >> >> >>>>>> compiles > > > > > > > >> >> >>>>>>>>>> app to job graph by run the main method with a > > > > > hijacked > > > > > > > exec > > > > > > > >> >> >>>> env, > > > > > > > >> >> >>>>>>>>>> which constrain the main method further. > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>>>> I'd like to write down a few of notes on > > > > configs/args > > > > > > pass > > > > > > > >> >> >> and > > > > > > > >> >> >>>>>>> respect, > > > > > > > >> >> >>>>>>>>>> as well as decoupling job compilation and > > > > submission. > > > > > > > Share > > > > > > > >> >> >> on > > > > > > > >> >> >>>>> this > > > > > > > >> >> >>>>>>>>>> thread later. > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>>>> Best, > > > > > > > >> >> >>>>>>>>>> tison. > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>>>> SHI Xiaogang <shixiaoga...@gmail.com> > > 于2019年6月17日周一 > > > > > > > >> >> >> 下午7:29写道： > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> Hi Jeff and Flavio, > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> Thanks Jeff a lot for proposing the design > > > > document. > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> We are also working on refactoring > > ClusterClient to > > > > > > allow > > > > > > > >> >> >>>>> flexible > > > > > > > >> >> >>>>>>> and > > > > > > > >> >> >>>>>>>>>>> efficient job management in our real-time > > platform. > > > > > > > >> >> >>>>>>>>>>> We would like to draft a document to share > our > > > > ideas > > > > > > with > > > > > > > >> >> >>> you. > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> I think it's a good idea to have something > like > > > > > Apache > > > > > > > Livy > > > > > > > >> >> >>> for > > > > > > > >> >> >>>>>>> Flink, > > > > > > > >> >> >>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>> the efforts discussed here will take a great > > step > > > > > > forward > > > > > > > >> >> >> to > > > > > > > >> >> >>>> it. > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> Regards, > > > > > > > >> >> >>>>>>>>>>> Xiaogang > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> Flavio Pompermaier <pomperma...@okkam.it> > > > > > > 于2019年6月17日周一 > > > > > > > >> >> >>>>> 下午7:13写道： > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>> Is there any possibility to have something > > like > > > > > Apache > > > > > > > >> >> >> Livy > > > > > > > >> >> >>>> [1] > > > > > > > >> >> >>>>>>> also > > > > > > > >> >> >>>>>>>>>>> for > > > > > > > >> >> >>>>>>>>>>>> Flink in the future? > > > > > > > >> >> >>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>> [1] https://livy.apache.org/ > > > > > > > >> >> >>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>> On Tue, Jun 11, 2019 at 5:23 PM Jeff Zhang < > > > > > > > >> >> >>> zjf...@gmail.com > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>>>>> wrote: > > > > > > > >> >> >>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> Any API we expose should not have > > dependencies > > > > > on > > > > > > > >> >> >>> the > > > > > > > >> >> >>>>>>> runtime > > > > > > > >> >> >>>>>>>>>>>>> (flink-runtime) package or other > > implementation > > > > > > > >> >> >> details. > > > > > > > >> >> >>> To > > > > > > > >> >> >>>>> me, > > > > > > > >> >> >>>>>>>> this > > > > > > > >> >> >>>>>>>>>>>> means > > > > > > > >> >> >>>>>>>>>>>>> that the current ClusterClient cannot be > > exposed > > > > to > > > > > > > >> >> >> users > > > > > > > >> >> >>>>>> because > > > > > > > >> >> >>>>>>>> it > > > > > > > >> >> >>>>>>>>>>>> uses > > > > > > > >> >> >>>>>>>>>>>>> quite some classes from the optimiser and > > runtime > > > > > > > >> >> >>> packages. > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> We should change ClusterClient from class > to > > > > > > interface. > > > > > > > >> >> >>>>>>>>>>>>> ExecutionEnvironment only use the interface > > > > > > > >> >> >> ClusterClient > > > > > > > >> >> >>>>> which > > > > > > > >> >> >>>>>>>>>>> should be > > > > > > > >> >> >>>>>>>>>>>>> in flink-clients while the concrete > > > > implementation > > > > > > > >> >> >> class > > > > > > > >> >> >>>>> could > > > > > > > >> >> >>>>>> be > > > > > > > >> >> >>>>>>>> in > > > > > > > >> >> >>>>>>>>>>>>> flink-runtime. > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> What happens when a failure/restart in > the > > > > > client > > > > > > > >> >> >>>>> happens? > > > > > > > >> >> >>>>>>>> There > > > > > > > >> >> >>>>>>>>>>> need > > > > > > > >> >> >>>>>>>>>>>>> to be a way of re-establishing the > > connection to > > > > > the > > > > > > > >> >> >> job, > > > > > > > >> >> >>>> set > > > > > > > >> >> >>>>>> up > > > > > > > >> >> >>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>> listeners again, etc. > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> Good point. First we need to define what > > does > > > > > > > >> >> >>>>> failure/restart > > > > > > > >> >> >>>>>> in > > > > > > > >> >> >>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>> client mean. IIUC, that usually mean > network > > > > > failure > > > > > > > >> >> >>> which > > > > > > > >> >> >>>>> will > > > > > > > >> >> >>>>>>>>>>> happen in > > > > > > > >> >> >>>>>>>>>>>>> class RestClient. If my understanding is > > correct, > > > > > > > >> >> >>>>> restart/retry > > > > > > > >> >> >>>>>>>>>>> mechanism > > > > > > > >> >> >>>>>>>>>>>>> should be done in RestClient. > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> Aljoscha Krettek <aljos...@apache.org> > > > > > 于2019年6月11日周二 > > > > > > > >> >> >>>>>> 下午11:10写道： > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> Some points to consider: > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> * Any API we expose should not have > > dependencies > > > > > on > > > > > > > >> >> >> the > > > > > > > >> >> >>>>>> runtime > > > > > > > >> >> >>>>>>>>>>>>>> (flink-runtime) package or other > > implementation > > > > > > > >> >> >>> details. > > > > > > > >> >> >>>> To > > > > > > > >> >> >>>>>> me, > > > > > > > >> >> >>>>>>>>>>> this > > > > > > > >> >> >>>>>>>>>>>>> means > > > > > > > >> >> >>>>>>>>>>>>>> that the current ClusterClient cannot be > > exposed > > > > > to > > > > > > > >> >> >>> users > > > > > > > >> >> >>>>>>> because > > > > > > > >> >> >>>>>>>>>>> it > > > > > > > >> >> >>>>>>>>>>>>> uses > > > > > > > >> >> >>>>>>>>>>>>>> quite some classes from the optimiser and > > > > runtime > > > > > > > >> >> >>>> packages. > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> * What happens when a failure/restart in > the > > > > > client > > > > > > > >> >> >>>>> happens? > > > > > > > >> >> >>>>>>>> There > > > > > > > >> >> >>>>>>>>>>> need > > > > > > > >> >> >>>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>> be a way of re-establishing the connection > > to > > > > the > > > > > > > >> >> >> job, > > > > > > > >> >> >>>> set > > > > > > > >> >> >>>>> up > > > > > > > >> >> >>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>> listeners > > > > > > > >> >> >>>>>>>>>>>>>> again, etc. > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> Aljoscha > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> On 29. May 2019, at 10:17, Jeff Zhang < > > > > > > > >> >> >>>> zjf...@gmail.com> > > > > > > > >> >> >>>>>>>> wrote: > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> Sorry folks, the design doc is late as > you > > > > > > > >> >> >> expected. > > > > > > > >> >> >>>>> Here's > > > > > > > >> >> >>>>>>> the > > > > > > > >> >> >>>>>>>>>>>> design > > > > > > > >> >> >>>>>>>>>>>>>> doc > > > > > > > >> >> >>>>>>>>>>>>>>> I drafted, welcome any comments and > > feedback. > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>> > > > > > > > >> >> >>> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1VavBrYn8vJeZs-Mhu5VzKO6xrWCF40aY0nlQ_UVVTRg/edit?usp=sharing > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> Stephan Ewen <se...@apache.org> > > 于2019年2月14日周四 > > > > > > > >> >> >>>> 下午8:43写道： > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> Nice that this discussion is happening. > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> In the FLIP, we could also revisit the > > entire > > > > > role > > > > > > > >> >> >>> of > > > > > > > >> >> >>>>> the > > > > > > > >> >> >>>>>>>>>>>> environments > > > > > > > >> >> >>>>>>>>>>>>>>>> again. > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> Initially, the idea was: > > > > > > > >> >> >>>>>>>>>>>>>>>> - the environments take care of the > > specific > > > > > > > >> >> >> setup > > > > > > > >> >> >>>> for > > > > > > > >> >> >>>>>>>>>>> standalone > > > > > > > >> >> >>>>>>>>>>>> (no > > > > > > > >> >> >>>>>>>>>>>>>>>> setup needed), yarn, mesos, etc. > > > > > > > >> >> >>>>>>>>>>>>>>>> - the session ones have control over the > > > > > session. > > > > > > > >> >> >>> The > > > > > > > >> >> >>>>>>>>>>> environment > > > > > > > >> >> >>>>>>>>>>>>> holds > > > > > > > >> >> >>>>>>>>>>>>>>>> the session client. > > > > > > > >> >> >>>>>>>>>>>>>>>> - running a job gives a "control" object > > for > > > > > that > > > > > > > >> >> >>>> job. > > > > > > > >> >> >>>>>> That > > > > > > > >> >> >>>>>>>>>>>> behavior > > > > > > > >> >> >>>>>>>>>>>>> is > > > > > > > >> >> >>>>>>>>>>>>>>>> the same in all environments. > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> The actual implementation diverged quite > > a bit > > > > > > > >> >> >> from > > > > > > > >> >> >>>>> that. > > > > > > > >> >> >>>>>>>> Happy > > > > > > > >> >> >>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>> see a > > > > > > > >> >> >>>>>>>>>>>>>>>> discussion about straitening this out a > > bit > > > > > more. > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> On Tue, Feb 12, 2019 at 4:58 AM Jeff > > Zhang < > > > > > > > >> >> >>>>>>> zjf...@gmail.com> > > > > > > > >> >> >>>>>>>>>>>> wrote: > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> Hi folks, > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> Sorry for late response, It seems we > > reach > > > > > > > >> >> >>> consensus > > > > > > > >> >> >>>> on > > > > > > > >> >> >>>>>>>> this, I > > > > > > > >> >> >>>>>>>>>>>> will > > > > > > > >> >> >>>>>>>>>>>>>>>> create > > > > > > > >> >> >>>>>>>>>>>>>>>>> FLIP for this with more detailed design > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> Thomas Weise <t...@apache.org> > > 于2018年12月21日周五 > > > > > > > >> >> >>>>> 上午11:43写道： > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Great to see this discussion seeded! > The > > > > > > > >> >> >> problems > > > > > > > >> >> >>>> you > > > > > > > >> >> >>>>>> face > > > > > > > >> >> >>>>>>>>>>> with > > > > > > > >> >> >>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Zeppelin integration are also > affecting > > > > other > > > > > > > >> >> >>>>> downstream > > > > > > > >> >> >>>>>>>>>>> projects, > > > > > > > >> >> >>>>>>>>>>>>>> like > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Beam. > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> We just enabled the savepoint restore > > option > > > > > in > > > > > > > >> >> >>>>>>>>>>>>>> RemoteStreamEnvironment > > > > > > > >> >> >>>>>>>>>>>>>>>>> [1] > > > > > > > >> >> >>>>>>>>>>>>>>>>>> and that was more difficult than it > > should > > > > be. > > > > > > > >> >> >> The > > > > > > > >> >> >>>>> main > > > > > > > >> >> >>>>>>>> issue > > > > > > > >> >> >>>>>>>>>>> is > > > > > > > >> >> >>>>>>>>>>>>> that > > > > > > > >> >> >>>>>>>>>>>>>>>>>> environment and cluster client aren't > > > > > decoupled. > > > > > > > >> >> >>>>> Ideally > > > > > > > >> >> >>>>>>> it > > > > > > > >> >> >>>>>>>>>>> should > > > > > > > >> >> >>>>>>>>>>>>> be > > > > > > > >> >> >>>>>>>>>>>>>>>>>> possible to just get the matching > > cluster > > > > > client > > > > > > > >> >> >>>> from > > > > > > > >> >> >>>>>> the > > > > > > > >> >> >>>>>>>>>>>>> environment > > > > > > > >> >> >>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>> then control the job through it > > (environment > > > > > as > > > > > > > >> >> >>>>> factory > > > > > > > >> >> >>>>>>> for > > > > > > > >> >> >>>>>>>>>>>> cluster > > > > > > > >> >> >>>>>>>>>>>>>>>>>> client). But note that the environment > > > > classes > > > > > > > >> >> >> are > > > > > > > >> >> >>>>> part > > > > > > > >> >> >>>>>> of > > > > > > > >> >> >>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>> public > > > > > > > >> >> >>>>>>>>>>>>>>>>> API, > > > > > > > >> >> >>>>>>>>>>>>>>>>>> and it is not straightforward to make > > larger > > > > > > > >> >> >>> changes > > > > > > > >> >> >>>>>>> without > > > > > > > >> >> >>>>>>>>>>>>> breaking > > > > > > > >> >> >>>>>>>>>>>>>>>>>> backward compatibility. > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> ClusterClient currently exposes > internal > > > > > classes > > > > > > > >> >> >>>> like > > > > > > > >> >> >>>>>>>>>>> JobGraph and > > > > > > > >> >> >>>>>>>>>>>>>>>>>> StreamGraph. But it should be possible > > to > > > > wrap > > > > > > > >> >> >>> this > > > > > > > >> >> >>>>>> with a > > > > > > > >> >> >>>>>>>> new > > > > > > > >> >> >>>>>>>>>>>>> public > > > > > > > >> >> >>>>>>>>>>>>>>>> API > > > > > > > >> >> >>>>>>>>>>>>>>>>>> that brings the required job control > > > > > > > >> >> >> capabilities > > > > > > > >> >> >>>> for > > > > > > > >> >> >>>>>>>>>>> downstream > > > > > > > >> >> >>>>>>>>>>>>>>>>> projects. > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Perhaps it is helpful to look at some > > of the > > > > > > > >> >> >>>>> interfaces > > > > > > > >> >> >>>>>> in > > > > > > > >> >> >>>>>>>>>>> Beam > > > > > > > >> >> >>>>>>>>>>>>> while > > > > > > > >> >> >>>>>>>>>>>>>>>>>> thinking about this: [2] for the > > portable > > > > job > > > > > > > >> >> >> API > > > > > > > >> >> >>>> and > > > > > > > >> >> >>>>>> [3] > > > > > > > >> >> >>>>>>>> for > > > > > > > >> >> >>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>> old > > > > > > > >> >> >>>>>>>>>>>>>>>>>> asynchronous job control from the Beam > > Java > > > > > SDK. > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> The backward compatibility discussion > > [4] is > > > > > > > >> >> >> also > > > > > > > >> >> >>>>>> relevant > > > > > > > >> >> >>>>>>>>>>> here. A > > > > > > > >> >> >>>>>>>>>>>>> new > > > > > > > >> >> >>>>>>>>>>>>>>>>> API > > > > > > > >> >> >>>>>>>>>>>>>>>>>> should shield downstream projects from > > > > > internals > > > > > > > >> >> >>> and > > > > > > > >> >> >>>>>> allow > > > > > > > >> >> >>>>>>>>>>> them to > > > > > > > >> >> >>>>>>>>>>>>>>>>>> interoperate with multiple future > Flink > > > > > versions > > > > > > > >> >> >>> in > > > > > > > >> >> >>>>> the > > > > > > > >> >> >>>>>>> same > > > > > > > >> >> >>>>>>>>>>>> release > > > > > > > >> >> >>>>>>>>>>>>>>>> line > > > > > > > >> >> >>>>>>>>>>>>>>>>>> without forced upgrades. > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Thanks, > > > > > > > >> >> >>>>>>>>>>>>>>>>>> Thomas > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> [1] > > > > https://github.com/apache/flink/pull/7249 > > > > > > > >> >> >>>>>>>>>>>>>>>>>> [2] > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>> > > > > > > > >> >> >>> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/beam_job_api.proto > > > > > > > >> >> >>>>>>>>>>>>>>>>>> [3] > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>> > > > > > > > >> >> >>> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineResult.java > > > > > > > >> >> >>>>>>>>>>>>>>>>>> [4] > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>> > > > > > > > >> >> >>> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/064c75c5d10f0806095b14f6d76942598917a14429c1acbddd151fe2@%3Cdev.flink.apache.org%3E > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> On Thu, Dec 20, 2018 at 6:15 PM Jeff > > Zhang < > > > > > > > >> >> >>>>>>>> zjf...@gmail.com> > > > > > > > >> >> >>>>>>>>>>>>> wrote: > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> I'm not so sure whether the user > > should > > > > be > > > > > > > >> >> >>> able > > > > > > > >> >> >>>> to > > > > > > > >> >> >>>>>>>> define > > > > > > > >> >> >>>>>>>>>>>> where > > > > > > > >> >> >>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> runs (in your example Yarn). This is > > > > actually > > > > > > > >> >> >>>>>> independent > > > > > > > >> >> >>>>>>>> of > > > > > > > >> >> >>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> development and is something which is > > > > decided > > > > > > > >> >> >> at > > > > > > > >> >> >>>>>>> deployment > > > > > > > >> >> >>>>>>>>>>> time. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> User don't need to specify execution > > mode > > > > > > > >> >> >>>>>>> programmatically. > > > > > > > >> >> >>>>>>>>>>> They > > > > > > > >> >> >>>>>>>>>>>>> can > > > > > > > >> >> >>>>>>>>>>>>>>>>> also > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> pass the execution mode from the > > arguments > > > > in > > > > > > > >> >> >>> flink > > > > > > > >> >> >>>>> run > > > > > > > >> >> >>>>>>>>>>> command. > > > > > > > >> >> >>>>>>>>>>>>> e.g. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> bin/flink run -m yarn-cluster .... > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> bin/flink run -m local ... > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> bin/flink run -m host:port ... > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Does this make sense to you ? > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> To me it makes sense that the > > > > > > > >> >> >>>> ExecutionEnvironment > > > > > > > >> >> >>>>>> is > > > > > > > >> >> >>>>>>>> not > > > > > > > >> >> >>>>>>>>>>>>>>>> directly > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> initialized by the user and instead > > context > > > > > > > >> >> >>>> sensitive > > > > > > > >> >> >>>>>> how > > > > > > > >> >> >>>>>>>> you > > > > > > > >> >> >>>>>>>>>>>> want > > > > > > > >> >> >>>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> execute your job (Flink CLI vs. IDE, > > for > > > > > > > >> >> >>> example). > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Right, currently I notice Flink would > > > > create > > > > > > > >> >> >>>>> different > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> ContextExecutionEnvironment based on > > > > > different > > > > > > > >> >> >>>>>> submission > > > > > > > >> >> >>>>>>>>>>>> scenarios > > > > > > > >> >> >>>>>>>>>>>>>>>>>> (Flink > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Cli vs IDE). To me this is kind of > hack > > > > > > > >> >> >> approach, > > > > > > > >> >> >>>> not > > > > > > > >> >> >>>>>> so > > > > > > > >> >> >>>>>>>>>>>>>>>>> straightforward. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> What I suggested above is that is > that > > > > flink > > > > > > > >> >> >>> should > > > > > > > >> >> >>>>>>> always > > > > > > > >> >> >>>>>>>>>>> create > > > > > > > >> >> >>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>> same > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> ExecutionEnvironment but with > different > > > > > > > >> >> >>>>> configuration, > > > > > > > >> >> >>>>>>> and > > > > > > > >> >> >>>>>>>>>>> based > > > > > > > >> >> >>>>>>>>>>>> on > > > > > > > >> >> >>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> configuration it would create the > > proper > > > > > > > >> >> >>>>> ClusterClient > > > > > > > >> >> >>>>>>> for > > > > > > > >> >> >>>>>>>>>>>>> different > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> behaviors. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Till Rohrmann <trohrm...@apache.org> > > > > > > > >> >> >>>> 于2018年12月20日周四 > > > > > > > >> >> >>>>>>>>>>> 下午11:18写道： > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> You are probably right that we have > > code > > > > > > > >> >> >>>> duplication > > > > > > > >> >> >>>>>>> when > > > > > > > >> >> >>>>>>>> it > > > > > > > >> >> >>>>>>>>>>>> comes > > > > > > > >> >> >>>>>>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> creation of the ClusterClient. This > > should > > > > > be > > > > > > > >> >> >>>>> reduced > > > > > > > >> >> >>>>>> in > > > > > > > >> >> >>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>> future. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> I'm not so sure whether the user > > should be > > > > > > > >> >> >> able > > > > > > > >> >> >>> to > > > > > > > >> >> >>>>>>> define > > > > > > > >> >> >>>>>>>>>>> where > > > > > > > >> >> >>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> runs (in your example Yarn). This is > > > > > actually > > > > > > > >> >> >>>>>>> independent > > > > > > > >> >> >>>>>>>>>>> of the > > > > > > > >> >> >>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> development and is something which > is > > > > > decided > > > > > > > >> >> >> at > > > > > > > >> >> >>>>>>>> deployment > > > > > > > >> >> >>>>>>>>>>>> time. > > > > > > > >> >> >>>>>>>>>>>>>>>> To > > > > > > > >> >> >>>>>>>>>>>>>>>>> me > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> it > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> makes sense that the > > ExecutionEnvironment > > > > is > > > > > > > >> >> >> not > > > > > > > >> >> >>>>>>> directly > > > > > > > >> >> >>>>>>>>>>>>>>>> initialized > > > > > > > >> >> >>>>>>>>>>>>>>>>>> by > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> the user and instead context > > sensitive how > > > > > you > > > > > > > >> >> >>>> want > > > > > > > >> >> >>>>> to > > > > > > > >> >> >>>>>>>>>>> execute > > > > > > > >> >> >>>>>>>>>>>>> your > > > > > > > >> >> >>>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> (Flink CLI vs. IDE, for example). > > > > However, I > > > > > > > >> >> >>> agree > > > > > > > >> >> >>>>>> that > > > > > > > >> >> >>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> ExecutionEnvironment should give you > > > > access > > > > > to > > > > > > > >> >> >>> the > > > > > > > >> >> >>>>>>>>>>> ClusterClient > > > > > > > >> >> >>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> job (maybe in the form of the > > JobGraph or > > > > a > > > > > > > >> >> >> job > > > > > > > >> >> >>>>> plan). > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> Cheers, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> Till > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> On Thu, Dec 13, 2018 at 4:36 AM Jeff > > > > Zhang < > > > > > > > >> >> >>>>>>>>>>> zjf...@gmail.com> > > > > > > > >> >> >>>>>>>>>>>>>>>> wrote: > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Hi Till, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. You are > > right > > > > > that I > > > > > > > >> >> >>>>> expect > > > > > > > >> >> >>>>>>>> better > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> programmatic > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> job submission/control api which > > could be > > > > > > > >> >> >> used > > > > > > > >> >> >>> by > > > > > > > >> >> >>>>>>>>>>> downstream > > > > > > > >> >> >>>>>>>>>>>>>>>>> project. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> And > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> it would benefit for the flink > > ecosystem. > > > > > > > >> >> >> When > > > > > > > >> >> >>> I > > > > > > > >> >> >>>>> look > > > > > > > >> >> >>>>>>> at > > > > > > > >> >> >>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>> code > > > > > > > >> >> >>>>>>>>>>>>>>>>> of > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> flink > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> scala-shell and sql-client (I > believe > > > > they > > > > > > > >> >> >> are > > > > > > > >> >> >>>> not > > > > > > > >> >> >>>>>> the > > > > > > > >> >> >>>>>>>>>>> core of > > > > > > > >> >> >>>>>>>>>>>>>>>>> flink, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> but > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> belong to the ecosystem of flink), > I > > find > > > > > > > >> >> >> many > > > > > > > >> >> >>>>>>> duplicated > > > > > > > >> >> >>>>>>>>>>> code > > > > > > > >> >> >>>>>>>>>>>>>>>> for > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> creating > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> ClusterClient from user provided > > > > > > > >> >> >> configuration > > > > > > > >> >> >>>>>>>>>>> (configuration > > > > > > > >> >> >>>>>>>>>>>>>>>>> format > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> may > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> be > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> different from scala-shell and > > > > sql-client) > > > > > > > >> >> >> and > > > > > > > >> >> >>>> then > > > > > > > >> >> >>>>>> use > > > > > > > >> >> >>>>>>>>>>> that > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> ClusterClient > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> to manipulate jobs. I don't think > > this is > > > > > > > >> >> >>>>> convenient > > > > > > > >> >> >>>>>>> for > > > > > > > >> >> >>>>>>>>>>>>>>>> downstream > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> projects. What I expect is that > > > > downstream > > > > > > > >> >> >>>> project > > > > > > > >> >> >>>>>> only > > > > > > > >> >> >>>>>>>>>>> needs > > > > > > > >> >> >>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> provide > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> necessary configuration info (maybe > > > > > > > >> >> >> introducing > > > > > > > >> >> >>>>> class > > > > > > > >> >> >>>>>>>>>>>> FlinkConf), > > > > > > > >> >> >>>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> then > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> build ExecutionEnvironment based on > > this > > > > > > > >> >> >>>> FlinkConf, > > > > > > > >> >> >>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment will create > the > > > > proper > > > > > > > >> >> >>>>>>>> ClusterClient. > > > > > > > >> >> >>>>>>>>>>> It > > > > > > > >> >> >>>>>>>>>>>> not > > > > > > > >> >> >>>>>>>>>>>>>>>>>> only > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> benefit for the downstream project > > > > > > > >> >> >> development > > > > > > > >> >> >>>> but > > > > > > > >> >> >>>>>> also > > > > > > > >> >> >>>>>>>> be > > > > > > > >> >> >>>>>>>>>>>>>>>> helpful > > > > > > > >> >> >>>>>>>>>>>>>>>>>> for > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> their integration test with flink. > > Here's > > > > > one > > > > > > > >> >> >>>>> sample > > > > > > > >> >> >>>>>>> code > > > > > > > >> >> >>>>>>>>>>>> snippet > > > > > > > >> >> >>>>>>>>>>>>>>>>>> that > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> I > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> expect. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> val conf = new > > FlinkConf().mode("yarn") > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> val env = new > > ExecutionEnvironment(conf) > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> val jobId = env.submit(...) > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> val jobStatus = > > > > > > > >> >> >>>>>>>>>>> env.getClusterClient().queryJobStatus(jobId) > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > env.getClusterClient().cancelJob(jobId) > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> What do you think ? > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Till Rohrmann < > trohrm...@apache.org> > > > > > > > >> >> >>>>> 于2018年12月11日周二 > > > > > > > >> >> >>>>>>>>>>> 下午6:28写道： > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Hi Jeff, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> what you are proposing is to > > provide the > > > > > > > >> >> >> user > > > > > > > >> >> >>>> with > > > > > > > >> >> >>>>>>>> better > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> programmatic > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> control. There was actually an > > effort to > > > > > > > >> >> >>> achieve > > > > > > > >> >> >>>>>> this > > > > > > > >> >> >>>>>>>> but > > > > > > > >> >> >>>>>>>>>>> it > > > > > > > >> >> >>>>>>>>>>>>>>>> has > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> never > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> been > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> completed [1]. However, there are > > some > > > > > > > >> >> >>>> improvement > > > > > > > >> >> >>>>>> in > > > > > > > >> >> >>>>>>>> the > > > > > > > >> >> >>>>>>>>>>> code > > > > > > > >> >> >>>>>>>>>>>>>>>>> base > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> now. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Look for example at the > > NewClusterClient > > > > > > > >> >> >>>> interface > > > > > > > >> >> >>>>>>> which > > > > > > > >> >> >>>>>>>>>>>>>>>> offers a > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> non-blocking job submission. But I > > agree > > > > > > > >> >> >> that > > > > > > > >> >> >>> we > > > > > > > >> >> >>>>>> need > > > > > > > >> >> >>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>> improve > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Flink > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> in > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> this regard. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> I would not be in favour if > > exposing all > > > > > > > >> >> >>>>>> ClusterClient > > > > > > > >> >> >>>>>>>>>>> calls > > > > > > > >> >> >>>>>>>>>>>>>>>> via > > > > > > > >> >> >>>>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment because it > > would > > > > > > > >> >> >> clutter > > > > > > > >> >> >>>> the > > > > > > > >> >> >>>>>>> class > > > > > > > >> >> >>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>> would > > > > > > > >> >> >>>>>>>>>>>>>>>>>> not > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> be > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> a > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> good separation of concerns. > > Instead one > > > > > > > >> >> >> idea > > > > > > > >> >> >>>>> could > > > > > > > >> >> >>>>>> be > > > > > > > >> >> >>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>> retrieve > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> current ClusterClient from the > > > > > > > >> >> >>>>> ExecutionEnvironment > > > > > > > >> >> >>>>>>>> which > > > > > > > >> >> >>>>>>>>>>> can > > > > > > > >> >> >>>>>>>>>>>>>>>>> then > > > > > > > >> >> >>>>>>>>>>>>>>>>>> be > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> used > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> for cluster and job control. But > > before > > > > we > > > > > > > >> >> >>> start > > > > > > > >> >> >>>>> an > > > > > > > >> >> >>>>>>>> effort > > > > > > > >> >> >>>>>>>>>>>>>>>> here, > > > > > > > >> >> >>>>>>>>>>>>>>>>> we > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> need > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> agree and capture what > > functionality we > > > > > want > > > > > > > >> >> >>> to > > > > > > > >> >> >>>>>>> provide. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Initially, the idea was that we > > have the > > > > > > > >> >> >>>>>>>> ClusterDescriptor > > > > > > > >> >> >>>>>>>>>>>>>>>>>> describing > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> how > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> to talk to cluster manager like > > Yarn or > > > > > > > >> >> >> Mesos. > > > > > > > >> >> >>>> The > > > > > > > >> >> >>>>>>>>>>>>>>>>>> ClusterDescriptor > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> can > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> be > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> used for deploying Flink clusters > > (job > > > > and > > > > > > > >> >> >>>>> session) > > > > > > > >> >> >>>>>>> and > > > > > > > >> >> >>>>>>>>>>> gives > > > > > > > >> >> >>>>>>>>>>>>>>>>> you a > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> ClusterClient. The ClusterClient > > > > controls > > > > > > > >> >> >> the > > > > > > > >> >> >>>>>> cluster > > > > > > > >> >> >>>>>>>>>>> (e.g. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> submitting > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> jobs, listing all running jobs). > And > > > > then > > > > > > > >> >> >>> there > > > > > > > >> >> >>>>> was > > > > > > > >> >> >>>>>>> the > > > > > > > >> >> >>>>>>>>>>> idea > > > > > > > >> >> >>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> introduce a > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> JobClient which you obtain from > the > > > > > > > >> >> >>>> ClusterClient > > > > > > > >> >> >>>>> to > > > > > > > >> >> >>>>>>>>>>> trigger > > > > > > > >> >> >>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> specific > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> operations (e.g. taking a > savepoint, > > > > > > > >> >> >>> cancelling > > > > > > > >> >> >>>>> the > > > > > > > >> >> >>>>>>>> job). > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> [1] > > > > > > > >> >> >>>>>> https://issues.apache.org/jira/browse/FLINK-4272 > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Cheers, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> Till > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> On Tue, Dec 11, 2018 at 10:13 AM > > Jeff > > > > > Zhang > > > > > > > >> >> >> < > > > > > > > >> >> >>>>>>>>>>> zjf...@gmail.com > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> wrote: > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Hi Folks, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> I am trying to integrate flink > into > > > > > apache > > > > > > > >> >> >>>>> zeppelin > > > > > > > >> >> >>>>>>>>>>> which is > > > > > > > >> >> >>>>>>>>>>>>>>>> an > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> interactive > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> notebook. And I hit several > issues > > that > > > > > is > > > > > > > >> >> >>>> caused > > > > > > > >> >> >>>>>> by > > > > > > > >> >> >>>>>>>>>>> flink > > > > > > > >> >> >>>>>>>>>>>>>>>>> client > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> api. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> So > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> I'd like to proposal the > following > > > > > changes > > > > > > > >> >> >>> for > > > > > > > >> >> >>>>>> flink > > > > > > > >> >> >>>>>>>>>>> client > > > > > > > >> >> >>>>>>>>>>>>>>>>> api. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 1. Support nonblocking execution. > > > > > > > >> >> >> Currently, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment#execute > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> is a blocking method which would > > do 2 > > > > > > > >> >> >> things, > > > > > > > >> >> >>>>> first > > > > > > > >> >> >>>>>>>>>>> submit > > > > > > > >> >> >>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> then > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> wait for job until it is > finished. > > I'd > > > > > like > > > > > > > >> >> >>>>>>> introduce a > > > > > > > >> >> >>>>>>>>>>>>>>>>>> nonblocking > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> execution method like > > > > > > > >> >> >>>> ExecutionEnvironment#submit > > > > > > > >> >> >>>>>>> which > > > > > > > >> >> >>>>>>>>>>> only > > > > > > > >> >> >>>>>>>>>>>>>>>>>> submit > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> then return jobId to client. And > > allow > > > > > user > > > > > > > >> >> >>> to > > > > > > > >> >> >>>>>> query > > > > > > > >> >> >>>>>>>> the > > > > > > > >> >> >>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>> status > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> via > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> jobId. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 2. Add cancel api in > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >> ExecutionEnvironment/StreamExecutionEnvironment, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> currently the only way to cancel > > job is > > > > > via > > > > > > > >> >> >>> cli > > > > > > > >> >> >>>>>>>>>>> (bin/flink), > > > > > > > >> >> >>>>>>>>>>>>>>>>> this > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> is > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> not > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> convenient for downstream project > > to > > > > use > > > > > > > >> >> >> this > > > > > > > >> >> >>>>>>> feature. > > > > > > > >> >> >>>>>>>>>>> So I'd > > > > > > > >> >> >>>>>>>>>>>>>>>>>> like > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> add > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> cancel api in > ExecutionEnvironment > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 3. Add savepoint api in > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>> ExecutionEnvironment/StreamExecutionEnvironment. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> It > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> is similar as cancel api, we > > should use > > > > > > > >> >> >>>>>>>>>>> ExecutionEnvironment > > > > > > > >> >> >>>>>>>>>>>>>>>> as > > > > > > > >> >> >>>>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> unified > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> api for third party to integrate > > with > > > > > > > >> >> >> flink. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 4. Add listener for job execution > > > > > > > >> >> >> lifecycle. > > > > > > > >> >> >>>>>>> Something > > > > > > > >> >> >>>>>>>>>>> like > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> following, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> so > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> that downstream project can do > > custom > > > > > logic > > > > > > > >> >> >>> in > > > > > > > >> >> >>>>> the > > > > > > > >> >> >>>>>>>>>>> lifecycle > > > > > > > >> >> >>>>>>>>>>>>>>>> of > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> job. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> e.g. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Zeppelin would capture the jobId > > after > > > > > job > > > > > > > >> >> >> is > > > > > > > >> >> >>>>>>> submitted > > > > > > > >> >> >>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>> then > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> use > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> this > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> jobId to cancel it later when > > > > necessary. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> public interface JobListener { > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> void onJobSubmitted(JobID > jobId); > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> void > > onJobExecuted(JobExecutionResult > > > > > > > >> >> >>>>> jobResult); > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> void onJobCanceled(JobID jobId); > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> } > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 5. Enable session in > > > > > ExecutionEnvironment. > > > > > > > >> >> >>>>>> Currently > > > > > > > >> >> >>>>>>> it > > > > > > > >> >> >>>>>>>>>>> is > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> disabled, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> but > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> session is very convenient for > > third > > > > > party > > > > > > > >> >> >> to > > > > > > > >> >> >>>>>>>> submitting > > > > > > > >> >> >>>>>>>>>>> jobs > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> continually. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> I hope flink can enable it again. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 6. Unify all flink client api > into > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>> ExecutionEnvironment/StreamExecutionEnvironment. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> This is a long term issue which > > needs > > > > > more > > > > > > > >> >> >>>>> careful > > > > > > > >> >> >>>>>>>>>>> thinking > > > > > > > >> >> >>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> design. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Currently some of features of > > flink is > > > > > > > >> >> >>> exposed > > > > > > > >> >> >>>> in > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>> ExecutionEnvironment/StreamExecutionEnvironment, > > > > > > > >> >> >>>>>> but > > > > > > > >> >> >>>>>>>>>>> some are > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> exposed > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> in > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> cli instead of api, like the > > cancel and > > > > > > > >> >> >>>>> savepoint I > > > > > > > >> >> >>>>>>>>>>> mentioned > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> above. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> I > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> think the root cause is due to > that > > > > flink > > > > > > > >> >> >>>> didn't > > > > > > > >> >> >>>>>>> unify > > > > > > > >> >> >>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> interaction > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> with > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> flink. Here I list 3 scenarios of > > flink > > > > > > > >> >> >>>> operation > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> - Local job execution. Flink > will > > > > > create > > > > > > > >> >> >>>>>>>>>>> LocalEnvironment > > > > > > > >> >> >>>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> then > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> use > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> this LocalEnvironment to create > > > > > > > >> >> >>> LocalExecutor > > > > > > > >> >> >>>>> for > > > > > > > >> >> >>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>> execution. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> - Remote job execution. Flink > will > > > > > create > > > > > > > >> >> >>>>>>>> ClusterClient > > > > > > > >> >> >>>>>>>>>>>>>>>>> first > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> then > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> create ContextEnvironment based > > on the > > > > > > > >> >> >>>>>>> ClusterClient > > > > > > > >> >> >>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>> then > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> run > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> job. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> - Job cancelation. Flink will > > create > > > > > > > >> >> >>>>>> ClusterClient > > > > > > > >> >> >>>>>>>>>>> first > > > > > > > >> >> >>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> then > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> cancel > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> this job via this ClusterClient. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> As you can see in the above 3 > > > > scenarios. > > > > > > > >> >> >>> Flink > > > > > > > >> >> >>>>>> didn't > > > > > > > >> >> >>>>>>>>>>> use the > > > > > > > >> >> >>>>>>>>>>>>>>>>>> same > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> approach(code path) to interact > > with > > > > > flink > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> What I propose is following: > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Create the proper > > > > > > > >> >> >>>>>> LocalEnvironment/RemoteEnvironment > > > > > > > >> >> >>>>>>>>>>> (based > > > > > > > >> >> >>>>>>>>>>>>>>>> on > > > > > > > >> >> >>>>>>>>>>>>>>>>>> user > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> configuration) --> Use this > > Environment > > > > > to > > > > > > > >> >> >>>> create > > > > > > > >> >> >>>>>>>> proper > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> ClusterClient > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> (LocalClusterClient or > > > > RestClusterClient) > > > > > > > >> >> >> to > > > > > > > >> >> >>>>>>>> interactive > > > > > > > >> >> >>>>>>>>>>> with > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Flink ( > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> job > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> execution or cancelation) > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> This way we can unify the process > > of > > > > > local > > > > > > > >> >> >>>>>> execution > > > > > > > >> >> >>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>> remote > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> execution. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> And it is much easier for third > > party > > > > to > > > > > > > >> >> >>>>> integrate > > > > > > > >> >> >>>>>>> with > > > > > > > >> >> >>>>>>>>>>>>>>>> flink, > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> because > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment is the > unified > > > > entry > > > > > > > >> >> >>> point > > > > > > > >> >> >>>>> for > > > > > > > >> >> >>>>>>>>>>> flink. > > > > > > > >> >> >>>>>>>>>>>>>>>> What > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> third > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> party > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> needs to do is just pass > > configuration > > > > to > > > > > > > >> >> >>>>>>>>>>>>>>>> ExecutionEnvironment > > > > > > > >> >> >>>>>>>>>>>>>>>>>> and > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> ExecutionEnvironment will do the > > right > > > > > > > >> >> >> thing > > > > > > > >> >> >>>>> based > > > > > > > >> >> >>>>>> on > > > > > > > >> >> >>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> configuration. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Flink cli can also be considered > as > > > > flink > > > > > > > >> >> >> api > > > > > > > >> >> >>>>>>> consumer. > > > > > > > >> >> >>>>>>>>>>> it > > > > > > > >> >> >>>>>>>>>>>>>>>> just > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> pass > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> the > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> configuration to > > ExecutionEnvironment > > > > and > > > > > > > >> >> >> let > > > > > > > >> >> >>>>>>>>>>>>>>>>>> ExecutionEnvironment > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> create the proper ClusterClient > > instead > > > > > of > > > > > > > >> >> >>>>> letting > > > > > > > >> >> >>>>>>> cli > > > > > > > >> >> >>>>>>>> to > > > > > > > >> >> >>>>>>>>>>>>>>>>> create > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> ClusterClient directly. > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> 6 would involve large code > > refactoring, > > > > > so > > > > > > > >> >> >> I > > > > > > > >> >> >>>>> think > > > > > > > >> >> >>>>>> we > > > > > > > >> >> >>>>>>>> can > > > > > > > >> >> >>>>>>>>>>>>>>>> defer > > > > > > > >> >> >>>>>>>>>>>>>>>>>> it > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> for > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> future release, 1,2,3,4,5 could > be > > done > > > > > at > > > > > > > >> >> >>>> once I > > > > > > > >> >> >>>>>>>>>>> believe. > > > > > > > >> >> >>>>>>>>>>>>>>>> Let > > > > > > > >> >> >>>>>>>>>>>>>>>>> me > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> know > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> your > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> comments and feedback, thanks > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> -- > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Best Regards > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> Jeff Zhang > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> -- > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Best Regards > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> Jeff Zhang > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> -- > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Best Regards > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> Jeff Zhang > > > > > > > >> >> >>>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> -- > > > > > > > >> >> >>>>>>>>>>>>>>>>> Best Regards > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>>> Jeff Zhang > > > > > > > >> >> >>>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> -- > > > > > > > >> >> >>>>>>>>>>>>>>> Best Regards > > > > > > > >> >> >>>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>>> Jeff Zhang > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> -- > > > > > > > >> >> >>>>>>>>>>>>> Best Regards > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>>> Jeff Zhang > > > > > > > >> >> >>>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>>> > > > > > > > >> >> >>>>>>>>>> > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>> -- > > > > > > > >> >> >>>>>>>> Best Regards > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>>> Jeff Zhang > > > > > > > >> >> >>>>>>>> > > > > > > > >> >> >>>>>>> > > > > > > > >> >> >>>>>> > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>>> -- > > > > > > > >> >> >>>>> Best Regards > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>>> Jeff Zhang > > > > > > > >> >> >>>>> > > > > > > > >> >> >>>> > > > > > > > >> >> >>> > > > > > > > >> >> >> > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > -- > > > > > > > >> >> > Best Regards > > > > > > > >> >> > > > > > > > > >> >> > Jeff Zhang > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
