Hi Becket, Thanks for your suggestion. We will update the FLIP to add/enrich the following parts. * User cli option change, use "-R/--remote" to apply the cluster deploy mode * Configuration change, how to specify remote user jars and dependencies * The whole story about how "application mode" works, upload -> fetch -> submit job * The cluster lifecycle, when and how the Flink cluster is destroyed
Best, Yang Becket Qin <becket....@gmail.com> 于2020年3月9日周一 下午12:34写道： > Thanks for the reply, tison and Yang, > > Regarding the public interface, is "-R/--remote" option the only change? > Will the users also need to provide a remote location to upload and store > the jars, and a list of jars as dependencies to be uploaded? > > It would be important that the public interface section in the FLIP > includes all the user sensible changes including the CLI / configuration / > metrics, etc. Can we update the FLIP to include the conclusion we have here > in the ML? > > Thanks, > > Jiangjie (Becket) Qin > > On Mon, Mar 9, 2020 at 11:59 AM Yang Wang <danrtsey...@gmail.com> wrote: > >> Hi Becket, >> >> Thanks for jumping out and sharing your concerns. I second tison's answer >> and just >> make some additions. >> >> >> > job submission interface >> >> This FLIP will introduce an interface for running user `main()` on >> cluster, named as >> “ProgramDeployer”. However, it is not a public interface. It will be used >> in `CliFrontend` >> when the remote deploy option(-R/--remote-deploy) is specified. So the >> only changes >> on user side is about the cli option. >> >> >> > How to fetch the jars? >> >> The “local path” and “dfs path“ could be supported to fetch the user jars >> and dependencies. >> Just like tison has said, we could ship the user jar and dependencies >> from client side to >> HDFS and use the entrypoint to fetch. >> >> Also we have some other practical ways to use the new “application mode“. >> 1. Upload the user jars and dependencies to the DFS(e.g. HDFS, S3, Aliyun >> OSS) manually >> or some external deployer system. For K8s, the user jars and dependencies >> could also be >> built in the docker image. >> 2. Specify the remote/local user jar and dependencies in `flink run`. >> Usually this could also >> be done by the external deployer system. >> 3. When the `ClusterEntrypoint` is launched, it will fetch the jars and >> files automatically. We >> do not need any specific fetcher implementation. Since we could leverage >> flink `FileSystem` >> to do this. >> >> >> >> >> >> Best, >> Yang >> >> tison <wander4...@gmail.com> 于2020年3月9日周一 上午11:34写道： >> >>> Hi Becket, >>> >>> Thanks for your attention on FLIP-85! I answered your question inline. >>> >>> 1. What exactly the job submission interface will look like after this >>> FLIP? The FLIP template has a Public Interface section but was removed from >>> this FLIP. >>> >>> As Yang mentioned in this thread above: >>> >>> From user perspective, only a `-R/-- remote-deploy` cli option is >>> visible. They are not aware of the application mode. >>> >>> 2. How will the new ClusterEntrypoint fetch the jars from external >>> storage? What external storage will be supported out of the box? Will this >>> "jar fetcher" be pluggable? If so, how does the API look like and how will >>> users specify the custom "jar fetcher"? >>> >>> It depends actually. Here are several points: >>> >>> i. Currently, shipping user files is handled by Flink, dependencies >>> fetching can be handled by Flink. >>> ii. Current, we only support local file system shipfiles. When in >>> Application Mode, to support meaningful jar fetch we should support user to >>> configure richer shipfiles schema at first. >>> iii. Dependencies fetching varies from deployments. That is, on YARN, >>> its convention is through HDFS; on Kubernetes, its convention is configured >>> resource server and fetched by initContainer. >>> >>> Thus, in the First phase of Application Mode dependencies fetching is >>> totally handled within Flink. >>> >>> 3. It sounds that in this FLIP, the "session cluster" running the >>> application has the same lifecycle as the user application. How will the >>> session cluster be teared down after the application finishes? Will the >>> ClusterEntrypoint do that? Will there be an option of not tearing the >>> cluster down? >>> >>> The precondition we tear down the cluster is *both* >>> >>> i. user main reached to its end >>> ii. all jobs submitted(current, at most one) reached global terminate >>> state >>> >>> For the "how", it is an implementation topic, but conceptually it is >>> ClusterEntrypoint's responsibility. >>> >>> >Will there be an option of not tearing the cluster down? >>> >>> I think the answer is "No" because the cluster is designed to be bounded >>> with an Application. User logic that communicates with the job is always in >>> its `main`, and for history information we have history server. >>> >>> Best, >>> tison. >>> >>> >>> Becket Qin <becket....@gmail.com> 于2020年3月9日周一 上午8:12写道： >>> >>>> Hi Peter and Kostas, >>>> >>>> Thanks for creating this FLIP. Moving the JobGraph compilation to the >>>> cluster makes a lot of sense to me. FLIP-40 had the exactly same idea, but >>>> is currently dormant and can probably be superseded by this FLIP. After >>>> reading the FLIP, I still have a few questions. >>>> >>>> 1. What exactly the job submission interface will look like after this >>>> FLIP? The FLIP template has a Public Interface section but was removed from >>>> this FLIP. >>>> 2. How will the new ClusterEntrypoint fetch the jars from external >>>> storage? What external storage will be supported out of the box? Will this >>>> "jar fetcher" be pluggable? If so, how does the API look like and how will >>>> users specify the custom "jar fetcher"? >>>> 3. It sounds that in this FLIP, the "session cluster" running the >>>> application has the same lifecycle as the user application. How will the >>>> session cluster be teared down after the application finishes? Will the >>>> ClusterEntrypoint do that? Will there be an option of not tearing the >>>> cluster down? >>>> >>>> Maybe they have been discussed in the ML earlier, but I think they >>>> should be part of the FLIP also. >>>> >>>> Thanks, >>>> >>>> Jiangjie (Becket) Qin >>>> >>>> On Thu, Mar 5, 2020 at 10:09 PM Kostas Kloudas <kklou...@gmail.com> >>>> wrote: >>>> >>>>> Also from my side +1 to start voting. >>>>> >>>>> Cheers, >>>>> Kostas >>>>> >>>>> On Thu, Mar 5, 2020 at 7:45 AM tison <wander4...@gmail.com> wrote: >>>>> > >>>>> > +1 to star voting. >>>>> > >>>>> > Best, >>>>> > tison. >>>>> > >>>>> > >>>>> > Yang Wang <danrtsey...@gmail.com> 于2020年3月5日周四 下午2:29写道： >>>>> >> >>>>> >> Hi Peter, >>>>> >> Really thanks for your response. >>>>> >> >>>>> >> Hi all @Kostas Kloudas @Zili Chen @Peter Huang @Rong Rong >>>>> >> It seems that we have reached an agreement. The “application mode” >>>>> is regarded as the enhanced “per-job”. It is >>>>> >> orthogonal with “cluster deploy”. Currently, we bind the “per-job” >>>>> to `run-user-main-on-client` and “application mode” >>>>> >> to `run-user-main-on-cluster`. >>>>> >> >>>>> >> Do you have other concerns to moving FLIP-85 to voting? >>>>> >> >>>>> >> >>>>> >> Best, >>>>> >> Yang >>>>> >> >>>>> >> Peter Huang <huangzhenqiu0...@gmail.com> 于2020年3月5日周四 下午12:48写道： >>>>> >>> >>>>> >>> Hi Yang and Kostas, >>>>> >>> >>>>> >>> Thanks for the clarification. It makes more sense to me if the >>>>> long term goal is to replace per job mode to application mode >>>>> >>> in the future (at the time that multiple execute can be >>>>> supported). Before that, It will be better to keep the concept of >>>>> >>> application mode internally. As Yang suggested, User only need to >>>>> use a `-R/-- remote-deploy` cli option to launch >>>>> >>> a per job cluster with the main function executed in cluster >>>>> entry-point. +1 for the execution plan. >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> Best Regards >>>>> >>> Peter Huang >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> On Tue, Mar 3, 2020 at 7:11 AM Yang Wang <danrtsey...@gmail.com> >>>>> wrote: >>>>> >>>> >>>>> >>>> Hi Peter, >>>>> >>>> >>>>> >>>> Having the application mode does not mean we will drop the >>>>> cluster-deploy >>>>> >>>> option. I just want to share some thoughts about “Application >>>>> Mode”. >>>>> >>>> >>>>> >>>> >>>>> >>>> 1. The application mode could cover the per-job sematic. Its >>>>> lifecyle is bound >>>>> >>>> to the user `main()`. And all the jobs in the user main will be >>>>> executed in a same >>>>> >>>> Flink cluster. In first phase of FLIP-85 implementation, running >>>>> user main on the >>>>> >>>> cluster side could be supported in application mode. >>>>> >>>> >>>>> >>>> 2. Maybe in the future, we also need to support multiple >>>>> `execute()` on client side >>>>> >>>> in a same Flink cluster. Then the per-job mode will evolve to >>>>> application mode. >>>>> >>>> >>>>> >>>> 3. From user perspective, only a `-R/-- remote-deploy` cli option >>>>> is visible. They >>>>> >>>> are not aware of the application mode. >>>>> >>>> >>>>> >>>> 4. In the first phase, the application mode is working as >>>>> “per-job”(only one job in >>>>> >>>> the user main). We just leave more potential for the future. >>>>> >>>> >>>>> >>>> >>>>> >>>> I am not against with calling it “cluster deploy mode” if you all >>>>> think it is clearer for users. >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> Best, >>>>> >>>> Yang >>>>> >>>> >>>>> >>>> Kostas Kloudas <kklou...@gmail.com> 于2020年3月3日周二 下午6:49写道： >>>>> >>>>> >>>>> >>>>> Hi Peter, >>>>> >>>>> >>>>> >>>>> I understand your point. This is why I was also a bit torn about >>>>> the >>>>> >>>>> name and my proposal was a bit aligned with yours (something >>>>> along the >>>>> >>>>> lines of "cluster deploy" mode). >>>>> >>>>> >>>>> >>>>> But many of the other participants in the discussion suggested >>>>> the >>>>> >>>>> "Application Mode". I think that the reasoning is that now the >>>>> user's >>>>> >>>>> Application is more self-contained. >>>>> >>>>> It will be submitted to the cluster and the user can just >>>>> disconnect. >>>>> >>>>> In addition, as discussed briefly in the doc, in the future >>>>> there may >>>>> >>>>> be better support for multi-execute applications which will >>>>> bring us >>>>> >>>>> one step closer to the true "Application Mode". But this is how I >>>>> >>>>> interpreted their arguments, of course they can also express >>>>> their >>>>> >>>>> thoughts on the topic :) >>>>> >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> Kostas >>>>> >>>>> >>>>> >>>>> On Mon, Mar 2, 2020 at 6:15 PM Peter Huang < >>>>> huangzhenqiu0...@gmail.com> wrote: >>>>> >>>>> > >>>>> >>>>> > Hi Kostas, >>>>> >>>>> > >>>>> >>>>> > Thanks for updating the wiki. We have aligned with the >>>>> implementations in the doc. But I feel it is still a little bit confusing >>>>> of the naming from a user's perspective. It is well known that Flink >>>>> support per job cluster and session cluster. The concept is in the layer >>>>> of >>>>> how a job is managed within Flink. The method introduced util now is a >>>>> kind >>>>> of mixing job and session cluster to promising the implementation >>>>> complexity. We probably don't need to label it as Application Model as the >>>>> same layer of per job cluster and session cluster. Conceptually, I think >>>>> it >>>>> is still a cluster mode implementation for per job cluster. >>>>> >>>>> > >>>>> >>>>> > To minimize the confusion of users, I think it would be better >>>>> just an option of per job cluster for each type of cluster manager. How do >>>>> you think? >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > Best Regards >>>>> >>>>> > Peter Huang >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > On Mon, Mar 2, 2020 at 7:22 AM Kostas Kloudas < >>>>> kklou...@gmail.com> wrote: >>>>> >>>>> >> >>>>> >>>>> >> Hi Yang, >>>>> >>>>> >> >>>>> >>>>> >> The difference between per-job and application mode is that, >>>>> as you >>>>> >>>>> >> described, in the per-job mode the main is executed on the >>>>> client >>>>> >>>>> >> while in the application mode, the main is executed on the >>>>> cluster. >>>>> >>>>> >> I do not think we have to offer "application mode" with >>>>> running the >>>>> >>>>> >> main on the client side as this is exactly what the per-job >>>>> mode does >>>>> >>>>> >> currently and, as you described also, it would be redundant. >>>>> >>>>> >> >>>>> >>>>> >> Sorry if this was not clear in the document. >>>>> >>>>> >> >>>>> >>>>> >> Cheers, >>>>> >>>>> >> Kostas >>>>> >>>>> >> >>>>> >>>>> >> On Mon, Mar 2, 2020 at 3:17 PM Yang Wang < >>>>> danrtsey...@gmail.com> wrote: >>>>> >>>>> >> > >>>>> >>>>> >> > Hi Kostas, >>>>> >>>>> >> > >>>>> >>>>> >> > Thanks a lot for your conclusion and updating the FLIP-85 >>>>> WIKI. Currently, i have no more >>>>> >>>>> >> > questions about motivation, approach, fault tolerance and >>>>> the first phase implementation. >>>>> >>>>> >> > >>>>> >>>>> >> > I think the new title "Flink Application Mode" makes a lot >>>>> senses to me. Especially for the >>>>> >>>>> >> > containerized environment, the cluster deploy option will >>>>> be very useful. >>>>> >>>>> >> > >>>>> >>>>> >> > Just one concern, how do we introduce this new application >>>>> mode to our users? >>>>> >>>>> >> > Each user program(i.e. `main()`) is an application. >>>>> Currently, we intend to only support one >>>>> >>>>> >> > `execute()`. So what's the difference between per-job and >>>>> application mode? >>>>> >>>>> >> > >>>>> >>>>> >> > For per-job, user `main()` is always executed on client >>>>> side. And For application mode, user >>>>> >>>>> >> > `main()` could be executed on client or master >>>>> side(configured via cli option). >>>>> >>>>> >> > Right? We need to have a clear concept. Otherwise, the >>>>> users will be more and more confusing. >>>>> >>>>> >> > >>>>> >>>>> >> > >>>>> >>>>> >> > Best, >>>>> >>>>> >> > Yang >>>>> >>>>> >> > >>>>> >>>>> >> > Kostas Kloudas <kklou...@gmail.com> 于2020年3月2日周一 下午5:58写道： >>>>> >>>>> >> >> >>>>> >>>>> >> >> Hi all, >>>>> >>>>> >> >> >>>>> >>>>> >> >> I update >>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode >>>>> >>>>> >> >> based on the discussion we had here: >>>>> >>>>> >> >> >>>>> >>>>> >> >> >>>>> https://docs.google.com/document/d/1ji72s3FD9DYUyGuKnJoO4ApzV-nSsZa0-bceGXW7Ocw/edit# >>>>> >>>>> >> >> >>>>> >>>>> >> >> Please let me know what you think and please keep the >>>>> discussion in the ML :) >>>>> >>>>> >> >> >>>>> >>>>> >> >> Thanks for starting the discussion and I hope that soon we >>>>> will be >>>>> >>>>> >> >> able to vote on the FLIP. >>>>> >>>>> >> >> >>>>> >>>>> >> >> Cheers, >>>>> >>>>> >> >> Kostas >>>>> >>>>> >> >> >>>>> >>>>> >> >> On Thu, Jan 16, 2020 at 3:40 AM Yang Wang < >>>>> danrtsey...@gmail.com> wrote: >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > Hi all, >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > Thanks a lot for the feedback from @Kostas Kloudas. Your >>>>> all concerns are >>>>> >>>>> >> >> > on point. The FLIP-85 is mainly >>>>> >>>>> >> >> > focused on supporting cluster mode for per-job. Since it >>>>> is more urgent and >>>>> >>>>> >> >> > have much more use >>>>> >>>>> >> >> > cases both in Yarn and Kubernetes deployment. For >>>>> session cluster, we could >>>>> >>>>> >> >> > have more discussion >>>>> >>>>> >> >> > in a new thread later. >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > #1, How to download the user jars and dependencies for >>>>> per-job in cluster >>>>> >>>>> >> >> > mode? >>>>> >>>>> >> >> > For Yarn, we could register the user jars and >>>>> dependencies as >>>>> >>>>> >> >> > LocalResource. They will be distributed >>>>> >>>>> >> >> > by Yarn. And once the JobManager and TaskManager >>>>> launched, the jars are >>>>> >>>>> >> >> > already exists. >>>>> >>>>> >> >> > For Standalone per-job and K8s, we expect that the user >>>>> jars >>>>> >>>>> >> >> > and dependencies are built into the image. >>>>> >>>>> >> >> > Or the InitContainer could be used for downloading. It >>>>> is natively >>>>> >>>>> >> >> > distributed and we will not have bottleneck. >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > #2, Job graph recovery >>>>> >>>>> >> >> > We could have an optimization to store job graph on the >>>>> DFS. However, i >>>>> >>>>> >> >> > suggest building a new jobgraph >>>>> >>>>> >> >> > from the configuration is the default option. Since we >>>>> will not always have >>>>> >>>>> >> >> > a DFS store when deploying a >>>>> >>>>> >> >> > Flink per-job cluster. Of course, we assume that using >>>>> the same >>>>> >>>>> >> >> > configuration(e.g. job_id, user_jar, main_class, >>>>> >>>>> >> >> > main_args, parallelism, savepoint_settings, etc.) will >>>>> get a same job >>>>> >>>>> >> >> > graph. I think the standalone per-job >>>>> >>>>> >> >> > already has the similar behavior. >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > #3, What happens with jobs that have multiple execute >>>>> calls? >>>>> >>>>> >> >> > Currently, it is really a problem. Even we use a local >>>>> client on Flink >>>>> >>>>> >> >> > master side, it will have different behavior with >>>>> >>>>> >> >> > client mode. For client mode, if we execute multiple >>>>> times, then we will >>>>> >>>>> >> >> > deploy multiple Flink clusters for each execute. >>>>> >>>>> >> >> > I am not pretty sure whether it is reasonable. However, >>>>> i still think using >>>>> >>>>> >> >> > the local client is a good choice. We could >>>>> >>>>> >> >> > continue the discussion in a new thread. @Zili Chen < >>>>> wander4...@gmail.com> Do >>>>> >>>>> >> >> > you want to drive this? >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > Best, >>>>> >>>>> >> >> > Yang >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > Peter Huang <huangzhenqiu0...@gmail.com> 于2020年1月16日周四 >>>>> 上午1:55写道： >>>>> >>>>> >> >> > >>>>> >>>>> >> >> > > Hi Kostas, >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > > Thanks for this feedback. I can't agree more about the >>>>> opinion. The >>>>> >>>>> >> >> > > cluster mode should be added >>>>> >>>>> >> >> > > first in per job cluster. >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > > 1) For job cluster implementation >>>>> >>>>> >> >> > > 1. Job graph recovery from configuration or store as >>>>> static job graph as >>>>> >>>>> >> >> > > session cluster. I think the static one will be better >>>>> for less recovery >>>>> >>>>> >> >> > > time. >>>>> >>>>> >> >> > > Let me update the doc for details. >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > > 2. For job execute multiple times, I think @Zili Chen >>>>> >>>>> >> >> > > <wander4...@gmail.com> has proposed the local client >>>>> solution that can >>>>> >>>>> >> >> > > the run program actually in the cluster entry point. >>>>> We can put the >>>>> >>>>> >> >> > > implementation in the second stage, >>>>> >>>>> >> >> > > or even a new FLIP for further discussion. >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > > 2) For session cluster implementation >>>>> >>>>> >> >> > > We can disable the cluster mode for the session >>>>> cluster in the first >>>>> >>>>> >> >> > > stage. I agree the jar downloading will be a painful >>>>> thing. >>>>> >>>>> >> >> > > We can consider about PoC and performance evaluation >>>>> first. If the end to >>>>> >>>>> >> >> > > end experience is good enough, then we can consider >>>>> >>>>> >> >> > > proceeding with the solution. >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > > Looking forward to more opinions from @Yang Wang < >>>>> danrtsey...@gmail.com> @Zili >>>>> >>>>> >> >> > > Chen <wander4...@gmail.com> @Dian Fu < >>>>> dian0511...@gmail.com>. >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > > Best Regards >>>>> >>>>> >> >> > > Peter Huang >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > > On Wed, Jan 15, 2020 at 7:50 AM Kostas Kloudas < >>>>> kklou...@gmail.com> wrote: >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> > >> Hi all, >>>>> >>>>> >> >> > >> >>>>> >>>>> >> >> > >> I am writing here as the discussion on the Google Doc >>>>> seems to be a >>>>> >>>>> >> >> > >> bit difficult to follow. >>>>> >>>>> >> >> > >> >>>>> >>>>> >> >> > >> I think that in order to be able to make progress, it >>>>> would be helpful >>>>> >>>>> >> >> > >> to focus on per-job mode for now. >>>>> >>>>> >> >> > >> The reason is that: >>>>> >>>>> >> >> > >> 1) making the (unique) JobSubmitHandler responsible >>>>> for creating the >>>>> >>>>> >> >> > >> jobgraphs, >>>>> >>>>> >> >> > >> which includes downloading dependencies, is not an >>>>> optimal solution >>>>> >>>>> >> >> > >> 2) even if we put the responsibility on the >>>>> JobMaster, currently each >>>>> >>>>> >> >> > >> job has its own >>>>> >>>>> >> >> > >> JobMaster but they all run on the same process, so >>>>> we have again a >>>>> >>>>> >> >> > >> single entity. >>>>> >>>>> >> >> > >> >>>>> >>>>> >> >> > >> Of course after this is done, and if we feel >>>>> comfortable with the >>>>> >>>>> >> >> > >> solution, then we can go to the session mode. >>>>> >>>>> >> >> > >> >>>>> >>>>> >> >> > >> A second comment has to do with fault-tolerance in >>>>> the per-job, >>>>> >>>>> >> >> > >> cluster-deploy mode. >>>>> >>>>> >> >> > >> In the document, it is suggested that upon recovery, >>>>> the JobMaster of >>>>> >>>>> >> >> > >> each job re-creates the JobGraph. >>>>> >>>>> >> >> > >> I am just wondering if it is better to create and >>>>> store the jobGraph >>>>> >>>>> >> >> > >> upon submission and only fetch it >>>>> >>>>> >> >> > >> upon recovery so that we have a static jobGraph. >>>>> >>>>> >> >> > >> >>>>> >>>>> >> >> > >> Finally, I have a question which is what happens with >>>>> jobs that have >>>>> >>>>> >> >> > >> multiple execute calls? >>>>> >>>>> >> >> > >> The semantics seem to change compared to the current >>>>> behaviour, right? >>>>> >>>>> >> >> > >> >>>>> >>>>> >> >> > >> Cheers, >>>>> >>>>> >> >> > >> Kostas >>>>> >>>>> >> >> > >> >>>>> >>>>> >> >> > >> On Wed, Jan 8, 2020 at 8:05 PM tison < >>>>> wander4...@gmail.com> wrote: >>>>> >>>>> >> >> > >> > >>>>> >>>>> >> >> > >> > not always, Yang Wang is also not yet a committer >>>>> but he can join the >>>>> >>>>> >> >> > >> > channel. I cannot find the id by clicking “Add new >>>>> member in channel” so >>>>> >>>>> >> >> > >> > come to you and ask for try out the link. Possibly >>>>> I will find other >>>>> >>>>> >> >> > >> ways >>>>> >>>>> >> >> > >> > but the original purpose is that the slack channel >>>>> is a public area we >>>>> >>>>> >> >> > >> > discuss about developing... >>>>> >>>>> >> >> > >> > Best, >>>>> >>>>> >> >> > >> > tison. >>>>> >>>>> >> >> > >> > >>>>> >>>>> >> >> > >> > >>>>> >>>>> >> >> > >> > Peter Huang <huangzhenqiu0...@gmail.com> >>>>> 于2020年1月9日周四 上午2:44写道： >>>>> >>>>> >> >> > >> > >>>>> >>>>> >> >> > >> > > Hi Tison, >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> > > I am not the committer of Flink yet. I think I >>>>> can't join it also. >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> > > Best Regards >>>>> >>>>> >> >> > >> > > Peter Huang >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> > > On Wed, Jan 8, 2020 at 9:39 AM tison < >>>>> wander4...@gmail.com> wrote: >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> > > > Hi Peter, >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > > Could you try out this link? >>>>> >>>>> >> >> > >> > > https://the-asf.slack.com/messages/CNA3ADZPH >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > > Best, >>>>> >>>>> >> >> > >> > > > tison. >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > > Peter Huang <huangzhenqiu0...@gmail.com> >>>>> 于2020年1月9日周四 上午1:22写道： >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > > > Hi Tison, >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > > I can't join the group with shared link. >>>>> Would you please add me >>>>> >>>>> >> >> > >> into >>>>> >>>>> >> >> > >> > > the >>>>> >>>>> >> >> > >> > > > > group? My slack account is huangzhenqiu0825. >>>>> >>>>> >> >> > >> > > > > Thank you in advance. >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > > Best Regards >>>>> >>>>> >> >> > >> > > > > Peter Huang >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > > On Wed, Jan 8, 2020 at 12:02 AM tison < >>>>> wander4...@gmail.com> >>>>> >>>>> >> >> > >> wrote: >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > > > Hi Peter, >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > > As described above, this effort should get >>>>> attention from people >>>>> >>>>> >> >> > >> > > > > developing >>>>> >>>>> >> >> > >> > > > > > FLIP-73 a.k.a. Executor abstractions. I >>>>> recommend you to join >>>>> >>>>> >> >> > >> the >>>>> >>>>> >> >> > >> > > > public >>>>> >>>>> >> >> > >> > > > > > slack channel[1] for Flink Client API >>>>> Enhancement and you can >>>>> >>>>> >> >> > >> try to >>>>> >>>>> >> >> > >> > > > > share >>>>> >>>>> >> >> > >> > > > > > you detailed thoughts there. It possibly >>>>> gets more concrete >>>>> >>>>> >> >> > >> > > attentions. >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > > Best, >>>>> >>>>> >> >> > >> > > > > > tison. >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > > [1] >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> >>>>> https://slack.com/share/IS21SJ75H/Rk8HhUly9FuEHb7oGwBZ33uL/enQtODg2MDYwNjE5MTg3LTA2MjIzNDc1M2ZjZDVlMjdlZjk1M2RkYmJhNjAwMTk2ZDZkODQ4NmY5YmI4OGRhNWJkYTViMTM1NzlmMzc4OWM >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > > Peter Huang <huangzhenqiu0...@gmail.com> >>>>> 于2020年1月7日周二 上午5:09写道： >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > > > Dear All, >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > Happy new year! According to existing >>>>> feedback from the >>>>> >>>>> >> >> > >> community, >>>>> >>>>> >> >> > >> > > we >>>>> >>>>> >> >> > >> > > > > > > revised the doc with the consideration of >>>>> session cluster >>>>> >>>>> >> >> > >> support, >>>>> >>>>> >> >> > >> > > > and >>>>> >>>>> >> >> > >> > > > > > > concrete interface changes needed and >>>>> execution plan. Please >>>>> >>>>> >> >> > >> take >>>>> >>>>> >> >> > >> > > one >>>>> >>>>> >> >> > >> > > > > > more >>>>> >>>>> >> >> > >> > > > > > > round of review at your most convenient >>>>> time. >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> >>>>> https://docs.google.com/document/d/1aAwVjdZByA-0CHbgv16Me-vjaaDMCfhX7TzVVTuifYM/edit# >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > Best Regards >>>>> >>>>> >> >> > >> > > > > > > Peter Huang >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > On Thu, Jan 2, 2020 at 11:29 AM Peter >>>>> Huang < >>>>> >>>>> >> >> > >> > > > > huangzhenqiu0...@gmail.com> >>>>> >>>>> >> >> > >> > > > > > > wrote: >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > Hi Dian, >>>>> >>>>> >> >> > >> > > > > > > > Thanks for giving us valuable feedbacks. >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > 1) It's better to have a whole design >>>>> for this feature >>>>> >>>>> >> >> > >> > > > > > > > For the suggestion of enabling the >>>>> cluster mode also session >>>>> >>>>> >> >> > >> > > > > cluster, I >>>>> >>>>> >> >> > >> > > > > > > > think Flink already supported it. >>>>> WebSubmissionExtension >>>>> >>>>> >> >> > >> already >>>>> >>>>> >> >> > >> > > > > allows >>>>> >>>>> >> >> > >> > > > > > > > users to start a job with the specified >>>>> jar by using web UI. >>>>> >>>>> >> >> > >> > > > > > > > But we need to enable the feature from >>>>> CLI for both local >>>>> >>>>> >> >> > >> jar, >>>>> >>>>> >> >> > >> > > > remote >>>>> >>>>> >> >> > >> > > > > > > jar. >>>>> >>>>> >> >> > >> > > > > > > > I will align with Yang Wang first about >>>>> the details and >>>>> >>>>> >> >> > >> update >>>>> >>>>> >> >> > >> > > the >>>>> >>>>> >> >> > >> > > > > > design >>>>> >>>>> >> >> > >> > > > > > > > doc. >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > 2) It's better to consider the >>>>> convenience for users, such >>>>> >>>>> >> >> > >> as >>>>> >>>>> >> >> > >> > > > > debugging >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > I am wondering whether we can store the >>>>> exception in >>>>> >>>>> >> >> > >> jobgragh >>>>> >>>>> >> >> > >> > > > > > > > generation in application master. As no >>>>> streaming graph can >>>>> >>>>> >> >> > >> be >>>>> >>>>> >> >> > >> > > > > > scheduled >>>>> >>>>> >> >> > >> > > > > > > in >>>>> >>>>> >> >> > >> > > > > > > > this case, there will be no more TM >>>>> will be requested from >>>>> >>>>> >> >> > >> > > FlinkRM. >>>>> >>>>> >> >> > >> > > > > > > > If the AM is still running, users can >>>>> still query it from >>>>> >>>>> >> >> > >> CLI. As >>>>> >>>>> >> >> > >> > > > it >>>>> >>>>> >> >> > >> > > > > > > > requires more change, we can get some >>>>> feedback from < >>>>> >>>>> >> >> > >> > > > > > aljos...@apache.org >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > and @zjf...@gmail.com <zjf...@gmail.com >>>>> >. >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > 3) It's better to consider the impact >>>>> to the stability of >>>>> >>>>> >> >> > >> the >>>>> >>>>> >> >> > >> > > > cluster >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > I agree with Yang Wang's opinion. >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > Best Regards >>>>> >>>>> >> >> > >> > > > > > > > Peter Huang >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > > On Sun, Dec 29, 2019 at 9:44 PM Dian Fu >>>>> < >>>>> >>>>> >> >> > >> dian0511...@gmail.com> >>>>> >>>>> >> >> > >> > > > > wrote: >>>>> >>>>> >> >> > >> > > > > > > > >>>>> >>>>> >> >> > >> > > > > > > >> Hi all, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> Sorry to jump into this discussion. >>>>> Thanks everyone for the >>>>> >>>>> >> >> > >> > > > > > discussion. >>>>> >>>>> >> >> > >> > > > > > > >> I'm very interested in this topic >>>>> although I'm not an >>>>> >>>>> >> >> > >> expert in >>>>> >>>>> >> >> > >> > > > this >>>>> >>>>> >> >> > >> > > > > > > part. >>>>> >>>>> >> >> > >> > > > > > > >> So I'm glad to share my thoughts as >>>>> following: >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> 1) It's better to have a whole design >>>>> for this feature >>>>> >>>>> >> >> > >> > > > > > > >> As we know, there are two deployment >>>>> modes: per-job mode >>>>> >>>>> >> >> > >> and >>>>> >>>>> >> >> > >> > > > session >>>>> >>>>> >> >> > >> > > > > > > >> mode. I'm wondering which mode really >>>>> needs this feature. >>>>> >>>>> >> >> > >> As the >>>>> >>>>> >> >> > >> > > > > > design >>>>> >>>>> >> >> > >> > > > > > > doc >>>>> >>>>> >> >> > >> > > > > > > >> mentioned, per-job mode is more used >>>>> for streaming jobs and >>>>> >>>>> >> >> > >> > > > session >>>>> >>>>> >> >> > >> > > > > > > mode is >>>>> >>>>> >> >> > >> > > > > > > >> usually used for batch jobs(Of course, >>>>> the job types and >>>>> >>>>> >> >> > >> the >>>>> >>>>> >> >> > >> > > > > > deployment >>>>> >>>>> >> >> > >> > > > > > > >> modes are orthogonal). Usually >>>>> streaming job is only >>>>> >>>>> >> >> > >> needed to >>>>> >>>>> >> >> > >> > > be >>>>> >>>>> >> >> > >> > > > > > > submitted >>>>> >>>>> >> >> > >> > > > > > > >> once and it will run for days or >>>>> weeks, while batch jobs >>>>> >>>>> >> >> > >> will be >>>>> >>>>> >> >> > >> > > > > > > submitted >>>>> >>>>> >> >> > >> > > > > > > >> more frequently compared with >>>>> streaming jobs. This means >>>>> >>>>> >> >> > >> that >>>>> >>>>> >> >> > >> > > > maybe >>>>> >>>>> >> >> > >> > > > > > > session >>>>> >>>>> >> >> > >> > > > > > > >> mode also needs this feature. However, >>>>> if we support this >>>>> >>>>> >> >> > >> > > feature >>>>> >>>>> >> >> > >> > > > in >>>>> >>>>> >> >> > >> > > > > > > >> session mode, the application master >>>>> will become the new >>>>> >>>>> >> >> > >> > > > centralized >>>>> >>>>> >> >> > >> > > > > > > >> service(which should be solved). So in >>>>> this case, it's >>>>> >>>>> >> >> > >> better to >>>>> >>>>> >> >> > >> > > > > have >>>>> >>>>> >> >> > >> > > > > > a >>>>> >>>>> >> >> > >> > > > > > > >> complete design for both per-job mode >>>>> and session mode. >>>>> >>>>> >> >> > >> > > > Furthermore, >>>>> >>>>> >> >> > >> > > > > > > even >>>>> >>>>> >> >> > >> > > > > > > >> if we can do it phase by phase, we >>>>> need to have a whole >>>>> >>>>> >> >> > >> picture >>>>> >>>>> >> >> > >> > > of >>>>> >>>>> >> >> > >> > > > > how >>>>> >>>>> >> >> > >> > > > > > > it >>>>> >>>>> >> >> > >> > > > > > > >> works in both per-job mode and session >>>>> mode. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> 2) It's better to consider the >>>>> convenience for users, such >>>>> >>>>> >> >> > >> as >>>>> >>>>> >> >> > >> > > > > > debugging >>>>> >>>>> >> >> > >> > > > > > > >> After we finish this feature, the job >>>>> graph will be >>>>> >>>>> >> >> > >> compiled in >>>>> >>>>> >> >> > >> > > > the >>>>> >>>>> >> >> > >> > > > > > > >> application master, which means that >>>>> users cannot easily >>>>> >>>>> >> >> > >> get the >>>>> >>>>> >> >> > >> > > > > > > exception >>>>> >>>>> >> >> > >> > > > > > > >> message synchorousely in the job >>>>> client if there are >>>>> >>>>> >> >> > >> problems >>>>> >>>>> >> >> > >> > > > during >>>>> >>>>> >> >> > >> > > > > > the >>>>> >>>>> >> >> > >> > > > > > > >> job graph compiling (especially for >>>>> platform users), such >>>>> >>>>> >> >> > >> as the >>>>> >>>>> >> >> > >> > > > > > > resource >>>>> >>>>> >> >> > >> > > > > > > >> path is incorrect, the user program >>>>> itself has some >>>>> >>>>> >> >> > >> problems, >>>>> >>>>> >> >> > >> > > etc. >>>>> >>>>> >> >> > >> > > > > > What >>>>> >>>>> >> >> > >> > > > > > > I'm >>>>> >>>>> >> >> > >> > > > > > > >> thinking is that maybe we should throw >>>>> the exceptions as >>>>> >>>>> >> >> > >> early >>>>> >>>>> >> >> > >> > > as >>>>> >>>>> >> >> > >> > > > > > > possible >>>>> >>>>> >> >> > >> > > > > > > >> (during job submission stage). >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> 3) It's better to consider the impact >>>>> to the stability of >>>>> >>>>> >> >> > >> the >>>>> >>>>> >> >> > >> > > > > cluster >>>>> >>>>> >> >> > >> > > > > > > >> If we perform the compiling in the >>>>> application master, we >>>>> >>>>> >> >> > >> should >>>>> >>>>> >> >> > >> > > > > > > consider >>>>> >>>>> >> >> > >> > > > > > > >> the impact of the compiling errors. >>>>> Although YARN could >>>>> >>>>> >> >> > >> resume >>>>> >>>>> >> >> > >> > > the >>>>> >>>>> >> >> > >> > > > > > > >> application master in case of >>>>> failures, but in some case >>>>> >>>>> >> >> > >> the >>>>> >>>>> >> >> > >> > > > > compiling >>>>> >>>>> >> >> > >> > > > > > > >> failure may be a waste of cluster >>>>> resource and may impact >>>>> >>>>> >> >> > >> the >>>>> >>>>> >> >> > >> > > > > > stability >>>>> >>>>> >> >> > >> > > > > > > the >>>>> >>>>> >> >> > >> > > > > > > >> cluster and the other jobs in the >>>>> cluster, such as the >>>>> >>>>> >> >> > >> resource >>>>> >>>>> >> >> > >> > > > path >>>>> >>>>> >> >> > >> > > > > > is >>>>> >>>>> >> >> > >> > > > > > > >> incorrect, the user program itself has >>>>> some problems(in >>>>> >>>>> >> >> > >> this >>>>> >>>>> >> >> > >> > > case, >>>>> >>>>> >> >> > >> > > > > job >>>>> >>>>> >> >> > >> > > > > > > >> failover cannot solve this kind of >>>>> problems) etc. In the >>>>> >>>>> >> >> > >> current >>>>> >>>>> >> >> > >> > > > > > > >> implemention, the compiling errors are >>>>> handled in the >>>>> >>>>> >> >> > >> client >>>>> >>>>> >> >> > >> > > side >>>>> >>>>> >> >> > >> > > > > and >>>>> >>>>> >> >> > >> > > > > > > there >>>>> >>>>> >> >> > >> > > > > > > >> is no impact to the cluster at all. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> Regarding to 1), it's clearly pointed >>>>> in the design doc >>>>> >>>>> >> >> > >> that >>>>> >>>>> >> >> > >> > > only >>>>> >>>>> >> >> > >> > > > > > > per-job >>>>> >>>>> >> >> > >> > > > > > > >> mode will be supported. However, I >>>>> think it's better to >>>>> >>>>> >> >> > >> also >>>>> >>>>> >> >> > >> > > > > consider >>>>> >>>>> >> >> > >> > > > > > > the >>>>> >>>>> >> >> > >> > > > > > > >> session mode in the design doc. >>>>> >>>>> >> >> > >> > > > > > > >> Regarding to 2) and 3), I have not >>>>> seen related sections >>>>> >>>>> >> >> > >> in the >>>>> >>>>> >> >> > >> > > > > design >>>>> >>>>> >> >> > >> > > > > > > >> doc. It will be good if we can cover >>>>> them in the design >>>>> >>>>> >> >> > >> doc. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> Feel free to correct me If there is >>>>> anything I >>>>> >>>>> >> >> > >> misunderstand. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> Regards, >>>>> >>>>> >> >> > >> > > > > > > >> Dian >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> > 在 2019年12月27日，上午3:13，Peter Huang < >>>>> >>>>> >> >> > >> huangzhenqiu0...@gmail.com> >>>>> >>>>> >> >> > >> > > > 写道： >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > Hi Yang, >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > I can't agree more. The effort >>>>> definitely needs to align >>>>> >>>>> >> >> > >> with >>>>> >>>>> >> >> > >> > > > the >>>>> >>>>> >> >> > >> > > > > > > final >>>>> >>>>> >> >> > >> > > > > > > >> > goal of FLIP-73. >>>>> >>>>> >> >> > >> > > > > > > >> > I am thinking about whether we can >>>>> achieve the goal with >>>>> >>>>> >> >> > >> two >>>>> >>>>> >> >> > >> > > > > phases. >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > 1) Phase I >>>>> >>>>> >> >> > >> > > > > > > >> > As the CLiFrontend will not be >>>>> depreciated soon. We can >>>>> >>>>> >> >> > >> still >>>>> >>>>> >> >> > >> > > > use >>>>> >>>>> >> >> > >> > > > > > the >>>>> >>>>> >> >> > >> > > > > > > >> > deployMode flag there, >>>>> >>>>> >> >> > >> > > > > > > >> > pass the program info through Flink >>>>> configuration, use >>>>> >>>>> >> >> > >> the >>>>> >>>>> >> >> > >> > > > > > > >> > ClassPathJobGraphRetriever >>>>> >>>>> >> >> > >> > > > > > > >> > to generate the job graph in >>>>> ClusterEntrypoints of yarn >>>>> >>>>> >> >> > >> and >>>>> >>>>> >> >> > >> > > > > > > Kubernetes. >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > 2) Phase II >>>>> >>>>> >> >> > >> > > > > > > >> > In AbstractJobClusterExecutor, the >>>>> job graph is >>>>> >>>>> >> >> > >> generated in >>>>> >>>>> >> >> > >> > > > the >>>>> >>>>> >> >> > >> > > > > > > >> execute >>>>> >>>>> >> >> > >> > > > > > > >> > function. We can still >>>>> >>>>> >> >> > >> > > > > > > >> > use the deployMode in it. With >>>>> deployMode = cluster, the >>>>> >>>>> >> >> > >> > > execute >>>>> >>>>> >> >> > >> > > > > > > >> function >>>>> >>>>> >> >> > >> > > > > > > >> > only starts the cluster. >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > When >>>>> {Yarn/Kuberneates}PerJobClusterEntrypoint starts, >>>>> >>>>> >> >> > >> It will >>>>> >>>>> >> >> > >> > > > > start >>>>> >>>>> >> >> > >> > > > > > > the >>>>> >>>>> >> >> > >> > > > > > > >> > dispatch first, then we can use >>>>> >>>>> >> >> > >> > > > > > > >> > a ClusterEnvironment similar to >>>>> ContextEnvironment to >>>>> >>>>> >> >> > >> submit >>>>> >>>>> >> >> > >> > > the >>>>> >>>>> >> >> > >> > > > > job >>>>> >>>>> >> >> > >> > > > > > > >> with >>>>> >>>>> >> >> > >> > > > > > > >> > jobName the local >>>>> >>>>> >> >> > >> > > > > > > >> > dispatcher. For the details, we need >>>>> more investigation. >>>>> >>>>> >> >> > >> Let's >>>>> >>>>> >> >> > >> > > > > wait >>>>> >>>>> >> >> > >> > > > > > > >> > for @Aljoscha >>>>> >>>>> >> >> > >> > > > > > > >> > Krettek <aljos...@apache.org> @Till >>>>> Rohrmann < >>>>> >>>>> >> >> > >> > > > > trohrm...@apache.org >>>>> >>>>> >> >> > >> > > > > > >'s >>>>> >>>>> >> >> > >> > > > > > > >> > feedback after the holiday season. >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > Thank you in advance. Merry Chrismas >>>>> and Happy New >>>>> >>>>> >> >> > >> Year!!! >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > Best Regards >>>>> >>>>> >> >> > >> > > > > > > >> > Peter Huang >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> > On Wed, Dec 25, 2019 at 1:08 AM Yang >>>>> Wang < >>>>> >>>>> >> >> > >> > > > danrtsey...@gmail.com> >>>>> >>>>> >> >> > >> > > > > > > >> wrote: >>>>> >>>>> >> >> > >> > > > > > > >> > >>>>> >>>>> >> >> > >> > > > > > > >> >> Hi Peter, >>>>> >>>>> >> >> > >> > > > > > > >> >> >>>>> >>>>> >> >> > >> > > > > > > >> >> I think we need to reconsider >>>>> tison's suggestion >>>>> >>>>> >> >> > >> seriously. >>>>> >>>>> >> >> > >> > > > After >>>>> >>>>> >> >> > >> > > > > > > >> FLIP-73, >>>>> >>>>> >> >> > >> > > > > > > >> >> the deployJobCluster has >>>>> >>>>> >> >> > >> > > > > > > >> >> beenmoved into >>>>> `JobClusterExecutor#execute`. It should >>>>> >>>>> >> >> > >> not be >>>>> >>>>> >> >> > >> > > > > > > perceived >>>>> >>>>> >> >> > >> > > > > > > >> >> for `CliFrontend`. That >>>>> >>>>> >> >> > >> > > > > > > >> >> means the user program will >>>>> *ALWAYS* be executed on >>>>> >>>>> >> >> > >> client >>>>> >>>>> >> >> > >> > > > side. >>>>> >>>>> >> >> > >> > > > > > This >>>>> >>>>> >> >> > >> > > > > > > >> is >>>>> >>>>> >> >> > >> > > > > > > >> >> the by design behavior. >>>>> >>>>> >> >> > >> > > > > > > >> >> So, we could not just add >>>>> `if(client mode) .. else >>>>> >>>>> >> >> > >> if(cluster >>>>> >>>>> >> >> > >> > > > > mode) >>>>> >>>>> >> >> > >> > > > > > > >> ...` >>>>> >>>>> >> >> > >> > > > > > > >> >> codes in `CliFrontend` to bypass >>>>> >>>>> >> >> > >> > > > > > > >> >> the executor. We need to find a >>>>> clean way to decouple >>>>> >>>>> >> >> > >> > > executing >>>>> >>>>> >> >> > >> > > > > > user >>>>> >>>>> >> >> > >> > > > > > > >> >> program and deploying per-job >>>>> >>>>> >> >> > >> > > > > > > >> >> cluster. Based on this, we could >>>>> support to execute user >>>>> >>>>> >> >> > >> > > > program >>>>> >>>>> >> >> > >> > > > > on >>>>> >>>>> >> >> > >> > > > > > > >> client >>>>> >>>>> >> >> > >> > > > > > > >> >> or master side. >>>>> >>>>> >> >> > >> > > > > > > >> >> >>>>> >>>>> >> >> > >> > > > > > > >> >> Maybe Aljoscha and Jeff could give >>>>> some good >>>>> >>>>> >> >> > >> suggestions. >>>>> >>>>> >> >> > >> > > > > > > >> >> >>>>> >>>>> >> >> > >> > > > > > > >> >> >>>>> >>>>> >> >> > >> > > > > > > >> >> >>>>> >>>>> >> >> > >> > > > > > > >> >> Best, >>>>> >>>>> >> >> > >> > > > > > > >> >> Yang >>>>> >>>>> >> >> > >> > > > > > > >> >> >>>>> >>>>> >> >> > >> > > > > > > >> >> Peter Huang < >>>>> huangzhenqiu0...@gmail.com> 于2019年12月25日周三 >>>>> >>>>> >> >> > >> > > > > 上午4:03写道： >>>>> >>>>> >> >> > >> > > > > > > >> >> >>>>> >>>>> >> >> > >> > > > > > > >> >>> Hi Jingjing, >>>>> >>>>> >> >> > >> > > > > > > >> >>> >>>>> >>>>> >> >> > >> > > > > > > >> >>> The improvement proposed is a >>>>> deployment option for >>>>> >>>>> >> >> > >> CLI. For >>>>> >>>>> >> >> > >> > > > SQL >>>>> >>>>> >> >> > >> > > > > > > based >>>>> >>>>> >> >> > >> > > > > > > >> >>> Flink application, It is more >>>>> convenient to use the >>>>> >>>>> >> >> > >> existing >>>>> >>>>> >> >> > >> > > > > model >>>>> >>>>> >> >> > >> > > > > > > in >>>>> >>>>> >> >> > >> > > > > > > >> >>> SqlClient in which >>>>> >>>>> >> >> > >> > > > > > > >> >>> the job graph is generated within >>>>> SqlClient. After >>>>> >>>>> >> >> > >> adding >>>>> >>>>> >> >> > >> > > the >>>>> >>>>> >> >> > >> > > > > > > delayed >>>>> >>>>> >> >> > >> > > > > > > >> job >>>>> >>>>> >> >> > >> > > > > > > >> >>> graph generation, I think there is >>>>> no change is needed >>>>> >>>>> >> >> > >> for >>>>> >>>>> >> >> > >> > > > your >>>>> >>>>> >> >> > >> > > > > > > side. >>>>> >>>>> >> >> > >> > > > > > > >> >>> >>>>> >>>>> >> >> > >> > > > > > > >> >>> >>>>> >>>>> >> >> > >> > > > > > > >> >>> Best Regards >>>>> >>>>> >> >> > >> > > > > > > >> >>> Peter Huang >>>>> >>>>> >> >> > >> > > > > > > >> >>> >>>>> >>>>> >> >> > >> > > > > > > >> >>> >>>>> >>>>> >> >> > >> > > > > > > >> >>> On Wed, Dec 18, 2019 at 6:01 AM >>>>> jingjing bai < >>>>> >>>>> >> >> > >> > > > > > > >> baijingjing7...@gmail.com> >>>>> >>>>> >> >> > >> > > > > > > >> >>> wrote: >>>>> >>>>> >> >> > >> > > > > > > >> >>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>> hi peter: >>>>> >>>>> >> >> > >> > > > > > > >> >>>> we had extension SqlClent to >>>>> support sql job >>>>> >>>>> >> >> > >> submit in >>>>> >>>>> >> >> > >> > > web >>>>> >>>>> >> >> > >> > > > > > base >>>>> >>>>> >> >> > >> > > > > > > on >>>>> >>>>> >> >> > >> > > > > > > >> >>>> flink 1.9. we support submit to >>>>> yarn on per job >>>>> >>>>> >> >> > >> mode too. >>>>> >>>>> >> >> > >> > > > > > > >> >>>> in this case, the job graph >>>>> generated on client >>>>> >>>>> >> >> > >> side >>>>> >>>>> >> >> > >> > > . I >>>>> >>>>> >> >> > >> > > > > > think >>>>> >>>>> >> >> > >> > > > > > > >> >>> this >>>>> >>>>> >> >> > >> > > > > > > >> >>>> discuss Mainly to improve api >>>>> programme. but in my >>>>> >>>>> >> >> > >> case , >>>>> >>>>> >> >> > >> > > > > there >>>>> >>>>> >> >> > >> > > > > > is >>>>> >>>>> >> >> > >> > > > > > > >> no >>>>> >>>>> >> >> > >> > > > > > > >> >>>> jar to upload but only a sql >>>>> string . >>>>> >>>>> >> >> > >> > > > > > > >> >>>> do u had more suggestion to >>>>> improve for sql mode >>>>> >>>>> >> >> > >> or it >>>>> >>>>> >> >> > >> > > is >>>>> >>>>> >> >> > >> > > > > > only a >>>>> >>>>> >> >> > >> > > > > > > >> >>>> switch for api programme？ >>>>> >>>>> >> >> > >> > > > > > > >> >>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>> best >>>>> >>>>> >> >> > >> > > > > > > >> >>>> bai jj >>>>> >>>>> >> >> > >> > > > > > > >> >>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>> Yang Wang <danrtsey...@gmail.com> >>>>> 于2019年12月18日周三 >>>>> >>>>> >> >> > >> 下午7:21写道： >>>>> >>>>> >> >> > >> > > > > > > >> >>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> I just want to revive this >>>>> discussion. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> Recently, i am thinking about >>>>> how to natively run >>>>> >>>>> >> >> > >> flink >>>>> >>>>> >> >> > >> > > > > per-job >>>>> >>>>> >> >> > >> > > > > > > >> >>> cluster on >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> Kubernetes. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> The per-job mode on Kubernetes >>>>> is very different >>>>> >>>>> >> >> > >> from on >>>>> >>>>> >> >> > >> > > > Yarn. >>>>> >>>>> >> >> > >> > > > > > And >>>>> >>>>> >> >> > >> > > > > > > >> we >>>>> >>>>> >> >> > >> > > > > > > >> >>> will >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> have >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> the same deployment requirements >>>>> to the client and >>>>> >>>>> >> >> > >> entry >>>>> >>>>> >> >> > >> > > > > point. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> 1. Flink client not always need >>>>> a local jar to start >>>>> >>>>> >> >> > >> a >>>>> >>>>> >> >> > >> > > Flink >>>>> >>>>> >> >> > >> > > > > > > per-job >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> cluster. We could >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> support multiple schemas. For >>>>> example, >>>>> >>>>> >> >> > >> > > > file:///path/of/my.jar >>>>> >>>>> >> >> > >> > > > > > > means >>>>> >>>>> >> >> > >> > > > > > > >> a >>>>> >>>>> >> >> > >> > > > > > > >> >>> jar >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> located >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> at client side, >>>>> >>>>> >> >> > >> hdfs://myhdfs/user/myname/flink/my.jar >>>>> >>>>> >> >> > >> > > > means a >>>>> >>>>> >> >> > >> > > > > > jar >>>>> >>>>> >> >> > >> > > > > > > >> >>> located >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> at >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> remote hdfs, >>>>> local:///path/in/image/my.jar means a >>>>> >>>>> >> >> > >> jar >>>>> >>>>> >> >> > >> > > > located >>>>> >>>>> >> >> > >> > > > > > at >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> jobmanager side. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> 2. Support running user program >>>>> on master side. This >>>>> >>>>> >> >> > >> also >>>>> >>>>> >> >> > >> > > > > means >>>>> >>>>> >> >> > >> > > > > > > the >>>>> >>>>> >> >> > >> > > > > > > >> >>> entry >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> point >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> will generate the job graph on >>>>> master side. We could >>>>> >>>>> >> >> > >> use >>>>> >>>>> >> >> > >> > > the >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> ClasspathJobGraphRetriever >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> or start a local Flink client to >>>>> achieve this >>>>> >>>>> >> >> > >> purpose. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> cc tison, Aljoscha & Kostas Do >>>>> you think this is the >>>>> >>>>> >> >> > >> right >>>>> >>>>> >> >> > >> > > > > > > >> direction we >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> need to work? >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> tison <wander4...@gmail.com> >>>>> 于2019年12月12日周四 >>>>> >>>>> >> >> > >> 下午4:48写道： >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> A quick idea is that we >>>>> separate the deployment >>>>> >>>>> >> >> > >> from user >>>>> >>>>> >> >> > >> > > > > > program >>>>> >>>>> >> >> > >> > > > > > > >> >>> that >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> it >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> has always been done >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> outside the program. On user >>>>> program executed there >>>>> >>>>> >> >> > >> is >>>>> >>>>> >> >> > >> > > > > always a >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> ClusterClient that communicates >>>>> with >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> an existing cluster, remote or >>>>> local. It will be >>>>> >>>>> >> >> > >> another >>>>> >>>>> >> >> > >> > > > > thread >>>>> >>>>> >> >> > >> > > > > > > so >>>>> >>>>> >> >> > >> > > > > > > >> >>> just >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> for >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> your information. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> Best, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> tison. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> tison <wander4...@gmail.com> >>>>> 于2019年12月12日周四 >>>>> >>>>> >> >> > >> 下午4:40写道： >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> Hi Peter, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> Another concern I realized >>>>> recently is that with >>>>> >>>>> >> >> > >> current >>>>> >>>>> >> >> > >> > > > > > > Executors >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> abstraction(FLIP-73) >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> I'm afraid that user program >>>>> is designed to ALWAYS >>>>> >>>>> >> >> > >> run >>>>> >>>>> >> >> > >> > > on >>>>> >>>>> >> >> > >> > > > > the >>>>> >>>>> >> >> > >> > > > > > > >> >>> client >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> side. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> Specifically, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> we deploy the job in executor >>>>> when env.execute >>>>> >>>>> >> >> > >> called. >>>>> >>>>> >> >> > >> > > > This >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> abstraction >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> possibly prevents >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> Flink runs user program on the >>>>> cluster side. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> For your proposal, in this >>>>> case we already >>>>> >>>>> >> >> > >> compiled the >>>>> >>>>> >> >> > >> > > > > > program >>>>> >>>>> >> >> > >> > > > > > > >> and >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> run >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> on >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> the client side, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> even we deploy a cluster and >>>>> retrieve job graph >>>>> >>>>> >> >> > >> from >>>>> >>>>> >> >> > >> > > > program >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> metadata, it >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> doesn't make >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> many sense. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> cc Aljoscha & Kostas what do >>>>> you think about this >>>>> >>>>> >> >> > >> > > > > constraint? >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> Best, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> tison. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> Peter Huang < >>>>> huangzhenqiu0...@gmail.com> >>>>> >>>>> >> >> > >> 于2019年12月10日周二 >>>>> >>>>> >> >> > >> > > > > > > >> 下午12:45写道： >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> Hi Tison, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> Yes, you are right. I think I >>>>> made the wrong >>>>> >>>>> >> >> > >> argument >>>>> >>>>> >> >> > >> > > in >>>>> >>>>> >> >> > >> > > > > the >>>>> >>>>> >> >> > >> > > > > > > doc. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> Basically, the packaging jar >>>>> problem is only for >>>>> >>>>> >> >> > >> > > platform >>>>> >>>>> >> >> > >> > > > > > > users. >>>>> >>>>> >> >> > >> > > > > > > >> >>> In >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> our >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> internal deploy service, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> we further optimized the >>>>> deployment latency by >>>>> >>>>> >> >> > >> letting >>>>> >>>>> >> >> > >> > > > > users >>>>> >>>>> >> >> > >> > > > > > to >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> packaging >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> flink-runtime together with >>>>> the uber jar, so that >>>>> >>>>> >> >> > >> we >>>>> >>>>> >> >> > >> > > > don't >>>>> >>>>> >> >> > >> > > > > > need >>>>> >>>>> >> >> > >> > > > > > > >> to >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> consider >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> multiple flink version >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> support for now. In the >>>>> session client mode, as >>>>> >>>>> >> >> > >> Flink >>>>> >>>>> >> >> > >> > > > libs >>>>> >>>>> >> >> > >> > > > > > will >>>>> >>>>> >> >> > >> > > > > > > >> be >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> shipped >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> anyway as local resources of >>>>> yarn. Users actually >>>>> >>>>> >> >> > >> don't >>>>> >>>>> >> >> > >> > > > > need >>>>> >>>>> >> >> > >> > > > > > to >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> package >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> those libs into job jar. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> Best Regards >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> Peter Huang >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> On Mon, Dec 9, 2019 at 8:35 >>>>> PM tison < >>>>> >>>>> >> >> > >> > > > wander4...@gmail.com >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > > > >> >>> wrote: >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> 3. What do you mean about >>>>> the package? Do users >>>>> >>>>> >> >> > >> need >>>>> >>>>> >> >> > >> > > to >>>>> >>>>> >> >> > >> > > > > > > >> >>> compile >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> their >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> jars >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> inlcuding flink-clients, >>>>> flink-optimizer, >>>>> >>>>> >> >> > >> flink-table >>>>> >>>>> >> >> > >> > > > > codes? >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> The answer should be no >>>>> because they exist in >>>>> >>>>> >> >> > >> system >>>>> >>>>> >> >> > >> > > > > > > classpath. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> Best, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> tison. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> Yang Wang < >>>>> danrtsey...@gmail.com> 于2019年12月10日周二 >>>>> >>>>> >> >> > >> > > > > 下午12:18写道： >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> Hi Peter, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> Thanks a lot for starting >>>>> this discussion. I >>>>> >>>>> >> >> > >> think >>>>> >>>>> >> >> > >> > > this >>>>> >>>>> >> >> > >> > > > > is >>>>> >>>>> >> >> > >> > > > > > a >>>>> >>>>> >> >> > >> > > > > > > >> >>> very >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> useful >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> feature. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> Not only for Yarn, i am >>>>> focused on flink on >>>>> >>>>> >> >> > >> > > Kubernetes >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> integration >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> and >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> come >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> across the same >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> problem. I do not want the >>>>> job graph generated >>>>> >>>>> >> >> > >> on >>>>> >>>>> >> >> > >> > > > client >>>>> >>>>> >> >> > >> > > > > > > side. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> Instead, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> the >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> user jars are built in >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> a user-defined image. When >>>>> the job manager >>>>> >>>>> >> >> > >> launched, >>>>> >>>>> >> >> > >> > > we >>>>> >>>>> >> >> > >> > > > > > just >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> need to >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> generate the job graph >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> based on local user jars. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> I have some small >>>>> suggestion about this. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> 1. >>>>> `ProgramJobGraphRetriever` is very similar to >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> `ClasspathJobGraphRetriever`, the differences >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> are the former needs >>>>> `ProgramMetadata` and the >>>>> >>>>> >> >> > >> latter >>>>> >>>>> >> >> > >> > > > > needs >>>>> >>>>> >> >> > >> > > > > > > >> >>> some >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> arguments. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> Is it possible to >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> have an unified >>>>> `JobGraphRetriever` to support >>>>> >>>>> >> >> > >> both? >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> 2. Is it possible to not >>>>> use a local user jar to >>>>> >>>>> >> >> > >> > > start >>>>> >>>>> >> >> > >> > > > a >>>>> >>>>> >> >> > >> > > > > > > >> >>> per-job >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> cluster? >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> In your case, the user jars >>>>> has >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> existed on hdfs already and >>>>> we do need to >>>>> >>>>> >> >> > >> download >>>>> >>>>> >> >> > >> > > the >>>>> >>>>> >> >> > >> > > > > jars >>>>> >>>>> >> >> > >> > > > > > > to >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> deployer >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> service. Currently, we >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> always need a local user >>>>> jar to start a flink >>>>> >>>>> >> >> > >> > > cluster. >>>>> >>>>> >> >> > >> > > > It >>>>> >>>>> >> >> > >> > > > > > is >>>>> >>>>> >> >> > >> > > > > > > >> >>> be >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> great >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> if >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> we >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> could support remote user >>>>> jars. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>>> In the implementation, we >>>>> assume users package >>>>> >>>>> >> >> > >> > > > > > > >> >>> flink-clients, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> flink-optimizer, >>>>> flink-table together within >>>>> >>>>> >> >> > >> the job >>>>> >>>>> >> >> > >> > > > jar. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> Otherwise, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> the >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> job graph generation within >>>>> >>>>> >> >> > >> JobClusterEntryPoint will >>>>> >>>>> >> >> > >> > > > > fail. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> 3. What do you mean about >>>>> the package? Do users >>>>> >>>>> >> >> > >> need >>>>> >>>>> >> >> > >> > > to >>>>> >>>>> >> >> > >> > > > > > > >> >>> compile >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> their >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> jars >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> inlcuding flink-clients, >>>>> flink-optimizer, >>>>> >>>>> >> >> > >> flink-table >>>>> >>>>> >> >> > >> > > > > > codes? >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> Best, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> Yang >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> Peter Huang < >>>>> huangzhenqiu0...@gmail.com> >>>>> >>>>> >> >> > >> > > > 于2019年12月10日周二 >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> 上午2:37写道： >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Dear All, >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Recently, the Flink >>>>> community starts to >>>>> >>>>> >> >> > >> improve the >>>>> >>>>> >> >> > >> > > > yarn >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> cluster >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> descriptor >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> to make job jar and config >>>>> files configurable >>>>> >>>>> >> >> > >> from >>>>> >>>>> >> >> > >> > > > CLI. >>>>> >>>>> >> >> > >> > > > > It >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> improves >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> the >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> flexibility of Flink >>>>> deployment Yarn Per Job >>>>> >>>>> >> >> > >> Mode. >>>>> >>>>> >> >> > >> > > > For >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> platform >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> users >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> who >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> manage tens of hundreds of >>>>> streaming pipelines >>>>> >>>>> >> >> > >> for >>>>> >>>>> >> >> > >> > > the >>>>> >>>>> >> >> > >> > > > > > whole >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> org >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> or >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> company, we found the job >>>>> graph generation in >>>>> >>>>> >> >> > >> > > > > client-side >>>>> >>>>> >> >> > >> > > > > > is >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> another >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> pinpoint. Thus, we want to >>>>> propose a >>>>> >>>>> >> >> > >> configurable >>>>> >>>>> >> >> > >> > > > > feature >>>>> >>>>> >> >> > >> > > > > > > >> >>> for >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> FlinkYarnSessionCli. The >>>>> feature can allow >>>>> >>>>> >> >> > >> users to >>>>> >>>>> >> >> > >> > > > > choose >>>>> >>>>> >> >> > >> > > > > > > >> >>> the >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> job >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> graph >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> generation in Flink >>>>> ClusterEntryPoint so that >>>>> >>>>> >> >> > >> the >>>>> >>>>> >> >> > >> > > job >>>>> >>>>> >> >> > >> > > > > jar >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> doesn't >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> need >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> to >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> be locally for the job >>>>> graph generation. The >>>>> >>>>> >> >> > >> > > proposal >>>>> >>>>> >> >> > >> > > > is >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> organized >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> as a >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> FLIP >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> >>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Delayed+JobGraph+Generation >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> . >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Any questions and >>>>> suggestions are welcomed. >>>>> >>>>> >> >> > >> Thank >>>>> >>>>> >> >> > >> > > you >>>>> >>>>> >> >> > >> > > > in >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> advance. >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Best Regards >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Peter Huang >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>>> >>>>> >>>>> >> >> > >> > > > > > > >> >>> >>>>> >>>>> >> >> > >> > > > > > > >> >> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >> >>>>> >>>>> >> >> > >> > > > > > > >>>>> >>>>> >> >> > >> > > > > > >>>>> >>>>> >> >> > >> > > > > >>>>> >>>>> >> >> > >> > > > >>>>> >>>>> >> >> > >> > > >>>>> >>>>> >> >> > >> >>>>> >>>>> >> >> > > >>>>> >>>>> >> >> >>>>> >>>>
