+1 to star voting. Best, tison.
Yang Wang <danrtsey...@gmail.com> 于2020年3月5日周四 下午2:29写道： > Hi Peter, > Really thanks for your response. > > Hi all @Kostas Kloudas <kklou...@gmail.com> @Zili Chen > <wander4...@gmail.com> @Peter Huang <huangzhenqiu0...@gmail.com> @Rong > Rong <walter...@gmail.com> > It seems that we have reached an agreement. The “application mode” > is regarded as the enhanced “per-job”. It is > orthogonal with “cluster deploy”. Currently, we bind the “per-job” to > `run-user-main-on-client` and “application mode” > to `run-user-main-on-cluster`. > > Do you have other concerns to moving FLIP-85 to voting? > > > Best, > Yang > > Peter Huang <huangzhenqiu0...@gmail.com> 于2020年3月5日周四 下午12:48写道： > >> Hi Yang and Kostas, >> >> Thanks for the clarification. It makes more sense to me if the long term >> goal is to replace per job mode to application mode >> in the future (at the time that multiple execute can be supported). >> Before that, It will be better to keep the concept of >> application mode internally. As Yang suggested, User only need to use a >> `-R/-- remote-deploy` cli option to launch >> a per job cluster with the main function executed in cluster >> entry-point. +1 for the execution plan. >> >> >> >> Best Regards >> Peter Huang >> >> >> >> >> On Tue, Mar 3, 2020 at 7:11 AM Yang Wang <danrtsey...@gmail.com> wrote: >> >>> Hi Peter, >>> >>> Having the application mode does not mean we will drop the cluster-deploy >>> option. I just want to share some thoughts about “Application Mode”. >>> >>> >>> 1. The application mode could cover the per-job sematic. Its lifecyle is >>> bound >>> to the user `main()`. And all the jobs in the user main will be executed >>> in a same >>> Flink cluster. In first phase of FLIP-85 implementation, running user >>> main on the >>> cluster side could be supported in application mode. >>> >>> 2. Maybe in the future, we also need to support multiple `execute()` on >>> client side >>> in a same Flink cluster. Then the per-job mode will evolve to >>> application mode. >>> >>> 3. From user perspective, only a `-R/-- remote-deploy` cli option is >>> visible. They >>> are not aware of the application mode. >>> >>> 4. In the first phase, the application mode is working as “per-job”(only >>> one job in >>> the user main). We just leave more potential for the future. >>> >>> >>> I am not against with calling it “cluster deploy mode” if you all think >>> it is clearer for users. >>> >>> >>> >>> Best, >>> Yang >>> >>> Kostas Kloudas <kklou...@gmail.com> 于2020年3月3日周二 下午6:49写道： >>> >>>> Hi Peter, >>>> >>>> I understand your point. This is why I was also a bit torn about the >>>> name and my proposal was a bit aligned with yours (something along the >>>> lines of "cluster deploy" mode). >>>> >>>> But many of the other participants in the discussion suggested the >>>> "Application Mode". I think that the reasoning is that now the user's >>>> Application is more self-contained. >>>> It will be submitted to the cluster and the user can just disconnect. >>>> In addition, as discussed briefly in the doc, in the future there may >>>> be better support for multi-execute applications which will bring us >>>> one step closer to the true "Application Mode". But this is how I >>>> interpreted their arguments, of course they can also express their >>>> thoughts on the topic :) >>>> >>>> Cheers, >>>> Kostas >>>> >>>> On Mon, Mar 2, 2020 at 6:15 PM Peter Huang <huangzhenqiu0...@gmail.com> >>>> wrote: >>>> > >>>> > Hi Kostas, >>>> > >>>> > Thanks for updating the wiki. We have aligned with the >>>> implementations in the doc. But I feel it is still a little bit confusing >>>> of the naming from a user's perspective. It is well known that Flink >>>> support per job cluster and session cluster. The concept is in the layer of >>>> how a job is managed within Flink. The method introduced util now is a kind >>>> of mixing job and session cluster to promising the implementation >>>> complexity. We probably don't need to label it as Application Model as the >>>> same layer of per job cluster and session cluster. Conceptually, I think it >>>> is still a cluster mode implementation for per job cluster. >>>> > >>>> > To minimize the confusion of users, I think it would be better just >>>> an option of per job cluster for each type of cluster manager. How do you >>>> think? >>>> > >>>> > >>>> > Best Regards >>>> > Peter Huang >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > On Mon, Mar 2, 2020 at 7:22 AM Kostas Kloudas <kklou...@gmail.com> >>>> wrote: >>>> >> >>>> >> Hi Yang, >>>> >> >>>> >> The difference between per-job and application mode is that, as you >>>> >> described, in the per-job mode the main is executed on the client >>>> >> while in the application mode, the main is executed on the cluster. >>>> >> I do not think we have to offer "application mode" with running the >>>> >> main on the client side as this is exactly what the per-job mode does >>>> >> currently and, as you described also, it would be redundant. >>>> >> >>>> >> Sorry if this was not clear in the document. >>>> >> >>>> >> Cheers, >>>> >> Kostas >>>> >> >>>> >> On Mon, Mar 2, 2020 at 3:17 PM Yang Wang <danrtsey...@gmail.com> >>>> wrote: >>>> >> > >>>> >> > Hi Kostas, >>>> >> > >>>> >> > Thanks a lot for your conclusion and updating the FLIP-85 WIKI. >>>> Currently, i have no more >>>> >> > questions about motivation, approach, fault tolerance and the >>>> first phase implementation. >>>> >> > >>>> >> > I think the new title "Flink Application Mode" makes a lot senses >>>> to me. Especially for the >>>> >> > containerized environment, the cluster deploy option will be very >>>> useful. >>>> >> > >>>> >> > Just one concern, how do we introduce this new application mode to >>>> our users? >>>> >> > Each user program(i.e. `main()`) is an application. Currently, we >>>> intend to only support one >>>> >> > `execute()`. So what's the difference between per-job and >>>> application mode? >>>> >> > >>>> >> > For per-job, user `main()` is always executed on client side. And >>>> For application mode, user >>>> >> > `main()` could be executed on client or master side(configured via >>>> cli option). >>>> >> > Right? We need to have a clear concept. Otherwise, the users will >>>> be more and more confusing. >>>> >> > >>>> >> > >>>> >> > Best, >>>> >> > Yang >>>> >> > >>>> >> > Kostas Kloudas <kklou...@gmail.com> 于2020年3月2日周一 下午5:58写道： >>>> >> >> >>>> >> >> Hi all, >>>> >> >> >>>> >> >> I update >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode >>>> >> >> based on the discussion we had here: >>>> >> >> >>>> >> >> >>>> https://docs.google.com/document/d/1ji72s3FD9DYUyGuKnJoO4ApzV-nSsZa0-bceGXW7Ocw/edit# >>>> >> >> >>>> >> >> Please let me know what you think and please keep the discussion >>>> in the ML :) >>>> >> >> >>>> >> >> Thanks for starting the discussion and I hope that soon we will be >>>> >> >> able to vote on the FLIP. >>>> >> >> >>>> >> >> Cheers, >>>> >> >> Kostas >>>> >> >> >>>> >> >> On Thu, Jan 16, 2020 at 3:40 AM Yang Wang <danrtsey...@gmail.com> >>>> wrote: >>>> >> >> > >>>> >> >> > Hi all, >>>> >> >> > >>>> >> >> > Thanks a lot for the feedback from @Kostas Kloudas. Your all >>>> concerns are >>>> >> >> > on point. The FLIP-85 is mainly >>>> >> >> > focused on supporting cluster mode for per-job. Since it is >>>> more urgent and >>>> >> >> > have much more use >>>> >> >> > cases both in Yarn and Kubernetes deployment. For session >>>> cluster, we could >>>> >> >> > have more discussion >>>> >> >> > in a new thread later. >>>> >> >> > >>>> >> >> > #1, How to download the user jars and dependencies for per-job >>>> in cluster >>>> >> >> > mode? >>>> >> >> > For Yarn, we could register the user jars and dependencies as >>>> >> >> > LocalResource. They will be distributed >>>> >> >> > by Yarn. And once the JobManager and TaskManager launched, the >>>> jars are >>>> >> >> > already exists. >>>> >> >> > For Standalone per-job and K8s, we expect that the user jars >>>> >> >> > and dependencies are built into the image. >>>> >> >> > Or the InitContainer could be used for downloading. It is >>>> natively >>>> >> >> > distributed and we will not have bottleneck. >>>> >> >> > >>>> >> >> > #2, Job graph recovery >>>> >> >> > We could have an optimization to store job graph on the DFS. >>>> However, i >>>> >> >> > suggest building a new jobgraph >>>> >> >> > from the configuration is the default option. Since we will not >>>> always have >>>> >> >> > a DFS store when deploying a >>>> >> >> > Flink per-job cluster. Of course, we assume that using the same >>>> >> >> > configuration(e.g. job_id, user_jar, main_class, >>>> >> >> > main_args, parallelism, savepoint_settings, etc.) will get a >>>> same job >>>> >> >> > graph. I think the standalone per-job >>>> >> >> > already has the similar behavior. >>>> >> >> > >>>> >> >> > #3, What happens with jobs that have multiple execute calls? >>>> >> >> > Currently, it is really a problem. Even we use a local client >>>> on Flink >>>> >> >> > master side, it will have different behavior with >>>> >> >> > client mode. For client mode, if we execute multiple times, >>>> then we will >>>> >> >> > deploy multiple Flink clusters for each execute. >>>> >> >> > I am not pretty sure whether it is reasonable. However, i still >>>> think using >>>> >> >> > the local client is a good choice. We could >>>> >> >> > continue the discussion in a new thread. @Zili Chen < >>>> wander4...@gmail.com> Do >>>> >> >> > you want to drive this? >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > Best, >>>> >> >> > Yang >>>> >> >> > >>>> >> >> > Peter Huang <huangzhenqiu0...@gmail.com> 于2020年1月16日周四 >>>> 上午1:55写道： >>>> >> >> > >>>> >> >> > > Hi Kostas, >>>> >> >> > > >>>> >> >> > > Thanks for this feedback. I can't agree more about the >>>> opinion. The >>>> >> >> > > cluster mode should be added >>>> >> >> > > first in per job cluster. >>>> >> >> > > >>>> >> >> > > 1) For job cluster implementation >>>> >> >> > > 1. Job graph recovery from configuration or store as static >>>> job graph as >>>> >> >> > > session cluster. I think the static one will be better for >>>> less recovery >>>> >> >> > > time. >>>> >> >> > > Let me update the doc for details. >>>> >> >> > > >>>> >> >> > > 2. For job execute multiple times, I think @Zili Chen >>>> >> >> > > <wander4...@gmail.com> has proposed the local client >>>> solution that can >>>> >> >> > > the run program actually in the cluster entry point. We can >>>> put the >>>> >> >> > > implementation in the second stage, >>>> >> >> > > or even a new FLIP for further discussion. >>>> >> >> > > >>>> >> >> > > 2) For session cluster implementation >>>> >> >> > > We can disable the cluster mode for the session cluster in >>>> the first >>>> >> >> > > stage. I agree the jar downloading will be a painful thing. >>>> >> >> > > We can consider about PoC and performance evaluation first. >>>> If the end to >>>> >> >> > > end experience is good enough, then we can consider >>>> >> >> > > proceeding with the solution. >>>> >> >> > > >>>> >> >> > > Looking forward to more opinions from @Yang Wang < >>>> danrtsey...@gmail.com> @Zili >>>> >> >> > > Chen <wander4...@gmail.com> @Dian Fu <dian0511...@gmail.com>. >>>> >> >> > > >>>> >> >> > > >>>> >> >> > > Best Regards >>>> >> >> > > Peter Huang >>>> >> >> > > >>>> >> >> > > On Wed, Jan 15, 2020 at 7:50 AM Kostas Kloudas < >>>> kklou...@gmail.com> wrote: >>>> >> >> > > >>>> >> >> > >> Hi all, >>>> >> >> > >> >>>> >> >> > >> I am writing here as the discussion on the Google Doc seems >>>> to be a >>>> >> >> > >> bit difficult to follow. >>>> >> >> > >> >>>> >> >> > >> I think that in order to be able to make progress, it would >>>> be helpful >>>> >> >> > >> to focus on per-job mode for now. >>>> >> >> > >> The reason is that: >>>> >> >> > >> 1) making the (unique) JobSubmitHandler responsible for >>>> creating the >>>> >> >> > >> jobgraphs, >>>> >> >> > >> which includes downloading dependencies, is not an optimal >>>> solution >>>> >> >> > >> 2) even if we put the responsibility on the JobMaster, >>>> currently each >>>> >> >> > >> job has its own >>>> >> >> > >> JobMaster but they all run on the same process, so we have >>>> again a >>>> >> >> > >> single entity. >>>> >> >> > >> >>>> >> >> > >> Of course after this is done, and if we feel comfortable >>>> with the >>>> >> >> > >> solution, then we can go to the session mode. >>>> >> >> > >> >>>> >> >> > >> A second comment has to do with fault-tolerance in the >>>> per-job, >>>> >> >> > >> cluster-deploy mode. >>>> >> >> > >> In the document, it is suggested that upon recovery, the >>>> JobMaster of >>>> >> >> > >> each job re-creates the JobGraph. >>>> >> >> > >> I am just wondering if it is better to create and store the >>>> jobGraph >>>> >> >> > >> upon submission and only fetch it >>>> >> >> > >> upon recovery so that we have a static jobGraph. >>>> >> >> > >> >>>> >> >> > >> Finally, I have a question which is what happens with jobs >>>> that have >>>> >> >> > >> multiple execute calls? >>>> >> >> > >> The semantics seem to change compared to the current >>>> behaviour, right? >>>> >> >> > >> >>>> >> >> > >> Cheers, >>>> >> >> > >> Kostas >>>> >> >> > >> >>>> >> >> > >> On Wed, Jan 8, 2020 at 8:05 PM tison <wander4...@gmail.com> >>>> wrote: >>>> >> >> > >> > >>>> >> >> > >> > not always, Yang Wang is also not yet a committer but he >>>> can join the >>>> >> >> > >> > channel. I cannot find the id by clicking “Add new member >>>> in channel” so >>>> >> >> > >> > come to you and ask for try out the link. Possibly I will >>>> find other >>>> >> >> > >> ways >>>> >> >> > >> > but the original purpose is that the slack channel is a >>>> public area we >>>> >> >> > >> > discuss about developing... >>>> >> >> > >> > Best, >>>> >> >> > >> > tison. >>>> >> >> > >> > >>>> >> >> > >> > >>>> >> >> > >> > Peter Huang <huangzhenqiu0...@gmail.com> 于2020年1月9日周四 >>>> 上午2:44写道： >>>> >> >> > >> > >>>> >> >> > >> > > Hi Tison, >>>> >> >> > >> > > >>>> >> >> > >> > > I am not the committer of Flink yet. I think I can't >>>> join it also. >>>> >> >> > >> > > >>>> >> >> > >> > > >>>> >> >> > >> > > Best Regards >>>> >> >> > >> > > Peter Huang >>>> >> >> > >> > > >>>> >> >> > >> > > On Wed, Jan 8, 2020 at 9:39 AM tison < >>>> wander4...@gmail.com> wrote: >>>> >> >> > >> > > >>>> >> >> > >> > > > Hi Peter, >>>> >> >> > >> > > > >>>> >> >> > >> > > > Could you try out this link? >>>> >> >> > >> > > https://the-asf.slack.com/messages/CNA3ADZPH >>>> >> >> > >> > > > >>>> >> >> > >> > > > Best, >>>> >> >> > >> > > > tison. >>>> >> >> > >> > > > >>>> >> >> > >> > > > >>>> >> >> > >> > > > Peter Huang <huangzhenqiu0...@gmail.com> 于2020年1月9日周四 >>>> 上午1:22写道： >>>> >> >> > >> > > > >>>> >> >> > >> > > > > Hi Tison, >>>> >> >> > >> > > > > >>>> >> >> > >> > > > > I can't join the group with shared link. Would you >>>> please add me >>>> >> >> > >> into >>>> >> >> > >> > > the >>>> >> >> > >> > > > > group? My slack account is huangzhenqiu0825. >>>> >> >> > >> > > > > Thank you in advance. >>>> >> >> > >> > > > > >>>> >> >> > >> > > > > >>>> >> >> > >> > > > > Best Regards >>>> >> >> > >> > > > > Peter Huang >>>> >> >> > >> > > > > >>>> >> >> > >> > > > > On Wed, Jan 8, 2020 at 12:02 AM tison < >>>> wander4...@gmail.com> >>>> >> >> > >> wrote: >>>> >> >> > >> > > > > >>>> >> >> > >> > > > > > Hi Peter, >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > > As described above, this effort should get >>>> attention from people >>>> >> >> > >> > > > > developing >>>> >> >> > >> > > > > > FLIP-73 a.k.a. Executor abstractions. I recommend >>>> you to join >>>> >> >> > >> the >>>> >> >> > >> > > > public >>>> >> >> > >> > > > > > slack channel[1] for Flink Client API Enhancement >>>> and you can >>>> >> >> > >> try to >>>> >> >> > >> > > > > share >>>> >> >> > >> > > > > > you detailed thoughts there. It possibly gets more >>>> concrete >>>> >> >> > >> > > attentions. >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > > Best, >>>> >> >> > >> > > > > > tison. >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > > [1] >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > >>>> >> >> > >> > > > >>>> >> >> > >> > > >>>> >> >> > >> >>>> https://slack.com/share/IS21SJ75H/Rk8HhUly9FuEHb7oGwBZ33uL/enQtODg2MDYwNjE5MTg3LTA2MjIzNDc1M2ZjZDVlMjdlZjk1M2RkYmJhNjAwMTk2ZDZkODQ4NmY5YmI4OGRhNWJkYTViMTM1NzlmMzc4OWM >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > > Peter Huang <huangzhenqiu0...@gmail.com> >>>> 于2020年1月7日周二 上午5:09写道： >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > > > Dear All, >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > Happy new year! According to existing feedback >>>> from the >>>> >> >> > >> community, >>>> >> >> > >> > > we >>>> >> >> > >> > > > > > > revised the doc with the consideration of >>>> session cluster >>>> >> >> > >> support, >>>> >> >> > >> > > > and >>>> >> >> > >> > > > > > > concrete interface changes needed and execution >>>> plan. Please >>>> >> >> > >> take >>>> >> >> > >> > > one >>>> >> >> > >> > > > > > more >>>> >> >> > >> > > > > > > round of review at your most convenient time. >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > >>>> >> >> > >> > > > >>>> >> >> > >> > > >>>> >> >> > >> >>>> https://docs.google.com/document/d/1aAwVjdZByA-0CHbgv16Me-vjaaDMCfhX7TzVVTuifYM/edit# >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > Best Regards >>>> >> >> > >> > > > > > > Peter Huang >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > On Thu, Jan 2, 2020 at 11:29 AM Peter Huang < >>>> >> >> > >> > > > > huangzhenqiu0...@gmail.com> >>>> >> >> > >> > > > > > > wrote: >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > > > Hi Dian, >>>> >> >> > >> > > > > > > > Thanks for giving us valuable feedbacks. >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > 1) It's better to have a whole design for this >>>> feature >>>> >> >> > >> > > > > > > > For the suggestion of enabling the cluster >>>> mode also session >>>> >> >> > >> > > > > cluster, I >>>> >> >> > >> > > > > > > > think Flink already supported it. >>>> WebSubmissionExtension >>>> >> >> > >> already >>>> >> >> > >> > > > > allows >>>> >> >> > >> > > > > > > > users to start a job with the specified jar by >>>> using web UI. >>>> >> >> > >> > > > > > > > But we need to enable the feature from CLI for >>>> both local >>>> >> >> > >> jar, >>>> >> >> > >> > > > remote >>>> >> >> > >> > > > > > > jar. >>>> >> >> > >> > > > > > > > I will align with Yang Wang first about the >>>> details and >>>> >> >> > >> update >>>> >> >> > >> > > the >>>> >> >> > >> > > > > > design >>>> >> >> > >> > > > > > > > doc. >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > 2) It's better to consider the convenience for >>>> users, such >>>> >> >> > >> as >>>> >> >> > >> > > > > debugging >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > I am wondering whether we can store the >>>> exception in >>>> >> >> > >> jobgragh >>>> >> >> > >> > > > > > > > generation in application master. As no >>>> streaming graph can >>>> >> >> > >> be >>>> >> >> > >> > > > > > scheduled >>>> >> >> > >> > > > > > > in >>>> >> >> > >> > > > > > > > this case, there will be no more TM will be >>>> requested from >>>> >> >> > >> > > FlinkRM. >>>> >> >> > >> > > > > > > > If the AM is still running, users can still >>>> query it from >>>> >> >> > >> CLI. As >>>> >> >> > >> > > > it >>>> >> >> > >> > > > > > > > requires more change, we can get some feedback >>>> from < >>>> >> >> > >> > > > > > aljos...@apache.org >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > and @zjf...@gmail.com <zjf...@gmail.com>. >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > 3) It's better to consider the impact to the >>>> stability of >>>> >> >> > >> the >>>> >> >> > >> > > > cluster >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > I agree with Yang Wang's opinion. >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > Best Regards >>>> >> >> > >> > > > > > > > Peter Huang >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > > On Sun, Dec 29, 2019 at 9:44 PM Dian Fu < >>>> >> >> > >> dian0511...@gmail.com> >>>> >> >> > >> > > > > wrote: >>>> >> >> > >> > > > > > > > >>>> >> >> > >> > > > > > > >> Hi all, >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> Sorry to jump into this discussion. Thanks >>>> everyone for the >>>> >> >> > >> > > > > > discussion. >>>> >> >> > >> > > > > > > >> I'm very interested in this topic although >>>> I'm not an >>>> >> >> > >> expert in >>>> >> >> > >> > > > this >>>> >> >> > >> > > > > > > part. >>>> >> >> > >> > > > > > > >> So I'm glad to share my thoughts as following: >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> 1) It's better to have a whole design for >>>> this feature >>>> >> >> > >> > > > > > > >> As we know, there are two deployment modes: >>>> per-job mode >>>> >> >> > >> and >>>> >> >> > >> > > > session >>>> >> >> > >> > > > > > > >> mode. I'm wondering which mode really needs >>>> this feature. >>>> >> >> > >> As the >>>> >> >> > >> > > > > > design >>>> >> >> > >> > > > > > > doc >>>> >> >> > >> > > > > > > >> mentioned, per-job mode is more used for >>>> streaming jobs and >>>> >> >> > >> > > > session >>>> >> >> > >> > > > > > > mode is >>>> >> >> > >> > > > > > > >> usually used for batch jobs(Of course, the >>>> job types and >>>> >> >> > >> the >>>> >> >> > >> > > > > > deployment >>>> >> >> > >> > > > > > > >> modes are orthogonal). Usually streaming job >>>> is only >>>> >> >> > >> needed to >>>> >> >> > >> > > be >>>> >> >> > >> > > > > > > submitted >>>> >> >> > >> > > > > > > >> once and it will run for days or weeks, while >>>> batch jobs >>>> >> >> > >> will be >>>> >> >> > >> > > > > > > submitted >>>> >> >> > >> > > > > > > >> more frequently compared with streaming jobs. >>>> This means >>>> >> >> > >> that >>>> >> >> > >> > > > maybe >>>> >> >> > >> > > > > > > session >>>> >> >> > >> > > > > > > >> mode also needs this feature. However, if we >>>> support this >>>> >> >> > >> > > feature >>>> >> >> > >> > > > in >>>> >> >> > >> > > > > > > >> session mode, the application master will >>>> become the new >>>> >> >> > >> > > > centralized >>>> >> >> > >> > > > > > > >> service(which should be solved). So in this >>>> case, it's >>>> >> >> > >> better to >>>> >> >> > >> > > > > have >>>> >> >> > >> > > > > > a >>>> >> >> > >> > > > > > > >> complete design for both per-job mode and >>>> session mode. >>>> >> >> > >> > > > Furthermore, >>>> >> >> > >> > > > > > > even >>>> >> >> > >> > > > > > > >> if we can do it phase by phase, we need to >>>> have a whole >>>> >> >> > >> picture >>>> >> >> > >> > > of >>>> >> >> > >> > > > > how >>>> >> >> > >> > > > > > > it >>>> >> >> > >> > > > > > > >> works in both per-job mode and session mode. >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> 2) It's better to consider the convenience >>>> for users, such >>>> >> >> > >> as >>>> >> >> > >> > > > > > debugging >>>> >> >> > >> > > > > > > >> After we finish this feature, the job graph >>>> will be >>>> >> >> > >> compiled in >>>> >> >> > >> > > > the >>>> >> >> > >> > > > > > > >> application master, which means that users >>>> cannot easily >>>> >> >> > >> get the >>>> >> >> > >> > > > > > > exception >>>> >> >> > >> > > > > > > >> message synchorousely in the job client if >>>> there are >>>> >> >> > >> problems >>>> >> >> > >> > > > during >>>> >> >> > >> > > > > > the >>>> >> >> > >> > > > > > > >> job graph compiling (especially for platform >>>> users), such >>>> >> >> > >> as the >>>> >> >> > >> > > > > > > resource >>>> >> >> > >> > > > > > > >> path is incorrect, the user program itself >>>> has some >>>> >> >> > >> problems, >>>> >> >> > >> > > etc. >>>> >> >> > >> > > > > > What >>>> >> >> > >> > > > > > > I'm >>>> >> >> > >> > > > > > > >> thinking is that maybe we should throw the >>>> exceptions as >>>> >> >> > >> early >>>> >> >> > >> > > as >>>> >> >> > >> > > > > > > possible >>>> >> >> > >> > > > > > > >> (during job submission stage). >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> 3) It's better to consider the impact to the >>>> stability of >>>> >> >> > >> the >>>> >> >> > >> > > > > cluster >>>> >> >> > >> > > > > > > >> If we perform the compiling in the >>>> application master, we >>>> >> >> > >> should >>>> >> >> > >> > > > > > > consider >>>> >> >> > >> > > > > > > >> the impact of the compiling errors. Although >>>> YARN could >>>> >> >> > >> resume >>>> >> >> > >> > > the >>>> >> >> > >> > > > > > > >> application master in case of failures, but >>>> in some case >>>> >> >> > >> the >>>> >> >> > >> > > > > compiling >>>> >> >> > >> > > > > > > >> failure may be a waste of cluster resource >>>> and may impact >>>> >> >> > >> the >>>> >> >> > >> > > > > > stability >>>> >> >> > >> > > > > > > the >>>> >> >> > >> > > > > > > >> cluster and the other jobs in the cluster, >>>> such as the >>>> >> >> > >> resource >>>> >> >> > >> > > > path >>>> >> >> > >> > > > > > is >>>> >> >> > >> > > > > > > >> incorrect, the user program itself has some >>>> problems(in >>>> >> >> > >> this >>>> >> >> > >> > > case, >>>> >> >> > >> > > > > job >>>> >> >> > >> > > > > > > >> failover cannot solve this kind of problems) >>>> etc. In the >>>> >> >> > >> current >>>> >> >> > >> > > > > > > >> implemention, the compiling errors are >>>> handled in the >>>> >> >> > >> client >>>> >> >> > >> > > side >>>> >> >> > >> > > > > and >>>> >> >> > >> > > > > > > there >>>> >> >> > >> > > > > > > >> is no impact to the cluster at all. >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> Regarding to 1), it's clearly pointed in the >>>> design doc >>>> >> >> > >> that >>>> >> >> > >> > > only >>>> >> >> > >> > > > > > > per-job >>>> >> >> > >> > > > > > > >> mode will be supported. However, I think it's >>>> better to >>>> >> >> > >> also >>>> >> >> > >> > > > > consider >>>> >> >> > >> > > > > > > the >>>> >> >> > >> > > > > > > >> session mode in the design doc. >>>> >> >> > >> > > > > > > >> Regarding to 2) and 3), I have not seen >>>> related sections >>>> >> >> > >> in the >>>> >> >> > >> > > > > design >>>> >> >> > >> > > > > > > >> doc. It will be good if we can cover them in >>>> the design >>>> >> >> > >> doc. >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> Feel free to correct me If there is anything I >>>> >> >> > >> misunderstand. >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> Regards, >>>> >> >> > >> > > > > > > >> Dian >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> > 在 2019年12月27日，上午3:13，Peter Huang < >>>> >> >> > >> huangzhenqiu0...@gmail.com> >>>> >> >> > >> > > > 写道： >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > Hi Yang, >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > I can't agree more. The effort definitely >>>> needs to align >>>> >> >> > >> with >>>> >> >> > >> > > > the >>>> >> >> > >> > > > > > > final >>>> >> >> > >> > > > > > > >> > goal of FLIP-73. >>>> >> >> > >> > > > > > > >> > I am thinking about whether we can achieve >>>> the goal with >>>> >> >> > >> two >>>> >> >> > >> > > > > phases. >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > 1) Phase I >>>> >> >> > >> > > > > > > >> > As the CLiFrontend will not be depreciated >>>> soon. We can >>>> >> >> > >> still >>>> >> >> > >> > > > use >>>> >> >> > >> > > > > > the >>>> >> >> > >> > > > > > > >> > deployMode flag there, >>>> >> >> > >> > > > > > > >> > pass the program info through Flink >>>> configuration, use >>>> >> >> > >> the >>>> >> >> > >> > > > > > > >> > ClassPathJobGraphRetriever >>>> >> >> > >> > > > > > > >> > to generate the job graph in >>>> ClusterEntrypoints of yarn >>>> >> >> > >> and >>>> >> >> > >> > > > > > > Kubernetes. >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > 2) Phase II >>>> >> >> > >> > > > > > > >> > In AbstractJobClusterExecutor, the job >>>> graph is >>>> >> >> > >> generated in >>>> >> >> > >> > > > the >>>> >> >> > >> > > > > > > >> execute >>>> >> >> > >> > > > > > > >> > function. We can still >>>> >> >> > >> > > > > > > >> > use the deployMode in it. With deployMode = >>>> cluster, the >>>> >> >> > >> > > execute >>>> >> >> > >> > > > > > > >> function >>>> >> >> > >> > > > > > > >> > only starts the cluster. >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > When >>>> {Yarn/Kuberneates}PerJobClusterEntrypoint starts, >>>> >> >> > >> It will >>>> >> >> > >> > > > > start >>>> >> >> > >> > > > > > > the >>>> >> >> > >> > > > > > > >> > dispatch first, then we can use >>>> >> >> > >> > > > > > > >> > a ClusterEnvironment similar to >>>> ContextEnvironment to >>>> >> >> > >> submit >>>> >> >> > >> > > the >>>> >> >> > >> > > > > job >>>> >> >> > >> > > > > > > >> with >>>> >> >> > >> > > > > > > >> > jobName the local >>>> >> >> > >> > > > > > > >> > dispatcher. For the details, we need more >>>> investigation. >>>> >> >> > >> Let's >>>> >> >> > >> > > > > wait >>>> >> >> > >> > > > > > > >> > for @Aljoscha >>>> >> >> > >> > > > > > > >> > Krettek <aljos...@apache.org> @Till >>>> Rohrmann < >>>> >> >> > >> > > > > trohrm...@apache.org >>>> >> >> > >> > > > > > >'s >>>> >> >> > >> > > > > > > >> > feedback after the holiday season. >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > Thank you in advance. Merry Chrismas and >>>> Happy New >>>> >> >> > >> Year!!! >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > Best Regards >>>> >> >> > >> > > > > > > >> > Peter Huang >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> > On Wed, Dec 25, 2019 at 1:08 AM Yang Wang < >>>> >> >> > >> > > > danrtsey...@gmail.com> >>>> >> >> > >> > > > > > > >> wrote: >>>> >> >> > >> > > > > > > >> > >>>> >> >> > >> > > > > > > >> >> Hi Peter, >>>> >> >> > >> > > > > > > >> >> >>>> >> >> > >> > > > > > > >> >> I think we need to reconsider tison's >>>> suggestion >>>> >> >> > >> seriously. >>>> >> >> > >> > > > After >>>> >> >> > >> > > > > > > >> FLIP-73, >>>> >> >> > >> > > > > > > >> >> the deployJobCluster has >>>> >> >> > >> > > > > > > >> >> beenmoved into >>>> `JobClusterExecutor#execute`. It should >>>> >> >> > >> not be >>>> >> >> > >> > > > > > > perceived >>>> >> >> > >> > > > > > > >> >> for `CliFrontend`. That >>>> >> >> > >> > > > > > > >> >> means the user program will *ALWAYS* be >>>> executed on >>>> >> >> > >> client >>>> >> >> > >> > > > side. >>>> >> >> > >> > > > > > This >>>> >> >> > >> > > > > > > >> is >>>> >> >> > >> > > > > > > >> >> the by design behavior. >>>> >> >> > >> > > > > > > >> >> So, we could not just add `if(client mode) >>>> .. else >>>> >> >> > >> if(cluster >>>> >> >> > >> > > > > mode) >>>> >> >> > >> > > > > > > >> ...` >>>> >> >> > >> > > > > > > >> >> codes in `CliFrontend` to bypass >>>> >> >> > >> > > > > > > >> >> the executor. We need to find a clean way >>>> to decouple >>>> >> >> > >> > > executing >>>> >> >> > >> > > > > > user >>>> >> >> > >> > > > > > > >> >> program and deploying per-job >>>> >> >> > >> > > > > > > >> >> cluster. Based on this, we could support >>>> to execute user >>>> >> >> > >> > > > program >>>> >> >> > >> > > > > on >>>> >> >> > >> > > > > > > >> client >>>> >> >> > >> > > > > > > >> >> or master side. >>>> >> >> > >> > > > > > > >> >> >>>> >> >> > >> > > > > > > >> >> Maybe Aljoscha and Jeff could give some >>>> good >>>> >> >> > >> suggestions. >>>> >> >> > >> > > > > > > >> >> >>>> >> >> > >> > > > > > > >> >> >>>> >> >> > >> > > > > > > >> >> >>>> >> >> > >> > > > > > > >> >> Best, >>>> >> >> > >> > > > > > > >> >> Yang >>>> >> >> > >> > > > > > > >> >> >>>> >> >> > >> > > > > > > >> >> Peter Huang <huangzhenqiu0...@gmail.com> >>>> 于2019年12月25日周三 >>>> >> >> > >> > > > > 上午4:03写道： >>>> >> >> > >> > > > > > > >> >> >>>> >> >> > >> > > > > > > >> >>> Hi Jingjing, >>>> >> >> > >> > > > > > > >> >>> >>>> >> >> > >> > > > > > > >> >>> The improvement proposed is a deployment >>>> option for >>>> >> >> > >> CLI. For >>>> >> >> > >> > > > SQL >>>> >> >> > >> > > > > > > based >>>> >> >> > >> > > > > > > >> >>> Flink application, It is more convenient >>>> to use the >>>> >> >> > >> existing >>>> >> >> > >> > > > > model >>>> >> >> > >> > > > > > > in >>>> >> >> > >> > > > > > > >> >>> SqlClient in which >>>> >> >> > >> > > > > > > >> >>> the job graph is generated within >>>> SqlClient. After >>>> >> >> > >> adding >>>> >> >> > >> > > the >>>> >> >> > >> > > > > > > delayed >>>> >> >> > >> > > > > > > >> job >>>> >> >> > >> > > > > > > >> >>> graph generation, I think there is no >>>> change is needed >>>> >> >> > >> for >>>> >> >> > >> > > > your >>>> >> >> > >> > > > > > > side. >>>> >> >> > >> > > > > > > >> >>> >>>> >> >> > >> > > > > > > >> >>> >>>> >> >> > >> > > > > > > >> >>> Best Regards >>>> >> >> > >> > > > > > > >> >>> Peter Huang >>>> >> >> > >> > > > > > > >> >>> >>>> >> >> > >> > > > > > > >> >>> >>>> >> >> > >> > > > > > > >> >>> On Wed, Dec 18, 2019 at 6:01 AM jingjing >>>> bai < >>>> >> >> > >> > > > > > > >> baijingjing7...@gmail.com> >>>> >> >> > >> > > > > > > >> >>> wrote: >>>> >> >> > >> > > > > > > >> >>> >>>> >> >> > >> > > > > > > >> >>>> hi peter: >>>> >> >> > >> > > > > > > >> >>>> we had extension SqlClent to support >>>> sql job >>>> >> >> > >> submit in >>>> >> >> > >> > > web >>>> >> >> > >> > > > > > base >>>> >> >> > >> > > > > > > on >>>> >> >> > >> > > > > > > >> >>>> flink 1.9. we support submit to yarn >>>> on per job >>>> >> >> > >> mode too. >>>> >> >> > >> > > > > > > >> >>>> in this case, the job graph >>>> generated on client >>>> >> >> > >> side >>>> >> >> > >> > > . I >>>> >> >> > >> > > > > > think >>>> >> >> > >> > > > > > > >> >>> this >>>> >> >> > >> > > > > > > >> >>>> discuss Mainly to improve api >>>> programme. but in my >>>> >> >> > >> case , >>>> >> >> > >> > > > > there >>>> >> >> > >> > > > > > is >>>> >> >> > >> > > > > > > >> no >>>> >> >> > >> > > > > > > >> >>>> jar to upload but only a sql string . >>>> >> >> > >> > > > > > > >> >>>> do u had more suggestion to improve >>>> for sql mode >>>> >> >> > >> or it >>>> >> >> > >> > > is >>>> >> >> > >> > > > > > only a >>>> >> >> > >> > > > > > > >> >>>> switch for api programme？ >>>> >> >> > >> > > > > > > >> >>>> >>>> >> >> > >> > > > > > > >> >>>> >>>> >> >> > >> > > > > > > >> >>>> best >>>> >> >> > >> > > > > > > >> >>>> bai jj >>>> >> >> > >> > > > > > > >> >>>> >>>> >> >> > >> > > > > > > >> >>>> >>>> >> >> > >> > > > > > > >> >>>> Yang Wang <danrtsey...@gmail.com> >>>> 于2019年12月18日周三 >>>> >> >> > >> 下午7:21写道： >>>> >> >> > >> > > > > > > >> >>>> >>>> >> >> > >> > > > > > > >> >>>>> I just want to revive this discussion. >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>>>> Recently, i am thinking about how to >>>> natively run >>>> >> >> > >> flink >>>> >> >> > >> > > > > per-job >>>> >> >> > >> > > > > > > >> >>> cluster on >>>> >> >> > >> > > > > > > >> >>>>> Kubernetes. >>>> >> >> > >> > > > > > > >> >>>>> The per-job mode on Kubernetes is very >>>> different >>>> >> >> > >> from on >>>> >> >> > >> > > > Yarn. >>>> >> >> > >> > > > > > And >>>> >> >> > >> > > > > > > >> we >>>> >> >> > >> > > > > > > >> >>> will >>>> >> >> > >> > > > > > > >> >>>>> have >>>> >> >> > >> > > > > > > >> >>>>> the same deployment requirements to the >>>> client and >>>> >> >> > >> entry >>>> >> >> > >> > > > > point. >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>>>> 1. Flink client not always need a local >>>> jar to start >>>> >> >> > >> a >>>> >> >> > >> > > Flink >>>> >> >> > >> > > > > > > per-job >>>> >> >> > >> > > > > > > >> >>>>> cluster. We could >>>> >> >> > >> > > > > > > >> >>>>> support multiple schemas. For example, >>>> >> >> > >> > > > file:///path/of/my.jar >>>> >> >> > >> > > > > > > means >>>> >> >> > >> > > > > > > >> a >>>> >> >> > >> > > > > > > >> >>> jar >>>> >> >> > >> > > > > > > >> >>>>> located >>>> >> >> > >> > > > > > > >> >>>>> at client side, >>>> >> >> > >> hdfs://myhdfs/user/myname/flink/my.jar >>>> >> >> > >> > > > means a >>>> >> >> > >> > > > > > jar >>>> >> >> > >> > > > > > > >> >>> located >>>> >> >> > >> > > > > > > >> >>>>> at >>>> >> >> > >> > > > > > > >> >>>>> remote hdfs, >>>> local:///path/in/image/my.jar means a >>>> >> >> > >> jar >>>> >> >> > >> > > > located >>>> >> >> > >> > > > > > at >>>> >> >> > >> > > > > > > >> >>>>> jobmanager side. >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>>>> 2. Support running user program on >>>> master side. This >>>> >> >> > >> also >>>> >> >> > >> > > > > means >>>> >> >> > >> > > > > > > the >>>> >> >> > >> > > > > > > >> >>> entry >>>> >> >> > >> > > > > > > >> >>>>> point >>>> >> >> > >> > > > > > > >> >>>>> will generate the job graph on master >>>> side. We could >>>> >> >> > >> use >>>> >> >> > >> > > the >>>> >> >> > >> > > > > > > >> >>>>> ClasspathJobGraphRetriever >>>> >> >> > >> > > > > > > >> >>>>> or start a local Flink client to >>>> achieve this >>>> >> >> > >> purpose. >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>>>> cc tison, Aljoscha & Kostas Do you >>>> think this is the >>>> >> >> > >> right >>>> >> >> > >> > > > > > > >> direction we >>>> >> >> > >> > > > > > > >> >>>>> need to work? >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>>>> tison <wander4...@gmail.com> >>>> 于2019年12月12日周四 >>>> >> >> > >> 下午4:48写道： >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>>>>> A quick idea is that we separate the >>>> deployment >>>> >> >> > >> from user >>>> >> >> > >> > > > > > program >>>> >> >> > >> > > > > > > >> >>> that >>>> >> >> > >> > > > > > > >> >>>>> it >>>> >> >> > >> > > > > > > >> >>>>>> has always been done >>>> >> >> > >> > > > > > > >> >>>>>> outside the program. On user program >>>> executed there >>>> >> >> > >> is >>>> >> >> > >> > > > > always a >>>> >> >> > >> > > > > > > >> >>>>>> ClusterClient that communicates with >>>> >> >> > >> > > > > > > >> >>>>>> an existing cluster, remote or local. >>>> It will be >>>> >> >> > >> another >>>> >> >> > >> > > > > thread >>>> >> >> > >> > > > > > > so >>>> >> >> > >> > > > > > > >> >>> just >>>> >> >> > >> > > > > > > >> >>>>> for >>>> >> >> > >> > > > > > > >> >>>>>> your information. >>>> >> >> > >> > > > > > > >> >>>>>> >>>> >> >> > >> > > > > > > >> >>>>>> Best, >>>> >> >> > >> > > > > > > >> >>>>>> tison. >>>> >> >> > >> > > > > > > >> >>>>>> >>>> >> >> > >> > > > > > > >> >>>>>> >>>> >> >> > >> > > > > > > >> >>>>>> tison <wander4...@gmail.com> >>>> 于2019年12月12日周四 >>>> >> >> > >> 下午4:40写道： >>>> >> >> > >> > > > > > > >> >>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>> Hi Peter, >>>> >> >> > >> > > > > > > >> >>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>> Another concern I realized recently >>>> is that with >>>> >> >> > >> current >>>> >> >> > >> > > > > > > Executors >>>> >> >> > >> > > > > > > >> >>>>>>> abstraction(FLIP-73) >>>> >> >> > >> > > > > > > >> >>>>>>> I'm afraid that user program is >>>> designed to ALWAYS >>>> >> >> > >> run >>>> >> >> > >> > > on >>>> >> >> > >> > > > > the >>>> >> >> > >> > > > > > > >> >>> client >>>> >> >> > >> > > > > > > >> >>>>>> side. >>>> >> >> > >> > > > > > > >> >>>>>>> Specifically, >>>> >> >> > >> > > > > > > >> >>>>>>> we deploy the job in executor when >>>> env.execute >>>> >> >> > >> called. >>>> >> >> > >> > > > This >>>> >> >> > >> > > > > > > >> >>>>> abstraction >>>> >> >> > >> > > > > > > >> >>>>>>> possibly prevents >>>> >> >> > >> > > > > > > >> >>>>>>> Flink runs user program on the >>>> cluster side. >>>> >> >> > >> > > > > > > >> >>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>> For your proposal, in this case we >>>> already >>>> >> >> > >> compiled the >>>> >> >> > >> > > > > > program >>>> >> >> > >> > > > > > > >> and >>>> >> >> > >> > > > > > > >> >>>>> run >>>> >> >> > >> > > > > > > >> >>>>>> on >>>> >> >> > >> > > > > > > >> >>>>>>> the client side, >>>> >> >> > >> > > > > > > >> >>>>>>> even we deploy a cluster and retrieve >>>> job graph >>>> >> >> > >> from >>>> >> >> > >> > > > program >>>> >> >> > >> > > > > > > >> >>>>> metadata, it >>>> >> >> > >> > > > > > > >> >>>>>>> doesn't make >>>> >> >> > >> > > > > > > >> >>>>>>> many sense. >>>> >> >> > >> > > > > > > >> >>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>> cc Aljoscha & Kostas what do you >>>> think about this >>>> >> >> > >> > > > > constraint? >>>> >> >> > >> > > > > > > >> >>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>> Best, >>>> >> >> > >> > > > > > > >> >>>>>>> tison. >>>> >> >> > >> > > > > > > >> >>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>> Peter Huang < >>>> huangzhenqiu0...@gmail.com> >>>> >> >> > >> 于2019年12月10日周二 >>>> >> >> > >> > > > > > > >> 下午12:45写道： >>>> >> >> > >> > > > > > > >> >>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>> Hi Tison, >>>> >> >> > >> > > > > > > >> >>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>> Yes, you are right. I think I made >>>> the wrong >>>> >> >> > >> argument >>>> >> >> > >> > > in >>>> >> >> > >> > > > > the >>>> >> >> > >> > > > > > > doc. >>>> >> >> > >> > > > > > > >> >>>>>>>> Basically, the packaging jar problem >>>> is only for >>>> >> >> > >> > > platform >>>> >> >> > >> > > > > > > users. >>>> >> >> > >> > > > > > > >> >>> In >>>> >> >> > >> > > > > > > >> >>>>> our >>>> >> >> > >> > > > > > > >> >>>>>>>> internal deploy service, >>>> >> >> > >> > > > > > > >> >>>>>>>> we further optimized the deployment >>>> latency by >>>> >> >> > >> letting >>>> >> >> > >> > > > > users >>>> >> >> > >> > > > > > to >>>> >> >> > >> > > > > > > >> >>>>>> packaging >>>> >> >> > >> > > > > > > >> >>>>>>>> flink-runtime together with the uber >>>> jar, so that >>>> >> >> > >> we >>>> >> >> > >> > > > don't >>>> >> >> > >> > > > > > need >>>> >> >> > >> > > > > > > >> to >>>> >> >> > >> > > > > > > >> >>>>>>>> consider >>>> >> >> > >> > > > > > > >> >>>>>>>> multiple flink version >>>> >> >> > >> > > > > > > >> >>>>>>>> support for now. In the session >>>> client mode, as >>>> >> >> > >> Flink >>>> >> >> > >> > > > libs >>>> >> >> > >> > > > > > will >>>> >> >> > >> > > > > > > >> be >>>> >> >> > >> > > > > > > >> >>>>>> shipped >>>> >> >> > >> > > > > > > >> >>>>>>>> anyway as local resources of yarn. >>>> Users actually >>>> >> >> > >> don't >>>> >> >> > >> > > > > need >>>> >> >> > >> > > > > > to >>>> >> >> > >> > > > > > > >> >>>>> package >>>> >> >> > >> > > > > > > >> >>>>>>>> those libs into job jar. >>>> >> >> > >> > > > > > > >> >>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>> Best Regards >>>> >> >> > >> > > > > > > >> >>>>>>>> Peter Huang >>>> >> >> > >> > > > > > > >> >>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>> On Mon, Dec 9, 2019 at 8:35 PM tison >>>> < >>>> >> >> > >> > > > wander4...@gmail.com >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > > > >> >>> wrote: >>>> >> >> > >> > > > > > > >> >>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> 3. What do you mean about the >>>> package? Do users >>>> >> >> > >> need >>>> >> >> > >> > > to >>>> >> >> > >> > > > > > > >> >>> compile >>>> >> >> > >> > > > > > > >> >>>>>> their >>>> >> >> > >> > > > > > > >> >>>>>>>>> jars >>>> >> >> > >> > > > > > > >> >>>>>>>>> inlcuding flink-clients, >>>> flink-optimizer, >>>> >> >> > >> flink-table >>>> >> >> > >> > > > > codes? >>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>> The answer should be no because >>>> they exist in >>>> >> >> > >> system >>>> >> >> > >> > > > > > > classpath. >>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>> Best, >>>> >> >> > >> > > > > > > >> >>>>>>>>> tison. >>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>> Yang Wang <danrtsey...@gmail.com> >>>> 于2019年12月10日周二 >>>> >> >> > >> > > > > 下午12:18写道： >>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> Hi Peter, >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> Thanks a lot for starting this >>>> discussion. I >>>> >> >> > >> think >>>> >> >> > >> > > this >>>> >> >> > >> > > > > is >>>> >> >> > >> > > > > > a >>>> >> >> > >> > > > > > > >> >>> very >>>> >> >> > >> > > > > > > >> >>>>>>>> useful >>>> >> >> > >> > > > > > > >> >>>>>>>>>> feature. >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> Not only for Yarn, i am focused on >>>> flink on >>>> >> >> > >> > > Kubernetes >>>> >> >> > >> > > > > > > >> >>>>> integration >>>> >> >> > >> > > > > > > >> >>>>>> and >>>> >> >> > >> > > > > > > >> >>>>>>>>> come >>>> >> >> > >> > > > > > > >> >>>>>>>>>> across the same >>>> >> >> > >> > > > > > > >> >>>>>>>>>> problem. I do not want the job >>>> graph generated >>>> >> >> > >> on >>>> >> >> > >> > > > client >>>> >> >> > >> > > > > > > side. >>>> >> >> > >> > > > > > > >> >>>>>>>> Instead, >>>> >> >> > >> > > > > > > >> >>>>>>>>> the >>>> >> >> > >> > > > > > > >> >>>>>>>>>> user jars are built in >>>> >> >> > >> > > > > > > >> >>>>>>>>>> a user-defined image. When the job >>>> manager >>>> >> >> > >> launched, >>>> >> >> > >> > > we >>>> >> >> > >> > > > > > just >>>> >> >> > >> > > > > > > >> >>>>> need to >>>> >> >> > >> > > > > > > >> >>>>>>>>>> generate the job graph >>>> >> >> > >> > > > > > > >> >>>>>>>>>> based on local user jars. >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> I have some small suggestion about >>>> this. >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> 1. `ProgramJobGraphRetriever` is >>>> very similar to >>>> >> >> > >> > > > > > > >> >>>>>>>>>> `ClasspathJobGraphRetriever`, the >>>> differences >>>> >> >> > >> > > > > > > >> >>>>>>>>>> are the former needs >>>> `ProgramMetadata` and the >>>> >> >> > >> latter >>>> >> >> > >> > > > > needs >>>> >> >> > >> > > > > > > >> >>> some >>>> >> >> > >> > > > > > > >> >>>>>>>>> arguments. >>>> >> >> > >> > > > > > > >> >>>>>>>>>> Is it possible to >>>> >> >> > >> > > > > > > >> >>>>>>>>>> have an unified >>>> `JobGraphRetriever` to support >>>> >> >> > >> both? >>>> >> >> > >> > > > > > > >> >>>>>>>>>> 2. Is it possible to not use a >>>> local user jar to >>>> >> >> > >> > > start >>>> >> >> > >> > > > a >>>> >> >> > >> > > > > > > >> >>> per-job >>>> >> >> > >> > > > > > > >> >>>>>>>> cluster? >>>> >> >> > >> > > > > > > >> >>>>>>>>>> In your case, the user jars has >>>> >> >> > >> > > > > > > >> >>>>>>>>>> existed on hdfs already and we do >>>> need to >>>> >> >> > >> download >>>> >> >> > >> > > the >>>> >> >> > >> > > > > jars >>>> >> >> > >> > > > > > > to >>>> >> >> > >> > > > > > > >> >>>>>>>> deployer >>>> >> >> > >> > > > > > > >> >>>>>>>>>> service. Currently, we >>>> >> >> > >> > > > > > > >> >>>>>>>>>> always need a local user jar to >>>> start a flink >>>> >> >> > >> > > cluster. >>>> >> >> > >> > > > It >>>> >> >> > >> > > > > > is >>>> >> >> > >> > > > > > > >> >>> be >>>> >> >> > >> > > > > > > >> >>>>>> great >>>> >> >> > >> > > > > > > >> >>>>>>>> if >>>> >> >> > >> > > > > > > >> >>>>>>>>> we >>>> >> >> > >> > > > > > > >> >>>>>>>>>> could support remote user jars. >>>> >> >> > >> > > > > > > >> >>>>>>>>>>>> In the implementation, we assume >>>> users package >>>> >> >> > >> > > > > > > >> >>> flink-clients, >>>> >> >> > >> > > > > > > >> >>>>>>>>>> flink-optimizer, flink-table >>>> together within >>>> >> >> > >> the job >>>> >> >> > >> > > > jar. >>>> >> >> > >> > > > > > > >> >>>>> Otherwise, >>>> >> >> > >> > > > > > > >> >>>>>>>> the >>>> >> >> > >> > > > > > > >> >>>>>>>>>> job graph generation within >>>> >> >> > >> JobClusterEntryPoint will >>>> >> >> > >> > > > > fail. >>>> >> >> > >> > > > > > > >> >>>>>>>>>> 3. What do you mean about the >>>> package? Do users >>>> >> >> > >> need >>>> >> >> > >> > > to >>>> >> >> > >> > > > > > > >> >>> compile >>>> >> >> > >> > > > > > > >> >>>>>> their >>>> >> >> > >> > > > > > > >> >>>>>>>>> jars >>>> >> >> > >> > > > > > > >> >>>>>>>>>> inlcuding flink-clients, >>>> flink-optimizer, >>>> >> >> > >> flink-table >>>> >> >> > >> > > > > > codes? >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> Best, >>>> >> >> > >> > > > > > > >> >>>>>>>>>> Yang >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> Peter Huang < >>>> huangzhenqiu0...@gmail.com> >>>> >> >> > >> > > > 于2019年12月10日周二 >>>> >> >> > >> > > > > > > >> >>>>> 上午2:37写道： >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Dear All, >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Recently, the Flink community >>>> starts to >>>> >> >> > >> improve the >>>> >> >> > >> > > > yarn >>>> >> >> > >> > > > > > > >> >>>>> cluster >>>> >> >> > >> > > > > > > >> >>>>>>>>>> descriptor >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> to make job jar and config files >>>> configurable >>>> >> >> > >> from >>>> >> >> > >> > > > CLI. >>>> >> >> > >> > > > > It >>>> >> >> > >> > > > > > > >> >>>>>> improves >>>> >> >> > >> > > > > > > >> >>>>>>>> the >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> flexibility of Flink deployment >>>> Yarn Per Job >>>> >> >> > >> Mode. >>>> >> >> > >> > > > For >>>> >> >> > >> > > > > > > >> >>>>> platform >>>> >> >> > >> > > > > > > >> >>>>>>>> users >>>> >> >> > >> > > > > > > >> >>>>>>>>>> who >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> manage tens of hundreds of >>>> streaming pipelines >>>> >> >> > >> for >>>> >> >> > >> > > the >>>> >> >> > >> > > > > > whole >>>> >> >> > >> > > > > > > >> >>>>> org >>>> >> >> > >> > > > > > > >> >>>>>> or >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> company, we found the job graph >>>> generation in >>>> >> >> > >> > > > > client-side >>>> >> >> > >> > > > > > is >>>> >> >> > >> > > > > > > >> >>>>>> another >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> pinpoint. Thus, we want to >>>> propose a >>>> >> >> > >> configurable >>>> >> >> > >> > > > > feature >>>> >> >> > >> > > > > > > >> >>> for >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> FlinkYarnSessionCli. The feature >>>> can allow >>>> >> >> > >> users to >>>> >> >> > >> > > > > choose >>>> >> >> > >> > > > > > > >> >>> the >>>> >> >> > >> > > > > > > >> >>>>> job >>>> >> >> > >> > > > > > > >> >>>>>>>>> graph >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> generation in Flink >>>> ClusterEntryPoint so that >>>> >> >> > >> the >>>> >> >> > >> > > job >>>> >> >> > >> > > > > jar >>>> >> >> > >> > > > > > > >> >>>>> doesn't >>>> >> >> > >> > > > > > > >> >>>>>>>> need >>>> >> >> > >> > > > > > > >> >>>>>>>>> to >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> be locally for the job graph >>>> generation. The >>>> >> >> > >> > > proposal >>>> >> >> > >> > > > is >>>> >> >> > >> > > > > > > >> >>>>> organized >>>> >> >> > >> > > > > > > >> >>>>>>>> as a >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> FLIP >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>> >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>> >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > >>>> >> >> > >> > > > >>>> >> >> > >> > > >>>> >> >> > >> >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Delayed+JobGraph+Generation >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> . >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Any questions and suggestions are >>>> welcomed. >>>> >> >> > >> Thank >>>> >> >> > >> > > you >>>> >> >> > >> > > > in >>>> >> >> > >> > > > > > > >> >>>>> advance. >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Best Regards >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> Peter Huang >>>> >> >> > >> > > > > > > >> >>>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>>> >>>> >> >> > >> > > > > > > >> >>>>>> >>>> >> >> > >> > > > > > > >> >>>>> >>>> >> >> > >> > > > > > > >> >>>> >>>> >> >> > >> > > > > > > >> >>> >>>> >> >> > >> > > > > > > >> >> >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >> >>>> >> >> > >> > > > > > > >>>> >> >> > >> > > > > > >>>> >> >> > >> > > > > >>>> >> >> > >> > > > >>>> >> >> > >> > > >>>> >> >> > >> >>>> >> >> > > >>>> >> >> >>>> >>>
