Hi Yang, Thanks for the suggestion. Actually I forgot to share the original google doc with you. Feel free to comment directly on it, so that I may revise it based on people's feedback before syncing to confluence. https://docs.google.com/document/d/1aAwVjdZByA-0CHbgv16Me-vjaaDMCfhX7TzVVTuifYM/edit
1) Whether can define a unified `JobGraphRetriever` to support both I think it is a feasible and good suggestion. I will revise the description. 2) Whether can use remote jar together delayed job graph generation. The goal of supporting job graph generation is for launch job with a remote jar, so that remote jar doesn't need to be downloaded to local for job graph generation. As the feature will be a flag in session cli, the user can enable the flag and also use a remote jar for deployment at the same time. Then, users can achieve the most an efficient way of job submission. 3) What do you mean about the package? Original, if the job graph generation is in client, a job only need flink runtime in each container. But if we postpone job graph generation into job manager, then the AM will need flink-clients, flink-optimizer, flink-table (for job use table api/sql) in classpath. It means the user needs to package these libraries within the job jar. Best Regards Peter Huang On Mon, Dec 9, 2019 at 8:18 PM Yang Wang <danrtsey...@gmail.com> wrote: > Hi Peter, > > Thanks a lot for starting this discussion. I think this is a very useful > feature. > > Not only for Yarn, i am focused on flink on Kubernetes integration and come > across the same > problem. I do not want the job graph generated on client side. Instead, the > user jars are built in > a user-defined image. When the job manager launched, we just need to > generate the job graph > based on local user jars. > > I have some small suggestion about this. > > 1. `ProgramJobGraphRetriever` is very similar to > `ClasspathJobGraphRetriever`, the differences > are the former needs `ProgramMetadata` and the latter needs some arguments. > Is it possible to > have an unified `JobGraphRetriever` to support both? > 2. Is it possible to not use a local user jar to start a per-job cluster? > In your case, the user jars has > existed on hdfs already and we do need to download the jars to deployer > service. Currently, we > always need a local user jar to start a flink cluster. It is be great if we > could support remote user jars. > >> In the implementation, we assume users package flink-clients, > flink-optimizer, flink-table together within the job jar. Otherwise, the > job graph generation within JobClusterEntryPoint will fail. > 3. What do you mean about the package? Do users need to compile their jars > inlcuding flink-clients, flink-optimizer, flink-table codes? > > > > Best, > Yang > > Peter Huang <huangzhenqiu0...@gmail.com> 于2019年12月10日周二 上午2:37写道： > > > Dear All, > > > > Recently, the Flink community starts to improve the yarn cluster > descriptor > > to make job jar and config files configurable from CLI. It improves the > > flexibility of Flink deployment Yarn Per Job Mode. For platform users > who > > manage tens of hundreds of streaming pipelines for the whole org or > > company, we found the job graph generation in client-side is another > > pinpoint. Thus, we want to propose a configurable feature for > > FlinkYarnSessionCli. The feature can allow users to choose the job graph > > generation in Flink ClusterEntryPoint so that the job jar doesn't need to > > be locally for the job graph generation. The proposal is organized as a > > FLIP > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Delayed+JobGraph+Generation > > . > > > > Any questions and suggestions are welcomed. Thank you in advance. > > > > > > Best Regards > > Peter Huang > > >
