Hi Biao Geng: Thanks for your feedback, I'm +1 to go with option#2. It's a good point that
we should improve the error message debugging for the session job, I think it can be a follow up work as an improvement after we support the session job operation. Best, Aitozi. Geng Biao <biaoge...@gmail.com> 于2022年3月17日周四 10:55写道： > Thanks Aitozi for the work! > > I lean to option#2 of using JarRunHeaders with uber job jar as well. As > Yang said, the user defined dependencies may be better supported in > upstream flink. > A follow-up thought: I think we should care the potential influence on > user experiences: as the job graph is generated in JM, when the generation > fails due to some issues in the main() method, we should do some work on > showing such error messages in this proposal or the later k8s operator > implementation. Reason for this question is that if users submit many jobs > to one same session cluster, it may be not easy for them to find relevant > error logs about main() method of a specific job. The FLINK-25715 could > help us later. > > > Best, > Biao Geng > > > 发件人: Aitozi <gjying1...@gmail.com> > 日期: 星期三, 2022年3月16日 下午5:19 > 收件人: dev@flink.apache.org <dev@flink.apache.org> > 主题: Re: [DISCUSS] Support the session job management in kubernetes operator > Hi Yang Wang > Thanks for your feedback, Provide the local and http implementation for > the first version makes sense to me. > +1 for it. > > Best, > Aitozi > > Yang Wang <danrtsey...@gmail.com> 于2022年3月16日周三 16:44写道： > > > # How to download the user jars > > I agree with Gyula that it will be a burden if we bundle the flink > > filesystem dependencies in the operator image. > > Maybe we could have a *ArtifactFetcher* interface in the > > flink-kubernetes-operator. By default, we provide the local and http > > implementation, > > which means we could get the user jars from local files or HTTP URLs. > Flink > > filesystem support could be done as a follow-up based on the feedback. > > > > If the user wants to use the local implementation, they need to mount a > > PV(aka persist volume) to the operator first and then put their jars into > > the PV. > > > > # How to talk to session JobManager to submit the job > > After more consideration, I also prefer the second approach, via REST API > > /jars/:jarid/run. If we have strong requirements to support dependencies > > jars and > > artifacts, we could try to support this in the upstream project. > > > > Best, > > Yang > > > > > > Aitozi <gjying1...@gmail.com> 于2022年3月16日周三 16:11写道： > > > > > Hi Gyula > > > Thanks for your quick response. Regarding the different filesystems > > > dependency, > > > I think we can make it optional and pluggable, and let it choose by > user > > > when building > > > their operator image. Users can build their image from the base > operator > > > image and > > > add filesystem dependency they want to use to it. BTW, we can support > the > > > http URI > > > by default. > > > > > > Thanks, > > > Aitozi. > > > > > > Gyula Fóra <gyula.f...@gmail.com> 于2022年3月16日周三 15:53写道： > > > > > > > Thank you Aitozi! > > > > > > > > I think this will be a very nice (and simple) addition to enable > these > > > > use-cases. > > > > > > > > I have 2 comments regarding the proposal: > > > > > > > > 1. I think if we want to support different filesystems to download > jars > > > > from, we probably need some clever ways to add external operator > > > > dependencies (jars, configs). > > > > I would prefer not to bundle them into the base operator image. > > > > > > > > 2. I think we should avoid creating the jobgraphs on the operator > side > > > and > > > > use the jar upload/run rest api instead as you suggested. This will > > avoid > > > > flink version and dependency conflicts. > > > > > > > > Cheers, > > > > Gyula > > > > > > > > On Wed, Mar 16, 2022 at 8:41 AM Aitozi <gjying1...@gmail.com> wrote: > > > > > > > > > Hi Guys: > > > > > > > > > > I would like to open a discussion for support session job > > > management > > > > in > > > > > kubernetes operator. It’s intended to enhance the > > > > flink-kubernetes-operator > > > > > to manage the session job with k8s tooling. I have drafted the > design > > > > > doc[1]. Please refer to it and give me some feedback . > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1WPGbur1eT3H_5gN-kyXfp7EDjdbJUURx6jN8nt6UT-s/edit# > < > https://docs.google.com/document/d/1WPGbur1eT3H_5gN-kyXfp7EDjdbJUURx6jN8nt6UT-s/edit > > > > > > > > > > > > Best, > > > > > > > > > > Aitozi. > > > > > > > > > > > > > > >
