Thanks Aitozi for the work!
I lean to option#2 of using JarRunHeaders with uber job jar as well. As Yang
said, the user defined dependencies may be better supported in upstream flink.
A follow-up thought: I think we should care the potential influence on user
experiences: as the job graph is generated in JM, when the generation fails due
to some issues in the main() method, we should do some work on showing such
error messages in this proposal or the later k8s operator implementation.
Reason for this question is that if users submit many jobs to one same session
cluster, it may be not easy for them to find relevant error logs about main()
method of a specific job. The FLINK-25715 could help us later.
Best,
Biao Geng
发件人: Aitozi <gjying1...@gmail.com>
日期: 星期三, 2022年3月16日 下午5:19
收件人: dev@flink.apache.org <dev@flink.apache.org>
主题: Re: [DISCUSS] Support the session job management in kubernetes operator
Hi Yang Wang
Thanks for your feedback, Provide the local and http implementation for
the first version makes sense to me.
+1 for it.
Best,
Aitozi
Yang Wang <danrtsey...@gmail.com> 于2022年3月16日周三 16:44写道：
> # How to download the user jars
> I agree with Gyula that it will be a burden if we bundle the flink
> filesystem dependencies in the operator image.
> Maybe we could have a *ArtifactFetcher* interface in the
> flink-kubernetes-operator. By default, we provide the local and http
> implementation,
> which means we could get the user jars from local files or HTTP URLs. Flink
> filesystem support could be done as a follow-up based on the feedback.
>
> If the user wants to use the local implementation, they need to mount a
> PV(aka persist volume) to the operator first and then put their jars into
> the PV.
>
> # How to talk to session JobManager to submit the job
> After more consideration, I also prefer the second approach, via REST API
> /jars/:jarid/run. If we have strong requirements to support dependencies
> jars and
> artifacts, we could try to support this in the upstream project.
>
> Best,
> Yang
>
>
> Aitozi <gjying1...@gmail.com> 于2022年3月16日周三 16:11写道：
>
> > Hi Gyula
> > Thanks for your quick response. Regarding the different filesystems
> > dependency,
> > I think we can make it optional and pluggable, and let it choose by user
> > when building
> > their operator image. Users can build their image from the base operator
> > image and
> > add filesystem dependency they want to use to it. BTW, we can support the
> > http URI
> > by default.
> >
> > Thanks,
> > Aitozi.
> >
> > Gyula Fóra <gyula.f...@gmail.com> 于2022年3月16日周三 15:53写道：
> >
> > > Thank you Aitozi!
> > >
> > > I think this will be a very nice (and simple) addition to enable these
> > > use-cases.
> > >
> > > I have 2 comments regarding the proposal:
> > >
> > > 1. I think if we want to support different filesystems to download jars
> > > from, we probably need some clever ways to add external operator
> > > dependencies (jars, configs).
> > > I would prefer not to bundle them into the base operator image.
> > >
> > > 2. I think we should avoid creating the jobgraphs on the operator side
> > and
> > > use the jar upload/run rest api instead as you suggested. This will
> avoid
> > > flink version and dependency conflicts.
> > >
> > > Cheers,
> > > Gyula
> > >
> > > On Wed, Mar 16, 2022 at 8:41 AM Aitozi <gjying1...@gmail.com> wrote:
> > >
> > > > Hi Guys:
> > > >
> > > > I would like to open a discussion for support session job
> > management
> > > in
> > > > kubernetes operator. It’s intended to enhance the
> > > flink-kubernetes-operator
> > > > to manage the session job with k8s tooling. I have drafted the design
> > > > doc[1]. Please refer to it and give me some feedback .
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1WPGbur1eT3H_5gN-kyXfp7EDjdbJUURx6jN8nt6UT-s/edit#<https://docs.google.com/document/d/1WPGbur1eT3H_5gN-kyXfp7EDjdbJUURx6jN8nt6UT-s/edit>
> > > >
> > > > Best,
> > > >
> > > > Aitozi.
> > > >
> > >
> >
>
