Hi Yangze! This is not set in stone at the moment but the way I think it should work is that first class config options in the CR should always take precedence over the Flink config.
In general we should not introduce too many arbitrary config options that duplicate the flink configs without good reasons but the ones we introduce should overwrite flink configs. We should discuss and decide together what config options to keep in the flink conf and what to bring on the CR level. Resource related ones are difficult because on one hand they are integral to every application, on the other hand there are many expert settings that we should probably leave in the conf. Cheers, Gyula On Mon, Feb 7, 2022 at 8:28 AM Yangze Guo <karma...@gmail.com> wrote: > Thanks everyone for the great effort. The FLIP looks really good. > > I just want to make sure the configuration priority in the CR example. > It seems the requests resources or "taskManager. taskSlots" will be > transferred to Flink internal config, e.g. > "taskmanager.memory.process.size" and "taskmanager.numberOfTaskSlots", > and override the one in "flinkConfiguration". Am I understanding this > correctly? > > Best, > Yangze Guo > > On Mon, Feb 7, 2022 at 10:22 AM Xintong Song <tonysong...@gmail.com> > wrote: > > > > Sorry for the late reply. We were out due to the public holidays in > China. > > > > @Thomas, > > > > The intention is to support application management through operator and > CR, > > > which means there won't be any 2 step submission process, which as you > > > allude to would defeat the purpose of this project. The CR example > shows > > > the application part. Please note that the bare cluster support is an > > > *additional* feature for scenarios that require external job > management. Is > > > there anything on the FLIP page that creates a different impression? > > > > > > > Sounds good to me. I don't remember what created the impression of 2 step > > submission back then. I revisited the latest version of this FLIP and it > > looks good to me. > > > > @Gyula, > > > > Versioning: > > > Versioning will be independent from Flink and the operator will depend > on a > > > fixed flink version (in every given operator version). > > > This should be the exact same setup as with Stateful Functions ( > > > https://github.com/apache/flink-statefun). So independent release > cycle > > > but > > > still within the Flink umbrella. > > > > > > > Does this mean if someone wants to upgrade Flink to a version that is > > released after the operator version that is being used, he/she would need > > to upgrade the operator version first? > > I'm not questioning this, just trying to make sure I'm understanding this > > correctly. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Mon, Feb 7, 2022 at 3:14 AM Gyula Fóra <gyula.f...@gmail.com> wrote: > > > > > Thank you Alexis, > > > > > > Will definitely check this out. You are right, Kotlin makes it > difficult to > > > adopt pieces of this code directly but I think it will be good to get > > > inspiration for the architecture and look at how particular problems > have > > > been solved. It will be a great help for us I am sure. > > > > > > Cheers, > > > Gyula > > > > > > On Sat, Feb 5, 2022 at 12:28 PM Alexis Sarda-Espinosa < > > > alexis.sarda-espin...@microfocus.com> wrote: > > > > > > > Hi everyone, > > > > > > > > just wanted to mention that my employer agreed to open source the > PoC I > > > > developed: https://github.com/MicroFocus/opsb-flink-k8s-operator > > > > > > > > I understand the concern for maintainability, so Gradle & Kotlin > might > > > not > > > > be appealing to you, but at least it gives you another reference. The > > > Helm > > > > resources in particular might be useful. > > > > > > > > There are bits and pieces there referring to Flink sessions, but > those > > > are > > > > just placeholders, the functioning parts use application mode with > native > > > > integration. > > > > > > > > Regards, > > > > Alexis. > > > > > > > > ________________________________ > > > > From: Thomas Weise <t...@apache.org> > > > > Sent: Saturday, February 5, 2022 2:41 AM > > > > To: dev <dev@flink.apache.org> > > > > Subject: Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator > > > > > > > > Hi, > > > > > > > > Thanks for the continued feedback and discussion. Looks like we are > > > > ready to start a VOTE, I will initiate it shortly. > > > > > > > > In parallel it would be good to find the repository name. > > > > > > > > My suggestion would be: flink-kubernetes-operator > > > > > > > > I thought "flink-operator" could be a bit misleading since the term > > > > operator already has a meaning in Flink. > > > > > > > > I also considered "flink-k8s-operator" but that would be almost > > > > identical to existing operator implementations and could lead to > > > > confusion in the future. > > > > > > > > Thoughts? > > > > > > > > Thanks, > > > > Thomas > > > > > > > > > > > > > > > > On Fri, Feb 4, 2022 at 5:15 AM Gyula Fóra <gyula.f...@gmail.com> > wrote: > > > > > > > > > > Hi Danny, > > > > > > > > > > So far we have been focusing our dev efforts on the initial native > > > > > implementation with the team. > > > > > If the discussion and vote goes well for this FLIP we are looking > > > forward > > > > > to contributing the initial version sometime next week (fingers > > > crossed). > > > > > > > > > > At that point I think we can already start the dev work to support > the > > > > > standalone mode as well, especially if you can dedicate some > effort to > > > > > pushing that side. > > > > > Working together on this sounds like a great idea and we should > start > > > as > > > > > soon as possible! :) > > > > > > > > > > Cheers, > > > > > Gyula > > > > > > > > > > On Fri, Feb 4, 2022 at 2:07 PM Danny Cranmer < > dannycran...@apache.org> > > > > > wrote: > > > > > > > > > > > I have been discussing this one with my team. We are interested > in > > > the > > > > > > Standalone mode, and are willing to contribute towards the > > > > implementation. > > > > > > Potentially we can work together to support both modes in > parallel? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > On Wed, Feb 2, 2022 at 4:02 PM Gyula Fóra <gyula.f...@gmail.com> > > > > wrote: > > > > > > > > > > > > > Hi Danny! > > > > > > > > > > > > > > Thanks for the feedback :) > > > > > > > > > > > > > > Versioning: > > > > > > > Versioning will be independent from Flink and the operator will > > > > depend > > > > > > on a > > > > > > > fixed flink version (in every given operator version). > > > > > > > This should be the exact same setup as with Stateful Functions > ( > > > > > > > https://github.com/apache/flink-statefun). So independent > release > > > > cycle > > > > > > > but > > > > > > > still within the Flink umbrella. > > > > > > > > > > > > > > Deployment error handling: > > > > > > > I think that's a very good point, as general exception > handling for > > > > the > > > > > > > different failure scenarios is a tricky problem. I think the > > > > exception > > > > > > > classifiers and retry strategies could avoid a lot of manual > > > > intervention > > > > > > > from the user. We will definitely need to add something like > this. > > > > Once > > > > > > we > > > > > > > have the repo created with the initial operator code we should > open > > > > some > > > > > > > tickets for this and put it on the short term roadmap! > > > > > > > > > > > > > > Cheers, > > > > > > > Gyula > > > > > > > > > > > > > > On Wed, Feb 2, 2022 at 4:50 PM Danny Cranmer < > > > > dannycran...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > Hey team, > > > > > > > > > > > > > > > > Great work on the FLIP, I am looking forward to this one. I > agree > > > > that > > > > > > we > > > > > > > > can move forward to the voting stage. > > > > > > > > > > > > > > > > I have general feedback around how we will handle job > submission > > > > > > failure > > > > > > > > and retry. As discussed in the Rejected Alternatives > section, we > > > > can > > > > > > use > > > > > > > > Java to handle job submission failures from the Flink > client. It > > > > would > > > > > > be > > > > > > > > useful to have the ability to configure exception > classifiers and > > > > retry > > > > > > > > strategy as part of operator configuration. > > > > > > > > > > > > > > > > Given this will be in a separate Github repository I am > curious > > > how > > > > > > ther > > > > > > > > versioning strategy will work in relation to the Flink > version? > > > Do > > > > we > > > > > > > have > > > > > > > > any other components with a similar setup I can look at? > Will the > > > > > > > operator > > > > > > > > version track Flink or will it use its own versioning > strategy > > > > with a > > > > > > > Flink > > > > > > > > version support matrix, or similar? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 1, 2022 at 2:33 PM Márton Balassi < > > > > > > balassi.mar...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi team, > > > > > > > > > > > > > > > > > > Thank you for the great feedback, Thomas has updated the > FLIP > > > > page > > > > > > > > > accordingly. If you are comfortable with the currently > existing > > > > > > design > > > > > > > > and > > > > > > > > > depth in the FLIP [1] I suggest moving forward to the > voting > > > > stage - > > > > > > > once > > > > > > > > > that reaches a positive conclusion it lets us create the > > > separate > > > > > > code > > > > > > > > > repository under the flink project for the operator. > > > > > > > > > > > > > > > > > > I encourage everyone to keep improving the details in the > > > > meantime, > > > > > > > > however > > > > > > > > > I believe given the existing design and the general > sentiment > > > on > > > > this > > > > > > > > > thread that the most efficient path from here is starting > the > > > > > > > > > implementation so that we can collectively iterate over it. > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-212%3A+Introduce+Flink+Kubernetes+Operator > > > > > > > > > > > > > > > > > > On Mon, Jan 31, 2022 at 10:15 PM Thomas Weise < > t...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > HI Xintong, > > > > > > > > > > > > > > > > > > > > Thanks for the feedback and please see responses below > --> > > > > > > > > > > > > > > > > > > > > On Fri, Jan 28, 2022 at 12:21 AM Xintong Song < > > > > > > tonysong...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks Thomas for drafting this FLIP, and everyone for > the > > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > I also have a few questions and comments. > > > > > > > > > > > > > > > > > > > > > > ## Job Submission > > > > > > > > > > > Deploying a Flink session cluster via kubectl & CR and > then > > > > > > > > submitting > > > > > > > > > > jobs > > > > > > > > > > > to the cluster via Flink cli / REST is probably the > > > approach > > > > that > > > > > > > > > > requires > > > > > > > > > > > the least effort. However, I'd like to point out 2 > > > > weaknesses. > > > > > > > > > > > 1. A lot of users use Flink in perjob/application > modes. > > > For > > > > > > these > > > > > > > > > users, > > > > > > > > > > > having to run the job in two steps (deploy the > cluster, and > > > > > > submit > > > > > > > > the > > > > > > > > > > job) > > > > > > > > > > > is not that convenient. > > > > > > > > > > > 2. One of our motivations is being able to manage Flink > > > > > > > applications' > > > > > > > > > > > lifecycles with kubectl. Submitting jobs from cli > sounds > > > not > > > > > > > aligned > > > > > > > > > with > > > > > > > > > > > this motivation. > > > > > > > > > > > I think it's probably worth it to support submitting > jobs > > > via > > > > > > > > kubectl & > > > > > > > > > > CR > > > > > > > > > > > in the first version, both together with deploying the > > > > cluster > > > > > > like > > > > > > > > in > > > > > > > > > > > perjob/application mode and after deploying the cluster > > > like > > > > in > > > > > > > > session > > > > > > > > > > > mode. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The intention is to support application management > through > > > > operator > > > > > > > and > > > > > > > > > CR, > > > > > > > > > > which means there won't be any 2 step submission process, > > > > which as > > > > > > > you > > > > > > > > > > allude to would defeat the purpose of this project. The > CR > > > > example > > > > > > > > shows > > > > > > > > > > the application part. Please note that the bare cluster > > > > support is > > > > > > an > > > > > > > > > > *additional* feature for scenarios that require external > job > > > > > > > > management. > > > > > > > > > Is > > > > > > > > > > there anything on the FLIP page that creates a different > > > > > > impression? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Versioning > > > > > > > > > > > Which Flink versions does the operator plan to support? > > > > > > > > > > > 1. Native K8s deployment was firstly introduced in > Flink > > > 1.10 > > > > > > > > > > > 2. Native K8s HA was introduced in Flink 1.12 > > > > > > > > > > > 3. The Pod template support was introduced in Flink > 1.13 > > > > > > > > > > > 4. There was some changes to the Flink docker image > > > > entrypoint > > > > > > > script > > > > > > > > > in, > > > > > > > > > > > IIRC, Flink 1.13 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Great, thanks for providing this. It is important for the > > > > > > > compatibility > > > > > > > > > > going forward also. We are targeting Flink 1.14.x > upwards. > > > > Before > > > > > > the > > > > > > > > > > operator is ready there will be another Flink release. > Let's > > > > see if > > > > > > > > > anyone > > > > > > > > > > is interested in earlier versions? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ## Compatibility > > > > > > > > > > > What kind of API compatibility we can commit to? It's > > > > probably > > > > > > fine > > > > > > > > to > > > > > > > > > > have > > > > > > > > > > > alpha / beta version APIs that allow incompatible > future > > > > changes > > > > > > > for > > > > > > > > > the > > > > > > > > > > > first version. But eventually we would need to > guarantee > > > > > > backwards > > > > > > > > > > > compatibility, so that an early version CR can work > with a > > > > new > > > > > > > > version > > > > > > > > > > > operator. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another great point and please let me include that on the > > > FLIP > > > > > > page. > > > > > > > > ;-) > > > > > > > > > > > > > > > > > > > > I think we should allow incompatible changes for the > first > > > one > > > > or > > > > > > two > > > > > > > > > > versions, similar to how other major features have > evolved > > > > > > recently, > > > > > > > > such > > > > > > > > > > as FLIP-27. > > > > > > > > > > > > > > > > > > > > Would be great to get broader feedback on this one. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Thomas > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 28, 2022 at 1:18 PM Thomas Weise < > > > t...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > # 1 Flink Native vs Standalone integration > > > > > > > > > > > > > Maybe we should make this more clear in the FLIP > but we > > > > > > agreed > > > > > > > to > > > > > > > > > do > > > > > > > > > > > the > > > > > > > > > > > > > first version of the operator based on the native > > > > > > integration. > > > > > > > > > > > > > While this clearly does not cover all use-cases and > > > > > > > requirements, > > > > > > > > > it > > > > > > > > > > > > seems > > > > > > > > > > > > > this would lead to a much smaller initial effort > and a > > > > nicer > > > > > > > > first > > > > > > > > > > > > version. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm also leaning towards the native integration, as > long > > > > as it > > > > > > > > > reduces > > > > > > > > > > > the > > > > > > > > > > > > MVP effort. Ultimately the operator will need to also > > > > support > > > > > > the > > > > > > > > > > > > standalone mode. I would like to gain more confidence > > > that > > > > > > native > > > > > > > > > > > > integration reduces the effort. While it cuts the > effort > > > to > > > > > > > handle > > > > > > > > > the > > > > > > > > > > TM > > > > > > > > > > > > pod creation, some mapping code from the CR to the > native > > > > > > > > integration > > > > > > > > > > > > client and config needs to be created. As mentioned > in > > > the > > > > > > FLIP, > > > > > > > > > native > > > > > > > > > > > > integration requires the Flink job manager to have > access > > > > to > > > > > > the > > > > > > > > k8s > > > > > > > > > > API > > > > > > > > > > > to > > > > > > > > > > > > create pods, which in some scenarios may be seen as > > > > > > unfavorable. > > > > > > > > > > > > > > > > > > > > > > > > > > > # Pod Template > > > > > > > > > > > > > > > Is the pod template in CR same with what Flink > has > > > > > > already > > > > > > > > > > > > > supported[4]? > > > > > > > > > > > > > > > Then I am afraid not the arbitrary field(e.g. > > > > cpu/memory > > > > > > > > > > resources) > > > > > > > > > > > > > could > > > > > > > > > > > > > > > take effect. > > > > > > > > > > > > > > > > > > > > > > > > Yes, pod template would look almost identical. There > are > > > a > > > > few > > > > > > > > > settings > > > > > > > > > > > > that the operator will control (and that may need to > be > > > > > > > > blacklisted), > > > > > > > > > > but > > > > > > > > > > > > in general we would not want to place restrictions. I > > > > think a > > > > > > > > > mechanism > > > > > > > > > > > > where a pod template is merged from multiple layers > would > > > > also > > > > > > be > > > > > > > > > > > > interesting to make this more flexible. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Thomas > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
