Thanks. In my opinion, this FLIP can go into VOTEing.
On Mon, Aug 31, 2026 at 6:14 PM Dale Lane <[email protected]> wrote: > I've added a description of how the launcher could be implemented to the > same Google Doc at > https://docs.google.com/document/d/1dtGjPYcsBkx1vxHPs1QnDtPxeH_Acz_pl8gx4b1BLB4/edit?usp=sharing > > Having looked into it more, it'll be lot smaller and simpler to implement > than I first thought. > > Kind regards > > D > -- > dalelane.co.uk > > > On Thursday, 27 August 2026 at 19:42, Robert Metzger <[email protected]> > wrote: > > > I think the launcher can go into this FLIP. > > > > > > On Tue, Aug 25, 2026 at 8:44 PM Dale Lane < > [email protected]> > > wrote: > > > > > > 3. Launcher placement - Here I'd go the opposite way of the current > doc. > > > > Currently, every deployment shape keeps its bootstrap in the Flink > > > > distribution, and that convention is what decouples operator version > from > > > > Flink version. My opinion is that the MiniCluster launcher should > follow > > > > it. In practice, the launcher is close to `flink run -t local`, only > > > adding > > > > HA-aware resubmission and pod exit-code semantics on top, so it is > core > > > > code either way. I'd also say it makes sense to ship it in the > > > distribution > > > > (ideally with a bin/ script, so the same setup runs locally too) > because > > > > this means the operator depends on no MiniCluster Java API at all, > and it > > > > can manage the deployment in the same unified way as the other > shapes. > > > > > > I really like this idea, thanks - that's much better than what I'd > > > originally thought of. It simplifies a few things, such as migration > > > between MiniCluster and Deployment by letting you just use the same > image > > > for both as-is. I'll have a proper think through what it would need > and > > > update the doc. > > > > > > Do you think a Flink launcher would need a separate/dedicated FLIP to > get > > > broader scrutiny? > > > > > > Kind regards > > > > > > D > > > -- > > > dalelane.co.uk > > > > > > On Tuesday, 25 August 2026 at 14:10, Dennis-Mircea Ciupitu < > > > [email protected]> wrote: > > > > > > > Hi Dale, > > > > > > > > My response to the open points is the following: > > > > > > > > 1. MiniCluster vs 2 containers in one pod - Robert's numbers and > > > > argument make it clear to me. So, in order to ensure the lowest > > > possible > > > > resource consumption, we need to go with the MiniCluster approach > and > > > not > > > > rely on something intermediary that does not reach this end goal > and > > > is > > > > only better suited from the implementation/design perspective. > > > > 2. Separate CRD vs a third mode - Because the MiniCluster > approach is > > > > the way to go, I'd say that we should have an implementation that > > > > integrates with it in the best way possible. For me, I see a > separate > > > CRD > > > > as the best way to proceed here, as MiniCluster is different from > many > > > > points of view from a normal Flink cluster, and by going with a > third > > > mode > > > > I think we just complicate ourselves and we introduce too much > blast > > > radius. > > > > 3. Launcher placement - Here I'd go the opposite way of the > current > > > doc. > > > > Currently, every deployment shape keeps its bootstrap in the Flink > > > > distribution, and that convention is what decouples operator > version > > > from > > > > Flink version. My opinion is that the MiniCluster launcher should > > > follow > > > > it. In practice, the launcher is close to `flink run -t local`, > only > > > adding > > > > HA-aware resubmission and pod exit-code semantics on top, so it is > > > core > > > > code either way. I'd also say it makes sense to ship it in the > > > distribution > > > > (ideally with a bin/ script, so the same setup runs locally too) > > > because > > > > this means the operator depends on no MiniCluster Java API at all, > > > and it > > > > can manage the deployment in the same unified way as the other > shapes. > > > > 4. Promoting the MiniCluster - Here I say if the entrypoint lands > in > > > > Flink core, then I'd follow Gyula's and Robert's guidance on this > > > side, if > > > > there is something that can be done further here in order to > promote > > > it > > > > properly. > > > > 5. Resource and memory mapping - Deferring the exact ratios to > > > > implementation is fine with me. > > > > > > > > > > > > Best regards, > > > > Dennis > > > > > > > > > > > > On Mon, Aug 24, 2026 at 2:00 PM Dale Lane < > > > [email protected]> > > > > wrote: > > > > > > > > > > Whether it should be a different CRD I think should > > > > > > depend on the overlap with the FlinkDeployment. If > > > > > > we are exposing essentially the same spec (podTemplate, > > > > > > jobSpec, flinkConfig, ingress..) then it feels like we > > > > > > should reuse the CRD and simply introduce a new > > > > > > deployment mode next to native and standalone. > > > > > > > > > > @Gyula, @Dennis - How would you like to proceed with this question? > > > > > > > > > > I can absolutely see arguments for/against both approaches. In the > end > > > I > > > > > leaned more towards a separate CRD, partly from an API purity > > > perspective > > > > > (feeling that the separate topology deserved a separate API, and > avoids > > > > > describing the pod spec and status as "jobManager" or "taskManager" > > > when > > > > > it's both) and partly because it dodged the automated topology > > > migration. > > > > > > > > > > But maybe we don't mind the "a MiniCluster includes a Job Manager > so we > > > > > can just describe it as jobManager (and ignore the taskManager > > > references)" > > > > > API smell. It could change the manual migration path I describe to > > > > > transition jobs between MiniCluster and full distributed clusters > to > > > just > > > > > flipping a mode value. And we can always reject/block such a mode > > > switch in > > > > > the meantime unless/until the Operator is ready to support topology > > > > > transition/migration. > > > > > > > > > > Kind regards > > > > > > > > > > Dale > > > > > -- > > > > > dalelane.co.uk > > > > > > > > > > Sent with Proton Mail secure email. > > > > > > > > > > On Thursday, 20 August 2026 at 10:47, Gyula Fóra < > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Thanks @Robert Metzger for the explanation, > > > > > > > > > > > > Overall I think this is a good idea and hopefully we can reuse > most > > > of > > > > > > the controller flow that we already have. > > > > > > It would be nice to share the spec/status as much as reasonably > > > possible. > > > > > > > > > > > > Whether it should be a different CRD I think should depend on the > > > > > > overlap with the FlinkDeployment. If we are exposing essentially > the > > > > > > same spec (podTemplate, jobSpec, flinkConfig, ingress..) then it > > > feels > > > > > > like we should reuse the CRD and simply introduce a new > deployment > > > > > > mode next to native and standalone. > > > > > > > > > > > > Cheers > > > > > > Gyula > > > > > > > > > > > > On Thu, Aug 20, 2026 at 11:26 AM Dale Lane > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > This bit that I added to the FLIP is perhaps worthy of a bit > more > > > > > scrutiny, as it's based on my assumptions (misunderstandings?) of > how > > > the > > > > > FKO project works. > > > > > > > > > > > > > > In the section "Promoting MiniCluster to a stable API" I > asserted: > > > > > > > > > > > > > > > For the Flink Kubernetes Operator project to provide a > launcher > > > > > > > > as an example project would depend on the core Flink project > > > > > > > > promoting some MiniCluster components to @PublicEvolving ... > > > > > > > > This is not a technical requirement, as the current > > > > > > > > proof-of-concept demonstrates that the existing API is > sufficient > > > > > > > > - the promotion recommended here is of MiniCluster’s > existing > > > > > > > > submission surface. The requirement is about the Operator > project > > > > > > > > managing risk by building upon a stable contract. ... > > > > > > > > > > > > > > Is that really a requirement or have I just invented that? :) > > > > > > > > > > > > > > Looking more closely at the existing code, I can see we already > > > have > > > > > plenty of uses of flink-runtime classes that don't have @Public / > > > > > @PublicEvolving annotations, so my usage of flink-runtime classes > like > > > the > > > > > MiniCluster and MiniClusterConfiguration wouldn't be without > precedent. > > > > > > > > > > > > > > While I still think it'd be lovely to build this feature > solely on > > > > > stable APIs, maybe I'm creating an unreasonably high bar to clear > by > > > > > framing it this so strongly. > > > > > > > > > > > > > > What do you think? What are the norms here? > > > > > > > > > > > > > > Kind regards > > > > > > > > > > > > > > Dale > > > > > > > -- > > > > > > > dalelane.co.uk > > > > > > > > > > > > > > > > > > > > > On Wednesday, 5 August 2026 at 16:02, Dale Lane < > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > I'd like to start a discussion on > > > > > > > > FLIP-XXX : Running Flink jobs in MiniCluster using the > Kubernetes > > > > > Operator > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1dtGjPYcsBkx1vxHPs1QnDtPxeH_Acz_pl8gx4b1BLB4/edit?usp=sharing > > > > > > > > > > > > > > > > The aim of the FLIP is to extend the Flink Kubernetes > Operator to > > > > > offer a single-pod, light-weight deployment option for > low-throughput > > > jobs. > > > > > > > > > > > > > > > > From the motivation: > > > > > > > > A single-pod, self-contained Flink job that starts fast and > > > needs no > > > > > multi-pod coordination could be a good fit for low-throughput jobs > that > > > > > aren't suitable for session clusters because they need isolation. > > > > > > > > > > > > > > > > Looking forward to feedback, both on the general motivation > (Have > > > > > you seen a need for very small lightweight Flink jobs where fast > > > > > crash-consistent resume is good enough without a full distributed > Flink > > > > > cluster?) and the suggested implementation approach (Do you think > a new > > > > > custom resource kind is the best way to represent this capability?) > > > > > > > > > > > > > > > > Kind regards > > > > > > > > > > > > > > > > Dale > > > > > > > > -- > > > > > > > > dalelane.co.uk > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
