Hi Dale, all, Thanks for the FLIP, +1 on the motivation. To your first question: yes, I see this need regularly. Small or intermittent jobs that need isolation, where the two-pod baseline dominates the actual cost of the job.
On the open questions in the thread: 1. New kind vs extending FlinkDeployment (Robert's question): +1 for the separate kind. Beyond the union-shaped spec, a `mode: minicluster` field would turn a mode edit into an in-place topology migration inside the reconciler of the operator's most critical API. The upgrade machinery today treats spec diffs as upgrades within one topology. Tearing down a single pod and standing up a JM/TM pair mid-flight (or the reverse) is a different, riskier operation, better kept out of FlinkDeployment. 2. Recovery semantics: I would give this its own section in the FLIP. In a single pod the JobManager dies with every failure, so the resume path is always the same: pod restarts, launcher boots a fresh MiniCluster, HA recovers the job. Three things worth pinning down: * the job needs a fixed JobID, so checkpoint lineage survives pod restarts * the launcher behaviour when the dispatcher has already recovered the job from HA: tolerate DuplicateJobSubmissionException, like ApplicationDispatcherBootstrap does for application mode today * what `upgradeMode: last-state` means when the whole cluster dies with the pod: HA metadata availability, and storage guidance (PVC vs object storage) 3. Memory mapping: MiniClusterConfiguration applies TaskExecutorResourceUtils.adjustForLocalExecution, so memory inside the pod does not follow the distributed TM model. The mapping from spec.resources to Flink memory settings is worth specifying, for sizing guidance now and for tuning later. 4. On Robert's autoscaler question, the two halves differ: * Parallelism: the overrides are realized generically today (they land in spec.flinkConfiguration), and MiniCluster itself can run parallelism > 1. But in-place rescaling in the operator requires the adaptive scheduler and updates resource requirements over REST, and that path is only wired for the native service today. Even wired up, a single pod bounds it: slots are fixed at boot and the pod's CPU/memory are the ceiling. In practice a parallelism decision for a MiniCluster job translates into a full pod redeploy until in-process rescaling is supported end to end, and beyond the pod's capacity it becomes a promotion (point 5). * Vertical: for a one-pod topology, right-sizing the pod is the lever that actually matters, and it needs a tuning decision with its own trigger, decoupled from scaling decisions. That is exactly the decoupling I'm proposing in [1]. FlinkMiniCluster would be a natural future consumer of that decision point, with no new scope for either proposal. 5. Automatic transitions: I have spent some time researching cross-topology moves. JobID regeneration on restore, per-cluster HA metadata, and the differing jar distribution models make this a savepoint-boundary orchestration problem, not an in-place mutation. I think it belongs in a separate orchestrator kind in the same way as FlinkBlueGreenDeployment (transition state machine in status, owning the children it creates), building on exactly the manual migration workflow this FLIP defines. I would be happy to draft that follow-up FLIP once this one lands. So +1 also on keeping automatic promotion out of scope here. 6. Launcher and version support: I would go one step further than the pre-2.4 / 2.4 split. Make Flink 2.4 the minimum supported version for FlinkMiniCluster in the operator, so operator support begins exactly where the promoted API and the core launcher exist. That removes the separate per-version compatibility tracking the test plan describes, gives one clean contract (flinkVersion >= v2_4, enforced at validation), and matches how other operator features are version-gated already (in-place rescaling requires 1.18+). Example code can still serve pre-2.4 experimentation, just outside the operator's supported surface. [1] https://lists.apache.org/thread/vk6j8793rx2y78vhh1yvfjf4qntpfkm9 Best regards, Dennis On Thu, Aug 13, 2026 at 12:01 PM Robert Metzger <[email protected]> wrote: > Thanks a lot for kicking off a discussion on adding this to the Kubernetes > Operator! I've seen many Flink users who would benefit from this. > > Some comments: > * " jobManagerDeploymentStatus: READY" --> is "jobManagerDeploymentStatus" > still a good name within "kind: FlinkMiniCluster"? how about > miniClusterPodDeploymentStatus? I do like the idea of reusing the entire > status field. It's difficult :) But If you are going to propose to > introduce a new CR for FlinkMiniCluster for the purpose of having it > properly typed, then we should go all in? > * on the "Flink MiniCluster launcher: Or this could be a bootstrap class > added to core Flink, so that it is available out of the box in a regular > Flink image.": The proposal is fairly vague here. How about we provide the > required example code for pre-Flink 2.4? versions, and starting from 2.4 we > add the launcher to core and it works out of the box? > * It would be really nice to consider how hard it would be to integrate > this with the autoscaler, so that the autoscaler can scale down into a > minicluster, our out of the minicluster into a full cluster? Not using a > dedicated FlinkMiniCluster CR, but further extending the FlinkDeployment CR > would potentially make that easier? ... however, in that case, the > semantics of jobManager / taskManager resources would be messed up -- > because suddenly when transitioning from a regular Flink cluster to a > MiniCluster, the meaning of either a job manager or taskmanager resource > would change. We could introduce a 3rd "miniCluster" field for the > resources? It would probably be most elegant if the autoscaler could do the > transition from FlinkDeployment to FlinkMiniCluster automagically? > > CCing Gyula for additional feedback > > On Wed, Aug 5, 2026 at 5:11 PM Dale Lane <[email protected]> > wrote: > > > I'd like to start a discussion on > > FLIP-XXX : Running Flink jobs in MiniCluster using the Kubernetes > Operator > > > > > https://docs.google.com/document/d/1dtGjPYcsBkx1vxHPs1QnDtPxeH_Acz_pl8gx4b1BLB4/edit?usp=sharing > > > > The aim of the FLIP is to extend the Flink Kubernetes Operator to offer a > > single-pod, light-weight deployment option for low-throughput jobs. > > > > From the motivation: > > A single-pod, self-contained Flink job that starts fast and needs no > > multi-pod coordination could be a good fit for low-throughput jobs that > > aren't suitable for session clusters because they need isolation. > > > > Looking forward to feedback, both on the general motivation (Have you > seen > > a need for very small lightweight Flink jobs where fast crash-consistent > > resume is good enough without a full distributed Flink cluster?) and the > > suggested implementation approach (Do you think a new custom resource > kind > > is the best way to represent this capability?) > > > > Kind regards > > > > Dale > > -- > > dalelane.co.uk > > > > >
