Hi Yanis, I do not have an apples-to-apples production comparison between controlled restarts and GKE-driven eviction, but one Flink-side distinction may be useful when evaluating them. A TaskManager eviction is failure-driven rather than a transparent downscale. With the Adaptive Scheduler, the failure causes the job to restart, and partial failover is not supported. The job is restored from its latest completed checkpoint and may subsequently run with the resources that are available, but the existing execution does not continue uninterrupted with fewer slots. This is different from resource-change-driven rescaling, where the Adaptive Scheduler has checkpoint-aware trigger behavior. Therefore, I would not assume that a GKE eviction is automatically aligned with a newly completed checkpoint. A controlled process can still provide two useful controls: which jobs are disrupted and how many TaskManagers are removed concurrently. If it explicitly obtains a checkpoint or savepoint before removing TaskManagers, it may also provide a more predictable recovery point. The amount of capacity reclaimed, on the other hand, is mostly determined by the resulting Kubernetes placement rather than the Flink restart mechanism. For a fair comparison, I would keep the target reclaimed capacity the same and measure the number of affected jobs, concurrently lost TaskManagers, recovery and catch-up time, age of the restored checkpoint, checkpoint failures during consolidation, and reclaimed nodes/CPU/memory. Does your controlled restart process explicitly trigger a checkpoint or savepoint before removing TaskManagers, or does it currently only control the order and timing of the restarts?
Best regards, Zhe Wang Yanis Djeridi via dev <[email protected]> 于2026年8月31日周一 22:13写道： > Hi Dennis, > > Thanks for the detailed response. This aligns closely with the approach > we’ve been exploring. > > We already use GKE’s optimize-utilization profile. We haven’t marked > TaskManagers as safe-to-evict because we’re still testing a controlled > restart process and currently trigger consolidation manually. This gives us > more control over which jobs are disrupted, and a better understanding of > the problem until we move to an automated approach. > > Do you have experience comparing controlled restarts with GKE-driven > eviction using safe-to-evict, particularly in terms of reclaimed capacity > and disruption? > > Overall, your recommendation reinforces our current approach: shape pods > to fit the available machines and use controlled restarts to eliminate > accumulated fragmentation, while making recovery as smooth as possible. > > We’re also open to other approaches the community has found effective at > scale. > > Thanks again, > > Yanis > > On 2026/08/28 13:23:53 Dennis-Mircea Ciupitu wrote: > > Hi Yanis, > > > > Based on the information that you provided, I assume you run under this > > setup: Flink Kubernetes Operator in native mode, with deployments in > > application mode. Having this in mind, I'm framing the following > response. > > > > 1. Resource-shape mismatch > > > > I see this as a provisioning problem more than a Flink one. Either the > node > > shape has to match the JM + TM shape, or JM + TM must be designed to run > > under the ratio that the node provides. There is no Flink core or > operator > > feature that fixes this automatically today. > > > > The operator's memory autotuning goes in that direction, since it moves > TM > > memory requests toward real usage, meaning that in the end the job will > run > > at the best CPU/memory ratio, but right now it only runs on scaling > > decisions. Decoupling it from scaling is tracked in FLINK-40455 [1], and > > once that matures (to include CPU tuning as well), it should help exactly > > this case. > > > > 2. Fragmentation > > > > There are 2 important things here: > > > > - The default GKE scheduler prefers emptier / balanced nodes, so it > > spreads your TMs, and nothing ever moves them back together. > > - Once the TM pods are spread, nothing in Kubernetes ever moves them > > back together. What I would try, based on what I know about GKE and > how > > native mode behaves, is to switch the cluster autoscaler to the > packing > > profile (optimize-utilization) so new pods land on busy nodes, and by > > marking the TM pods as safe to evict and JM pods with the opposite > > annotation. Both are plain pod annotations, so the operator's > podTemplate > > can carry them. Eviction is workable for Flink because the JM notices > the > > lost TM and requests a replacement, which is then scheduled onto the > packed > > nodes. > > > > The real cost is that each job's task on the TM that was hosted on the > > consolidated nodes will need to be restarted from a checkpoint. With the > > adaptive scheduler in place, this will be made much smoother, since the > job > > can keep running with fewer slots while the replacement comes up. So it > > ends up as a tradeoff between wasted capacity and how much disruption > each > > job can accept. > > > > In the long term, I think the operator could play an important role here, > > as it already takes jobs through full redeploys for spec upgrades, and > > those moments recreate all the TM pods anyway, so they are natural > windows > > to let the cluster repack without paying any extra disruption. Besides > > this, as I said, there will be some nice features added within the > operator > > that will help those use cases. > > > > Best, > > Dennis > > > > [1] https://issues.apache.org/jira/browse/FLINK-40455 > > > > > > On Thu, Aug 27, 2026 at 2:14 AM Yanis Djeridi via dev < > [email protected]> > > wrote: > > > > > Hi all, > > > > > > We run hundreds of Flink pipelines across several shared GKE clusters > and > > > experience significant cluster overhead. By cluster overhead, we mean > > > allocatable CPU and memory available on active nodes that remain > > > unrequested but cannot be reclaimed because the nodes still host pods. > > > > > > > > > This comes mainly from two sources: > > > > > > > > > - > > > > > > TaskManager CPU/memory request ratios do not always align with our > > > machine shapes, leaving capacity stranded when either CPU or memory > is > > > exhausted first. > > > - > > > > > > Deployments and autoscaling gradually scatter TaskManagers across > nodes. > > > Because TaskManagers are bare pods, GKE cannot evict them for > > > consolidation, and Kubernetes does not proactively repack them. > > > > > > > > > As a result, partially occupied nodes can remain running indefinitely > > > despite substantial aggregate free capacity. > > > > > > Have others operating large shared Flink clusters encountered this? > How do > > > you manage resource-shape mismatch and fragmentation? > > > > > > > > > Thanks, > > > > > > Yanis > > > > >
