[
https://issues.apache.org/jira/browse/FLINK-34152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822623#comment-17822623
]
Yang LI commented on FLINK-34152:
---------------------------------
Hi [~mxm] after reviewing those 2 pr and especially this one
[https://github.com/apache/flink-kubernetes-operator/pull/786] , I have
understood better how memory tuning is supposed to handle the flink cluster's
scaling down.
To answer your comment :
{*}“{*}We may need to dynamically adjust the Kubernetes CPU and memory limits
for both the job manager and task manager eventually, to align with the
automatically tuned memory and CPU parameters and prevent unnecessary resource
allocation.{*}”{*}
What I'm trying to say is that I didn't notice any updated configuration for
the memory request and limit at the pod template level for the taskmanager.
Therefore, I assume that the pod's memory allocation won't automatically adjust
to reflect changes in the taskmanager's heap size, unless I've missed something.
{*}"{*}Autoscaling assumes some sort of Kubernetes Cluster Autoscaling to be
active. When fewer resources are allocated, that should result in fewer nodes,
but in practice it isn't quite that easy. It requires a bit of extra work for
nodes to get released when fewer resources are used. The default Kubernetes
scheduler doesn't bin-pack, but it can be reconfigured to do bin-packing as
opposed to its default behavior to evenly spread out pods.{*}"{*}
Indeed, by implementing bin-packing, we can optimize resource utilization,
which is now clearer to me. However, its management becomes more complex (K8s
upgrade, daily node restart/eviction) for sure, especially when there are other
application components in the same Kubernetes cluster IMO. But I am very
curious now to test this feature when it's fully released, it looks indeed
another important step to push flink to cloud native :)
> Tune TaskManager memory
> -----------------------
>
> Key: FLINK-34152
> URL: https://issues.apache.org/jira/browse/FLINK-34152
> Project: Flink
> Issue Type: Sub-task
> Components: Autoscaler, Kubernetes Operator
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.8.0
>
>
> The current autoscaling algorithm adjusts the parallelism of the job task
> vertices according to the processing needs. By adjusting the parallelism, we
> systematically scale the amount of CPU for a task. At the same time, we also
> indirectly change the amount of memory tasks have at their dispense. However,
> there are some problems with this.
> # Memory is overprovisioned: On scale up we may add more memory than we
> actually need. Even on scale down, the memory / cpu ratio can still be off
> and too much memory is used.
> # Memory is underprovisioned: For stateful jobs, we risk running into
> OutOfMemoryErrors on scale down. Even before running out of memory, too
> little memory can have a negative impact on the effectiveness of the scaling.
> We lack the capability to tune memory proportionally to the processing needs.
> In the same way that we measure CPU usage and size the tasks accordingly, we
> need to evaluate memory usage and adjust the heap memory size.
> https://docs.google.com/document/d/19GXHGL_FvN6WBgFvLeXpDABog2H_qqkw1_wrpamkFSc/edit
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
