[jira] [Commented] (FLINK-15648) Support to configure request or limit for CPU requirement
[
https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028616#comment-17028616
]
Yang Wang commented on FLINK-15648:
---
[~felixzheng] Thanks for your detailed comments. It really makes sense to me to
adding the new config options for cpu limit. So {{kubernetes.jobmanager.cpu}}
will be the request value of cpu and we will add a new config option
{{kubernetes.jobmanager.cpu.limit}} for limit value. We should comment for the
new added limit config option, the negative value means no limit will be set.
Since the negative value of {{kubernetes.jobmanager.cpu}} has a different
semantic.
> Support to configure request or limit for CPU requirement
> -
>
> Key: FLINK-15648
> URL: https://issues.apache.org/jira/browse/FLINK-15648
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Priority: Major
>
> The current branch use kubernetes.xx.cpu to configure request and limit
> resource requirement for a Container, it's an important improvement to
> separate these two configurations, we can use kubernetes.xx.request.cpu and
> kubernetes.xx.limit.cpu to specify request and limit resource
> requirements.{color:#6a8759}
> {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15648) Support to configure request or limit for CPU requirement
[
https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028017#comment-17028017
]
Canbin Zheng commented on FLINK-15648:
--
Thanks [~fly_in_gis] . This ticket is about the CPU resource requirement, let’s
narrow the scope.
>>> Could you share the key advantages of separate the request and limit
>>> configuration
As you said, one can set the resource limit larger than the request to make the
most of the resources, such as share CPU resource when the machine CPU load is
low. And of course, one can just disables the CPU limit, just like what you do
for the internal use case, so that the Pods could use the CPU resource from the
landing machine as much as possible if they need. However it’s not always
suitable(also not particular recommend) to disable the CPU limit in production,
especially for the machine with large specification and has many Pods running
on, because the potential CPU competition brings extra overhead due to the CPU
scheduling mechanism, so in some case one needs to provide specific value for
the limit configuration option, instead of just disabling the limit.
As to the limit configuration option, it can satisfy your internal use case
also, the typical usage is as below:
# if not set, fallback to the CPU request configuration option
# if set with number no more than zero, disable the CPU limit
# set with number larger than the CPU request configuration option
>>> Since it will make kubernetes configuration very different from
>>> others(standalone, Yarn, mesos).
Could you share more details why it will make K8s configuration very different
from the standalone and mesos cluster managers?
As to the Yarn cluster manager, it do not support well for the request and
limit CPU settings, so the \{{yarn.xxx.vcores}} configuration option is just
good enough.
Lastly, it could be further discussed whether we should deprecate the
\{{kubernets.xxx.cpu}} configuration option, anyway I propose to provide
setting entrypoint for the CPU limit.
> Support to configure request or limit for CPU requirement
> -
>
> Key: FLINK-15648
> URL: https://issues.apache.org/jira/browse/FLINK-15648
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Priority: Major
>
> The current branch use kubernetes.xx.cpu to configure request and limit
> resource requirement for a Container, it's an important improvement to
> separate these two configurations, we can use kubernetes.xx.request.cpu and
> kubernetes.xx.limit.cpu to specify request and limit resource
> requirements.{color:#6a8759}
> {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15648) Support to configure request or limit for CPU requirement
[
https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027552#comment-17027552
]
Yang Wang commented on FLINK-15648:
---
[~felixzheng] Could you share the key advantages of separate the request and
limit configuration? Since it will make kubernetes configuration very different
from others(standalone, Yarn, mesos).
For cpu, it may be easier. Users could set the limit value bigger than request
to share some cpu when the machine cpu loads is low.
However for memory, if the limit value is bigger than request, the extra memory
could only be used by off-heap memory. How do we suggest our users to set the
limit value?
In fact, in our internal use case, we add a config option to turn off the
strict resource mode, just like Yarn.
{code:java}
public static final ConfigOption ENABLE_RESOURCE_MEMORY_LIMIT =
key("kubernetes.enable.resource.memory.limit")
.defaultValue(true)
.withDescription("This flag determines whether flink cluster should run with
strict memory resource limits " +
"or be allowed to consume spare resources if they need them.");
public static final ConfigOption ENABLE_RESOURCE_CPU_LIMIT =
key("kubernetes.enable.resource.cpu.limit")
.defaultValue(false)
.withDescription("This flag determines whether flink cluster should run with
strict cpu resource limits " +
"or be allowed to consume spare resources if they need them.");
{code}
> Support to configure request or limit for CPU requirement
> -
>
> Key: FLINK-15648
> URL: https://issues.apache.org/jira/browse/FLINK-15648
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Priority: Major
>
> The current branch use kubernetes.xx.cpu to configure request and limit
> resource requirement for a Container, it's an important improvement to
> separate these two configurations, we can use kubernetes.xx.request.cpu and
> kubernetes.xx.limit.cpu to specify request and limit resource
> requirements.{color:#6a8759}
> {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15648) Support to configure request or limit for CPU requirement
[
https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027467#comment-17027467
]
Canbin Zheng commented on FLINK-15648:
--
Thanks for the reminder [~gjy] , it's a subtask of FLINK-15788 now.
> Support to configure request or limit for CPU requirement
> -
>
> Key: FLINK-15648
> URL: https://issues.apache.org/jira/browse/FLINK-15648
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Priority: Major
>
> The current branch use kubernetes.xx.cpu to configure request and limit
> resource requirement for a Container, it's an important improvement to
> separate these two configurations, we can use kubernetes.xx.request.cpu and
> kubernetes.xx.limit.cpu to specify request and limit resource
> requirements.{color:#6a8759}
> {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15648) Support to configure request or limit for CPU requirement
[
https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027350#comment-17027350
]
Gary Yao commented on FLINK-15648:
--
Don't you want to link this to FLINK-15788?
> Support to configure request or limit for CPU requirement
> -
>
> Key: FLINK-15648
> URL: https://issues.apache.org/jira/browse/FLINK-15648
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Priority: Major
>
> The current branch use kubernetes.xx.cpu to configure request and limit
> resource requirement for a Container, it's an important improvement to
> separate these two configurations, we can use kubernetes.xx.request.cpu and
> kubernetes.xx.limit.cpu to specify request and limit resource
> requirements.{color:#6a8759}
> {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15648) Support to configure request or limit for CPU requirement
[
https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026402#comment-17026402
]
Canbin Zheng commented on FLINK-15648:
--
Tkx [~ptmagic]. It seems that the CPU limit is a more advanced configuration
option than the CPU request, how about adding a new parameter
"kubernetes.jobmanager.cpu.limit" instead of
"kubernetes.jobmanager.pod.cpu.requests"? {color:#6a8759}
{color}
WDYT [~yangwang166]
> Support to configure request or limit for CPU requirement
> -
>
> Key: FLINK-15648
> URL: https://issues.apache.org/jira/browse/FLINK-15648
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Priority: Major
>
> The current branch use kubernetes.xx.cpu to configure request and limit
> resource requirement for a Container, it's an important improvement to
> separate these two configurations, we can use kubernetes.xx.request.cpu and
> kubernetes.xx.limit.cpu to specify request and limit resource
> requirements.{color:#6a8759}
> {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15648) Support to configure request or limit for CPU requirement
[
https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018841#comment-17018841
]
ptmagic commented on FLINK-15648:
-
Is it possible to add some command line parameters to solve the problem？
add this parameters
public static final ConfigOption JOB_MANAGER_POD_CPU_REQUESTS =
key("kubernetes.jobmanager.pod.cpu.requests")
.noDefaultValue()
.withDescription("The cpu requests to be set for JobManager pods. " +
"Specified as ");
> Support to configure request or limit for CPU requirement
> -
>
> Key: FLINK-15648
> URL: https://issues.apache.org/jira/browse/FLINK-15648
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Priority: Major
>
> The current branch use kubernetes.xx.cpu to configure request and limit
> resource requirement for a Container, it's an important improvement to
> separate these two configurations, we can use kubernetes.xx.request.cpu and
> kubernetes.xx.limit.cpu to specify request and limit resource
> requirements.{color:#6a8759}
> {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
