Hi Xintong, Thanks for answering my question! After discussing with my teammates, we decide to rely on k8s pod and an external control plane to restrict the CPU usage of a job.
Thanks again for your help! Best, Fan ________________________________ From: Xintong Song <[email protected]> Sent: Thursday, May 6, 2021 8:39 PM To: [email protected] <[email protected]> Subject: Re: Question regarding cpu limit config in Flink standalone mode Hi Fan, For a java application, you cannot specify how many cpu a process should use. The JVM process will always try to use as much cpu time as it needs. The limitation can only come from external: hardware limit, OS scheduling, cgroups, etc. On Kubernetes, it is the pod's resource specifications that decide how many cpu resources a Flink JM/TM can use. - For the standalone kubernetes deployment, you can specify the pods' resources in your yaml files. - For the native kubernetes deployment, TM pods are requested by Flink's ResourceManager. Thus, the configuration option `kubernets.taskmanager.cpu` controls the cpu resource of pods Flink requests from Kubernetes. Thank you~ Xintong Song On Fri, May 7, 2021 at 10:35 AM Fan Xie <[email protected]<mailto:[email protected]>> wrote: Hi Flink Community, Recently I am working on an auto-scaling project that needs to dynamically adjust the cpu config of Flink standalone jobs . Our jobs will be running on standalone mode in a k8s cluster. After going through the configuration doc: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.13%2Fdocs%2Fdeployment%2Fconfig%2F&data=04%7C01%7Cfxie%40linkedin.com%7Ca71d36d7723b4a93022308d91109bfb7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637559555840356339%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YCVQnQlSBYu1lxTzI%2BBdCoQwri0%2Fz7he%2BQbMp7ljDCg%3D&reserved=0>, I can't find a config that can directly control the cpu of a standalone flink job. I can only see kubernetes.taskmanager.cpu, but looks like this config is only useful in native k8s mode. I also notice another config: taskmanager.numberOfTaskSlots that can control the cpu config in an indirect way. Is there any reason why we can't config the cpu for a standalone job directly? Thanks for answering my question. Best, Fan
