[
https://issues.apache.org/jira/browse/FLINK-41025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41025:
-----------------------------------
Labels: pull-request-available (was: )
> Add config option to control enableServiceLinks of JobManager and TaskManager
> pods
> ----------------------------------------------------------------------------------
>
> Key: FLINK-41025
> URL: https://issues.apache.org/jira/browse/FLINK-41025
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
> Reporter: Guotao Ding
> Priority: Major
> Labels: pull-request-available
>
> h3. Motivation
> Kubernetes enables {{enableServiceLinks}} in the pod spec by default. With it
> enabled, the kubelet injects several environment variables
> ({{<SVC>_SERVICE_HOST}}, {{<SVC>_SERVICE_PORT}}, {{<SVC>_PORT_*}}, ...) into
> every container for each Service in the same namespace.
> Every Flink native Kubernetes cluster creates its own rest and internal
> Services, so the number of injected environment variables grows with the
> number of Flink clusters in the namespace. In our production environment, a
> single JobManager/TaskManager pod ended up with more than 20,000 environment
> variables. In this case, submitting a Flink job on Kubernetes becomes very
> slow: the JobManager and TaskManager pods take a long time to start, since
> every shell invocation in the pod has to deal with the huge number of
> environment variables.
> We have disabled service links for all Flink pods in our production
> environment. Since then, the number of environment variables in a pod has
> dropped to less than 100, the startup time is back to normal, and no
> functional impact on Flink has been observed.
> Flink does not read these Service environment variables: components are
> reached via Service DNS names and ConfigMaps. {{KUBERNETES_SERVICE_HOST}} and
> {{KUBERNETES_SERVICE_PORT}}, which the Kubernetes client relies on, are
> injected regardless of {{enableServiceLinks}}.
> h3. Proposed Change
> * Add a config option {{kubernetes.pod.service-links.enabled}} (Boolean, no
> default value).
> * If it is configured, {{InitJobManagerDecorator}} and
> {{InitTaskManagerDecorator}} set {{enableServiceLinks}} of the JobManager and
> TaskManager pod spec to the configured value, overwriting the value from the
> pod template.
> * If it is not configured, the field is left untouched, i.e. the value from
> the pod template is kept, or the Kubernetes default ({{true}}) applies when
> the pod template does not define it either. There is no behavior change for
> existing deployments.
> * Document the option and add {{enableServiceLinks}} to the "Fields
> Overwritten by Flink" table of the native Kubernetes documentation.
> h3. Alternatives
> Setting {{enableServiceLinks: false}} in a pod template
> ({{kubernetes.pod-template-file.*}}) already works. A dedicated option is
> still proposed because:
> * The number of injected variables grows with the Services that Flink itself
> creates, so the issue shows up specifically in namespaces with many Flink
> clusters.
> * The symptom (slow pod startup) does not point to the environment variables,
> so the pod template workaround is hard to find. A documented option makes the
> cause and the fix discoverable.
> * Kubernetes is not going to change the default of {{enableServiceLinks}}
> since it would be a breaking change (see
> [kubernetes#121787|https://github.com/kubernetes/kubernetes/issues/121787]),
> so it has to be disabled by the application that creates the pods.
> * Other pod spec fields that can be set via pod template also have dedicated
> options, e.g. {{kubernetes.hostnetwork.enabled}},
> {{kubernetes.service-account}}, {{kubernetes.jobmanager.node-selector}} /
> {{kubernetes.taskmanager.node-selector}},
> {{kubernetes.jobmanager.tolerations}} /
> {{kubernetes.taskmanager.tolerations}}.
> h3. How to reproduce
> # Submit a large number of Flink jobs in native Kubernetes mode into one
> namespace.
> # Start a new Flink job in that namespace, and check {{env | wc -l}} and
> {{time bash -c true}} inside its JobManager/TaskManager pod.
> # Set {{enableServiceLinks: false}} via a pod template and repeat step 2.
> h3. References
> * [kubernetes#92615: 1000+ services in the same namespace will degrade pod
> start time & eventually prevent pods from
> starting|https://github.com/kubernetes/kubernetes/issues/92615]
> * [kubernetes#121787: Set enableServiceLinks to false as
> default|https://github.com/kubernetes/kubernetes/issues/121787]
> * [太多的 service 信息环境变量可能会导致容器中执行 bash
> 命令特别的慢|https://mozillazg.com/2020/06/kubernetes-k8s-too-many-service-environment-variables-cause-pod-container-start-bash-too-slow.html]
> * [任务执行龟速，原因竟然是......|https://cloud.tencent.com/developer/article/1916662]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
