Hi devs,
So far we keep developing the native K8s integration for three major
release cycles in Flink.
-
Flink 1.10
-
Beta version
-
Dynamic resource allocation
-
Session mode only
-
Flink 1.11
-
Application mode
-
Usability Improvements
-
GPU resource support
-
Flink 1.12
-
Security improvements(kerberos, K8s secret)
-
PyFlink support
-
K8s HA
Now the native K8s integration is almost production ready. We already have
the common features for deploying a Flink cluster on K8s cluster natively.
However, compared with the current supported features[1] in
flink-on-k8s-operator and the feedback from users(ML and JIRAs), I am
afraid that we still need some advanced features.
-
Init containers[2]
-
Sidecar containers[3]
-
Arbitrary volume(PVC, hostpath, emptyDir, ConfigMap) mount[4]
-
Set owner reference for the JobManager deployment[5]
-
Support customizing of containers for native kubernetes setup[6]
-
Pod Affinity and Anti-affinity
-
Pod Security Context
-
Pod Environment Variables Reference
Although they may not be used very frequently or we already have some work
around, I think they are still important for the production. So I am
considering whether we could introduce a pod template[7] for all the use
cases. Users could configure a local yaml file for JobManager/TaskManager.
This will let Flink create the JobManager/TaskManager pods with the
template first, not the empty. Of course, the Flink config options for
Kubernetes[8] have high priority and could overwrite these values. The pod
template could help us get rid of maintaining more and more K8s features.
Looking forward to your suggestions.
[1]. https://github.com/GoogleCloudPlatform/flink-on-k8s-operator#features
[2]. https://issues.apache.org/jira/browse/FLINK-15641
[3]. https://issues.apache.org/jira/browse/FLINK-15871
[4]. https://issues.apache.org/jira/browse/FLINK-15649
[5]. https://issues.apache.org/jira/browse/FLINK-20359
[6]. https://issues.apache.org/jira/browse/FLINK-20324
[7]. https://issues.apache.org/jira/browse/FLINK-15656
[8].
https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes
Best,
Yang
