Hi Yang Wang, thanks for starting this discussion. I like the idea of providing a general mechanism to customize Flink's K8s deployment instead of implementing each feature separately. This should lower the maintenance overhead and people coming from K8s should already be familiar with how to specify the pod template.
Cheers, Till On Wed, Dec 9, 2020 at 4:44 PM Yang Wang <danrtsey...@gmail.com> wrote: > Hi devs, > > So far we keep developing the native K8s integration for three major > release cycles in Flink. > > - > > Flink 1.10 > - > > Beta version > - > > Dynamic resource allocation > - > > Session mode only > - > > Flink 1.11 > - > > Application mode > - > > Usability Improvements > - > > GPU resource support > - > > Flink 1.12 > - > > Security improvements(kerberos, K8s secret) > - > > PyFlink support > - > > K8s HA > > Now the native K8s integration is almost production ready. We already have > the common features for deploying a Flink cluster on K8s cluster natively. > However, compared with the current supported features[1] in > flink-on-k8s-operator and the feedback from users(ML and JIRAs), I am > afraid that we still need some advanced features. > > - > > Init containers[2] > - > > Sidecar containers[3] > - > > Arbitrary volume(PVC, hostpath, emptyDir, ConfigMap) mount[4] > - > > Set owner reference for the JobManager deployment[5] > - > > Support customizing of containers for native kubernetes setup[6] > - > > Pod Affinity and Anti-affinity > - > > Pod Security Context > - > > Pod Environment Variables Reference > > Although they may not be used very frequently or we already have some work > around, I think they are still important for the production. So I am > considering whether we could introduce a pod template[7] for all the use > cases. Users could configure a local yaml file for JobManager/TaskManager. > This will let Flink create the JobManager/TaskManager pods with the > template first, not the empty. Of course, the Flink config options for > Kubernetes[8] have high priority and could overwrite these values. The pod > template could help us get rid of maintaining more and more K8s features. > > > Looking forward to your suggestions. > > > [1]. https://github.com/GoogleCloudPlatform/flink-on-k8s-operator#features > > [2]. https://issues.apache.org/jira/browse/FLINK-15641 > > [3]. https://issues.apache.org/jira/browse/FLINK-15871 > > [4]. https://issues.apache.org/jira/browse/FLINK-15649 > > [5]. https://issues.apache.org/jira/browse/FLINK-20359 > > [6]. https://issues.apache.org/jira/browse/FLINK-20324 > > [7]. https://issues.apache.org/jira/browse/FLINK-15656 > > [8]. > > https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes > > > Best, > > Yang >
