[jira] [Updated] (FLINK-14460) Active Kubernetes integration phase 2 - Advanced Features
[ https://issues.apache.org/jira/browse/FLINK-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Wang updated FLINK-14460: -- Description: This is phase 2 of active kubernetes integration. It is a umbrella jira to track all the advanced features and make Flink on Kubernetes production ready. Since the K8s is fast evolving and many new features will be supported in the future. Recently, i am thinking about which feature should be supported in flink configuration directly. And i have asked some suggestion from our users. I get the very rough conclusion here. > Very common use features. They should be directly supported in flink > configuration * Kube config, namespace, service account, image(pull-policy, secrets), service exposed type * Annotations, Labels, NodeSelector * Toleration * Mem and cpu resources for jobmanager and taskmanager * Jobmanager deployment replication, support multiple jobmanagers to get fast recovery > Uncommon, but required by few production users. Maybe pod template > FLINK-15656 is enough(will be supported in phase3). * Affinity and anti-affinity * Volume mount, from persistent volume, user created configmap, etc. * Dns policy * Sidecar container, will be used for debugging, log collecting, etc. In our internal production environment, we choose log4j2 custom appender to collect the jobmanager/taskmanger logs to persistent storage(HDFS, Aliyun OSS, etc.) * Init container, various initialization before user program running, download user jars and dependencies, register with external system, etc. We provide two ways for the image choice, user specified image with users jars baked in or official image with init container. was: This is phase 2 of active kubernetes integration. It is a umbrella jira to track all the advanced features and make Flink on Kubernetes production ready. Since the K8s is fast evolving and many new features will be supported in the future. Recently, i am thinking about which feature should be supported in flink configuration directly. And i have asked some suggestion from our users. I get the very rough conclusion here. > Very common use features. They should be directly supported in flink > configuration * Kube config, namespace, service account, image(pull-policy, secrets), service exposed type * Annotations, Labels, NodeSelector * Toleration * Mem and cpu resources for jobmanager and taskmanager * Jobmanager deployment replication, support multiple jobmanagers to get fast recovery > Uncommon, but required by few production users. Maybe pod template > FLINK-15656 is enough. * Affinity and anti-affinity * Volume mount, from persistent volume, user created configmap, etc. * Dns policy * Sidecar container, will be used for debugging, log collecting, etc. In our internal production environment, we choose log4j2 custom appender to collect the jobmanager/taskmanger logs to persistent storage(HDFS, Aliyun OSS, etc.) * Init container, various initialization before user program running, download user jars and dependencies, register with external system, etc. We provide two ways for the image choice, user specified image with users jars baked in or official image with init container. > Active Kubernetes integration phase 2 - Advanced Features > - > > Key: FLINK-14460 > URL: https://issues.apache.org/jira/browse/FLINK-14460 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > Fix For: 1.11.0 > > > This is phase 2 of active kubernetes integration. It is a umbrella jira to > track all the advanced features and make Flink on Kubernetes production ready. > > Since the K8s is fast evolving and many new features will be supported in the > future. Recently, i am thinking about which feature should be supported in > flink configuration directly. And i have asked some suggestion from our > users. I get the very rough conclusion here. > > Very common use features. They should be directly supported in flink > > configuration > * Kube config, namespace, service account, image(pull-policy, secrets), > service exposed type > * Annotations, Labels, NodeSelector > * Toleration > * Mem and cpu resources for jobmanager and taskmanager > * Jobmanager deployment replication, support multiple jobmanagers to get > fast recovery > > Uncommon, but required by few production users. Maybe pod template > > FLINK-15656 is enough(will be supported in phase3). > * Affinity and anti-affinity > * Volume mount, from persistent volume, user created configmap, etc. > * Dns policy > * Sidecar container, will be used for debugging, log collecting, etc. In our > internal production environment, we choose log4j2 custom appender to collect > t
[jira] [Updated] (FLINK-14460) Active Kubernetes integration phase 2 - Advanced Features
[ https://issues.apache.org/jira/browse/FLINK-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Wang updated FLINK-14460: -- Description: This is phase 2 of active kubernetes integration. It is a umbrella jira to track all the advanced features and make Flink on Kubernetes production ready. Since the K8s is fast evolving and many new features will be supported in the future. Recently, i am thinking about which feature should be supported in flink configuration directly. And i have asked some suggestion from our users. I get the very rough conclusion here. > Very common use features. They should be directly supported in flink > configuration * Kube config, namespace, service account, image(pull-policy, secrets), service exposed type * Annotations, Labels, NodeSelector * Toleration * Mem and cpu resources for jobmanager and taskmanager * Jobmanager deployment replication, support multiple jobmanagers to get fast recovery > Uncommon, but required by few production users. Maybe pod template > FLINK-15656 is enough. * Affinity and anti-affinity * Volume mount, from persistent volume, user created configmap, etc. * Dns policy * Sidecar container, will be used for debugging, log collecting, etc. In our internal production environment, we choose log4j2 custom appender to collect the jobmanager/taskmanger logs to persistent storage(HDFS, Aliyun OSS, etc.) * Init container, various initialization before user program running, download user jars and dependencies, register with external system, etc. We provide two ways for the image choice, user specified image with users jars baked in or official image with init container. was:This is phase 2 of active kubernetes integration. It is a umbrella jira to track all the advanced features and make Flink on Kubernetes production ready. > Active Kubernetes integration phase 2 - Advanced Features > - > > Key: FLINK-14460 > URL: https://issues.apache.org/jira/browse/FLINK-14460 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > Fix For: 1.11.0 > > > This is phase 2 of active kubernetes integration. It is a umbrella jira to > track all the advanced features and make Flink on Kubernetes production ready. > > Since the K8s is fast evolving and many new features will be supported in the > future. Recently, i am thinking about which feature should be supported in > flink configuration directly. And i have asked some suggestion from our > users. I get the very rough conclusion here. > > Very common use features. They should be directly supported in flink > > configuration > * Kube config, namespace, service account, image(pull-policy, secrets), > service exposed type > * Annotations, Labels, NodeSelector > * Toleration > * Mem and cpu resources for jobmanager and taskmanager > * Jobmanager deployment replication, support multiple jobmanagers to get > fast recovery > > Uncommon, but required by few production users. Maybe pod template > > FLINK-15656 is enough. > * Affinity and anti-affinity > * Volume mount, from persistent volume, user created configmap, etc. > * Dns policy > * Sidecar container, will be used for debugging, log collecting, etc. In our > internal production environment, we choose log4j2 custom appender to collect > the jobmanager/taskmanger logs to persistent storage(HDFS, Aliyun OSS, etc.) > * Init container, various initialization before user program running, > download user jars and dependencies, register with external system, etc. We > provide two ways for the image choice, user specified image with users jars > baked in or official image with init container. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14460) Active Kubernetes integration phase 2 - Advanced Features
[ https://issues.apache.org/jira/browse/FLINK-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Wang updated FLINK-14460: -- Fix Version/s: 1.11.0 > Active Kubernetes integration phase 2 - Advanced Features > - > > Key: FLINK-14460 > URL: https://issues.apache.org/jira/browse/FLINK-14460 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > Fix For: 1.11.0 > > > This is phase 2 of active kubernetes integration. It is a umbrella jira to > track all the advanced features and make Flink on Kubernetes production ready. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14460) Active Kubernetes integration phase 2 - Advanced Features
[ https://issues.apache.org/jira/browse/FLINK-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-14460: -- Summary: Active Kubernetes integration phase 2 - Advanced Features (was: Active Kubernetes integration phase2 - Advanced Features) > Active Kubernetes integration phase 2 - Advanced Features > - > > Key: FLINK-14460 > URL: https://issues.apache.org/jira/browse/FLINK-14460 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: Yang Wang >Priority: Major > > This is phase 2 of active kubernetes integration. It is a umbrella jira to > track all the advanced features and make Flink on Kubernetes production ready. -- This message was sent by Atlassian Jira (v8.3.4#803005)