Re: High availability data clean up
Thanks for the info, Yang! I'm using Finalizer and the labels to handle the deletion. On Mon, Oct 25, 2021 at 3:56 AM Yang Wang wrote: > Hi Weiqing, > > > Why does Flink not set the owner reference of HA related ConfigMaps to > JobManager deployment? It is easier to clean up for users. > The major reason is that simply deleting the HA related ConfigMaps will > make the HA data located in DFS leak. > > > How to delete the HA ConfigMap from external tools(e.g. kubectl, K8s > operator)? > All the HA ConfigMaps have specific labels > "app=,configmap-type=high-availability". So it is easy to clean > up them manually. > > kubectl delete cm > --selector='app=,configmap-type=high-availability' > > Best, > Yang > > Weiqing Yang 于2021年10月23日周六 上午9:10写道： > >> Thanks for the replies, Yangze and Vijay! >> >> We are using standalone Flink on K8s (we created a K8s operator to >> manage the life cycle of the flink clusters (session mode)). Seems there is >> no way for the operator to know when these HA related configMaps are >> created (if the operator somehow can know when these HA configMap are >> created, then we can add ownerRef for them). Please let me know if I missed >> anything and if you have any recommended way to clean these HA >> related data/configMaps when deleting a flink cluster. >> >> Best, >> Wq >> >> On Thu, Oct 21, 2021 at 11:05 PM Vijay Bhaskar >> wrote: >> >>> In HA mode the configMap will be retained after deletion of the >>> deployment: >>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/ >>> ( Refer High availability data clean up) >>> >>> >>> >>> On Fri, Oct 22, 2021 at 8:13 AM Yangze Guo wrote: >>> For application mode, when the job finished normally or be canceled, the ConfigMaps will be cleanup. For session mode, when you stop the session through [1], the ConfigMaps will be cleanup. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#stop-a-running-session-cluster Best, Yangze Guo On Thu, Oct 21, 2021 at 6:37 AM Weiqing Yang wrote: > > > Hi, > > Per the doc, `kubernetes.jobmanager.owner.reference` can be used to set up the owners of the job manager Deployment. If the owner is deleted, then the job manager and its related pods will be deleted. How about the HA related ConfigMaps? Are they also deleted when deleting the owner of the job manager Deployment? Per the wiki here, the HA data will be retained when deleting jobmanager Deployment. If we want to delete these HA related configMaps as well when deleting the job manager, what is the suggested way to do that? > > Thanks, > weiqing > >>>
Re: High availability data clean up
Hi Weiqing, > Why does Flink not set the owner reference of HA related ConfigMaps to JobManager deployment? It is easier to clean up for users. The major reason is that simply deleting the HA related ConfigMaps will make the HA data located in DFS leak. > How to delete the HA ConfigMap from external tools(e.g. kubectl, K8s operator)? All the HA ConfigMaps have specific labels "app=,configmap-type=high-availability". So it is easy to clean up them manually. kubectl delete cm --selector='app=,configmap-type=high-availability' Best, Yang Weiqing Yang 于2021年10月23日周六 上午9:10写道： > Thanks for the replies, Yangze and Vijay! > > We are using standalone Flink on K8s (we created a K8s operator to manage > the life cycle of the flink clusters (session mode)). Seems there is no way > for the operator to know when these HA related configMaps are created (if > the operator somehow can know when these HA configMap are created, then we > can add ownerRef for them). Please let me know if I missed anything and if > you have any recommended way to clean these HA related data/configMaps when > deleting a flink cluster. > > Best, > Wq > > On Thu, Oct 21, 2021 at 11:05 PM Vijay Bhaskar > wrote: > >> In HA mode the configMap will be retained after deletion of the >> deployment: >> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/ >> ( Refer High availability data clean up) >> >> >> >> On Fri, Oct 22, 2021 at 8:13 AM Yangze Guo wrote: >> >>> For application mode, when the job finished normally or be canceled, >>> the ConfigMaps will be cleanup. >>> For session mode, when you stop the session through [1], the >>> ConfigMaps will be cleanup. >>> >>> [1] >>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#stop-a-running-session-cluster >>> >>> Best, >>> Yangze Guo >>> >>> On Thu, Oct 21, 2021 at 6:37 AM Weiqing Yang >>> wrote: >>> > >>> > >>> > Hi, >>> > >>> > Per the doc, `kubernetes.jobmanager.owner.reference` can be used to >>> set up the owners of the job manager Deployment. If the owner is deleted, >>> then the job manager and its related pods will be deleted. How about the HA >>> related ConfigMaps? Are they also deleted when deleting the owner of the >>> job manager Deployment? Per the wiki here, the HA data will be retained >>> when deleting jobmanager Deployment. If we want to delete these HA related >>> configMaps as well when deleting the job manager, what is the suggested way >>> to do that? >>> > >>> > Thanks, >>> > weiqing >>> > >>> >>
Re: High availability data clean up
Thanks for the replies, Yangze and Vijay! We are using standalone Flink on K8s (we created a K8s operator to manage the life cycle of the flink clusters (session mode)). Seems there is no way for the operator to know when these HA related configMaps are created (if the operator somehow can know when these HA configMap are created, then we can add ownerRef for them). Please let me know if I missed anything and if you have any recommended way to clean these HA related data/configMaps when deleting a flink cluster. Best, Wq On Thu, Oct 21, 2021 at 11:05 PM Vijay Bhaskar wrote: > In HA mode the configMap will be retained after deletion of the > deployment: > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/ > ( Refer High availability data clean up) > > > > On Fri, Oct 22, 2021 at 8:13 AM Yangze Guo wrote: > >> For application mode, when the job finished normally or be canceled, >> the ConfigMaps will be cleanup. >> For session mode, when you stop the session through [1], the >> ConfigMaps will be cleanup. >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#stop-a-running-session-cluster >> >> Best, >> Yangze Guo >> >> On Thu, Oct 21, 2021 at 6:37 AM Weiqing Yang >> wrote: >> > >> > >> > Hi, >> > >> > Per the doc, `kubernetes.jobmanager.owner.reference` can be used to set >> up the owners of the job manager Deployment. If the owner is deleted, then >> the job manager and its related pods will be deleted. How about the HA >> related ConfigMaps? Are they also deleted when deleting the owner of the >> job manager Deployment? Per the wiki here, the HA data will be retained >> when deleting jobmanager Deployment. If we want to delete these HA related >> configMaps as well when deleting the job manager, what is the suggested way >> to do that? >> > >> > Thanks, >> > weiqing >> > >> >
Re: High availability data clean up
In HA mode the configMap will be retained after deletion of the deployment: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/ ( Refer High availability data clean up) On Fri, Oct 22, 2021 at 8:13 AM Yangze Guo wrote: > For application mode, when the job finished normally or be canceled, > the ConfigMaps will be cleanup. > For session mode, when you stop the session through [1], the > ConfigMaps will be cleanup. > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#stop-a-running-session-cluster > > Best, > Yangze Guo > > On Thu, Oct 21, 2021 at 6:37 AM Weiqing Yang > wrote: > > > > > > Hi, > > > > Per the doc, `kubernetes.jobmanager.owner.reference` can be used to set > up the owners of the job manager Deployment. If the owner is deleted, then > the job manager and its related pods will be deleted. How about the HA > related ConfigMaps? Are they also deleted when deleting the owner of the > job manager Deployment? Per the wiki here, the HA data will be retained > when deleting jobmanager Deployment. If we want to delete these HA related > configMaps as well when deleting the job manager, what is the suggested way > to do that? > > > > Thanks, > > weiqing > > >
Re: High availability data clean up
For application mode, when the job finished normally or be canceled, the ConfigMaps will be cleanup. For session mode, when you stop the session through [1], the ConfigMaps will be cleanup. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#stop-a-running-session-cluster Best, Yangze Guo On Thu, Oct 21, 2021 at 6:37 AM Weiqing Yang wrote: > > > Hi, > > Per the doc, `kubernetes.jobmanager.owner.reference` can be used to set up > the owners of the job manager Deployment. If the owner is deleted, then the > job manager and its related pods will be deleted. How about the HA related > ConfigMaps? Are they also deleted when deleting the owner of the job manager > Deployment? Per the wiki here, the HA data will be retained when deleting > jobmanager Deployment. If we want to delete these HA related configMaps as > well when deleting the job manager, what is the suggested way to do that? > > Thanks, > weiqing >
