SameerMesiah97 commented on PR #59160: URL: https://github.com/apache/airflow/pull/59160#issuecomment-3622616126
> @SameerMesiah97, I see you have taken condition > > ``` > self.on_finish_action == OnFinishAction.DELETE_ACTIVE_POD > and (pod.status.phase == PodPhase.RUNNING or pod.status.phase == PodPhase.PENDING) > ``` > > isnt this true for all the running tasks? > > Also, can we implement a timebased mechanism for this? as some pods might be in pending state due to resource constaints for 1-2 mins and we dont want to delete them. `def process_pod_deletion()` is not called arbitrarily but on cleanup i.e. when the task has terminated/completed. You are right that the condition applies to all running pods but the code path in the PR will not pick them up unless the overarching task triggers the clean_up function. In addition to this, there are 2 other scenarios where `def process_pod_deletion()` is called: 1) `reattach_on_restart` is set to 'True' and am existing pod is found. The 'on_finish_action' variable decides whether to delete this pod. 2) A pod with duplicate labels if found but this code path is only activated when 'on_finish_action' is set to 'delete_pod'. I think we should expand this to include 'delete_active_pods' but I wanted feedback from others before I implemented it. In both of these 2 scenarios, only the pod(s) related to the tasks calling these functions will be affected. Thank you for the feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
