auyer commented on issue #37681: URL: https://github.com/apache/airflow/issues/37681#issuecomment-1984210977
I had a different take. In my case, I only allow one task at a time. Sometimes after a failure, even when Airflow considered the task to be in error state, the Spark Task would still live on. One thing I did to deal with this, is to force the deletion old spark application with the same name. Airflow will always generate the same Spark Application name, and I run this code before every execution (with Python Operator). ```python from kubernetes import client, config def DeleteSparkApplicationByName(name, **context): try: config.load_incluster_config() except: try: config.load_kube_config() except Exception as e: raise (e) coa = client.CustomObjectsApi() group = "sparkoperator.k8s.io" version = "v1beta2" namespace = "spark-operator" plural = "SparkApplications".lower() obj = None try: obj = coa.get_namespaced_custom_object(group, version, namespace, plural, name) except: pass if obj: coa.delete_namespaced_custom_object(group, version, namespace, plural, name) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org