Re: Cancel a Running dag

We are exploring following approach for DAG cancellation. Please let us know if you see any issue with this 1) Set/create the xcom variable "cancel":"true". It would be set out of the band by updating the xcom Table in metadata store. 2) Operators would have the code to periodically check for t

Re: Cancel a Running dag

Thanks Bolke, Will it become part of Airflow 1.10 release. Is there any tentative timeline for same. -Raman Gupta On 2018/04/12 19:19:07, Bolke de Bruin wrote: > This is now fixed in master. Clearing tasks will now properly terminate a > running task. If you pause the dag run no new tasks will

Re: Cancel a Running dag

I have a similar use case, to support that I have implemented on_kill methods for all the operators. To cancel current run: Get all the running send a kill signal, in on_kill make sure current run for the task terminates gracefully, after the kill signal set state to failed(state should be set

Re: Cancel a Running dag

Like Bolke said, it has been fixed in master. One of the perquisites is support by the operator. For example, the Spark operator has implemented how to kill the Spark job on YARN, Local and Kubernetes. If you are running something else, you might want to check if this is implemented. Implemented o

Re: Cancel a Running dag

This is now fixed in master. Clearing tasks will now properly terminate a running task. If you pause the dag run no new tasks will be scheduled. B. Verstuurd vanaf mijn iPad > Op 12 apr. 2018 om 20:23 heeft Laura Lorenz het > volgende geschreven: > > That won't stop them if they are alread

Re: Cancel a Running dag

That won't stop them if they are already running in a celery worker or already in your messaging queue backend (e.g. rabbitmq; redis), but it will prevent the message to do them from being emitted again by the airflow scheduler to your messaging queue backend. To be thorough you have to do both - s

Re: Cancel a Running dag

Thanks Laura, We are using the CeleryExecutor. Just wondering if marking the TaskInstances as failed in metadata store would also work. -Raman On 2018/04/12 16:27:00, Laura Lorenz wrote: > I use the CeleryExecutor and have used a mix of `celery control` and > messaging queue purges to kill the

Re: Cancel a Running dag

I use the CeleryExecutor and have used a mix of `celery control` and messaging queue purges to kill the running tasks and prevent them from being picked up by workers again (respectively), and doctor the DagRun to failed to stop the scheduler from repopulating the message. I think if you are using

Re: Cancel a Running dag

I don't think killing a currently running task is possible today. Of course you can pause it from the CLI or web UI so that future runs don't get triggered, but it sounds like that's not what you're looking for. Best, Taylor *Taylor Edmiston* Blog | Stack Overflow CV

Cancel a Running dag

Hi All, We have a use case to cancel the already running DAG. So is there any recommended way to do so. Thanks, Raman