CyberDuck79 commented on issue #30264:
URL: https://github.com/apache/airflow/issues/30264#issuecomment-2550946737
> > I agree that this should be reopened. IMO skipped has ne sense for the
task instance state, as it was running but was terminating before it ends cause
of the dag run timeout. Beside that, we try a workaround using the
'on_skipped_callback' to set the TI state to failed, but it seems that the
callback is not executed in that scenario (airflow 2.9.3). Does someone knows
why ? Thanks
>
> We are running into the same issue too. In our case, we have our dags
timeout, which can cause some tasks to skip while running; these finish with
this:
>
> > [2024-11-22, 20:27:36 UTC] {taskinstance.py:3092} ERROR - Received
SIGTERM. Terminating subprocesses.
>
> We believe this means that `AirflowSkipException` is not being raised, so
`on_skip_callback` will not run at all. In order to actually call
`on_skipped_callback`, something would need to specifically raise
`AirflowSkipException`.
>
> Can anyone else confirm if our understanding is correct? At the end of the
day, we would like a way to have `on_kill' callback function that sends out an
email if any task is killed
Same behavior for us on MWAA, tasks are killed by SIGTERM, so it could not
launch any callback.
We discovered this behavior accidentally due to a bad configuration, but it
raised 3 questions:
- Is the SIGTERM normal behavior?
- Is skipped state the best state for that case?
- I suspected it, but are there no DAG-level callbacks?
For the DAG-level callbacks, it's not the first time we have encountered an
issue with our alerting system because tasks were killed (but it was due to the
MWAA Airflow internal architecture). So, it could be a good feature, even if I
think it is out of the scope of this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
