Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
AlvaroCavalcante commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2649657536 I was facing this issue, but I was able to call the on_failure_callback when I set this callback in the DAG definition, as stated here: https://stackoverflow.com/questions/69312630/airflow-triggering-the-on-failure-callback-when-the-dagrun-timeout-is-exceed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
morooshka commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2597125373 Also faced this. I see there one confusion - we fail the task on execution timeout but not fail on DAG timeout. To my mind the behaviour should be similar - fail the task P.S. In which case one should need to skip the task if the whole DAG times out? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
CyberDuck79 commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2551169610 Well, after experimentation, it seems there is confusion about callback configurations. There is DAG level callbacks: https://github.com/apache/airflow/discussions/38897 But we were also confused by the configurations: - In the DAG config in `default_args` (our old way of configuring callbacks) -> it configures the callbacks at the task level for all tasks. - In the DAG config directly (`on_failure_callback`, for example) -> callback triggered at dag level by dag state (so in case of dag run timeout, failed state -> failure callback) The second configuration would allow the callback to be triggered even if the task process is killed. For the question of the skipped status: Maybe the skipped status is not very clear because the task could have already started, but this information is included in the skipped callback so the DAG run timeout case can be handled. It seems that in Airflow, the meaning of the skipped state includes unfinished tasks. An unfinished state could be more clear, but is it worse to add a new state to handle? In regard to these discoveries, I think the issue should be closed because there is nothing that cannot be handled, and a conversation about a possible new "unfinished" state for those who find it interesting should be initiated. I hope my comments could help to clarify the confusions that I read in this conversation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
CyberDuck79 commented on issue #30264:
URL: https://github.com/apache/airflow/issues/30264#issuecomment-2550946737
> > I agree that this should be reopened. IMO skipped has ne sense for the
task instance state, as it was running but was terminating before it ends cause
of the dag run timeout. Beside that, we try a workaround using the
'on_skipped_callback' to set the TI state to failed, but it seems that the
callback is not executed in that scenario (airflow 2.9.3). Does someone knows
why ? Thanks
>
> We are running into the same issue too. In our case, we have our dags
timeout, which can cause some tasks to skip while running; these finish with
this:
>
> > [2024-11-22, 20:27:36 UTC] {taskinstance.py:3092} ERROR - Received
SIGTERM. Terminating subprocesses.
>
> We believe this means that `AirflowSkipException` is not being raised, so
`on_skip_callback` will not run at all. In order to actually call
`on_skipped_callback`, something would need to specifically raise
`AirflowSkipException`.
>
> Can anyone else confirm if our understanding is correct? At the end of the
day, we would like a way to have `on_kill' callback function that sends out an
email if any task is killed
Same behavior for us on MWAA, tasks are killed by SIGTERM, so it could not
launch any callback.
We discovered this behavior accidentally due to a bad configuration, but it
raised 3 questions:
- Is the SIGTERM normal behavior?
- Is skipped state the best state for that case?
- I suspected it, but are there no DAG-level callbacks?
For the DAG-level callbacks, it's not the first time we have encountered an
issue with our alerting system because tasks were killed (but it was due to the
MWAA Airflow internal architecture). So, it could be a good feature, even if I
think it is out of the scope of this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
github-actions[bot] commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2537466554 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
jrmidkiff commented on issue #30264:
URL: https://github.com/apache/airflow/issues/30264#issuecomment-2494767834
> I agree that this should be reopened. IMO skipped has ne sense for the
task instance state, as it was running but was terminating before it ends cause
of the dag run timeout. Beside that, we try a workaround using the
'on_skipped_callback' to set the TI state to failed, but it seems that the
callback is not executed in that scenario (airflow 2.9.3). Does someone knows
why ? Thanks
We are running into the same issue too. In our case, we have our dags
timeout, which can cause some tasks to skip while running; these finish with
this:
> [2024-11-22, 20:27:36 UTC] {taskinstance.py:3092} ERROR - Received
SIGTERM. Terminating subprocesses.
We believe this means that `AirflowSkipException` is not being raised, so
`on_skip_callback` will not run at all. In order to actually call
`on_skipped_callback`, something would need to specifically raise
`AirflowSkipException`.
Can anyone else confirm if our understanding is correct? At the end of the
day, we would like a way to have `on_kill' callback function that sends out an
email if any task is killed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
RNHTTR commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2453158616 Related: #42005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
github-actions[bot] commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2439657540 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
jledrumics commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2370803964 I agree that this should be reopen. IMO skipped has ne sense for the task instance state, as it was running but was terminating before it ends cause of the dag run timeout. Beside that, we try a workaround using the 'on_sipped_callback' to set the TI state to failed, but it seems that the callback is not executed in that scenario (airflow 2.9.3). Does someone knows why ? Thakns -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
github-actions[bot] commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2369824173 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
raphaelauv commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-2308387432 @pankajkoti could you please re-open the issue, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
raphaelauv commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-1926604905 I agree with @pankajkoti ``` Failed / Shutdown sounds more reasonable than the Skipped state. ``` I think we should re-open this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
github-actions[bot] closed issue #30264: Running tasks marked as skipped on DagRun timeout URL: https://github.com/apache/airflow/issues/30264 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
github-actions[bot] commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-1765457628 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Running tasks marked as skipped on DagRun timeout [airflow]
github-actions[bot] commented on issue #30264: URL: https://github.com/apache/airflow/issues/30264#issuecomment-1754095057 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
