kacpermuda opened a new issue, #48927: URL: https://github.com/apache/airflow/issues/48927
### Apache Airflow version main (development) ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? II've noticed that OpenLineage's listener is sending two FAIL events when task has some retries left but the problem is not OpenLineage-related - the listener_manager gets called twice. The problem is that [this](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/taskinstance.py#L3099) get_listener_manager().hook.on_task_instance_failed listener's call is done on the scheduler, exactly [here](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L890). This is because the task is falling into the if for tasks killed_externally, and the ti.handle_failure is called (and it has the listener's call inside). ### What you think should happen instead? I think that this simple task execution should not be treated as external task state change and for sure, the listener should be called once. ### How to reproduce 1. Run this DAG on latest main. 2. Look in the scheduler logs, you should find ERROR logs `Executor CeleryExecutor ... reported that the task instance ... finished with state success, but the task instance's state attribute is running` and then some logs about listener's on_task_instance_failed being called (on DEBUG level) ``` import datetime as dt from airflow import DAG from airflow.providers.standard.operators.bash import BashOperator with DAG( dag_id="dag_failure_wait", start_date=dt.datetime(2024, 7, 3), schedule=None, catchup=False, ) as dag: task_failure = BashOperator( task_id="task_failure", bash_command="sleep 2 && exit 1;", retry_delay=1, retries=1, ) ``` ### Operating System MacOS ### Versions of Apache Airflow Providers latest main ### Deployment Virtualenv installation ### Deployment details Breeze, with LocalExecutor / CeleryExecutor (tested both). `breeze start-airflow --integration openlineage -b postgres` `breeze start-airflow --integration openlineage -b postgres --executor=CeleryExecutor` ### Anything else? Logs for CeleryExecutor run: [log_celery_20250408_105902.txt](https://github.com/user-attachments/files/19647875/log_celery_20250408_105902.txt) [log_scheduler_20250408_105917.txt](https://github.com/user-attachments/files/19647876/log_scheduler_20250408_105917.txt) Logs for LocalExecutor run: [log_scheduler_20250408_110833.txt](https://github.com/user-attachments/files/19647906/log_scheduler_20250408_110833.txt) Logs for LocalExecutor without the OpenLineage integration: [log_scheduler_20250408_111105.txt](https://github.com/user-attachments/files/19647938/log_scheduler_20250408_111105.txt) OL events received for the run (notice two FAIL events for first task run):  ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
