GitHub user tzachii edited a discussion: Issue with on_failure_callback and
Downstream Retry in Airflow DAG
Hi,
I have a dag with on_failure_callback function that writes the error to a
BigQuery table.
The Airflow has 1 retry.
default_args = {
'owner': 'airflow',
'start_date': datetime(2023, 1, 1),
'retries': 1,
'retry_delay': timedelta(seconds=10),
'on_failure_callback': insert_failure_log_into_bq
}
with DAG(
dag_id='dag',
default_args=default_args,
schedule_interval=None,
catchup=False,
on_success_callback=None,
on_failure_callback=insert_failure_log_into_bq
) as dag:
def insert_failure_log_into_bq(context):
task_instance = context.get('task_instance')
dag_id = context.get('dag').dag_id
task_id = task_instance.task_id
....
task1 >> task2 >> ... Task10
The issue is that when the dag fails, for example on task2, for the first try
it captures task2 as the failed task. But for the second try, it captures one
of the downstream tasks as the failed task.
Why does the on_failure_callback capture a downstream task as the failed task
on the second retry?
I use PythonOperator and BigQueryOperators.
GitHub link: https://github.com/apache/airflow/discussions/46120
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]