ziyangRen commented on issue #52276:
URL: https://github.com/apache/airflow/issues/52276#issuecomment-3030192131
@sjyangkevin I apologize for the inaccurate description of the phenomenon.
In fact, the repeated execution of the task was not caused by multiple dag
runs. The repeated executions were all within the same dag run. Regarding the
start_date and catch_up you mentioned, we have a control group. I will put the
code below. Even with a more recent start_date and setting catch_up=true, this
problem can still be reproduced. Additionally, regarding the database
operations I mentioned coming from other nodes, our latest finding is that all
three Celery instances may execute this database operation. Finally, we
attempted to add logs for the airflow's remove operation, but even when we
enabled the debug log and printed the log at all places where
TaskInstanceState.REMOVED was executed, we did not receive any logs of this
operation in the executor or scheduler. This is really frustrating.
`with DAG(
dag_id="example_bash_operator1",
schedule="0 0 * * *",
start_date=pendulum.today('Asia/Shanghai').add(days=-3),
catchup=True,
#dagrun_timeout=datetime.timedelta(minutes=60),
tags=["example", "example2"],
params={"example_key": "example_value"},
) as dag:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
