BasPH commented on PR #35392: URL: https://github.com/apache/airflow/pull/35392#issuecomment-1803663947
I disagree with adding more options (enabled/disabled/ignore_first) because I think it introduces more unnecessary complexity. We should think ahead about the desired Airflow behavior >=2.8,<3 and >=3. To elaborate on the issue that users face, for `catchup` you can currently set `True` or `False`: - `True` is straightforward: when you unpause a DAG, it executes all intervals that should have been executed between the DAG's start_date and now. - `False`: Airflow doesn't execute DAGruns that should have executed in the past, but it does execute one DAGrun, which is the last complete interval. So say you have an `@daily` interval and unpause at 15:00 on jan 2nd, you will get one DAG run [jan 1st 00:00 - jan 2nd 00:00]. I have yet to find the first person that finds catchup's `False` current behavior logical: what people expect is that nothing happens until jan 3rd 00:00, at which they expect to execute the first interval [jan 2nd 00:00 - jan 3rd 00:00]. This is especially troublesome for users that do not use an interval start & end datetime in their jobs, but just want things to run at a certain point in time. When a DAG run starts at a different date/time than scheduled, this causes trouble. My suggestion would be: - Airflow >=2.8,<3: add a global config e.g.`ignore_first_catchup`: add it together with a deprecation warning for removal in Airflow 3. Clearly state that this is a temporary solution until a breaking change in Airflow 3 is introduced. I think a global parameter causes the least friction, supporting it as an argument on DAG-level would require people to change their code in 2.8 and change it again in 3. Default `False` would align with the current behavior, and setting it to `True` would not execute one DAG run from the past when unpausing a DAG. - Airflow >=3: remove `ignore_first_catchup` config, and make catchup=False not execute one DAG run upon unpausing. No code changes are required by users, only a behavioral change to be aware of. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
