Tegh25 commented on code in PR #70658: URL: https://github.com/apache/airflow/pull/70658#discussion_r3731345107
########## airflow-core/docs/authoring-and-scheduling/timetable.rst: ########## @@ -310,68 +311,121 @@ Airflow has two sets of timetables for cron and delta schedules: * CronTriggerTimetable_ and CronDataIntervalTimetable_ both accept a cron expression. * DeltaTriggerTimetable_ and DeltaDataIntervalTimetable_ both accept a timedelta or relativedelta. -- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) does not address the concept of *data interval*, while a "data interval" one (CronDataIntervalTimetable_ or DeltaDataIntervalTimetable_) does. -- The timestamp in the ``run_id``, the ``logical_date`` of the two timetable kinds are defined differently based on how they handle the data interval, as described in :ref:`timetables_run_id_logical_date`. - -Whether taking care of *Data Interval* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A trigger timetable *does not* include *data interval*. This means that the value of ``data_interval_start`` -and ``data_interval_end`` are the same; the time when a Dag run is triggered. - -For a data interval timetable, the value of ``data_interval_start`` and ``data_interval_end`` are different. -``data_interval_end`` is the time when a Dag run is triggered, while ``data_interval_start`` is the start of the interval. +In Airflow 3, a bare cron string such as ``@daily`` in ``schedule=`` resolves to +CronTriggerTimetable_ by default (``[scheduler] create_cron_data_intervals`` is +``False``). A bare ``timedelta`` resolves to DeltaTriggerTimetable_ when +``[scheduler] create_delta_data_intervals`` is ``False``. Set either flag to Review Comment: I believe it is worth documenting the intended behavior, especially since the PR with the code fix is open and may be merged soon. As a compromise, I reworded the section slightly to not promise anything when `create_delta_data_intervals` is set, only stating its default. I also added the note stating that `create_delta_data_intervals` is currently not read. ########## airflow-core/docs/authoring-and-scheduling/timetable.rst: ########## @@ -310,68 +311,121 @@ Airflow has two sets of timetables for cron and delta schedules: * CronTriggerTimetable_ and CronDataIntervalTimetable_ both accept a cron expression. * DeltaTriggerTimetable_ and DeltaDataIntervalTimetable_ both accept a timedelta or relativedelta. -- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) does not address the concept of *data interval*, while a "data interval" one (CronDataIntervalTimetable_ or DeltaDataIntervalTimetable_) does. -- The timestamp in the ``run_id``, the ``logical_date`` of the two timetable kinds are defined differently based on how they handle the data interval, as described in :ref:`timetables_run_id_logical_date`. - -Whether taking care of *Data Interval* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A trigger timetable *does not* include *data interval*. This means that the value of ``data_interval_start`` -and ``data_interval_end`` are the same; the time when a Dag run is triggered. - -For a data interval timetable, the value of ``data_interval_start`` and ``data_interval_end`` are different. -``data_interval_end`` is the time when a Dag run is triggered, while ``data_interval_start`` is the start of the interval. +In Airflow 3, a bare cron string such as ``@daily`` in ``schedule=`` resolves to +CronTriggerTimetable_ by default (``[scheduler] create_cron_data_intervals`` is +``False``). A bare ``timedelta`` resolves to DeltaTriggerTimetable_ when +``[scheduler] create_delta_data_intervals`` is ``False``. Set either flag to +``True``, or pass an explicit data-interval timetable class, to get contiguous +windows instead. + +- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) represents + each run as a point in time: by default ``data_interval_start`` and + ``data_interval_end`` are the same (the trigger time). You can optionally pass a + non-zero ``interval=`` so the data interval ends at the trigger time and spans + that duration. A data-interval timetable (CronDataIntervalTimetable_ or + DeltaDataIntervalTimetable_) always uses a contiguous non-zero window between + consecutive schedule boundaries. +- ``logical_date`` and the timestamp used in ``run_id`` differ between the two + kinds based on how they handle the data interval, as described in + :ref:`timetables_run_id_logical_date`. + +*Data Interval* Shape +~~~~~~~~~~~~~~~~~~~~~ + +A trigger timetable uses a *point* (zero-width) data interval by default. This +means that the values of ``data_interval_start`` and ``data_interval_end`` are +the same, the time when a Dag run is triggered. Passing a non-zero +``interval=`` makes the interval end at the trigger time and begin ``interval`` +earlier. + +For a data interval timetable, the values of ``data_interval_start`` and +``data_interval_end`` are different. ``data_interval_end`` is the time when a +Dag run is triggered (``run_after``), while ``data_interval_start`` is the start +of the contiguous window. ``logical_date`` is ``data_interval_start`` for both +kinds. *Catchup* behavior ^^^^^^^^^^^^^^^^^^ -By default, ``catchup`` is set to ``False``. This prevents running unnecessary Dags in the following scenarios: +By default, ``catchup`` is ``False`` (Airflow config +``[scheduler] catchup_by_default``). Missed scheduled run times between +``start_date`` and "now" are not backfilled when a Dag is activated or +re-enabled. The timetable instead selects the most recently applicable +scheduled run time: -- If you create a new Dag with a start date in the past, and don't want to run Dags for the past. If ``catchup`` is ``True``, Airflow runs all Dags that would have run in that time interval. -- If you pause an existing Dag, and then restart it at a later date, ``catchup`` being ``False`` means that Airflow does not run the Dags that would have run during the paused period. +- For CronTriggerTimetable_, the latest cron tick that is not after "now" and + not before ``start_date``. For DeltaTriggerTimetable_, pickup time itself — + a delta has no wall-clock tick to snap to. +- For a data-interval timetable, the most recently completed interval whose end + is not after "now". -In these scenarios, the ``logical_date`` in the ``run_id`` are based on how the timetable handles the data -interval. +If you set ``catchup=True``, the scheduler creates a Dag run for every scheduled +run time (or completed interval) between ``start_date`` and "now" that has not +yet run (or has been cleared). Runs are created in chronological order and may Review Comment: I removed "or has been cleared" and tightened the `max_active_runs` sentence. `dag-run.rst` no longer states "the scheduler will execute them sequentially", the "(or has been cleared)" statement in `dag-run.rst` can be followed up after this PR is resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
