kaxil commented on code in PR #70658: URL: https://github.com/apache/airflow/pull/70658#discussion_r3707318363
########## airflow-core/docs/authoring-and-scheduling/timetable.rst: ########## @@ -310,68 +311,121 @@ Airflow has two sets of timetables for cron and delta schedules: * CronTriggerTimetable_ and CronDataIntervalTimetable_ both accept a cron expression. * DeltaTriggerTimetable_ and DeltaDataIntervalTimetable_ both accept a timedelta or relativedelta. -- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) does not address the concept of *data interval*, while a "data interval" one (CronDataIntervalTimetable_ or DeltaDataIntervalTimetable_) does. -- The timestamp in the ``run_id``, the ``logical_date`` of the two timetable kinds are defined differently based on how they handle the data interval, as described in :ref:`timetables_run_id_logical_date`. - -Whether taking care of *Data Interval* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A trigger timetable *does not* include *data interval*. This means that the value of ``data_interval_start`` -and ``data_interval_end`` are the same; the time when a Dag run is triggered. - -For a data interval timetable, the value of ``data_interval_start`` and ``data_interval_end`` are different. -``data_interval_end`` is the time when a Dag run is triggered, while ``data_interval_start`` is the start of the interval. +In Airflow 3, a bare cron string such as ``@daily`` in ``schedule=`` resolves to +CronTriggerTimetable_ by default (``[scheduler] create_cron_data_intervals`` is +``False``). A bare ``timedelta`` resolves to DeltaTriggerTimetable_ when +``[scheduler] create_delta_data_intervals`` is ``False``. Set either flag to Review Comment: `create_delta_data_intervals` isn't read anywhere. `_create_timetable` checks `create_cron_data_intervals` in both branches, including the `timedelta`/`relativedelta` one: https://github.com/apache/airflow/blob/0bda2ffece97df4b90484a70dd4c62ff35993fda/task-sdk/src/airflow/sdk/definitions/dag.py#L149-L152. So setting `create_delta_data_intervals=True` on its own still gives you `DeltaTriggerTimetable`, and "set either flag to `True`" doesn't hold. The defaults happen to agree, so the default behaviour you describe is right. It looks like the flag was never wired up (2.11 hardcoded `DeltaDataIntervalTimetable` for deltas), so a code fix is probably the real answer, but the docs shouldn't promise the flag works in the meantime. ########## airflow-core/docs/authoring-and-scheduling/timetable.rst: ########## @@ -310,68 +311,121 @@ Airflow has two sets of timetables for cron and delta schedules: * CronTriggerTimetable_ and CronDataIntervalTimetable_ both accept a cron expression. * DeltaTriggerTimetable_ and DeltaDataIntervalTimetable_ both accept a timedelta or relativedelta. -- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) does not address the concept of *data interval*, while a "data interval" one (CronDataIntervalTimetable_ or DeltaDataIntervalTimetable_) does. -- The timestamp in the ``run_id``, the ``logical_date`` of the two timetable kinds are defined differently based on how they handle the data interval, as described in :ref:`timetables_run_id_logical_date`. - -Whether taking care of *Data Interval* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A trigger timetable *does not* include *data interval*. This means that the value of ``data_interval_start`` -and ``data_interval_end`` are the same; the time when a Dag run is triggered. - -For a data interval timetable, the value of ``data_interval_start`` and ``data_interval_end`` are different. -``data_interval_end`` is the time when a Dag run is triggered, while ``data_interval_start`` is the start of the interval. +In Airflow 3, a bare cron string such as ``@daily`` in ``schedule=`` resolves to +CronTriggerTimetable_ by default (``[scheduler] create_cron_data_intervals`` is +``False``). A bare ``timedelta`` resolves to DeltaTriggerTimetable_ when +``[scheduler] create_delta_data_intervals`` is ``False``. Set either flag to +``True``, or pass an explicit data-interval timetable class, to get contiguous +windows instead. + +- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) represents + each run as a point in time: by default ``data_interval_start`` and + ``data_interval_end`` are the same (the trigger time). You can optionally pass a + non-zero ``interval=`` so the data interval ends at the trigger time and spans + that duration. A data-interval timetable (CronDataIntervalTimetable_ or + DeltaDataIntervalTimetable_) always uses a contiguous non-zero window between + consecutive schedule boundaries. +- ``logical_date`` and the timestamp used in ``run_id`` differ between the two + kinds based on how they handle the data interval, as described in + :ref:`timetables_run_id_logical_date`. + +*Data Interval* Shape +~~~~~~~~~~~~~~~~~~~~~ + +A trigger timetable uses a *point* (zero-width) data interval by default. This +means that the values of ``data_interval_start`` and ``data_interval_end`` are +the same, the time when a Dag run is triggered. Passing a non-zero +``interval=`` makes the interval end at the trigger time and begin ``interval`` +earlier. + +For a data interval timetable, the values of ``data_interval_start`` and +``data_interval_end`` are different. ``data_interval_end`` is the time when a +Dag run is triggered (``run_after``), while ``data_interval_start`` is the start +of the contiguous window. ``logical_date`` is ``data_interval_start`` for both +kinds. *Catchup* behavior ^^^^^^^^^^^^^^^^^^ -By default, ``catchup`` is set to ``False``. This prevents running unnecessary Dags in the following scenarios: +By default, ``catchup`` is ``False`` (Airflow config +``[scheduler] catchup_by_default``). Missed scheduled run times between +``start_date`` and "now" are not backfilled when a Dag is activated or +re-enabled. The timetable instead selects the most recently applicable +scheduled run time: -- If you create a new Dag with a start date in the past, and don't want to run Dags for the past. If ``catchup`` is ``True``, Airflow runs all Dags that would have run in that time interval. -- If you pause an existing Dag, and then restart it at a later date, ``catchup`` being ``False`` means that Airflow does not run the Dags that would have run during the paused period. +- For CronTriggerTimetable_, the latest cron tick that is not after "now" and + not before ``start_date``. For DeltaTriggerTimetable_, pickup time itself — + a delta has no wall-clock tick to snap to. +- For a data-interval timetable, the most recently completed interval whose end + is not after "now". -In these scenarios, the ``logical_date`` in the ``run_id`` are based on how the timetable handles the data -interval. +If you set ``catchup=True``, the scheduler creates a Dag run for every scheduled +run time (or completed interval) between ``start_date`` and "now" that has not +yet run (or has been cleared). Runs are created in chronological order and may Review Comment: "or has been cleared" doesn't hold. The next run is derived by advancing from the last automated run (`calculate_dagrun_date_fields(..., reference_run=created_run)`, https://github.com/apache/airflow/blob/0bda2ffece97df4b90484a70dd4c62ff35993fda/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2601), so clearing an existing run's tasks never makes the scheduler create a run, and a run deleted from the middle of the sequence isn't refilled either. Catchup closes the gap between the latest automated run (or `start_date`) and now, nothing else. Separately, the `max_active_runs` sentence now contradicts the page you link three lines down: `dag-run.rst` still says "the scheduler will execute them sequentially", and it's also where the "(or has been cleared)" wording comes from. Folding that in here, or a follow-up? ########## airflow-core/docs/authoring-and-scheduling/timetable.rst: ########## @@ -310,68 +311,121 @@ Airflow has two sets of timetables for cron and delta schedules: * CronTriggerTimetable_ and CronDataIntervalTimetable_ both accept a cron expression. * DeltaTriggerTimetable_ and DeltaDataIntervalTimetable_ both accept a timedelta or relativedelta. -- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) does not address the concept of *data interval*, while a "data interval" one (CronDataIntervalTimetable_ or DeltaDataIntervalTimetable_) does. -- The timestamp in the ``run_id``, the ``logical_date`` of the two timetable kinds are defined differently based on how they handle the data interval, as described in :ref:`timetables_run_id_logical_date`. - -Whether taking care of *Data Interval* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A trigger timetable *does not* include *data interval*. This means that the value of ``data_interval_start`` -and ``data_interval_end`` are the same; the time when a Dag run is triggered. - -For a data interval timetable, the value of ``data_interval_start`` and ``data_interval_end`` are different. -``data_interval_end`` is the time when a Dag run is triggered, while ``data_interval_start`` is the start of the interval. +In Airflow 3, a bare cron string such as ``@daily`` in ``schedule=`` resolves to +CronTriggerTimetable_ by default (``[scheduler] create_cron_data_intervals`` is +``False``). A bare ``timedelta`` resolves to DeltaTriggerTimetable_ when +``[scheduler] create_delta_data_intervals`` is ``False``. Set either flag to +``True``, or pass an explicit data-interval timetable class, to get contiguous +windows instead. + +- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) represents + each run as a point in time: by default ``data_interval_start`` and + ``data_interval_end`` are the same (the trigger time). You can optionally pass a + non-zero ``interval=`` so the data interval ends at the trigger time and spans + that duration. A data-interval timetable (CronDataIntervalTimetable_ or + DeltaDataIntervalTimetable_) always uses a contiguous non-zero window between + consecutive schedule boundaries. +- ``logical_date`` and the timestamp used in ``run_id`` differ between the two + kinds based on how they handle the data interval, as described in + :ref:`timetables_run_id_logical_date`. + +*Data Interval* Shape +~~~~~~~~~~~~~~~~~~~~~ + +A trigger timetable uses a *point* (zero-width) data interval by default. This +means that the values of ``data_interval_start`` and ``data_interval_end`` are +the same, the time when a Dag run is triggered. Passing a non-zero +``interval=`` makes the interval end at the trigger time and begin ``interval`` +earlier. + +For a data interval timetable, the values of ``data_interval_start`` and +``data_interval_end`` are different. ``data_interval_end`` is the time when a +Dag run is triggered (``run_after``), while ``data_interval_start`` is the start +of the contiguous window. ``logical_date`` is ``data_interval_start`` for both +kinds. *Catchup* behavior ^^^^^^^^^^^^^^^^^^ Review Comment: Not from your change, but you're reshaping this hierarchy anyway: this underline is `^`, the same level as "Differences between "trigger" and "data interval" timetables", so catchup isn't a subsection of the comparison, and the two `~` sections after it ("The time when a Dag run is triggered" and "Switching between trigger and data interval timetables") end up nested under Catchup in the TOC. Dropping this to `~` puts all four at one level under the comparison section. ########## airflow-core/docs/authoring-and-scheduling/timetable.rst: ########## @@ -310,68 +311,121 @@ Airflow has two sets of timetables for cron and delta schedules: * CronTriggerTimetable_ and CronDataIntervalTimetable_ both accept a cron expression. * DeltaTriggerTimetable_ and DeltaDataIntervalTimetable_ both accept a timedelta or relativedelta. -- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) does not address the concept of *data interval*, while a "data interval" one (CronDataIntervalTimetable_ or DeltaDataIntervalTimetable_) does. -- The timestamp in the ``run_id``, the ``logical_date`` of the two timetable kinds are defined differently based on how they handle the data interval, as described in :ref:`timetables_run_id_logical_date`. - -Whether taking care of *Data Interval* -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A trigger timetable *does not* include *data interval*. This means that the value of ``data_interval_start`` -and ``data_interval_end`` are the same; the time when a Dag run is triggered. - -For a data interval timetable, the value of ``data_interval_start`` and ``data_interval_end`` are different. -``data_interval_end`` is the time when a Dag run is triggered, while ``data_interval_start`` is the start of the interval. +In Airflow 3, a bare cron string such as ``@daily`` in ``schedule=`` resolves to +CronTriggerTimetable_ by default (``[scheduler] create_cron_data_intervals`` is +``False``). A bare ``timedelta`` resolves to DeltaTriggerTimetable_ when +``[scheduler] create_delta_data_intervals`` is ``False``. Set either flag to +``True``, or pass an explicit data-interval timetable class, to get contiguous +windows instead. + +- A trigger timetable (CronTriggerTimetable_ or DeltaTriggerTimetable_) represents + each run as a point in time: by default ``data_interval_start`` and + ``data_interval_end`` are the same (the trigger time). You can optionally pass a + non-zero ``interval=`` so the data interval ends at the trigger time and spans + that duration. A data-interval timetable (CronDataIntervalTimetable_ or + DeltaDataIntervalTimetable_) always uses a contiguous non-zero window between + consecutive schedule boundaries. +- ``logical_date`` and the timestamp used in ``run_id`` differ between the two + kinds based on how they handle the data interval, as described in + :ref:`timetables_run_id_logical_date`. + +*Data Interval* Shape +~~~~~~~~~~~~~~~~~~~~~ + +A trigger timetable uses a *point* (zero-width) data interval by default. This +means that the values of ``data_interval_start`` and ``data_interval_end`` are +the same, the time when a Dag run is triggered. Passing a non-zero +``interval=`` makes the interval end at the trigger time and begin ``interval`` +earlier. + +For a data interval timetable, the values of ``data_interval_start`` and +``data_interval_end`` are different. ``data_interval_end`` is the time when a +Dag run is triggered (``run_after``), while ``data_interval_start`` is the start +of the contiguous window. ``logical_date`` is ``data_interval_start`` for both +kinds. *Catchup* behavior ^^^^^^^^^^^^^^^^^^ -By default, ``catchup`` is set to ``False``. This prevents running unnecessary Dags in the following scenarios: +By default, ``catchup`` is ``False`` (Airflow config +``[scheduler] catchup_by_default``). Missed scheduled run times between +``start_date`` and "now" are not backfilled when a Dag is activated or +re-enabled. The timetable instead selects the most recently applicable +scheduled run time: -- If you create a new Dag with a start date in the past, and don't want to run Dags for the past. If ``catchup`` is ``True``, Airflow runs all Dags that would have run in that time interval. -- If you pause an existing Dag, and then restart it at a later date, ``catchup`` being ``False`` means that Airflow does not run the Dags that would have run during the paused period. +- For CronTriggerTimetable_, the latest cron tick that is not after "now" and + not before ``start_date``. For DeltaTriggerTimetable_, pickup time itself — + a delta has no wall-clock tick to snap to. +- For a data-interval timetable, the most recently completed interval whose end + is not after "now". -In these scenarios, the ``logical_date`` in the ``run_id`` are based on how the timetable handles the data -interval. +If you set ``catchup=True``, the scheduler creates a Dag run for every scheduled +run time (or completed interval) between ``start_date`` and "now" that has not +yet run (or has been cleared). Runs are created in chronological order and may +run concurrently up to the Dag's ``max_active_runs`` (defaulted from :ref:`config:core__max_active_runs_per_dag`). -You can change the default ``catchup`` behavior using the Airflow config ``[scheduler] catchup_by_default``. +Catchup also applies when you pause a Dag for a period and then re-enable it. -See :ref:`dag-catchup` for more information about how Dag runs are triggered when using ``catchup``. +See :ref:`dag-catchup` for a worked example with ``@daily``. .. _timetables_run_id_logical_date: The time when a Dag run is triggered ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Both trigger and data interval timetables trigger Dag runs at the same time. However, the timestamp for the -``run_id`` is different for each. This is because ``run_id`` is based on ``logical_date``. - -For example, suppose there is a cron expression ``@daily`` or ``0 0 * * *``, which is scheduled to run at 12AM every day. If you enable Dags using the two timetables at 3PM on January -31st, - -- `CronTriggerTimetable`_ creates a new Dag run at 12AM on February 1st. The ``run_id`` timestamp is midnight, on February 1st. -- `CronDataIntervalTimetable`_ immediately creates a new Dag run, because a Dag run for the daily time interval beginning at 12AM on January 31st did not occur yet. The ``run_id`` timestamp is midnight, on January 31st, since that is the beginning of the data interval. - -The following is another example showing the difference in the case of skipping Dag runs: - -Suppose there are two running Dags with a cron expression ``@daily`` or ``0 0 * * *`` that use the two different timetables. If you pause the Dags at 3PM on January 31st and re-enable them at 3PM on February 2nd, - -- `CronTriggerTimetable`_ skips the Dag runs that were supposed to trigger on February 1st and 2nd. The next Dag run will be triggered at 12AM on February 3rd. -- `CronDataIntervalTimetable`_ skips the Dag runs that were supposed to trigger on February 1st only. A Dag run for February 2nd is immediately triggered after you re-enable the Dag. - -In these examples, you see how a trigger timetable creates Dag runs more intuitively and similar to what -people expect a workflow to behave, while a data interval timetable is designed heavily around the data -interval it processes, and does not reflect a workflow's own properties. +Both trigger and data interval timetables can create the first Dag run +immediately when ``catchup=False`` and ``start_date`` is in the past. What Review Comment: This is true with a `start_date`, but `start_date` is optional in Airflow 3 (only required when `catchup=True`). With no `start_date` on the Dag or any task, `restriction.earliest` is `None` and the trigger timetable falls into the `_calc_first_run()` branch instead (https://github.com/apache/airflow/blob/0bda2ffece97df4b90484a70dd4c62ff35993fda/airflow-core/src/airflow/timetables/trigger.py#L98-L106), which with the default `run_immediately` picks the next future tick. For the example below that's midnight February 1st, not January 31st, which is the answer the old text gave. A sentence covering the no-`start_date` case would keep both readers right. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
