Pebble32 commented on code in PR #69705:
URL: https://github.com/apache/airflow/pull/69705#discussion_r3657165640
##########
airflow-core/src/airflow/timetables/trigger.py:
##########
@@ -598,3 +599,83 @@ def generate_run_id(
suffix = f"{suffix}__{partition_key}"
suffix = f"{suffix}__{get_random_string()}"
return run_type.generate_run_id(suffix=suffix)
+
+
+class JitteredCronTimetable(CronTriggerTimetable):
+ """
+ A :class:`CronTriggerTimetable` that offsets each run by a deterministic,
per-DAG jitter.
+
+ Behaves exactly like ``CronTriggerTimetable`` but shifts every fire time
by a fixed offset
+ derived from ``seed`` and spread across ``[0, max_jitter)``. This avoids
the "thundering
+ herd" where many DAGs sharing a cron expression (e.g. ``@daily`` -> ``0 0
* * *``) all fire
+ at the same instant and overload the scheduler and workers at that
boundary.
+
+ The offset is deterministic: the same ``seed`` always maps to the same
offset, so runs are
+ stable and predictable across scheduler restarts and timetable
serialization. DAGs with
+ different seeds land in different slots; collisions are possible but
harmless (two DAGs
+ sharing one minute still beats every DAG firing at once).
+
+ All ``data_interval`` and ``logical_date`` semantics are inherited from
+ ``CronTriggerTimetable`` -- only the wall-clock fire time is shifted.
Review Comment:
Went with rewording over pinning logical_date. It derives from the fire time
in `CronTriggerTimetable`, so shifting it with the offset is intended behavior,
not a bug to fix. Doc now says `logical_date`/`data_interval` shift with the
offset. Let me know if pinning is preferred
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
