GitHub user Pebble32 added a comment to the discussion: Proposal: deterministic schedule jitter so daily DAGs don't all fire at the same time
Totally fair, and those knobs are the right tools when the work genuinely has to run at midnight. The one thing I would add: all of those limit or queue execution, but the DAGs are still scheduled at the same instant. And in the case I'm thinking about here is that they are at midnight by accident, not by need. `@daily` just resolves to `0 0 * * *`, so a bunch of jobs that only need to run once a day all land on the same second. Jitter is really about removing that accidental pile up so you do not have to over provision or hand tune pools for a spike that did not need to exist. I thought that a deterministic way of shuttering DAGs around would be a nice quality of life feature. That said, I get that this may not be something core wants to take on, and that is completely reasonable. I will build it as a small opt in plugin so anyone who hits the same thing can use it. If a maintainer ever feels it would be a welcome addition to core, I am happy to bring a PR with tests and docs. Thanks for talking it through. <3 GitHub link: https://github.com/apache/airflow/discussions/69027#discussioncomment-17472811 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
