GitHub user Pebble32 added a comment to the discussion: Proposal: deterministic schedule jitter so daily DAGs don't all fire at the same time
Thanks for the reply, that is fair and pools are a good call for capping how many tasks run at once. The gap I am trying to close is a little different though. Pools throttle execution, but every DAG is still scheduled at the exact same instant, so the scheduler still gets the midnight spike and the work just queues instead of spreading out in time. For anything tied to a specific moment (an external API, or a database that is busy right at midnight) spreading the start times helps in a way a concurrency cap does not. On the "put something in place for your specific case" part: I have actually already built a working version, and I am happy to write it up properly with tests and docs and maintain it going forward. Maintaining it is genuinely not a problem for me, I'm big fan of the project and would love to help. I just think it would help plenty of other deployments that hit the same midnight pile up, which is why I would rather contribute it than keep it local. Would a small deterministic jitter timetable be welcome, either in core or as a community plugin? Happy to go whichever route you prefer. GitHub link: https://github.com/apache/airflow/discussions/69027#discussioncomment-17469916 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
