Pebble32 commented on PR #69705: URL: https://github.com/apache/airflow/pull/69705#issuecomment-5217082043
Fair points. You are right it was not a scheduler error, the scheduler created the runs fine like your 1000 case. The failures were downstream, [tasks hitting X all at once]. And I agree concurrency limits and pools are the correct tool for capacity, and that jitter does not help backfill, clear or new dags. Those need concurrency control. Where I still see jitter helping is narrower: peak shaving for the recurring scheduled case, opt in, when timing is not critical. It shrinks the spike at the source so you lean on throttling less, rather than replacing it. Same reason Jenkins has H for cron, as @uranusjr mentioned. Complementary to pools, not a substitute. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
