raphaelauv commented on PR #69705: URL: https://github.com/apache/airflow/pull/69705#issuecomment-5210130314
hey about "enough to cause task failures" if it can happen then it will happen again ( backfill , new dags , big clear ... ) your tasks fail because you did not put correct/perfect limit on concurrency. I know it's not easy : yes airflow pools are too simple for many use-case yes airflow is not kubernetes ressource aware ( he do not know your max hardware scaling limits and don't queue tasks regarding this ) --- "but never spread the fire times apart at the source." I've a stack triggering more than a thousand of dag_run at midnight , yes it take almost 30 seconds to the scheduler to do so , but no errors , did you encounter a scheduler error ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
