ayman-albaz commented on issue #28372: URL: https://github.com/apache/airflow/issues/28372#issuecomment-1355872345
> You can limit the number of concurrent task run by airflow thanks to airflow pool > > or concurrency settings on the dag Ya perhaps this is not necessarily a bug from airflow then but more of a feature request. What you suggested is a workaround to my problem, thanks for the suggestion. I think it would be pretty cool if Airflow scheduler was aware of resources + auto scaling capability of a cluster, and then schedule accordingly (i.e. keep running jobs, and schedule the remainder that no resources can possibly be allocated for). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
