potiuk commented on issue #35803: URL: https://github.com/apache/airflow/issues/35803#issuecomment-1826298106
Just to set expectations here. This is something that is going to be super complex and I am not sure if we are ever going to do that. Scheduler performs selection on which tasks should be scheduled as select query on the database where it checks how many tasks are there in the pool. Dynamically seeting the pool size while tasks are running might have a lot of undesireable side effects, because the execution of tasks and scheduling are run completely independently from each other. There might be many side effects of changing pool side on-the-flight while tasks are already being scheduled for it - tasks not scheduled where they should, over-subscribing pools, deadlocks etc. etc. As @uranusjr explained in https://github.com/apache/airflow/issues/33657 - while theorethically possible, it could only be done if someone would heavily change the pool mechanism to allow such dynamic behaviour (basically it means that whether to choose task as candidate for scheduling the selection cannot be based on **rather static** calculation of pool capacity there. So, if you are counting on it as something to be implemented - this is unlikely to happen any time soon, unless someone will take on the task of re-implementing pool mechanism. Of course if you would like to be that someone @Digoya - feel free on doing it, but if you have no skills/capacity/experience/will/understanding of Airflow, this one will have to wait for **somone** who has it all and picks it up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
