MatthewStrickland commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1913515530
> Just for curiosity, did you try to create a pool for this scenario? I have an issue based when i consume an API and hit the rate limit, pools are good to manage this. Add the pool to default argument and you will have the same effect that you described. https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/pools.html @mariotaddeucci I hadn't looked at pools before but just took a look now. If I specify the pool at the DAG level (inside `default_args` I guess) will it block all other dags using the same pool from starting their execution until the first dag completes? That would suit me if so. However if it applies only at the task level would dags potentially compete for the slot? In which case it might not be suitable because my ETL workload runs across several tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
