Re: [I] Dag grouping [airflow]

2024-02-20 Thread via GitHub
github-actions[bot] commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1955456558 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Dag grouping [airflow]

2024-02-20 Thread via GitHub
github-actions[bot] closed issue #37042: Dag grouping URL: https://github.com/apache/airflow/issues/37042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [I] Dag grouping [airflow]

2024-02-12 Thread via GitHub
github-actions[bot] commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1939829382 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [I] Dag grouping [airflow]

2024-01-29 Thread via GitHub
potiuk commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1914536959 Just as a good example and showing how it can be done - here is a great article (published today) of on of our users who contributed something they needed. A Journey from the User t

Re: [I] Dag grouping [airflow]

2024-01-29 Thread via GitHub
potiuk commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1914532515 Airlfow does not have any "per dag" enforcement. It works pretty much exclusively on "task" level. Scheduling is not based on DAGs it's based on task. Schedulers look at tasks and p

Re: [I] Dag grouping [airflow]

2024-01-29 Thread via GitHub
MatthewStrickland commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1914203949 > Could you use a [cluster DAG policy](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#)? You can add the desire

Re: [I] Dag grouping [airflow]

2024-01-28 Thread via GitHub
RNHTTR commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1913636521 Could you use a [cluster DAG policy](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#)? You can add the desired DAGs into a

Re: [I] Dag grouping [airflow]

2024-01-28 Thread via GitHub
MatthewStrickland commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1913515530 > Just for curiosity, did you try to create a pool for this scenario? I have an issue based when i consume an API and hit the rate limit, pools are good to manage this. A

Re: [I] Dag grouping [airflow]

2024-01-27 Thread via GitHub
mariotaddeucci commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1913293229 Just for curiosity, did you try to create a pool for this scenario? I have an issue based when i consume an API and hit the rate limit, pools are good to manage this. A

Re: [I] Dag grouping [airflow]

2024-01-27 Thread via GitHub
boring-cyborg[bot] commented on issue #37042: URL: https://github.com/apache/airflow/issues/37042#issuecomment-1913097414 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

[I] Dag grouping [airflow]

2024-01-27 Thread via GitHub
MatthewStrickland opened a new issue, #37042: URL: https://github.com/apache/airflow/issues/37042 ### Description Support grouping dags, where a dag can be part of many groups, with a configuration similar to max_active_runs across the whole group. (I'm sure this must have bee