dheerajturaga commented on issue #73678: URL: https://github.com/apache/airflow/issues/73678#issuecomment-6042923094
Thanks @rloredo. Many people use TaskGroups as folders, and you're not the only one affected. The same came up on #73087 (one group per database schema) and #73724 (dbt models in a group with a Spark task between two of them). Here's what the dev list agreed ([discussion](https://lists.apache.org/thread/sossl7b2w2ftyk4028qrhps2tcdxj2px), [result](https://lists.apache.org/thread/5fpwlxg4y6jhyrw7n2m3jz3o7t6xvhlk)): - **3.4.0:** Grid and Graph render these Dags again (#73724). Parsing them shows a Dag warning in the UI and issues a `TaskGroupCycleDeprecationWarning` (#73746). They keep parsing and running. - **3.5.0 (planned):** these Dags fail to parse (#73087). Until 3.4.0, only Grid and Graph are affected. Tasks keep running and the REST API still works. The reason for the change is that planned TaskGroup features, such as loops, dynamic TaskGroups, retrying a group and waiting for a group to finish, need each group to come either before or after its neighbours. A cycle makes that impossible. The rule: a dependency into or out of any task in a group counts as a dependency of the whole group. So a path that leaves a group and comes back into it (`g.a >> x >> g.b`) is a cycle. **How to restructure.** One of these usually works without changing any task dependencies: 1. Move the task between the two group tasks into the group, or move the downstream task out of it. 2. Split the group along the direction of the data flow. If you use one group per schema and have dependencies across schemas, group by stage first and by schema second: ```python with TaskGroup("load"): with TaskGroup("sales"): load_sales = EmptyOperator(task_id="load") with TaskGroup("customers"): load_customers = EmptyOperator(task_id="load") with TaskGroup("report"): with TaskGroup("sales"): report_sales = EmptyOperator(task_id="report") with TaskGroup("customers"): report_customers = EmptyOperator(task_id="report") [load_sales, load_customers] >> report_sales [load_sales, load_customers] >> report_customers ``` This changes task IDs (for example, `sales.load` becomes `load.sales.load`). Check anything that refers to tasks by ID, such as sensors or `xcom_pull` calls. Once 3.4.0 is out, you can find affected Dags in CI by running your Dag tests with `-W error::airflow.sdk.exceptions.TaskGroupCycleDeprecationWarning`. **A folder-only alternative.** That's a fair ask, and Airflow doesn't have one today. Rejection in 3.5 is the plan. The dev list discussion also said we should check how common this pattern is and reconsider if it would break many Dags. Options include letting a Dag opt out of group-level features, or a grouping that's only for display. The 3.4 deprecation period is the time to collect that information. If you're affected, please reply here with: - an example of your Dag's shape, with any sensitive names removed (or which of the two shapes in the issue description it matches), - roughly how many of your Dags are affected, - whether the restructuring above works for you, and if it doesn't, why not. If enough users can't restructure, I'll raise it on the dev list again before 3.5. --- Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
