jason810496 opened a new pull request, #74230: URL: https://github.com/apache/airflow/pull/74230
- depends on: #71189 - Diff for early review: https://github.com/jason810496/airflow/compare/feature/java-sdk-native-dag...feature/java-sdk-task-group > **Merge order:** > 1. #71189 — Declare a Dag's task graph with @Builder.Deps > 2. **PRNUM — Group a native Dag's tasks with task groups** **(current one)** > 3. #71190 — Serialize native Dags to DagSerialization v3 > 4. #74096 — Pack each native Dag's source file into the bundle JAR > > Every PR targets `main` because GitHub cannot base a pull request on a branch that exists only on a fork, so these diffs are cumulative — the compare link above shows only this layer. #71188, #73594, #73595 and #73596, which this stack was built on, are merged, as are #69757 and #71057. #73796 was folded into #73595. ## Why A native Java Dag could name its tasks but not gather them, so a Dag with staging and publish phases showed as one flat list in the UI. Python has `TaskGroup` for this, and a Java Dag had no equivalent. ## Example Interface surface: ```java var staging = dag.taskGroup("staging"); var stage = staging.task("stage", Stage.class); // task "staging.stage" staging.taskGroup("checks").task("nulls", Nulls.class).after(stage); // "staging.checks.nulls" extract.before(staging); ``` Annotation surface, where a task names its group and the wiring class orders whole groups: ```java @Builder.Task(group = "staging") public void stage() { ... } @Builder.Task(id = "nulls", group = "staging.checks") public void checkNulls() { ... } @Builder.Deps static class Wiring implements EtlPipelineDeps { void depends() { extract().before(group("staging")); stage().before(checkNulls()); } } ``` ## How - Everything declared in a group carries the group's ID as a prefix, so `stage` in `staging` is the task `staging.stage`. A group ID is letters, digits, underscores or dashes, and cannot collide with a task or another group. - A group stands at either end of `before`, `after` and `Flow.of`. As an upstream it stands for its leaves, as a downstream for its roots, matching Python's `TaskGroup`. A group holding no tasks is stepped over to the tasks beyond it. - Group edges are expanded onto the task graph when the Dag is registered on a `Bundle`, so a group's edges can be drawn before its tasks are declared. Each group also records its own edges, which is what the serialized `task_group` object carries. Serialization of task groups, and the `task-group` capability flag, land in #71190, which is where a native Dag is serialized at all. `prefix_group_id = false` and the display options (tooltip, colours, display name) are left for a follow-up. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code Opus 5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
