jason810496 commented on code in PR #70059: URL: https://github.com/apache/airflow/pull/70059#discussion_r3612734439
########## java-sdk/adr/0006-no-lang-sdk-source-display.md: ########## @@ -0,0 +1,140 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# ADR-0006: No Lang-SDK Source Code Display for Mixed-Language (`@task.stub`) Dags + +## Status + +Proposed + +> **Note:** This ADR records a *negative* decision. It exists so that future feature requests +> or PRs proposing multi-file / multi-language source display for stub-backed Dags can be +> pointed here instead of re-litigating the design. Tracked operationally in +> [apache/airflow#67260](https://github.com/apache/airflow/issues/67260), which is closed as +> won't-fix referencing this document. + +## Context + +Per [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ), Java tasks are declared as +`@task.stub` operators in ordinary Python Dag files: the Dag is defined in Python, while the +task implementation lives in a Java-SDK build artifact (a JAR) shipped in the same bundle. We +call this a **mixed-language Dag**. This is distinct from pure Java Dags +([ADR-0003](0003-pure-java-dags.md)), where the Dag itself is defined in Java and the +coordinator extracts the packed `.java` source via the `Airflow-Java-SDK-Dag-Code` manifest +attribute for display — a mechanism that was removed from AIP-108 scope along with pure Java +Dag authoring. + +During AIP-108 testing and demos, users asked whether the UI Code view for a +mixed-language Dag could show the Java-SDK-side source *in addition to* the Python file +containing the stub declarations +([apache/airflow#67260](https://github.com/apache/airflow/issues/67260)) — for example, as a +second tab next to the Python source. + +A design sketch was drafted to make this work by extending the +[AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) `DagImporter` interface: + +1. A `StubDagImporter` subclassing the Python importer, which after parsing resolves each stub + Dag's backing artifact within the bundle and captures its source. +2. `get_source_code` becoming **per-Dag and multi-file** + (`get_source_code(file_path, dag_id, *, bundle_path, bundle_name) -> dict[str, DagSource]`), + so sibling Dags defined in one Python file do not leak each other's artifacts into the + Code view. +3. A schema change allowing **multiple `DagCode` rows per `DagVersion`** — dropping the unique + constraint on `DagCode.dag_version_id`, adding `source_file_name`, `language`, and an + entrypoint marker — with artifact digests participating in `dag_hash` so that a JAR rebuild + produces a new `DagVersion`. +4. REST API and UI changes: `GET /dagSources/{dag_id}` gaining a file dimension, and the Code + tab gaining a file tab bar with per-file language instead of the hardcoded Python. + +After discussion, we decided not to pursue this. + +## Decision + +**We will not support viewing the Lang-SDK-side source code for mixed-language Dags.** For a +Python Dag containing `@task.stub` tasks backed by a Lang-SDK artifact, the Code view shows +only the Python Dag file — the existing single-source, per-`DagVersion` behavior. + +Concretely, none of the following is implemented: + +- A `StubDagImporter` (or `CrossLangDagImporter`) that resolves and captures Lang-SDK artifact + source for stub-backed Dags. +- Per-Dag, multi-file semantics for `get_source_code`. +- Multiple `DagCode` rows per `DagVersion`, or `source_file_name` / `language` / entrypoint + columns on `DagCode`. +- A file dimension on `GET /dagSources/{dag_id}` or a multi-file Code tab in the UI for this + use case. +- Stamping artifact digests onto the serialized Dag model *for the purpose of source display*. + (Whether artifact digests should participate in Dag versioning for execution correctness is + a separate question, out of scope here.) + +### Why Not + +1. **A Dag with stub tasks is not a distinct Dag category.** A stub operator is just another + operator inside an ordinary Python Dag. A single Dag may soon mix tasks in several + languages, so there is no principled, bounded set of "the other side's" files to display. + Designing importer interfaces around a special stub-Dag concept bakes an artificial + distinction into AIP-85. +2. **It degenerates into storing a directory tree in `dag_code`.** Showing anything genuinely + useful for the Java side means more than one file — Java code in particular tends to split + each task into its own small file/class. Followed to its conclusion, this means persisting + an entire directory tree in the `dag_code` table, which is not what that table is for. +3. **The source view is already best-effort in pure Python.** Dags produced by factory Review Comment: > so the Code view still shows real Python from the bundle that users can navigate from. The detail page also tells the bundle and the `fileloc` as well. https://airflow.apache.org/docs/apache-airflow/stable/ui.html#details-tab > With @task.stub the body is just def extract(): ... - there is no reference to the artifact/class anywhere However, even we support this feature, it only shows the entypoint file of Lang-SDK. For Java-SDK, it only show something like (IMO, showing the entrypoint factory doesn't help much. If I'm user, I'd like to see each Dag definition, but it's still impossible with the current design mentioned in the ADR that we consider. ) ```java package org.apache.airflow.example; import java.util.List; import org.apache.airflow.sdk.*; import org.jetbrains.annotations.NotNull; public class ExampleBundleBuilder implements BundleBuilder { @NotNull @Override public Iterable<Dag> getDags() { return List.of( InterfaceExampleBuilder.build(), AnnotationExampleBuilder.build(), XComCastingExampleBuilder.build()); } public static void main(String[] args) { var bundle = new ExampleBundleBuilder().build(); Server.create(args).serve(bundle); } } ``` If we're going to show all the `InterfaceExampleBuilder`, `AnnotationExampleBuilder` ..., we will list out the directory tree for each the Lang-SDK involved in the Dag (don't forget it's valid to have both Python, Go, Java, Ts tasks in the same Dag). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
