jason810496 opened a new pull request, #74036: URL: https://github.com/apache/airflow/pull/74036
Replaces #73843, which GitHub closed as merged into a stack branch when the stack was reordered. Stack (bottom to top): #73841, #74004, #73842, **#73843**, #73844, #73845, #73846, #73847 - related: #71929 - Part of the native Dag e2e stack. Builds on the native Dag parse layer ([new commits only](https://github.com/apache/airflow/compare/jason/lang-sdk-e2e/03-native-dag-parse...jason/lang-sdk-e2e/04-java-dag-importer)). - Merge after #71190, which makes the Java runtime answer the Dag-parsing request. ## Why With this PR, `JavaCoordinator` parses the native Dags a bundle JAR declares, and the Code view shows their Java source. ## What changes - **Which JARs.** A `JavaCoordinator` without `jars_root` parses each `.jar` whose manifest sets `Main-Class` (matching `main_class` if set) in the Dag bundles it reads, skipping thin-bundle dependency JARs. An unreadable JAR is kept so its parse reports the error. - **Parse command.** The task command, `java -classpath <bundle JARs> <jvm_args> <Main-Class>`, with the same schema version lookup. A JAR it cannot run gets an import error. - **Manifest reader.** Tasks and parses now read manifests per the JAR specification (CR, LF or CRLF, folded lines, main section only). The old `email` parser broke a folded `Main-Class`. - **Dag source.** The Gradle plugin packs the main class's `.java` file, or `airflowBundle.dagSource`, into every bundle JAR, fat or thin, and names it in the manifest. `get_source_code` returns it, up to 1 MiB, or a placeholder. A JAR embeds one source for all its Dags, so the Java importer ignores the Dag id. The Java runtime is unchanged here. The Java SDK layer above makes it answer the parse request, with a payload that passes `validate_serialized_dag`. ```ini [sdk] coordinators = { "java-native": { "classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"java_executable": "/usr/lib/jvm/java-17-openjdk/bin/java"} } } queue_to_coordinator = {"java-native": "java-native"} ``` ```groovy airflowBundle { mainClass = "com.example.Main" dagSource = file("src/main/java/com/example/MyDag.java") // optional } ``` Example bundle manifest (`./gradlew bundle` in `java-sdk/example`): ```text Manifest-Version: 1.0 Airflow-Java-SDK-Dag-Code: META-INF/airflow/dag-code/org/apache/airflow/ example/ExampleBundleBuilder.java Airflow-Supervisor-Schema-Version: 2026-10-30 Main-Class: org.apache.airflow.example.ExampleBundleBuilder Multi-Release: true ``` ## Deviations and limitations - #71929 leaves `get_source_code` open for native Dags. This returns the source embedded as the pure Java Dags ADR (#70156) proposes, but under `META-INF/airflow/dag-code/` instead of the JAR root to avoid fat JAR collisions, and without its `airflow-metadata.yaml`, since the parse runs the JVM. - With several executable JARs in a bundle and no `main_class`, each is parsed, but a task runs the first one in walk order (#71134). - The Java SDK (#71190) still writes the stock values of `max_active_tasks`, `max_active_runs`, `max_consecutive_failed_dag_runs`, `catchup` and `disable_bundle_versioning` when a Dag leaves them unset, so a native Java Dag does not get the Airflow config values that #73842 fills in. - Documented: tasks need `queue` set, no Dag in both Java and Python in one bundle, no cluster policies, `airflow dags reserialize` does not store a JAR's Dags, and `airflow dags test`, `tasks test`, `tasks render` and `tasks list` refuse a native Java Dag. ## Verification - `task-sdk/tests/task_sdk/coordinators/java`: 88 passed. Java SDK `./gradlew test` at this layer passed. - The Maven snippet added to the docs was not built. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code Opus 5.5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
