jason810496 opened a new pull request, #65959: URL: https://github.com/apache/airflow/pull/65959
# Add CI, E2E Tests, and Pre-commit Hooks for Java SDK > **Merge order:** This is PR **3 of 3** — merge after `task-sdk/feature/coordinator-interface`. > 1. `feature/java-sdk` — Java SDK > 2. `task-sdk/feature/coordinator-interface` — Language Coordinator Layer + Java Provider > 3. **`ci/java-hardening`** (this PR) — CI, E2E Tests, and Pre-commit Hooks > > **Try it out:** A combined PoC branch with all changes cherry-picked is available at [`astronomer:airflow:feature/java-all`](https://github.com/apache/airflow/compare/main...astronomer:airflow:feature/java-all) for reviewers who want to test the full integration end-to-end. ## Why The Java SDK and Language Coordinator Layer (PRs 1 and 2) introduce cross-language serialization and a new execution path. Without CI coverage, regressions in the serialization contract or the end-to-end execution flow would go undetected. ## What ### E2E Tests (`airflow-e2e-tests/`) - Add `java-sdk.yml` Docker Compose overlay that provisions the Java SDK, builds the example JAR bundle, and configures Airflow with the Java provider - Add `test_java_sdk.py` with end-to-end tests for both integration patterns: - **Python stub DAG** (`stub_dag.py`): `python_task_1` → `extract` (Java) → `transform` (Java) → `python_task_2` — validates Python-Java XCom interop, `getConnection`, `getVariable`, and bidirectional data flow - **Pure Java DAG** (`JavaExample.java`): `extract` → `transform` → `load` — validates Java-only DAG parsing, task execution, and expected failure handling (`load` throws `RuntimeException`) - Extend `conftest.py` and `constants.py` with Java SDK fixtures, DAG IDs, and test utilities ### Pre-commit Hook (`scripts/ci/prek/`) - Add `check_java_serialization_compatibility.py` pre-commit hook that: 1. Runs the Java SDK `SerializationCompatibilityTest` via Gradle to produce `serialized_java.json` 2. Runs `serialize_python.py` to produce `serialized_python.json` 3. Runs `compare.py` to deep-diff the two outputs 4. Fails the commit if any DAG serializes differently across the two SDKs - Register the hook in `.pre-commit-config.yaml` ### Breeze Integration (`dev/breeze/`, `scripts/`) - Add `--sdk java` flag to `breeze start-airflow` for one-command Java development setup - Add `java_sdk_build.sh` — builds the Java SDK and example JAR from source inside the Breeze container - Add `java_sdk_setup.sh` — installs JDK, builds JARs, and configures DAG bundle paths - Cache Gradle dependencies in Breeze to speed up repeated builds - Extend `entrypoint_ci.sh` with Java SDK environment initialization --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code Ops 4.6 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) Co-authored-by: Tzu-ping Chung <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
