The GitHub Actions job "Tests (ARM)" on airflow.git/main has succeeded. Run started by GitHub user potiuk (triggered by henry3260).
Head commit for run: d0b456cdf9c730049f91499cb113cc406584ab8c / Jason(Zhe-You) Liu <[email protected]> Add Scala Spark ETL example to Java-SDK e2e tests (#68939) * Add Scala Spark ETL example bundle to the Java SDK e2e tests ## Why Demonstrate and regression-test that the Java SDK can run a real Scala + Apache Spark workload, with task logs routed into Airflow via Log4j 2. ## What - Add `java-sdk/scala_spark_example`: a standalone Scala + Spark 3.5 (local mode) ETL bundle whose three tasks pass scalar results over XCom and log through Log4j 2 (`airflow-sdk-log4j2`). - Run it inside the existing `java_sdk` e2e via a second coordinator and queue (`scala-jdk` / `scala`) with its own `jars_root`, keeping the Java example bundle Spark-free. - Pin the e2e worker JRE to Java 17 and pass Spark's `--add-opens` JVM args. - Add `TestJavaSDKScalaSparkExample` asserting the tasks succeed and the XComs match the fixed dataset (5 rows, total revenue 1000). * Avoid copying jars for docker build * Verify the Scala Spark transform stage in the Java-SDK e2e test The e2e test asserted only the extract and load XComs, so the aggregation stage in the middle of the pipeline could regress without the test noticing. Assert its XCom as well, drop the unused dataset constants, and note why the transform reads the upstream count it does not reuse. * Carry Spark's full Java 17 module options in the Scala Spark e2e test The Scala Spark coordinator launched the bundle JVM with a hand-curated subset of Spark's Java 17 module openings. Spark normally injects its full default set through its own launcher, which the raw JavaCoordinator launch bypasses. The subset is enough for the toy aggregation but omits openings that real Spark code paths need (Kryo reflection, off-heap cleaner, charset decoding, Kerberos), so the example would mislead anyone copying it for a non-trivial Spark workload. Mirror Spark 3.5.8's full default module option set instead. * Standardize Scala Spark example layout and document Spark JVM options Review feedback on the Scala Spark e2e example asked to make the bundle source layout consistent -- it mixed the Maven-standard src/main/resources with flat src/scala and src/resources -- so the whole tree now follows the standard src/main/{scala,resources} convention. It also asked that the JDK module options Spark needs under the Java SDK be documented where users can find them, not only buried in the e2e test, and that comments restating the code be trimmed. Report URL: https://github.com/apache/airflow/actions/runs/28505232508 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
