The GitHub Actions job "Required Checks" on texera.git/main has succeeded. Run started by GitHub user github-merge-queue[bot] (triggered by github-merge-queue[bot]).
Head commit for run: 1094a656953df71244ac1060c8082568e7bdcc58 / Yicong Huang <[email protected]> chore(deps): upgrade Apache Hadoop to 3.4.3 to resolve hadoop-common CVEs (#6201) ### What changes were proposed in this PR? Upgrade `org.apache.hadoop:hadoop-common` to 3.4.3 in both places it is pinned (`common/workflow-core/build.sbt` was at 3.3.1, `amber/build.sbt` at 3.3.3), and remove `hadoop-mapreduce-client-core` from workflow-core. This resolves 4 open Dependabot alerts on `hadoop-common`: - [#656](https://github.com/apache/texera/security/dependabot/656) CVE-2022-25168 (critical, argument injection in `FileUtil.unTar`, fixed in 3.3.3) - [#732](https://github.com/apache/texera/security/dependabot/732) CVE-2022-26612 (critical, fixed in 3.3.3) - [#654](https://github.com/apache/texera/security/dependabot/654) CVE-2021-37404 (critical, heap overflow in libhdfs, fixed in 3.3.2) - [#700](https://github.com/apache/texera/security/dependabot/700) CVE-2024-23454 (low, fixed in 3.4.0) The hadoop dependency itself cannot be removed: Iceberg's `HadoopCatalog`/`HadoopFileIO` (used by `IcebergUtil` in workflow-core) and `HDFSRecordStorage` (amber) require hadoop-common at runtime. `hadoop-mapreduce-client-core` has no compile-time references in the codebase, but iceberg-parquet's read path loads parquet-hadoop classes that extend `org.apache.hadoop.mapreduce` types (amber e2e tests fail with `NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/input/FileInputFormat` without it), so it is upgraded to 3.4.3 alongside hadoop-common rather than dropped. hadoop 3.4 gives it a direct `netty-all` dependency, which is excluded — the netty artifacts workflow-core needs are already declared explicitly, and this keeps the netty fat-jar set out of every dist. Side effects visible in the binary manifests (all previously bundled jars, now newer): the hadoop 3.3 baggage leaves the dists (log4j 1.2.17, netty 3.10.6, htrace, kerby 1.x server jars, okhttp 2.7.5), and the per-service `LICENSE-binary`(-java) / `NOTICE-binary` files are refreshed from freshly built dists using the CI scripts. ### Any related issues, documentation, discussions? Resolves #6200. Dependabot alert https://github.com/apache/texera/security/dependabot/656 (also 732, 654, 700). ### How was this PR tested? Binary licensing manifests were refreshed the same way CI checks them: built the four affected dists locally (`sbt <svc>/dist`), updated each service's `LICENSE-binary` from `check_binary_deps.py` findings, and regenerated `NOTICE-binary` with `generate_notice_binary.py` (amber with `--extras amber/NOTICE-binary-python`). `check_binary_deps.py` now passes locally for all three platform services in both PR mode and strict (nightly) mode. Tested with existing test cases: - `sbt WorkflowCore/test`: 458 tests passed, 0 failed. (5 suites — S3/LakeFS/LargeBinary specs — abort because they need external MinIO/LakeFS services; verified they abort identically on unmodified `main`, unrelated to this change.) - `sbt WorkflowExecutionService/Test/compile` and `WorkflowExecutionService/testOnly ...SequentialRecordStorageSpec` (covers the `HDFSRecordStorage` code path): 9 tests passed. - `IcebergUtilSpec` and `DocumentFactorySpec` (Iceberg + hadoop `Configuration` usage) pass, confirming nothing needed `hadoop-mapreduce-client-core`. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5) --------- Co-authored-by: Claude Fable 5 <[email protected]> Report URL: https://github.com/apache/texera/actions/runs/28762881533 With regards, GitHub Actions via GitBox
