This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new f383424d8891 chore(ci): fold hadoop-mr job into utilities-rest (#19933)
f383424d8891 is described below
commit f383424d8891a1253962fced88a3a0b08a962077
Author: voonhous <[email protected]>
AuthorDate: Thu Nov 19 10:56:51 2026 +0000
chore(ci): fold hadoop-mr job into utilities-rest (#19933)
* chore(ci): fold hadoop-mr job into utilities-rest
test-hudi-hadoop-mr-and-hudi-java-client ran 8 minutes, most of them
checkout and build, on the same JDK 11 lane as test-utilities-rest
(18 minutes). Its test step moves into that job, whose build now also
covers hudi-hadoop-mr and hudi-client/hudi-java-client. One job and
one build fewer per run; the merged job stays well under the run's
longest jobs at about 42 minutes.
The merged job uploads one report under both of its flags,
utilities,hadoop-mr-java-client, so hadoop-mr-java-client keeps
getting fresh reports instead of being carried forward. The fork
coverage bridge accepted only one flag per artifact; it now takes a
comma-separated list whose flags are all known.
* chore(ci): keep hadoop-mr tests and coverage flags apart
Run the hudi-hadoop-mr and hudi-java-client unit tests whenever the
build succeeded, so a hudi-utilities test failure no longer skips
them; they had a job of their own before the fold.
Upload one coverage report per flag: utilities from the hudi-utilities
exec files, hadoop-mr-java-client from the hudi-hadoop-mr and
hudi-java-client ones, so each flag keeps the coverage of its own
tests. generate_merged_coverage_report.sh and
merge_jacoco_exec_files.sh take optional exec directories for this.
The fork coverage bridge no longer needs comma-separated flags, so
that change is reverted.
* chore(ci): scope the hadoop-mr coverage report
The hadoop-mr-java-client report took its classes and sources from the
whole combined workspace, so hudi-utilities' classes showed up as
uncovered under that flag. The report step now lists the modules of
the reactor that job built before the fold (hudi-hadoop-mr,
hudi-java-client and their upstream modules) and passes them to
generate_merged_coverage_report.sh through COVERAGE_MODULE_DIRS. The
utilities reactor is the whole combined workspace, so its report
keeps the default.
generate_merged_coverage_report.sh recreates the aggregate class and
source directories on every call. merge_jacoco_exec_files.sh fails
when it finds no exec files, instead of writing an empty merge, and
fails after its last retry instead of exiting 0.
* chore(ci): drop the utilities report before the next
If listing the hadoop-mr reactor failed, the report step stopped
before generate_merged_coverage_report.sh removed the utilities
report, and the always() upload that follows published it under
hadoop-mr-java-client. The step now removes jacoco-report.xml before
the Maven command, so a failed step leaves no report to upload.
---
.github/workflows/README.md | 2 +-
.github/workflows/bot.yml | 98 ++++++++++-------------
scripts/jacoco/generate_merged_coverage_report.sh | 49 ++++++++++--
scripts/jacoco/merge_jacoco_exec_files.sh | 20 ++++-
4 files changed, 99 insertions(+), 70 deletions(-)
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 0508b8a7c281..b825123b2151 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -3,7 +3,7 @@
When updating the pr template, you must consider if updates need to be made to
scripts/pr_compliance.py
## What are the files in workflows?
-- bot.yml: Java CI. Spark lanes (JDK 11 on Spark 3.5, JDK 17 on Spark 4.2),
Flink, bundle validation and integration tests. Runs on PRs, merge-queue groups
and branch pushes; only push runs are path-filtered. Forks run it on every
branch push; upstream it skips every job on pushes to branches other than
master, release-* and branch-0.x, and every job but the aggregate on PRs from
forks to master (see [Pull requests from forks](#pull-requests-from-forks)).
Every test job depends only on t [...]
+- bot.yml: Java CI. Spark lanes (JDK 11 on Spark 3.5, JDK 17 on Spark 4.2),
Flink, bundle validation and integration tests. Runs on PRs, merge-queue groups
and branch pushes; only push runs are path-filtered. Forks run it on every
branch push; upstream it skips every job on pushes to branches other than
master, release-* and branch-0.x, and every job but the aggregate on PRs from
forks to master (see [Pull requests from forks](#pull-requests-from-forks)).
Every test job depends only on t [...]
- fork_ci_status.yml: Fork CI status. Mirrors the fork's bot.yml run for the
head commit of each open fork PR to master onto the PR as the `CI (fork)`
check, which also asks for action when the head commit has no fork run although
the PR changes more than docs and images. Runs when an upstream Java CI run
completes and every 15 minutes for late runs and reruns. It never checks out PR
code. Its tests, .github/ci/scripts/test-fork-ci-status.cjs, also cover the
routing and concurrency of bot.yml
- fork_coverage_bridge.yml: Fork coverage bridge. Every 15 minutes, uploads to
Codecov the JaCoCo reports that the fork's bot.yml runs stored as artifacts
(through .github/actions/upload-coverage) for the head commit of each recently
updated fork PR to master, and records each uploaded commit as a `Coverage
(fork)` check. It never checks out or runs PR code; its tests are
.github/ci/scripts/test-fork-coverage-bridge.cjs
- validate_source.yml: PR and Source Validation. One job,
`validate-pr-and-source`, one of the two required status checks with `Java CI
result`: it checks the PR description with scripts/pr_compliance.py, runs that
script's own tests and the fork_ci_status.yml and fork_coverage_bridge.yml
tests, validates the PR title, rejects commits with a Claude co-author trailer,
and runs the source release checks (RAT, binary files, copyright). Never
path-filtered, because a required check must repo [...]
diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml
index 36ab7f8ea13c..c8867e553e97 100644
--- a/.github/workflows/bot.yml
+++ b/.github/workflows/bot.yml
@@ -283,7 +283,7 @@ jobs:
flags: utilities
token: ${{ secrets.CODECOV_TOKEN }}
- test-utilities-rest:
+ test-utilities-rest-hadoop-mr-java-client:
if: *route
needs: changes
runs-on: ubuntu-24.04
@@ -305,12 +305,13 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
+ id: build
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
FLINK_PROFILE: ${{ matrix.flinkProfile }}
run:
- mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service -Pthrift-gen-source
$MVN_ARGS -am -pl hudi-utilities
+ mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service -Pthrift-gen-source
$MVN_ARGS -am -pl hudi-utilities,hudi-hadoop-mr,hudi-client/hudi-java-client
- name: UT - hudi-utilities (non-TestHoodieDeltaStreamer)
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
@@ -325,18 +326,50 @@ jobs:
FLINK_PROFILE: ${{ matrix.flinkProfile }}
run:
mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -fae -Dsurefire.failIfNoSpecifiedTests=false -pl
hudi-utilities $MVN_ARGS -Djacoco.skip=false
+ # These modules had a job of their own, so a hudi-utilities test failure
must not skip them.
+ - name: UT - hudi-hadoop-mr and hudi-client/hudi-java-client
+ if: ${{ !cancelled() && steps.build.outcome == 'success' }}
+ env:
+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ run:
+ mvn test -Punit-tests -fae -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-hadoop-mr,hudi-client/hudi-java-client $MVN_ARGS
-Djacoco.skip=false
- name: Test timing summary
if: always()
run: python3 scripts/ci/test_timing_summary.py --title "${{ github.job
}}"
- - name: Generate merged coverage report
+ # One report per flag, each from its own modules' exec files, so each
flag keeps the coverage
+ # of its own tests. Each report is uploaded before the next call
replaces it.
+ - name: Generate hudi-utilities coverage report
if: always() && github.event_name != 'merge_group'
- run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- - name: Upload coverage to Codecov
+ run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE $GITHUB_WORKSPACE/hudi-utilities
+ - name: Upload hudi-utilities coverage to Codecov
if: always() && github.event_name != 'merge_group'
uses: ./.github/actions/upload-coverage
with:
flags: utilities
token: ${{ secrets.CODECOV_TOKEN }}
+ # Classes and sources come from the reactor this flag's job built on its
own (hudi-hadoop-mr,
+ # hudi-java-client and their upstream modules), so hudi-utilities'
classes stay out of its report.
+ # exec-maven-plugin is pinned to the version the repo's poms already use.
+ - name: Generate hudi-hadoop-mr and hudi-java-client coverage report
+ if: always() && github.event_name != 'merge_group'
+ env:
+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ run: |
+ # Remove the utilities report first: if this step fails before the
generator runs, the
+ # always() upload below must not publish it under this flag.
+ rm -f jacoco-report.xml
+ module_dirs=$(mvn -q -B -ntp -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -Phudi-platform-service -am -pl
hudi-hadoop-mr,hudi-client/hudi-java-client
org.codehaus.mojo:exec-maven-plugin:1.6.0:exec -Dexec.executable=pwd)
+ COVERAGE_MODULE_DIRS="$module_dirs"
./scripts/jacoco/generate_merged_coverage_report.sh $GITHUB_WORKSPACE
$GITHUB_WORKSPACE/hudi-hadoop-mr $GITHUB_WORKSPACE/hudi-client/hudi-java-client
+ - name: Upload hudi-hadoop-mr and hudi-java-client coverage to Codecov
+ if: always() && github.event_name != 'merge_group'
+ uses: ./.github/actions/upload-coverage
+ with:
+ flags: hadoop-mr-java-client
+ token: ${{ secrets.CODECOV_TOKEN }}
test-common-and-other-modules:
if: *route
@@ -746,54 +779,6 @@ jobs:
# flags: spark-scala-tests
# token: ${{ secrets.CODECOV_TOKEN }}
- test-hudi-hadoop-mr-and-hudi-java-client:
- if: *route
- needs: changes
- runs-on: ubuntu-24.04
- timeout-minutes: 40
- strategy:
- matrix:
- include:
- - scalaProfile: "scala-2.12"
- sparkProfile: "spark3.5"
- flinkProfile: "flink2.2"
-
- steps:
- - uses: actions/checkout@v5
- - name: Set up JDK 11
- uses: actions/setup-java@v5
- with:
- java-version: '11'
- distribution: 'temurin'
- architecture: x64
- cache: maven
- - name: Build Project
- env:
- SCALA_PROFILE: ${{ matrix.scalaProfile }}
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- run:
- mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service $MVN_ARGS -am -pl
hudi-hadoop-mr,hudi-client/hudi-java-client
- - name: UT - hudi-hadoop-mr and hudi-client/hudi-java-client
- env:
- SCALA_PROFILE: ${{ matrix.scalaProfile }}
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- run:
- mvn test -Punit-tests -fae -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-hadoop-mr,hudi-client/hudi-java-client $MVN_ARGS
-Djacoco.skip=false
- - name: Test timing summary
- if: always()
- run: python3 scripts/ci/test_timing_summary.py --title "${{ github.job
}}"
- - name: Generate merged coverage report
- if: always() && github.event_name != 'merge_group'
- run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- - name: Upload coverage to Codecov
- if: always() && github.event_name != 'merge_group'
- uses: ./.github/actions/upload-coverage
- with:
- flags: hadoop-mr-java-client
- token: ${{ secrets.CODECOV_TOKEN }}
-
# How the Spark JDK 17 jobs below are sharded. The minutes come from master
run
# 34314808833 on 2026-09-09, before the two-fork surefire change earlier on
this branch,
# so treat them as relative weights rather than the wall time a job takes
today.
@@ -2093,8 +2078,8 @@ jobs:
# cancelled one. Adding or resharding a job means adding its id to `needs`
below and
# `needs: changes` and `if: *route` to the job, and nothing else. The ASF
GitHub Actions policy
# caps the jobs running at once in a workflow at 20 (15 recommended),
counted across all
- # matrices. The test jobs total 23 instances; `max-parallel` on
validate-bundles and
- # validate-bundle-spark4 holds back 3, so the test jobs can currently run 20
at once.
+ # matrices. The test jobs total 22 instances; `max-parallel` on
validate-bundles and
+ # validate-bundle-spark4 holds back 3, so the test jobs can currently run 19
at once.
# test-fork-ci-status.cjs checks that number, counting `max-parallel`, and
that no test job
# waits on another. GitHub Actions has no workflow-level job cap, so a
change that pushes the
# number past 20 must lower `max-parallel` or serialize jobs through
`needs`, and make that check
@@ -2115,9 +2100,8 @@ jobs:
needs:
- test-spark-client-and-hadoop-common
- test-utilities-streamer
- - test-utilities-rest
+ - test-utilities-rest-hadoop-mr-java-client
- test-common-and-other-modules
- - test-hudi-hadoop-mr-and-hudi-java-client
- test-spark-java17-ut-functional
- test-spark-java17-ut-other
- test-spark-java17-ut-fat-and-ftc
diff --git a/scripts/jacoco/generate_merged_coverage_report.sh
b/scripts/jacoco/generate_merged_coverage_report.sh
index a2b8b67276ff..882a2f704653 100755
--- a/scripts/jacoco/generate_merged_coverage_report.sh
+++ b/scripts/jacoco/generate_merged_coverage_report.sh
@@ -17,30 +17,63 @@
# specific language governing permissions and limitations
# under the License.
#
-# Generates a merged JaCoCo coverage report from all exec files in the
workspace.
+# Generates a merged JaCoCo coverage report from the exec files in the
workspace.
# Produces jacoco-report.xml suitable for upload to Codecov.
#
-# Usage: ./scripts/jacoco/generate_merged_coverage_report.sh [workspace_dir]
+# Usage: ./scripts/jacoco/generate_merged_coverage_report.sh [workspace_dir
[exec_dir ...]]
# workspace_dir: root of the source tree (defaults to current directory)
+# exec_dir: only the exec files under these directories go into the report
(defaults to
+# workspace_dir). A job that uploads one report per Codecov flag calls
this once per flag,
+# uploading each report before the next.
+# Environment:
+# COVERAGE_MODULE_DIRS: optional newline-separated module directories. When
set, only these
+# modules' classes and sources go into the report; otherwise every module
in the workspace does.
set -euo pipefail
WORKSPACE_DIR="${1:-.}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+EXEC_DIRS=("${@:2}")
+if [ ${#EXEC_DIRS[@]} -eq 0 ]; then
+ EXEC_DIRS=("$WORKSPACE_DIR")
+fi
+
+# A report left by an earlier call must not be uploaded again under another
flag.
+rm -f "$WORKSPACE_DIR/jacoco-report.xml"
# Skip if no exec files exist (e.g., tests were skipped for this matrix entry)
-if ! find "$WORKSPACE_DIR" -path '*/jacoco-agent/*.exec' -type f | grep -q .;
then
+if ! find "${EXEC_DIRS[@]}" -path '*/jacoco-agent/*.exec' -type f | grep -q .;
then
echo "No JaCoCo exec files found — skipping coverage report"
exit 0
fi
-"$SCRIPT_DIR/download_jacoco.sh"
-"$SCRIPT_DIR/merge_jacoco_exec_files.sh" jacoco-lib/lib/jacococli.jar
"$WORKSPACE_DIR"
+# An earlier call already downloaded the CLI, and unzip would stop to ask
before overwriting it.
+if [ ! -f jacoco-lib/lib/jacococli.jar ]; then
+ "$SCRIPT_DIR/download_jacoco.sh"
+fi
+"$SCRIPT_DIR/merge_jacoco_exec_files.sh" jacoco-lib/lib/jacococli.jar
"${EXEC_DIRS[@]}"
mv merged-jacoco.exec jacoco.exec
+# Recreated on every call, so a report never picks up classes or sources
copied for an earlier one.
+rm -rf aggregate-classes aggregate-sources
mkdir -p aggregate-classes aggregate-sources
-find "$WORKSPACE_DIR" -path '*/target/classes' -type d | while read d; do cp
-r "$d"/. aggregate-classes/ 2>/dev/null; done
-find "$WORKSPACE_DIR" -path '*/src/main/java' -type d | while read d; do cp -r
"$d"/. aggregate-sources/ 2>/dev/null; done
-find "$WORKSPACE_DIR" -path '*/src/main/scala' -type d | while read d; do cp
-r "$d"/. aggregate-sources/ 2>/dev/null; done
+if [ -n "${COVERAGE_MODULE_DIRS:-}" ]; then
+ while read -r d; do
+ # Skips pom-only modules, which have no classes, and any line that is not
a module directory.
+ if [ ! -d "$d/target/classes" ]; then
+ continue
+ fi
+ cp -r "$d/target/classes"/. aggregate-classes/
+ for src in "$d/src/main/java" "$d/src/main/scala"; do
+ if [ -d "$src" ]; then
+ cp -r "$src"/. aggregate-sources/
+ fi
+ done
+ done <<< "$COVERAGE_MODULE_DIRS"
+else
+ find "$WORKSPACE_DIR" -path '*/target/classes' -type d | while read d; do cp
-r "$d"/. aggregate-classes/ 2>/dev/null; done
+ find "$WORKSPACE_DIR" -path '*/src/main/java' -type d | while read d; do cp
-r "$d"/. aggregate-sources/ 2>/dev/null; done
+ find "$WORKSPACE_DIR" -path '*/src/main/scala' -type d | while read d; do cp
-r "$d"/. aggregate-sources/ 2>/dev/null; done
+fi
"$SCRIPT_DIR/generate_jacoco_coverage_report.sh" jacoco-lib/lib/jacococli.jar
"$WORKSPACE_DIR"
diff --git a/scripts/jacoco/merge_jacoco_exec_files.sh
b/scripts/jacoco/merge_jacoco_exec_files.sh
index 4f6b4048b935..a3f7ec086fa6 100755
--- a/scripts/jacoco/merge_jacoco_exec_files.sh
+++ b/scripts/jacoco/merge_jacoco_exec_files.sh
@@ -18,13 +18,23 @@
# under the License.
#
/bin/bash --version
-shopt -s globstar
+# nullglob drops a directory with no exec files instead of passing its pattern
to merge.
+shopt -s globstar nullglob
echo "Jacoco CLI jar: $1"
-echo "Hudi source directory: $2"
+echo "Hudi source directories: ${*:2}"
+exec_files=()
+for dir in "${@:2}"; do
+ exec_files+=($dir/**/jacoco-agent/**/*.exec)
+done
+# Without this check, merge with no input files succeeds and writes an empty
exec file.
+if [[ ${#exec_files[@]} -eq 0 ]]; then
+ echo "No JaCoCo exec files found under: ${*:2}" >&2
+ exit 1
+fi
retry_count=0
while [[ $retry_count -lt 3 ]]; do
- ls -l $2/**/jacoco-agent/**/*.exec
- java -jar $1 merge $2/**/jacoco-agent/**/*.exec --destfile merged-jacoco.exec
+ ls -l "${exec_files[@]}"
+ java -jar $1 merge "${exec_files[@]}" --destfile merged-jacoco.exec
exit_status=$?
if [[ $exit_status -eq 0 ]]; then
@@ -35,3 +45,5 @@ while [[ $retry_count -lt 3 ]]; do
retry_count=$((retry_count + 1))
sleep 10
done
+echo "Jacoco merge failed after $retry_count attempts" >&2
+exit 1