This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/master/pr-19933-498e5c26b152c119ee0381b923f402407d962c31 in repository https://gitbox.apache.org/repos/asf/hudi.git
commit b690550bba3bd1d1aa398860b5bb986a6a2bc4bd Author: voonhous <[email protected]> AuthorDate: Thu Nov 19 10:56:51 2026 +0000 chore(ci): fold hadoop-mr job into utilities-rest (#19933) * chore(ci): fold hadoop-mr job into utilities-rest test-hudi-hadoop-mr-and-hudi-java-client ran 8 minutes, most of them checkout and build, on the same JDK 11 lane as test-utilities-rest (18 minutes). Its test step moves into that job, whose build now also covers hudi-hadoop-mr and hudi-client/hudi-java-client. One job and one build fewer per run; the merged job stays well under the run's longest jobs at about 42 minutes. The merged job uploads one report under both of its flags, utilities,hadoop-mr-java-client, so hadoop-mr-java-client keeps getting fresh reports instead of being carried forward. The fork coverage bridge accepted only one flag per artifact; it now takes a comma-separated list whose flags are all known. * chore(ci): keep hadoop-mr tests and coverage flags apart Run the hudi-hadoop-mr and hudi-java-client unit tests whenever the build succeeded, so a hudi-utilities test failure no longer skips them; they had a job of their own before the fold. Upload one coverage report per flag: utilities from the hudi-utilities exec files, hadoop-mr-java-client from the hudi-hadoop-mr and hudi-java-client ones, so each flag keeps the coverage of its own tests. generate_merged_coverage_report.sh and merge_jacoco_exec_files.sh take optional exec directories for this. The fork coverage bridge no longer needs comma-separated flags, so that change is reverted. * chore(ci): scope the hadoop-mr coverage report The hadoop-mr-java-client report took its classes and sources from the whole combined workspace, so hudi-utilities' classes showed up as uncovered under that flag. The report step now lists the modules of the reactor that job built before the fold (hudi-hadoop-mr, hudi-java-client and their upstream modules) and passes them to generate_merged_coverage_report.sh through COVERAGE_MODULE_DIRS. The utilities reactor is the whole combined workspace, so its report keeps the default. generate_merged_coverage_report.sh recreates the aggregate class and source directories on every call. merge_jacoco_exec_files.sh fails when it finds no exec files, instead of writing an empty merge, and fails after its last retry instead of exiting 0. * chore(ci): drop the utilities report before the next If listing the hadoop-mr reactor failed, the report step stopped before generate_merged_coverage_report.sh removed the utilities report, and the always() upload that follows published it under hadoop-mr-java-client. The step now removes jacoco-report.xml before the Maven command, so a failed step leaves no report to upload. --- .github/workflows/README.md | 2 +- .github/workflows/bot.yml | 98 ++++++++++------------- scripts/jacoco/generate_merged_coverage_report.sh | 49 ++++++++++-- scripts/jacoco/merge_jacoco_exec_files.sh | 20 ++++- 4 files changed, 99 insertions(+), 70 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 0508b8a7c281..b825123b2151 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -3,7 +3,7 @@ When updating the pr template, you must consider if updates need to be made to scripts/pr_compliance.py ## What are the files in workflows? -- bot.yml: Java CI. Spark lanes (JDK 11 on Spark 3.5, JDK 17 on Spark 4.2), Flink, bundle validation and integration tests. Runs on PRs, merge-queue groups and branch pushes; only push runs are path-filtered. Forks run it on every branch push; upstream it skips every job on pushes to branches other than master, release-* and branch-0.x, and every job but the aggregate on PRs from forks to master (see [Pull requests from forks](#pull-requests-from-forks)). Every test job depends only on t [...] +- bot.yml: Java CI. Spark lanes (JDK 11 on Spark 3.5, JDK 17 on Spark 4.2), Flink, bundle validation and integration tests. Runs on PRs, merge-queue groups and branch pushes; only push runs are path-filtered. Forks run it on every branch push; upstream it skips every job on pushes to branches other than master, release-* and branch-0.x, and every job but the aggregate on PRs from forks to master (see [Pull requests from forks](#pull-requests-from-forks)). Every test job depends only on t [...] - fork_ci_status.yml: Fork CI status. Mirrors the fork's bot.yml run for the head commit of each open fork PR to master onto the PR as the `CI (fork)` check, which also asks for action when the head commit has no fork run although the PR changes more than docs and images. Runs when an upstream Java CI run completes and every 15 minutes for late runs and reruns. It never checks out PR code. Its tests, .github/ci/scripts/test-fork-ci-status.cjs, also cover the routing and concurrency of bot.yml - fork_coverage_bridge.yml: Fork coverage bridge. Every 15 minutes, uploads to Codecov the JaCoCo reports that the fork's bot.yml runs stored as artifacts (through .github/actions/upload-coverage) for the head commit of each recently updated fork PR to master, and records each uploaded commit as a `Coverage (fork)` check. It never checks out or runs PR code; its tests are .github/ci/scripts/test-fork-coverage-bridge.cjs - validate_source.yml: PR and Source Validation. One job, `validate-pr-and-source`, one of the two required status checks with `Java CI result`: it checks the PR description with scripts/pr_compliance.py, runs that script's own tests and the fork_ci_status.yml and fork_coverage_bridge.yml tests, validates the PR title, rejects commits with a Claude co-author trailer, and runs the source release checks (RAT, binary files, copyright). Never path-filtered, because a required check must repo [...] diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml index a05bb99d3b6e..b82ca74fcfba 100644 --- a/.github/workflows/bot.yml +++ b/.github/workflows/bot.yml @@ -283,7 +283,7 @@ jobs: flags: utilities token: ${{ secrets.CODECOV_TOKEN }} - test-utilities-rest: + test-utilities-rest-hadoop-mr-java-client: if: *route needs: changes runs-on: ubuntu-24.04 @@ -305,12 +305,13 @@ jobs: architecture: x64 cache: maven - name: Build Project + id: build env: SCALA_PROFILE: ${{ matrix.scalaProfile }} SPARK_PROFILE: ${{ matrix.sparkProfile }} FLINK_PROFILE: ${{ matrix.flinkProfile }} run: - mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service -Pthrift-gen-source $MVN_ARGS -am -pl hudi-utilities + mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service -Pthrift-gen-source $MVN_ARGS -am -pl hudi-utilities,hudi-hadoop-mr,hudi-client/hudi-java-client - name: UT - hudi-utilities (non-TestHoodieDeltaStreamer) env: SCALA_PROFILE: ${{ matrix.scalaProfile }} @@ -325,18 +326,50 @@ jobs: FLINK_PROFILE: ${{ matrix.flinkProfile }} run: mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -D"$FLINK_PROFILE" -fae -Dsurefire.failIfNoSpecifiedTests=false -pl hudi-utilities $MVN_ARGS -Djacoco.skip=false + # These modules had a job of their own, so a hudi-utilities test failure must not skip them. + - name: UT - hudi-hadoop-mr and hudi-client/hudi-java-client + if: ${{ !cancelled() && steps.build.outcome == 'success' }} + env: + SCALA_PROFILE: ${{ matrix.scalaProfile }} + SPARK_PROFILE: ${{ matrix.sparkProfile }} + FLINK_PROFILE: ${{ matrix.flinkProfile }} + run: + mvn test -Punit-tests -fae -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -D"$FLINK_PROFILE" -pl hudi-hadoop-mr,hudi-client/hudi-java-client $MVN_ARGS -Djacoco.skip=false - name: Test timing summary if: always() run: python3 scripts/ci/test_timing_summary.py --title "${{ github.job }}" - - name: Generate merged coverage report + # One report per flag, each from its own modules' exec files, so each flag keeps the coverage + # of its own tests. Each report is uploaded before the next call replaces it. + - name: Generate hudi-utilities coverage report if: always() && github.event_name != 'merge_group' - run: ./scripts/jacoco/generate_merged_coverage_report.sh $GITHUB_WORKSPACE - - name: Upload coverage to Codecov + run: ./scripts/jacoco/generate_merged_coverage_report.sh $GITHUB_WORKSPACE $GITHUB_WORKSPACE/hudi-utilities + - name: Upload hudi-utilities coverage to Codecov if: always() && github.event_name != 'merge_group' uses: ./.github/actions/upload-coverage with: flags: utilities token: ${{ secrets.CODECOV_TOKEN }} + # Classes and sources come from the reactor this flag's job built on its own (hudi-hadoop-mr, + # hudi-java-client and their upstream modules), so hudi-utilities' classes stay out of its report. + # exec-maven-plugin is pinned to the version the repo's poms already use. + - name: Generate hudi-hadoop-mr and hudi-java-client coverage report + if: always() && github.event_name != 'merge_group' + env: + SCALA_PROFILE: ${{ matrix.scalaProfile }} + SPARK_PROFILE: ${{ matrix.sparkProfile }} + FLINK_PROFILE: ${{ matrix.flinkProfile }} + run: | + # Remove the utilities report first: if this step fails before the generator runs, the + # always() upload below must not publish it under this flag. + rm -f jacoco-report.xml + module_dirs=$(mvn -q -B -ntp -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -D"$FLINK_PROFILE" -Phudi-platform-service -am -pl hudi-hadoop-mr,hudi-client/hudi-java-client org.codehaus.mojo:exec-maven-plugin:1.6.0:exec -Dexec.executable=pwd) + COVERAGE_MODULE_DIRS="$module_dirs" ./scripts/jacoco/generate_merged_coverage_report.sh $GITHUB_WORKSPACE $GITHUB_WORKSPACE/hudi-hadoop-mr $GITHUB_WORKSPACE/hudi-client/hudi-java-client + - name: Upload hudi-hadoop-mr and hudi-java-client coverage to Codecov + if: always() && github.event_name != 'merge_group' + uses: ./.github/actions/upload-coverage + with: + flags: hadoop-mr-java-client + token: ${{ secrets.CODECOV_TOKEN }} test-common-and-other-modules: if: *route @@ -746,54 +779,6 @@ jobs: # flags: spark-scala-tests # token: ${{ secrets.CODECOV_TOKEN }} - test-hudi-hadoop-mr-and-hudi-java-client: - if: *route - needs: changes - runs-on: ubuntu-24.04 - timeout-minutes: 40 - strategy: - matrix: - include: - - scalaProfile: "scala-2.12" - sparkProfile: "spark3.5" - flinkProfile: "flink2.2" - - steps: - - uses: actions/checkout@v5 - - name: Set up JDK 11 - uses: actions/setup-java@v5 - with: - java-version: '11' - distribution: 'temurin' - architecture: x64 - cache: maven - - name: Build Project - env: - SCALA_PROFILE: ${{ matrix.scalaProfile }} - SPARK_PROFILE: ${{ matrix.sparkProfile }} - FLINK_PROFILE: ${{ matrix.flinkProfile }} - run: - mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service $MVN_ARGS -am -pl hudi-hadoop-mr,hudi-client/hudi-java-client - - name: UT - hudi-hadoop-mr and hudi-client/hudi-java-client - env: - SCALA_PROFILE: ${{ matrix.scalaProfile }} - SPARK_PROFILE: ${{ matrix.sparkProfile }} - FLINK_PROFILE: ${{ matrix.flinkProfile }} - run: - mvn test -Punit-tests -fae -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -D"$FLINK_PROFILE" -pl hudi-hadoop-mr,hudi-client/hudi-java-client $MVN_ARGS -Djacoco.skip=false - - name: Test timing summary - if: always() - run: python3 scripts/ci/test_timing_summary.py --title "${{ github.job }}" - - name: Generate merged coverage report - if: always() && github.event_name != 'merge_group' - run: ./scripts/jacoco/generate_merged_coverage_report.sh $GITHUB_WORKSPACE - - name: Upload coverage to Codecov - if: always() && github.event_name != 'merge_group' - uses: ./.github/actions/upload-coverage - with: - flags: hadoop-mr-java-client - token: ${{ secrets.CODECOV_TOKEN }} - # How the Spark JDK 17 jobs below are sharded. The minutes come from master run # 34314808833 on 2026-09-09, before the two-fork surefire change earlier on this branch, # so treat them as relative weights rather than the wall time a job takes today. @@ -2096,8 +2081,8 @@ jobs: # cancelled one. Adding or resharding a job means adding its id to `needs` below and # `needs: changes` and `if: *route` to the job, and nothing else. The ASF GitHub Actions policy # caps the jobs running at once in a workflow at 20 (15 recommended), counted across all - # matrices. The test jobs total 23 instances; `max-parallel` on validate-bundles and - # validate-bundle-spark4 holds back 3, so the test jobs can currently run 20 at once. + # matrices. The test jobs total 22 instances; `max-parallel` on validate-bundles and + # validate-bundle-spark4 holds back 3, so the test jobs can currently run 19 at once. # test-fork-ci-status.cjs checks that number, counting `max-parallel`, and that no test job # waits on another. GitHub Actions has no workflow-level job cap, so a change that pushes the # number past 20 must lower `max-parallel` or serialize jobs through `needs`, and make that check @@ -2118,9 +2103,8 @@ jobs: needs: - test-spark-client-and-hadoop-common - test-utilities-streamer - - test-utilities-rest + - test-utilities-rest-hadoop-mr-java-client - test-common-and-other-modules - - test-hudi-hadoop-mr-and-hudi-java-client - test-spark-java17-ut-functional - test-spark-java17-ut-other - test-spark-java17-ut-fat-and-ftc diff --git a/scripts/jacoco/generate_merged_coverage_report.sh b/scripts/jacoco/generate_merged_coverage_report.sh index a2b8b67276ff..882a2f704653 100755 --- a/scripts/jacoco/generate_merged_coverage_report.sh +++ b/scripts/jacoco/generate_merged_coverage_report.sh @@ -17,30 +17,63 @@ # specific language governing permissions and limitations # under the License. # -# Generates a merged JaCoCo coverage report from all exec files in the workspace. +# Generates a merged JaCoCo coverage report from the exec files in the workspace. # Produces jacoco-report.xml suitable for upload to Codecov. # -# Usage: ./scripts/jacoco/generate_merged_coverage_report.sh [workspace_dir] +# Usage: ./scripts/jacoco/generate_merged_coverage_report.sh [workspace_dir [exec_dir ...]] # workspace_dir: root of the source tree (defaults to current directory) +# exec_dir: only the exec files under these directories go into the report (defaults to +# workspace_dir). A job that uploads one report per Codecov flag calls this once per flag, +# uploading each report before the next. +# Environment: +# COVERAGE_MODULE_DIRS: optional newline-separated module directories. When set, only these +# modules' classes and sources go into the report; otherwise every module in the workspace does. set -euo pipefail WORKSPACE_DIR="${1:-.}" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +EXEC_DIRS=("${@:2}") +if [ ${#EXEC_DIRS[@]} -eq 0 ]; then + EXEC_DIRS=("$WORKSPACE_DIR") +fi + +# A report left by an earlier call must not be uploaded again under another flag. +rm -f "$WORKSPACE_DIR/jacoco-report.xml" # Skip if no exec files exist (e.g., tests were skipped for this matrix entry) -if ! find "$WORKSPACE_DIR" -path '*/jacoco-agent/*.exec' -type f | grep -q .; then +if ! find "${EXEC_DIRS[@]}" -path '*/jacoco-agent/*.exec' -type f | grep -q .; then echo "No JaCoCo exec files found — skipping coverage report" exit 0 fi -"$SCRIPT_DIR/download_jacoco.sh" -"$SCRIPT_DIR/merge_jacoco_exec_files.sh" jacoco-lib/lib/jacococli.jar "$WORKSPACE_DIR" +# An earlier call already downloaded the CLI, and unzip would stop to ask before overwriting it. +if [ ! -f jacoco-lib/lib/jacococli.jar ]; then + "$SCRIPT_DIR/download_jacoco.sh" +fi +"$SCRIPT_DIR/merge_jacoco_exec_files.sh" jacoco-lib/lib/jacococli.jar "${EXEC_DIRS[@]}" mv merged-jacoco.exec jacoco.exec +# Recreated on every call, so a report never picks up classes or sources copied for an earlier one. +rm -rf aggregate-classes aggregate-sources mkdir -p aggregate-classes aggregate-sources -find "$WORKSPACE_DIR" -path '*/target/classes' -type d | while read d; do cp -r "$d"/. aggregate-classes/ 2>/dev/null; done -find "$WORKSPACE_DIR" -path '*/src/main/java' -type d | while read d; do cp -r "$d"/. aggregate-sources/ 2>/dev/null; done -find "$WORKSPACE_DIR" -path '*/src/main/scala' -type d | while read d; do cp -r "$d"/. aggregate-sources/ 2>/dev/null; done +if [ -n "${COVERAGE_MODULE_DIRS:-}" ]; then + while read -r d; do + # Skips pom-only modules, which have no classes, and any line that is not a module directory. + if [ ! -d "$d/target/classes" ]; then + continue + fi + cp -r "$d/target/classes"/. aggregate-classes/ + for src in "$d/src/main/java" "$d/src/main/scala"; do + if [ -d "$src" ]; then + cp -r "$src"/. aggregate-sources/ + fi + done + done <<< "$COVERAGE_MODULE_DIRS" +else + find "$WORKSPACE_DIR" -path '*/target/classes' -type d | while read d; do cp -r "$d"/. aggregate-classes/ 2>/dev/null; done + find "$WORKSPACE_DIR" -path '*/src/main/java' -type d | while read d; do cp -r "$d"/. aggregate-sources/ 2>/dev/null; done + find "$WORKSPACE_DIR" -path '*/src/main/scala' -type d | while read d; do cp -r "$d"/. aggregate-sources/ 2>/dev/null; done +fi "$SCRIPT_DIR/generate_jacoco_coverage_report.sh" jacoco-lib/lib/jacococli.jar "$WORKSPACE_DIR" diff --git a/scripts/jacoco/merge_jacoco_exec_files.sh b/scripts/jacoco/merge_jacoco_exec_files.sh index 4f6b4048b935..a3f7ec086fa6 100755 --- a/scripts/jacoco/merge_jacoco_exec_files.sh +++ b/scripts/jacoco/merge_jacoco_exec_files.sh @@ -18,13 +18,23 @@ # under the License. # /bin/bash --version -shopt -s globstar +# nullglob drops a directory with no exec files instead of passing its pattern to merge. +shopt -s globstar nullglob echo "Jacoco CLI jar: $1" -echo "Hudi source directory: $2" +echo "Hudi source directories: ${*:2}" +exec_files=() +for dir in "${@:2}"; do + exec_files+=($dir/**/jacoco-agent/**/*.exec) +done +# Without this check, merge with no input files succeeds and writes an empty exec file. +if [[ ${#exec_files[@]} -eq 0 ]]; then + echo "No JaCoCo exec files found under: ${*:2}" >&2 + exit 1 +fi retry_count=0 while [[ $retry_count -lt 3 ]]; do - ls -l $2/**/jacoco-agent/**/*.exec - java -jar $1 merge $2/**/jacoco-agent/**/*.exec --destfile merged-jacoco.exec + ls -l "${exec_files[@]}" + java -jar $1 merge "${exec_files[@]}" --destfile merged-jacoco.exec exit_status=$? if [[ $exit_status -eq 0 ]]; then @@ -35,3 +45,5 @@ while [[ $retry_count -lt 3 ]]; do retry_count=$((retry_count + 1)) sleep 10 done +echo "Jacoco merge failed after $retry_count attempts" >&2 +exit 1
