This is an automated email from the ASF dual-hosted git repository.
wombatu-kun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 8c4a8360aad6 chore(ci): flatten the Java CI job graph (#19890)
8c4a8360aad6 is described below
commit 8c4a8360aad614d60287797372cc53eb6205efb0
Author: voonhous <[email protected]>
AuthorDate: Thu Oct 22 14:35:32 2026 +0800
chore(ci): flatten the Java CI job graph (#19890)
Part of #19524.
Remove the changes job and every needs and step-level if that
depended on it; the docs-only skip is a trigger-level paths filter,
evaluated before any job exists. validate-source and
validate-ci-baseline move verbatim to validate_source.yml, which is
never path filtered because both are required status checks. The
Flink, bundle-validation and integration-test jobs move verbatim to
java_ci_engines.yml so each workflow stays under the ASF target of
15 concurrent jobs. The concurrency group carries the workflow name
so the three workflows cancel only their own superseded runs.
Review follow-ups: .github/** added to the positive paths list, a
note in .asf.yaml on why only always-on workflows may contribute
required contexts, and a shorter Codecov range comment.
---
.asf.yaml | 4 +
.codecov.yml | 7 +-
.github/workflows/README.md | 4 +-
.github/workflows/bot.yml | 906 +++---------------------
.github/workflows/java_ci_engines.yml | 615 ++++++++++++++++
.github/workflows/validate_source.yml | 72 ++
README.md | 1 +
hudi-integ-test/pom.xml | 2 +-
packaging/bundle-validation/README.md | 2 +-
scripts/jacoco/README.md | 2 +-
scripts/release/validate_bundle_ui_assets.sh | 2 +-
scripts/release/validate_source_binary_files.sh | 2 +-
12 files changed, 782 insertions(+), 837 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index 3f12c34d3a0b..83f585c6b998 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -51,6 +51,10 @@ github:
# is what made every change to the CI matrix in
.github/workflows/bot.yml a
# two-sided edit, and a context left without a job to report it
blocks merges
# indefinitely. Reviewers gate on the checks the PR shows.
+ # Only contexts from always-on workflows may be listed. bot.yml and
+ # java_ci_engines.yml are path-filtered: on a PR the filter skips,
GitHub never
+ # instantiates them, so any context of theirs would stay pending
forever
+ # (#18598 hit exactly this and worked around it with a gating job).
- validate-ci-baseline
- validate-source
# validate-pr also runs the PR title and commit co-author checks.
diff --git a/.codecov.yml b/.codecov.yml
index f7b071456b6c..0533f7cf23ce 100644
--- a/.codecov.yml
+++ b/.codecov.yml
@@ -28,8 +28,7 @@ codecov:
coverage:
precision: 2
round: down
- # Dashboard bars turn green only at or above the upper bound, so an upper
- # bound of 100 can never be green. 80 is the project coverage target.
+ # Bars are green only at or above the upper bound; 100 never is, 80 is our
target.
range: "70...80"
status:
# Statuses are informational for now: they surface the delta on each PR
but do
@@ -84,9 +83,9 @@ comment:
# Carry the last known coverage forward for a flag when its CI job is skipped
by the
# path filter on a given PR, so a partial run does not zero out that flag's
coverage.
-# Flags uploaded by .github/workflows/bot.yml: spark-client-hadoop-common,
utilities,
+# Flags uploaded by .github/workflows/bot.yml and java_ci_engines.yml:
spark-client-hadoop-common, utilities,
# common-and-other-modules, spark-java-tests, spark-scala-tests,
hadoop-mr-java-client,
-# integration-tests.
+# integration-tests, flink-integration-tests.
flag_management:
default_rules:
carryforward: true
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index fd2a178ea9f8..d496b1a205bb 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -3,5 +3,7 @@
When updating the pr template, you must consider if updates need to be made to
scripts/pr_compliance.py
## What are the files in workflows?
-- bot.yml: runs the hudi unit tests against the scala, spark, and flink
versions in its CI matrix. That matrix is currently cut down to one
configuration per area to reduce runner usage; the jobs and matrix entries it
dropped are commented out in place and marked `[CI-TRIM]`
+- bot.yml: Java CI. Spark lanes (JDK 11 on Spark 3.5, JDK 17 on Spark 4.2).
Path-filtered at the trigger; no job depends on another. Dropped lanes are
commented in place and marked `[CI-TRIM]`
+- java_ci_engines.yml: Java CI Engines. Flink, bundle validation and
integration tests. Same trigger filter and layout rules as bot.yml
+- validate_source.yml: Source Validation. The two required status checks
(validate-source, validate-ci-baseline). Never path-filtered, because a
required check must report on every PR
- pr_validation.yml: one job (`validate-pr`) that checks the PR description
with scripts/pr_compliance.py, runs that script's own tests, validates the PR
title, and rejects commits with a Claude co-author trailer
diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml
index 3bcd4c7ad8af..d7eda45dc6da 100644
--- a/.github/workflows/bot.yml
+++ b/.github/workflows/bot.yml
@@ -1,19 +1,53 @@
name: Java CI
+# This filter replaces the former `changes` job. That job listed the changed
files and every test
+# job waited on it through `needs`, which made each run wait for a shared
runner twice: once for
+# the filter, once for the tests. GitHub evaluates `paths` before creating any
job, so the same
+# decision now costs nothing. A docs-only change creates no run of this
workflow at all, which is
+# fine because no required status check lives here (see validate_source.yml).
+# yml/yaml files are deliberately NOT ignored, unlike the old filter:
docker/compose/*.yml drives
+# the integration tests, several modules keep yml test resources, and the
workflow files themselves
+# need CI to run when they change.
on:
push:
branches:
- master
- 'release-*'
- branch-0.x
+ paths:
+ - '**'
+ # Listed explicitly so the filter does not depend on '**' matching
dot-prefixed paths.
+ - '.github/**'
+ - '!**.bmp'
+ - '!**.gif'
+ - '!**.jpg'
+ - '!**.jpeg'
+ - '!**.md'
+ - '!**.pdf'
+ - '!**.png'
+ - '!**.svg'
+ - '!.gitignore'
pull_request:
branches:
- master
- 'release-*'
- branch-0.x
+ paths:
+ - '**'
+ # Listed explicitly so the filter does not depend on '**' matching
dot-prefixed paths.
+ - '.github/**'
+ - '!**.bmp'
+ - '!**.gif'
+ - '!**.jpg'
+ - '!**.jpeg'
+ - '!**.md'
+ - '!**.pdf'
+ - '!**.png'
+ - '!**.svg'
+ - '!.gitignore'
concurrency:
- group: ${{ github.ref }}
+ group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ !contains(github.ref, 'master') &&
!contains(github.ref, 'branch-0.x') && !contains(github.ref, 'release-') }}
env:
@@ -22,98 +56,15 @@ env:
# serviceUnavailable lists the HTTP codes retried instead of failing the
build. 403 is deliberately
# absent: Maven Central answers 403 when its CDN has blocked the runner's
egress IP, and retrying
# that is what escalates the block.
+ # Keep MVN_ARGS in sync with java_ci_engines.yml.
MVN_ARGS: -e -ntp -B -V -Dgpg.skip -Djacoco.skip -Pwarn-log
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.shade=warn
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.dependency=warn
-Daether.connector.http.retryHandler.count=5
-Daether.connector.http.retryHandler.interval=3000
-Daether.connector.http.retryHandler.intervalMax=30000
-Daether.connector.http.retryHandler.serviceUnavailable=429,500,502,503,504
-Daether.connector.http.connectionMaxTtl=25 -Dorg.eclipse.jetty.LEVEL=WARN
SPARK_COMMON_MODULES:
hudi-spark-datasource/hudi-spark,hudi-spark-datasource/hudi-spark-common
JAVA_UT_FILTER1:
-Dtest=!TestCOWDataSource,!TestMORDataSource,!TestHoodieFileSystemViews
JAVA_UT_FILTER2:
-Dtest=TestCOWDataSource,TestMORDataSource,TestHoodieFileSystemViews
SCALA_TEST_DML_FILTER: -DwildcardSuites=org.apache.spark.sql.hudi.dml
SCALA_TEST_OTHERS_FILTER:
-DwildcardSuites=org.apache.hudi,org.apache.spark.hudi,org.apache.spark.sql.avro,org.apache.spark.sql.execution,org.apache.spark.sql.hive,org.apache.spark.sql.hudi.analysis,org.apache.spark.sql.hudi.blob,org.apache.spark.sql.hudi.catalog,org.apache.spark.sql.hudi.command,org.apache.spark.sql.hudi.common,org.apache.spark.sql.hudi.ddl,org.apache.spark.sql.hudi.procedure,org.apache.spark.sql.hudi.feature
- FLINK_IT_FILTER1: -Dit.test=ITTestHoodieDataSource
- FLINK_IT_FILTER2: -Dit.test=!ITTestHoodieDataSource
jobs:
- # [CI-TRIM] to revisit for CI improvement
- validate-ci-baseline:
- runs-on: ubuntu-latest
- steps:
- - name: Confirm CI ran against the reduced matrix
- run: echo "Ran against the reduced CI matrix."
-
- changes:
- runs-on: ubuntu-latest
- outputs:
- relevant: ${{ steps.filter.outputs.relevant }}
- steps:
- - name: Detect relevant changes
- id: filter
- env:
- GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- REPO: ${{ github.repository }}
- EVENT: ${{ github.event_name }}
- PR_NUMBER: ${{ github.event.pull_request.number }}
- BEFORE_SHA: ${{ github.event.before }}
- AFTER_SHA: ${{ github.sha }}
- run: |
- set -euo pipefail
- if [ "$EVENT" = "pull_request" ]; then
- FILES=$(gh api --paginate "repos/$REPO/pulls/$PR_NUMBER/files"
--jq '.[].filename')
- else
- FILES=$(gh api "repos/$REPO/compare/$BEFORE_SHA...$AFTER_SHA" --jq
'.files[].filename')
- fi
- echo "Changed files:"
- printf '%s\n' "$FILES"
- RELEVANT=false
- while IFS= read -r f; do
- [ -z "$f" ] && continue
- case "$f" in
-
*.bmp|*.gif|*.jpg|*.jpeg|*.md|*.pdf|*.png|*.svg|*.yaml|*.yml|.gitignore) ;;
- *) RELEVANT=true ;;
- esac
- done <<< "$FILES"
- echo "Relevant: $RELEVANT"
- echo "relevant=$RELEVANT" >> "$GITHUB_OUTPUT"
-
- validate-source:
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v5
- - name: Set up JDK 11
- uses: actions/setup-java@v5
- with:
- java-version: '11'
- distribution: 'temurin'
- architecture: x64
- # No `cache: maven` on purpose. All jobs share one cache key and the
first job to finish
- # wins the write. This one only runs `mvn apache-rat:check`,
finishes in well under a
- # minute, and would seal a ~10 MB local repo that every build job
then has to discard and
- # re-download from Maven Central.
- # Both checks describe the source release, not the git checkout, so they
run inside the
- # directory create_source_directory.sh produces - the same way
validate_staged_release.sh
- # runs them inside an extracted source tarball. On the raw checkout they
would see the
- # images under rfc/, docker/images/ and hudi-notebooks/, which the
source release excludes.
- #
- # Stage the copy outside the workspace. `mvn apache-rat:check` in the
RAT check step scans
- # the filesystem from the repository root, so a copy of the tree left
inside the workspace
- # gets scanned too, and every root-anchored exclude in the root pom
(such as `.github/**`)
- # silently stops applying to it.
- #
- # hudi-trino needs no RAT exclude: its files carry AL headers that RAT's
default matchers
- # accept. The Trino-style header check (airbase's license-maven-plugin)
runs only in the
- # upstream trinodb/trino build; nothing in this repo runs it since
RFC-105 dropped the
- # trino-root parent.
- - name: Create source release directory
- run: ./scripts/release/create_source_directory.sh "${{ runner.temp
}}/hudi-tmp-repo"
- - name: Check Binary Files
- run: |
- cd "${{ runner.temp }}/hudi-tmp-repo"
- ./scripts/release/validate_source_binary_files.sh
- - name: Check Copyright
- run: |
- cd "${{ runner.temp }}/hudi-tmp-repo"
- ./scripts/release/validate_source_copyright.sh
- - name: RAT check
- run: ./scripts/release/validate_source_rat.sh
-
# hudi-cli rides along with hudi-spark-client here rather than joining the
catch-all
# module lists in test-common-and-other-modules: its functional suite starts
a
# SparkSession per test class, and this job has the headroom for it while
that one does
@@ -122,7 +73,6 @@ jobs:
# job's UT_MODULES/FT_MODULES exclusions so the tests are not run twice.
test-spark-client-and-hadoop-common:
runs-on: ubuntu-latest
- needs: changes
timeout-minutes: 75
strategy:
matrix:
@@ -132,10 +82,8 @@ jobs:
flinkProfile: "flink2.2"
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 11
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '11'
@@ -143,7 +91,6 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -151,7 +98,6 @@ jobs:
run:
mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service $MVN_ARGS -am -pl
hudi-client/hudi-spark-client,hudi-cli
- name: UT - hudi-hadoop-common
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -159,7 +105,6 @@ jobs:
run:
mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -fae -pl hudi-hadoop-common $MVN_ARGS -Djacoco.skip=false
- name: UT - hudi-client/hudi-spark-client
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -167,7 +112,6 @@ jobs:
run:
mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -fae -pl hudi-client/hudi-spark-client $MVN_ARGS
-Djacoco.skip=false
- name: FT - hudi-client/hudi-spark-client
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -175,7 +119,6 @@ jobs:
run:
mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -fae -pl hudi-client/hudi-spark-client $MVN_ARGS
-Djacoco.skip=false
- name: UT - hudi-cli
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -183,7 +126,6 @@ jobs:
run:
mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -fae -pl hudi-cli $MVN_ARGS -Djacoco.skip=false
- name: FT - hudi-cli
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -191,10 +133,10 @@ jobs:
run:
mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -fae -pl hudi-cli $MVN_ARGS -Djacoco.skip=false
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -204,7 +146,6 @@ jobs:
test-utilities:
runs-on: ubuntu-latest
- needs: changes
timeout-minutes: 60
strategy:
matrix:
@@ -214,10 +155,8 @@ jobs:
flinkProfile: "flink2.2"
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 11
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '11'
@@ -225,7 +164,6 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -233,7 +171,6 @@ jobs:
run:
mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service -Pthrift-gen-source
$MVN_ARGS -am -pl hudi-utilities
- name: UT - hudi-utilities (TestHoodie*)
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -241,7 +178,6 @@ jobs:
run:
mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -fae -Dtest="TestHoodie*"
-Dsurefire.failIfNoSpecifiedTests=false -pl hudi-utilities $MVN_ARGS
-Djacoco.skip=false
- name: FT - hudi-utilities
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -249,10 +185,10 @@ jobs:
run:
mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -fae -Dsurefire.failIfNoSpecifiedTests=false -pl
hudi-utilities $MVN_ARGS -Djacoco.skip=false
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -262,7 +198,6 @@ jobs:
test-common-and-other-modules:
runs-on: ubuntu-latest
- needs: changes
timeout-minutes: 90
strategy:
matrix:
@@ -278,10 +213,8 @@ jobs:
!hudi-client/hudi-spark-client,!hudi-cli,!hudi-examples,!hudi-examples/hudi-examples-common,!hudi-examples/hudi-examples-flink,!hudi-examples/hudi-examples-java,!hudi-examples/hudi-examples-spark,!hudi-spark-datasource/hudi-spark,!hudi-utilities
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 11
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '11'
@@ -289,10 +222,8 @@ jobs:
architecture: x64
cache: maven
- name: Build Docker image
- if: needs.changes.outputs.relevant == 'true'
run: docker build -t hudi-ci .
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -306,7 +237,6 @@ jobs:
hudi-ci \
/bin/bash -c "MAVEN_OPTS='-Xmx8g' mvn clean install -T 2
-D$SCALA_PROFILE -D$SPARK_PROFILE -D$FLINK_PROFILE -Phudi-platform-service
-Pthrift-gen-source -DskipTests=true -Dmaven.compiler.maxmem=8192m
-Dcheckstyle.skip=true -Drat.skip=true $MVN_ARGS"
- name: UT - common and other modules
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -320,7 +250,6 @@ jobs:
hudi-ci \
/bin/bash -c "MAVEN_OPTS='-Xmx4g' mvn test -Punit-tests
-D$SCALA_PROFILE -D$SPARK_PROFILE -D$FLINK_PROFILE -fae -pl $UT_MODULES
-Dcheckstyle.skip=true -Drat.skip=true $MVN_ARGS -Djacoco.skip=false"
- name: UT - hudi-utilities (non-TestHoodie)
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -334,7 +263,6 @@ jobs:
hudi-ci \
/bin/bash -c "MAVEN_OPTS='-Xmx4g' mvn test -Punit-tests
-D$SCALA_PROFILE -D$SPARK_PROFILE -D$FLINK_PROFILE -fae -Dtest='!TestHoodie*'
-Dsurefire.failIfNoSpecifiedTests=false -pl hudi-utilities
-Dcheckstyle.skip=true -Drat.skip=true $MVN_ARGS -Djacoco.skip=false"
- name: FT - common and other modules
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -348,10 +276,10 @@ jobs:
hudi-ci \
/bin/bash -c "MAVEN_OPTS='-Xmx4g' mvn test -Pfunctional-tests
-D$SCALA_PROFILE -D$SPARK_PROFILE -D$FLINK_PROFILE -fae -pl $FT_MODULES
-Dcheckstyle.skip=true -Drat.skip=true $MVN_ARGS -Djacoco.skip=false"
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -362,7 +290,6 @@ jobs:
# [CI-TRIM] to revisit for CI improvement
# test-spark-java-tests-part1:
# runs-on: ubuntu-latest
-# needs: changes
# strategy:
# matrix:
# include:
@@ -379,10 +306,8 @@ jobs:
# sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
+# - uses: actions/checkout@v5
# - name: Set up JDK 11
-# if: needs.changes.outputs.relevant == 'true'
# uses: actions/setup-java@v5
# with:
# java-version: '11'
@@ -390,7 +315,6 @@ jobs:
# architecture: x64
# cache: maven
# - name: Build Project
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -398,7 +322,6 @@ jobs:
# run:
# mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,$SPARK_COMMON_MODULES,$SPARK_MODULES"
# - name: Java UT 1 - Common & Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -406,10 +329,10 @@ jobs:
# run:
# mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DwildcardSuites=skipScalaTests $JAVA_UT_FILTER1
-Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Generate merged coverage report
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
# - name: Upload coverage to Codecov
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
# with:
# files: ./jacoco-report.xml
@@ -420,7 +343,6 @@ jobs:
# [CI-TRIM] to revisit for CI improvement
# test-spark-java-tests-part2:
# runs-on: ubuntu-latest
-# needs: changes
# strategy:
# matrix:
# include:
@@ -437,10 +359,8 @@ jobs:
# sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
+# - uses: actions/checkout@v5
# - name: Set up JDK 11
-# if: needs.changes.outputs.relevant == 'true'
# uses: actions/setup-java@v5
# with:
# java-version: '11'
@@ -448,7 +368,6 @@ jobs:
# architecture: x64
# cache: maven
# - name: Build Project
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -456,14 +375,12 @@ jobs:
# run:
# mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,$SPARK_COMMON_MODULES,$SPARK_MODULES"
# - name: Quickstart Test
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
# run:
# mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -pl
hudi-examples/hudi-examples-spark $MVN_ARGS -Djacoco.skip=false
# - name: Java UT 2 - Common & Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -471,7 +388,6 @@ jobs:
# run:
# mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DwildcardSuites=skipScalaTests $JAVA_UT_FILTER2
-Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Java FTA - Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -479,10 +395,10 @@ jobs:
# run:
# mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DwildcardSuites=skipScalaTests -Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Generate merged coverage report
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
# - name: Upload coverage to Codecov
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
# with:
# files: ./jacoco-report.xml
@@ -493,7 +409,6 @@ jobs:
# [CI-TRIM] to revisit for CI improvement
# test-spark-java-tests-part3:
# runs-on: ubuntu-latest
-# needs: changes
# strategy:
# matrix:
# include:
@@ -510,10 +425,8 @@ jobs:
# sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
+# - uses: actions/checkout@v5
# - name: Set up JDK 11
-# if: needs.changes.outputs.relevant == 'true'
# uses: actions/setup-java@v5
# with:
# java-version: '11'
@@ -521,7 +434,6 @@ jobs:
# architecture: x64
# cache: maven
# - name: Build Project
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -529,7 +441,6 @@ jobs:
# run:
# mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,$SPARK_COMMON_MODULES,$SPARK_MODULES"
# - name: Java FTB - Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -537,7 +448,6 @@ jobs:
# run:
# mvn test -Pfunctional-tests-b -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DwildcardSuites=skipScalaTests -Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Java FTC - Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -545,10 +455,10 @@ jobs:
# run:
# mvn test -Pfunctional-tests-c -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DwildcardSuites=skipScalaTests -Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Generate merged coverage report
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
# - name: Upload coverage to Codecov
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
# with:
# files: ./jacoco-report.xml
@@ -559,7 +469,6 @@ jobs:
# [CI-TRIM] to revisit for CI improvement
# test-spark-scala-dml-tests:
# runs-on: ubuntu-latest
-# needs: changes
# strategy:
# matrix:
# include:
@@ -576,10 +485,8 @@ jobs:
# sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
+# - uses: actions/checkout@v5
# - name: Set up JDK 11
-# if: needs.changes.outputs.relevant == 'true'
# uses: actions/setup-java@v5
# with:
# java-version: '11'
@@ -587,7 +494,6 @@ jobs:
# architecture: x64
# cache: maven
# - name: Build Project
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -595,7 +501,6 @@ jobs:
# run:
# mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,$SPARK_COMMON_MODULES,$SPARK_MODULES"
# - name: Scala UT - Common & Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -603,7 +508,6 @@ jobs:
# run:
# mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-Dtest=skipJavaTests $SCALA_TEST_DML_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Scala FT - Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -611,10 +515,10 @@ jobs:
# run:
# mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-Dtest=skipJavaTests $SCALA_TEST_DML_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Generate merged coverage report
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
# - name: Upload coverage to Codecov
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
# with:
# files: ./jacoco-report.xml
@@ -625,7 +529,6 @@ jobs:
# [CI-TRIM] to revisit for CI improvement
# test-spark-scala-other-tests:
# runs-on: ubuntu-latest
-# needs: changes
# strategy:
# matrix:
# include:
@@ -642,10 +545,8 @@ jobs:
# sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
+# - uses: actions/checkout@v5
# - name: Set up JDK 11
-# if: needs.changes.outputs.relevant == 'true'
# uses: actions/setup-java@v5
# with:
# java-version: '11'
@@ -653,7 +554,6 @@ jobs:
# architecture: x64
# cache: maven
# - name: Build Project
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -661,7 +561,6 @@ jobs:
# run:
# mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,$SPARK_COMMON_MODULES,$SPARK_MODULES"
# - name: Scala UT - Common & Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -669,7 +568,6 @@ jobs:
# run:
# mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-Dtest=skipJavaTests $SCALA_TEST_OTHERS_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Scala FT - Spark
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -677,10 +575,10 @@ jobs:
# run:
# mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-Dtest=skipJavaTests $SCALA_TEST_OTHERS_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
# - name: Generate merged coverage report
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
# - name: Upload coverage to Codecov
-# if: always() && needs.changes.outputs.relevant == 'true'
+# if: always()
# uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
# with:
# files: ./jacoco-report.xml
@@ -690,7 +588,6 @@ jobs:
test-hudi-hadoop-mr-and-hudi-java-client:
runs-on: ubuntu-latest
- needs: changes
timeout-minutes: 40
strategy:
matrix:
@@ -700,10 +597,8 @@ jobs:
flinkProfile: "flink2.2"
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 11
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '11'
@@ -711,7 +606,6 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -719,7 +613,6 @@ jobs:
run:
mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service $MVN_ARGS -am -pl
hudi-hadoop-mr,hudi-client/hudi-java-client
- name: UT - hudi-hadoop-mr and hudi-client/hudi-java-client
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -727,10 +620,10 @@ jobs:
run:
mvn test -Punit-tests -fae -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-hadoop-mr,hudi-client/hudi-java-client $MVN_ARGS
-Djacoco.skip=false
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -740,7 +633,6 @@ jobs:
test-spark-java17-java-tests-part1:
runs-on: ubuntu-latest
- needs: changes
strategy:
matrix:
include:
@@ -762,10 +654,8 @@ jobs:
sparkModules:
"hudi-spark-datasource/hudi-spark4-common,hudi-spark-datasource/hudi-spark4.2.x"
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 17
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '17'
@@ -773,7 +663,6 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -781,7 +670,6 @@ jobs:
run:
mvn clean install -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES"
- name: Java UT 1 - Common & Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -789,10 +677,10 @@ jobs:
run:
mvn test -Punit-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DwildcardSuites=skipScalaTests $JAVA_UT_FILTER1
-Dsurefire.failIfNoSpecifiedTests=false -pl
"hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -802,7 +690,6 @@ jobs:
test-spark-java17-java-tests-part2:
runs-on: ubuntu-latest
- needs: changes
strategy:
matrix:
include:
@@ -821,10 +708,8 @@ jobs:
sparkModules: "hudi-spark-datasource/hudi-spark4.2.x"
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 17
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '17'
@@ -832,7 +717,6 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -840,14 +724,12 @@ jobs:
run:
mvn clean install -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES"
- name: Quickstart Test
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
run:
mvn test -Punit-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DwildcardSuites=skipScalaTests -Dsurefire.failIfNoSpecifiedTests=false -pl
hudi-examples/hudi-examples-spark $MVN_ARGS -Djacoco.skip=false
- name: Java UT 2 - Common & Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -855,7 +737,6 @@ jobs:
run:
mvn test -Punit-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DwildcardSuites=skipScalaTests $JAVA_UT_FILTER2
-Dsurefire.failIfNoSpecifiedTests=false -pl
"hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
- name: Java FTA - Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -863,10 +744,10 @@ jobs:
run:
mvn test -Pfunctional-tests -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -pl "$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS
-Djacoco.skip=false
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -876,7 +757,6 @@ jobs:
test-spark-java17-java-tests-part3:
runs-on: ubuntu-latest
- needs: changes
strategy:
matrix:
include:
@@ -895,10 +775,8 @@ jobs:
sparkModules: "hudi-spark-datasource/hudi-spark4.2.x"
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 17
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '17'
@@ -906,7 +784,6 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -914,7 +791,6 @@ jobs:
run:
mvn clean install -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES"
- name: Java FTB - Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -922,7 +798,6 @@ jobs:
run:
mvn test -Pfunctional-tests-b -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -pl "$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS
-Djacoco.skip=false
- name: Java FTC - Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -930,10 +805,10 @@ jobs:
run:
mvn test -Pfunctional-tests-c -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -pl "$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS
-Djacoco.skip=false
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -943,7 +818,6 @@ jobs:
test-spark-java17-scala-dml-tests:
runs-on: ubuntu-latest
- needs: changes
strategy:
matrix:
include:
@@ -962,10 +836,8 @@ jobs:
sparkModules: "hudi-spark-datasource/hudi-spark4.2.x"
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 17
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '17'
@@ -973,7 +845,6 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -981,7 +852,6 @@ jobs:
run:
mvn clean install -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES"
- name: Scala UT - Common & Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -989,7 +859,6 @@ jobs:
run:
mvn test -Punit-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-Dtest=skipJavaTests $SCALA_TEST_DML_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
- name: Scala FT - Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -997,10 +866,10 @@ jobs:
run:
mvn test -Pfunctional-tests -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -Dtest=skipJavaTests $SCALA_TEST_DML_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -1010,7 +879,6 @@ jobs:
test-spark-java17-scala-other-tests:
runs-on: ubuntu-latest
- needs: changes
strategy:
matrix:
include:
@@ -1029,10 +897,8 @@ jobs:
sparkModules: "hudi-spark-datasource/hudi-spark4.2.x"
steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
+ - uses: actions/checkout@v5
- name: Set up JDK 17
- if: needs.changes.outputs.relevant == 'true'
uses: actions/setup-java@v5
with:
java-version: '17'
@@ -1040,7 +906,6 @@ jobs:
architecture: x64
cache: maven
- name: Build Project
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -1048,7 +913,6 @@ jobs:
run:
mvn clean install -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES"
- name: Scala UT - Common & Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -1056,7 +920,6 @@ jobs:
run:
mvn test -Punit-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-Dtest=skipJavaTests $SCALA_TEST_OTHERS_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"hudi-common,$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
- name: Scala FT - Spark
- if: needs.changes.outputs.relevant == 'true'
env:
SCALA_PROFILE: ${{ matrix.scalaProfile }}
SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -1064,10 +927,10 @@ jobs:
run:
mvn test -Pfunctional-tests -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -Dtest=skipJavaTests $SCALA_TEST_OTHERS_FILTER
-Dsurefire.failIfNoSpecifiedTests=false -pl
"$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS -Djacoco.skip=false
- name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
+ if: always()
uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
with:
files: ./jacoco-report.xml
@@ -1075,576 +938,9 @@ jobs:
flags: spark-scala-tests
token: ${{ secrets.CODECOV_TOKEN }}
- test-flink-1:
- runs-on: ubuntu-latest
- needs: changes
- strategy:
- matrix:
- include:
- # [CI-TRIM] to revisit for CI improvement
-# - flinkProfile: "flink1.18"
-# flinkAvroVersion: "1.11.4"
-# flinkParquetVersion: '1.13.1'
-# - flinkProfile: "flink1.19"
-# flinkAvroVersion: "1.11.4"
-# flinkParquetVersion: '1.13.1'
-# - flinkProfile: "flink1.20"
-# flinkAvroVersion: "1.11.4"
-# flinkParquetVersion: '1.13.1'
-# - flinkProfile: "flink2.0"
-# flinkAvroVersion: "1.11.4"
-# flinkParquetVersion: '1.14.4'
- - flinkProfile: "flink2.2"
- flinkAvroVersion: "1.11.4"
- flinkParquetVersion: '1.15.2'
- steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
- - name: Set up JDK 11
- if: needs.changes.outputs.relevant == 'true'
- uses: actions/setup-java@v5
- with:
- java-version: '11'
- distribution: 'temurin'
- architecture: x64
- cache: maven
- - name: Build Project
- if: needs.changes.outputs.relevant == 'true'
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- run:
- mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl
hudi-examples/hudi-examples-flink -am -Davro.version="$FLINK_AVRO_VERSION"
-Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
- - name: Quickstart Test
- if: needs.changes.outputs.relevant == 'true'
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- run:
- mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-pl hudi-examples/hudi-examples-flink $MVN_ARGS
- - name: Integration Test 1
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- if: ${{ endsWith(env.FLINK_PROFILE, '2.2') &&
needs.changes.outputs.relevant == 'true' }}
- run: |
- mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-DskipTests=true $MVN_ARGS
- mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
$FLINK_IT_FILTER1 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
-Djacoco.skip=false
- - name: Generate merged coverage report
- if: always() && endsWith(matrix.flinkProfile, '2.2') &&
needs.changes.outputs.relevant == 'true'
- run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- - name: Upload coverage to Codecov
- if: always() && endsWith(matrix.flinkProfile, '2.2') &&
needs.changes.outputs.relevant == 'true'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
- with:
- files: ./jacoco-report.xml
- disable_search: true
- flags: flink-integration-tests
- token: ${{ secrets.CODECOV_TOKEN }}
-
- test-flink-2:
- runs-on: ubuntu-latest
- needs: changes
- strategy:
- matrix:
- include:
- - flinkProfile: "flink2.2"
- flinkAvroVersion: "1.11.4"
- flinkParquetVersion: '1.15.2'
- steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
- - name: Set up JDK 11
- if: needs.changes.outputs.relevant == 'true'
- uses: actions/setup-java@v5
- with:
- java-version: '11'
- distribution: 'temurin'
- architecture: x64
- cache: maven
- - name: Build Project
- if: needs.changes.outputs.relevant == 'true'
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- run:
- mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl
hudi-examples/hudi-examples-flink -am -Davro.version="$FLINK_AVRO_VERSION"
-Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
- - name: Integration Test 2
- env:
- SCALA_PROFILE: 'scala-2.12'
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- if: ${{ endsWith(env.FLINK_PROFILE, '2.2') &&
needs.changes.outputs.relevant == 'true' }}
- run: |
- mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-DskipTests=true $MVN_ARGS
- mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
$FLINK_IT_FILTER2 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
-Djacoco.skip=false
- - name: Generate merged coverage report
- if: always() && endsWith(matrix.flinkProfile, '2.2') &&
needs.changes.outputs.relevant == 'true'
- run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- - name: Upload coverage to Codecov
- if: always() && endsWith(matrix.flinkProfile, '2.2') &&
needs.changes.outputs.relevant == 'true'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
- with:
- files: ./jacoco-report.xml
- disable_search: true
- flags: flink-integration-tests
- token: ${{ secrets.CODECOV_TOKEN }}
-
- # [CI-TRIM] to revisit for CI improvement
-# docker-java17-test:
-# runs-on: ubuntu-latest
-# needs: changes
-# strategy:
-# matrix:
-# include:
-# - scalaProfile: 'scala-2.13'
-# flinkProfile: 'flink2.1'
-# sparkProfile: 'spark3.5'
-# sparkRuntime: 'spark3.5.0'
-# - scalaProfile: 'scala-2.12'
-# flinkProfile: 'flink2.1'
-# sparkProfile: 'spark3.5'
-# sparkRuntime: 'spark3.5.0'
-# - scalaProfile: 'scala-2.13'
-# flinkProfile: 'flink2.1'
-# sparkProfile: 'spark4.0'
-# sparkRuntime: 'spark4.0.0'
-# - scalaProfile: 'scala-2.13'
-# flinkProfile: 'flink2.1'
-# sparkProfile: 'spark4.1'
-# sparkRuntime: 'spark4.1.1'
-# - scalaProfile: 'scala-2.13'
-# flinkProfile: 'flink1.20'
-# sparkProfile: 'spark4.2'
-# sparkRuntime: 'spark4.2.0'
-
-# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
-# - name: Set up JDK 17
-# if: needs.changes.outputs.relevant == 'true'
-# uses: actions/setup-java@v5
-# with:
-# java-version: '17'
-# distribution: 'temurin'
-# architecture: x64
-# # No `cache: maven` on purpose. This job runs no Maven on the host:
run_docker_java17.sh
-# # builds inside the container and never mounts $HOME/.m2. Caching
here seals a near-empty
-# # repo under the key all jobs share, which every later job then
discards and re-downloads.
-# - name: UT/FT - Docker Test - OpenJDK 17
-# env:
-# FLINK_PROFILE: ${{ matrix.flinkProfile }}
-# SPARK_PROFILE: ${{ matrix.sparkProfile }}
-# SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
-# SCALA_PROFILE: ${{ matrix.scalaProfile }}
-# # Only support Spark 3.4 for now
-# if: ${{ env.SPARK_PROFILE >= 'spark3.4' &&
needs.changes.outputs.relevant == 'true' }}
-# run: |
-# ./packaging/bundle-validation/run_docker_java17.sh
-
- validate-bundles:
- runs-on: ubuntu-latest
- needs: changes
- strategy:
- matrix:
- include:
- - scalaProfile: 'scala-2.12'
- flinkProfile: 'flink2.2'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.15.2'
- sparkProfile: 'spark3.5'
- sparkRuntime: 'spark3.5.1'
- # [CI-TRIM] to revisit for CI improvement
-# - scalaProfile: 'scala-2.13'
-# flinkProfile: 'flink1.19'
-# flinkAvroVersion: '1.11.4'
-# flinkParquetVersion: '1.13.1'
-# sparkProfile: 'spark3.5'
-# sparkRuntime: 'spark3.5.1'
-# - scalaProfile: 'scala-2.12'
-# flinkProfile: 'flink1.18'
-# flinkAvroVersion: '1.11.4'
-# flinkParquetVersion: '1.13.1'
-# sparkProfile: 'spark3.4'
-# sparkRuntime: 'spark3.4.3'
-# - scalaProfile: 'scala-2.12'
-# flinkProfile: 'flink1.18'
-# flinkAvroVersion: '1.11.4'
-# flinkParquetVersion: '1.13.1'
-# sparkProfile: 'spark3.3'
-# sparkRuntime: 'spark3.3.4'
-
- steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
- - name: Set up JDK 11
- if: needs.changes.outputs.relevant == 'true'
- uses: actions/setup-java@v5
- with:
- java-version: '11'
- distribution: 'temurin'
- architecture: x64
- cache: maven
- - name: Build Project
- if: needs.changes.outputs.relevant == 'true'
- env:
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- SCALA_PROFILE: ${{ matrix.scalaProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- run: |
- if [ "$SCALA_PROFILE" == "scala-2.13" ]; then
- mvn clean package -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DdeployArtifacts=true -DskipTests=true $MVN_ARGS -pl
packaging/hudi-hadoop-mr-bundle,packaging/hudi-spark-bundle,packaging/hudi-utilities-bundle,packaging/hudi-utilities-slim-bundle,packaging/hudi-cli-bundle
-am
- else
- mvn clean package -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -DdeployArtifacts=true -DskipTests=true $MVN_ARGS
- # TODO remove the sudo below. It's a needed workaround as detailed
in HUDI-5708.
- sudo chown -R "$USER:$(id -g -n)"
hudi-platform-service/hudi-metaserver/target/generated-sources
- mvn package -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-DdeployArtifacts=true -DskipTests=true $MVN_ARGS -pl
packaging/hudi-flink-bundle -am -Davro.version="$FLINK_AVRO_VERSION"
-Dparquet.version="$FLINK_PARQUET_VERSION"
- fi
- - name: Validate Bundle UI Assets
- if: needs.changes.outputs.relevant == 'true'
- run: ./scripts/release/validate_bundle_ui_assets.sh
- - name: IT - Bundle Validation - OpenJDK 11
- if: needs.changes.outputs.relevant == 'true'
- env:
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
- SCALA_PROFILE: ${{ matrix.scalaProfile }}
- run: |
- HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
- ./packaging/bundle-validation/ci_run.sh hudi_docker_java11
$HUDI_VERSION openjdk11
- - name: IT - Bundle Validation - OpenJDK 17
- if: needs.changes.outputs.relevant == 'true'
- env:
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
- SCALA_PROFILE: ${{ matrix.scalaProfile }}
- run: |
- HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
- ./packaging/bundle-validation/ci_run.sh hudi_docker_java17
$HUDI_VERSION openjdk17
-
- validate-bundle-spark4:
- runs-on: ubuntu-latest
- needs: changes
- strategy:
- matrix:
- include:
- # [CI-TRIM] to revisit for CI improvement
-# - scalaProfile: 'scala-2.13'
-# flinkProfile: 'flink1.20'
-# flinkAvroVersion: '1.11.4'
-# flinkParquetVersion: '1.13.1'
-# sparkProfile: 'spark4.0'
-# sparkRuntime: 'spark4.0.0'
-# - scalaProfile: 'scala-2.13'
-# flinkProfile: 'flink1.20'
-# flinkAvroVersion: '1.11.4'
-# flinkParquetVersion: '1.13.1'
-# sparkProfile: 'spark4.1'
-# sparkRuntime: 'spark4.1.1'
- - scalaProfile: 'scala-2.13'
- flinkProfile: 'flink1.20'
- flinkAvroVersion: '1.11.4'
- flinkParquetVersion: '1.13.1'
- sparkProfile: 'spark4.2'
- sparkRuntime: 'spark4.2.0'
- steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
- - name: Set up JDK 17
- if: needs.changes.outputs.relevant == 'true'
- uses: actions/setup-java@v5
- with:
- java-version: '17'
- distribution: 'temurin'
- architecture: x64
- - name: Build Project
- if: needs.changes.outputs.relevant == 'true'
- env:
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- SCALA_PROFILE: ${{ matrix.scalaProfile }}
- FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
- FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
- run: |
- mvn clean package -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DdeployArtifacts=true -DskipTests=true $MVN_ARGS -pl
packaging/hudi-hadoop-mr-bundle,packaging/hudi-spark-bundle,packaging/hudi-utilities-bundle,packaging/hudi-utilities-slim-bundle,packaging/hudi-cli-bundle
-am
- - name: Validate Bundle UI Assets
- if: needs.changes.outputs.relevant == 'true'
- run: ./scripts/release/validate_bundle_ui_assets.sh
- - name: IT - Spark4 Bundle Validation - OpenJDK 17
- if: needs.changes.outputs.relevant == 'true'
- env:
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
- SCALA_PROFILE: ${{ matrix.scalaProfile }}
- run: |
- HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
- ./packaging/bundle-validation/ci_run.sh hudi_docker_java17
$HUDI_VERSION openjdk17
-
- # [CI-TRIM] to revisit for CI improvement
-# # flink 2.0/2.1 only support Java 11 and above version
-# validate-bundles-java11:
-# runs-on: ubuntu-latest
-# needs: changes
-# strategy:
-# matrix:
-# include:
-# - scalaProfile: 'scala-2.12'
-# flinkProfile: 'flink2.0'
-# flinkAvroVersion: '1.11.4'
-# flinkParquetVersion: '1.14.4'
-# sparkProfile: 'spark3.5'
-# sparkRuntime: 'spark3.5.1'
-# - scalaProfile: 'scala-2.12'
-# flinkProfile: 'flink2.1'
-# flinkAvroVersion: '1.11.4'
-# flinkParquetVersion: '1.15.2'
-# sparkProfile: 'spark3.5'
-# sparkRuntime: 'spark3.5.1'
-
-# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
-# - name: Set up JDK 11
-# if: needs.changes.outputs.relevant == 'true'
-# uses: actions/setup-java@v5
-# with:
-# java-version: '11'
-# distribution: 'temurin'
-# architecture: x64
-# cache: maven
-# - name: Build Project
-# if: needs.changes.outputs.relevant == 'true'
-# env:
-# FLINK_PROFILE: ${{ matrix.flinkProfile }}
-# SPARK_PROFILE: ${{ matrix.sparkProfile }}
-# SCALA_PROFILE: ${{ matrix.scalaProfile }}
-# FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
-# FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
-# run: |
-# mvn clean package -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-DskipTests=true $MVN_ARGS -pl packaging/hudi-flink-bundle -am
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-# - name: Validate Bundle UI Assets
-# if: needs.changes.outputs.relevant == 'true'
-# run: ./scripts/release/validate_bundle_ui_assets.sh
-# - name: IT - Bundle Validation - OpenJDK 11
-# if: needs.changes.outputs.relevant == 'true'
-# env:
-# FLINK_PROFILE: ${{ matrix.flinkProfile }}
-# SPARK_PROFILE: ${{ matrix.sparkProfile }}
-# SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
-# SCALA_PROFILE: ${{ matrix.scalaProfile }}
-# run: |
-# HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
-# ./packaging/bundle-validation/ci_run.sh hudi_docker_java11
$HUDI_VERSION openjdk11
-# - name: IT - Bundle Validation - OpenJDK 17
-# if: needs.changes.outputs.relevant == 'true'
-# env:
-# FLINK_PROFILE: ${{ matrix.flinkProfile }}
-# SPARK_PROFILE: ${{ matrix.sparkProfile }}
-# SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
-# SCALA_PROFILE: ${{ matrix.scalaProfile }}
-# run: |
-# HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
-# ./packaging/bundle-validation/ci_run.sh hudi_docker_java17
$HUDI_VERSION openjdk17
-
- integration-tests:
- runs-on: ubuntu-latest
- needs: changes
- strategy:
- matrix:
- include:
- - sparkProfile: 'spark3.5'
- flinkProfile: 'flink2.2'
- sparkArchive: 'spark-3.5.9/spark-3.5.9-bin-hadoop3.tgz'
- steps:
- - if: needs.changes.outputs.relevant == 'true'
- uses: actions/checkout@v5
- - name: Set up JDK 11
- if: needs.changes.outputs.relevant == 'true'
- uses: actions/setup-java@v5
- with:
- java-version: '11'
- distribution: 'temurin'
- architecture: x64
- cache: maven
- - name: Check disk space
- if: needs.changes.outputs.relevant == 'true'
- run: df -h
- - name: 'Free space'
- if: needs.changes.outputs.relevant == 'true'
- run: |
- sudo rm -rf /usr/share/dotnet
- sudo rm -rf /usr/local/lib/android
- sudo rm -rf /opt/ghc
- sudo rm -rf /usr/local/share/boost
- docker system prune --all --force --volumes
- - name: Check disk space after cleanup
- if: needs.changes.outputs.relevant == 'true'
- run: df -h
- - name: Build Project
- if: needs.changes.outputs.relevant == 'true'
- env:
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- FLINK_PROFILE: ${{ matrix.flinkProfile }}
- SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
- run:
- mvn clean install -T 2 $SCALA_PROFILE -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -Pintegration-tests -DskipTests=true $MVN_ARGS
-Ddocker.compose.skip=true
- - name: 'UT integ-test'
- if: needs.changes.outputs.relevant == 'true'
- env:
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
- run:
- mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
- - name: 'IT'
- if: needs.changes.outputs.relevant == 'true'
- env:
- SPARK_PROFILE: ${{ matrix.sparkProfile }}
- SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
- SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
- run: |
- # dlcdn only carries the current release of each line; fall back to
- # the archive for older pins (#19883). Plain --retry, not
- # --retry-all-errors, so a 404 on the CDN falls through immediately.
- # --speed-limit is a dead-connection detector, not a slowness one:
- # --retry truncates the output back to byte 0, so a floor set near
- # the archive's real throughput re-downloads 382MB per abort.
- DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
- downloaded=false
- for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
- echo "Downloading $SPARK_ARCHIVE from $base"
- if curl -fL --create-dirs -o "$DEST" \
- --retry 5 --retry-delay 10 \
- --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
- "$base/$SPARK_ARCHIVE"; then
- downloaded=true
- break
- fi
- echo "$base did not serve $SPARK_ARCHIVE"
- rm -f "$DEST"
- done
- if [ "$downloaded" != true ]; then
- echo "ERROR: could not download $SPARK_ARCHIVE from any source"
- exit 1
- fi
- tar -xf "$DEST" -C $GITHUB_WORKSPACE/
- mkdir /tmp/spark-events/
- SPARK_ARCHIVE_BASENAME=$(basename $SPARK_ARCHIVE)
- export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
- rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
- mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl
!hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
- - name: Generate merged coverage report
- if: always() && needs.changes.outputs.relevant == 'true'
- run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
- - name: Upload coverage to Codecov
- if: always() && needs.changes.outputs.relevant == 'true'
- uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
- with:
- files: ./jacoco-report.xml
- disable_search: true
- flags: integration-tests
- token: ${{ secrets.CODECOV_TOKEN }}
-
- # [CI-TRIM] to revisit for CI improvement
-# integration-tests-hive-sync:
-# # Testcontainers-based E2E hive sync coverage for Hudi's custom logical
types
-# # (VECTOR, BLOB) and the Spark 4.0+ VARIANT type. Runs the integ2
testcontainers
-# # suite (ITTestCustomTypeHiveSync) against a real Hive metastore on Spark
3.5.9,
-# # 4.0.2, and 4.1.1 stacks.
-# runs-on: ubuntu-latest
-# needs: changes
-# strategy:
-# fail-fast: false
-# matrix:
-# include:
-# - sparkProfile: 'spark3.5'
-# scalaProfile: '-Dscala-2.12 -Dscala.binary.version=2.12'
-# flinkProfile: 'flink1.20'
-# jdkVersion: '11'
-# composePrefix: 'docker-compose_hadoop284_hive2310_spark359'
-# sparkAdhocImage:
'apachehudi/hudi-hadoop_2.8.4-hive_2.3.10-sparkadhoc_3.5.9:latest'
-# - sparkProfile: 'spark4.0'
-# scalaProfile: '-Dscala-2.13 -Dscala.binary.version=2.13'
-# flinkProfile: 'flink1.20'
-# jdkVersion: '17'
-# composePrefix: 'docker-compose_hadoop340_hive2310_spark402'
-# sparkAdhocImage:
'apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-sparkadhoc_4.0.2:latest'
-# - sparkProfile: 'spark4.1'
-# scalaProfile: '-Dscala-2.13 -Dscala.binary.version=2.13'
-# flinkProfile: 'flink1.20'
-# jdkVersion: '17'
-# composePrefix: 'docker-compose_hadoop340_hive2310_spark411'
-# sparkAdhocImage:
'apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-sparkadhoc_4.1.1:latest'
-# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
-# - name: Set up JDK ${{ matrix.jdkVersion }}
-# if: needs.changes.outputs.relevant == 'true'
-# uses: actions/setup-java@v5
-# with:
-# java-version: ${{ matrix.jdkVersion }}
-# distribution: 'temurin'
-# architecture: x64
-# cache: maven
-# - name: Free disk space
-# if: needs.changes.outputs.relevant == 'true'
-# run: |
-# sudo rm -rf /usr/share/dotnet
-# sudo rm -rf /usr/local/lib/android
-# sudo rm -rf /opt/ghc
-# sudo rm -rf /usr/local/share/boost
-# docker system prune --all --force --volumes
-# - name: Pre-pull compose images (fails fast if not published)
-# if: needs.changes.outputs.relevant == 'true'
-# env:
-# SPARK_ADHOC_IMAGE: ${{ matrix.sparkAdhocImage }}
-# run: |
-# # Surface missing Spark 4.0.2 images before the 15-minute Maven
install.
-# # The remaining images in the compose stack are pulled by
docker-compose at
-# # test time.
-# docker pull "$SPARK_ADHOC_IMAGE"
-# - name: Build and install Hudi artifacts
-# if: needs.changes.outputs.relevant == 'true'
-# env:
-# SPARK_PROFILE: ${{ matrix.sparkProfile }}
-# FLINK_PROFILE: ${{ matrix.flinkProfile }}
-# SCALA_PROFILE: ${{ matrix.scalaProfile }}
-# run:
-# mvn clean install -T 2 $SCALA_PROFILE -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -Pintegration-tests -DskipTests=true
-Ddocker.compose.skip=true $MVN_ARGS
-# - name: Run integ2 testcontainers suite
-# if: needs.changes.outputs.relevant == 'true'
-# env:
-# SPARK_PROFILE: ${{ matrix.sparkProfile }}
-# SCALA_PROFILE: ${{ matrix.scalaProfile }}
-# COMPOSE_PREFIX: ${{ matrix.composePrefix }}
-# run: |
-# # -DskipITs=false overrides the spark4.0 profile's skipITs=true
default
-# # (see root pom.xml). Without it, failsafe skips all ITs on the
spark4.0 matrix row.
-# mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests \
-# -pl hudi-integ-test \
-# -DskipITs=false \
-# -Ddocker.compose.skip=true \
-# -Dit.test='ITTestCustomTypeHiveSync' \
-# -Dspark.docker.compose.prefix=$COMPOSE_PREFIX \
-# $MVN_ARGS
-
# [CI-TRIM] to revisit for CI improvement
# build-spark-java17:
# runs-on: ubuntu-latest
-# needs: changes
# strategy:
# matrix:
# include:
@@ -1661,10 +957,8 @@ jobs:
# sparkModules: "hudi-spark-datasource/hudi-spark3.5.x"
# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
+# - uses: actions/checkout@v5
# - name: Set up JDK 17
-# if: needs.changes.outputs.relevant == 'true'
# uses: actions/setup-java@v5
# with:
# java-version: '17'
@@ -1672,7 +966,6 @@ jobs:
# architecture: x64
# cache: maven
# - name: Build Project
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
@@ -1680,49 +973,8 @@ jobs:
# run:
# mvn clean install -T 2 -Djava17 -Djava.version=17
-D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -DskipTests=true $MVN_ARGS -am -pl
"hudi-examples/hudi-examples-spark,$SPARK_COMMON_MODULES,$SPARK_MODULES"
# - name: Quickstart Test
-# if: needs.changes.outputs.relevant == 'true'
# env:
# SCALA_PROFILE: ${{ matrix.scalaProfile }}
# SPARK_PROFILE: ${{ matrix.sparkProfile }}
# run:
# mvn test -Punit-tests -Djava17 -Djava.version=17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -pl hudi-examples/hudi-examples-spark $MVN_ARGS
-
- # [CI-TRIM] to revisit for CI improvement
-# build-flink-java17:
-# runs-on: ubuntu-latest
-# needs: changes
-# strategy:
-# matrix:
-# include:
-# - scalaProfile: "scala-2.12"
-# flinkProfile: "flink2.1"
-# flinkAvroVersion: '1.11.4'
-# flinkParquetVersion: '1.15.2'
-# steps:
-# - if: needs.changes.outputs.relevant == 'true'
-# uses: actions/checkout@v5
-# - name: Set up JDK 17
-# if: needs.changes.outputs.relevant == 'true'
-# uses: actions/setup-java@v5
-# with:
-# java-version: '17'
-# distribution: 'temurin'
-# architecture: x64
-# cache: maven
-# - name: Build Project
-# if: needs.changes.outputs.relevant == 'true'
-# env:
-# SCALA_PROFILE: ${{ matrix.scalaProfile }}
-# FLINK_PROFILE: ${{ matrix.flinkProfile }}
-# FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
-# FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
-# run:
-# mvn clean install -T 2 -Djava17 -Djava.version=17
-D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-examples/hudi-examples-flink -am
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-DskipTests=true $MVN_ARGS
-# - name: Quickstart Test
-# if: needs.changes.outputs.relevant == 'true'
-# env:
-# SCALA_PROFILE: ${{ matrix.scalaProfile }}
-# FLINK_PROFILE: ${{ matrix.flinkProfile }}
-# run:
-# mvn test -Punit-tests -Djava17 -Djava.version=17 -D"$SCALA_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-examples/hudi-examples-flink $MVN_ARGS
-
diff --git a/.github/workflows/java_ci_engines.yml
b/.github/workflows/java_ci_engines.yml
new file mode 100644
index 000000000000..9cab0b3c14af
--- /dev/null
+++ b/.github/workflows/java_ci_engines.yml
@@ -0,0 +1,615 @@
+name: Java CI Engines
+
+# This filter replaces the former `changes` job. That job listed the changed
files and every test
+# job waited on it through `needs`, which made each run wait for a shared
runner twice: once for
+# the filter, once for the tests. GitHub evaluates `paths` before creating any
job, so the same
+# decision now costs nothing. A docs-only change creates no run of this
workflow at all, which is
+# fine because no required status check lives here (see validate_source.yml).
+# yml/yaml files are deliberately NOT ignored, unlike the old filter:
docker/compose/*.yml drives
+# the integration tests, several modules keep yml test resources, and the
workflow files themselves
+# need CI to run when they change.
+on:
+ push:
+ branches:
+ - master
+ - 'release-*'
+ - branch-0.x
+ paths:
+ - '**'
+ # Listed explicitly so the filter does not depend on '**' matching
dot-prefixed paths.
+ - '.github/**'
+ - '!**.bmp'
+ - '!**.gif'
+ - '!**.jpg'
+ - '!**.jpeg'
+ - '!**.md'
+ - '!**.pdf'
+ - '!**.png'
+ - '!**.svg'
+ - '!.gitignore'
+ pull_request:
+ branches:
+ - master
+ - 'release-*'
+ - branch-0.x
+ paths:
+ - '**'
+ # Listed explicitly so the filter does not depend on '**' matching
dot-prefixed paths.
+ - '.github/**'
+ - '!**.bmp'
+ - '!**.gif'
+ - '!**.jpg'
+ - '!**.jpeg'
+ - '!**.md'
+ - '!**.pdf'
+ - '!**.png'
+ - '!**.svg'
+ - '!.gitignore'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ !contains(github.ref, 'master') &&
!contains(github.ref, 'branch-0.x') && !contains(github.ref, 'release-') }}
+
+env:
+ # The retry/timeout knobs below are maven-resolver (aether.*) properties.
Maven 3.9 resolves
+ # artifacts through the native resolver HTTP transport, which ignores the
older maven.wagon.* names.
+ # serviceUnavailable lists the HTTP codes retried instead of failing the
build. 403 is deliberately
+ # absent: Maven Central answers 403 when its CDN has blocked the runner's
egress IP, and retrying
+ # that is what escalates the block.
+ # Keep MVN_ARGS in sync with bot.yml.
+ MVN_ARGS: -e -ntp -B -V -Dgpg.skip -Djacoco.skip -Pwarn-log
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.shade=warn
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.dependency=warn
-Daether.connector.http.retryHandler.count=5
-Daether.connector.http.retryHandler.interval=3000
-Daether.connector.http.retryHandler.intervalMax=30000
-Daether.connector.http.retryHandler.serviceUnavailable=429,500,502,503,504
-Daether.connector.http.connectionMaxTtl=25 -Dorg.eclipse.jetty.LEVEL=WARN
+ FLINK_IT_FILTER1: -Dit.test=ITTestHoodieDataSource
+ FLINK_IT_FILTER2: -Dit.test=!ITTestHoodieDataSource
+
+jobs:
+ test-flink-1:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ include:
+ # [CI-TRIM] to revisit for CI improvement
+# - flinkProfile: "flink1.18"
+# flinkAvroVersion: "1.11.4"
+# flinkParquetVersion: '1.13.1'
+# - flinkProfile: "flink1.19"
+# flinkAvroVersion: "1.11.4"
+# flinkParquetVersion: '1.13.1'
+# - flinkProfile: "flink1.20"
+# flinkAvroVersion: "1.11.4"
+# flinkParquetVersion: '1.13.1'
+# - flinkProfile: "flink2.0"
+# flinkAvroVersion: "1.11.4"
+# flinkParquetVersion: '1.14.4'
+ - flinkProfile: "flink2.2"
+ flinkAvroVersion: "1.11.4"
+ flinkParquetVersion: '1.15.2'
+ steps:
+ - uses: actions/checkout@v5
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: 'temurin'
+ architecture: x64
+ cache: maven
+ - name: Build Project
+ env:
+ SCALA_PROFILE: 'scala-2.12'
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+ FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+ run:
+ mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl
hudi-examples/hudi-examples-flink -am -Davro.version="$FLINK_AVRO_VERSION"
-Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
+ - name: Quickstart Test
+ env:
+ SCALA_PROFILE: 'scala-2.12'
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+ FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+ run:
+ mvn test -Punit-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-pl hudi-examples/hudi-examples-flink $MVN_ARGS
+ - name: Integration Test 1
+ env:
+ SCALA_PROFILE: 'scala-2.12'
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+ FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+ if: ${{ endsWith(env.FLINK_PROFILE, '2.2') }}
+ run: |
+ mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-DskipTests=true $MVN_ARGS
+ mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
$FLINK_IT_FILTER1 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
-Djacoco.skip=false
+ - name: Generate merged coverage report
+ if: always() && endsWith(matrix.flinkProfile, '2.2')
+ run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
+ - name: Upload coverage to Codecov
+ if: always() && endsWith(matrix.flinkProfile, '2.2')
+ uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ with:
+ files: ./jacoco-report.xml
+ disable_search: true
+ flags: flink-integration-tests
+ token: ${{ secrets.CODECOV_TOKEN }}
+
+ test-flink-2:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ include:
+ - flinkProfile: "flink2.2"
+ flinkAvroVersion: "1.11.4"
+ flinkParquetVersion: '1.15.2'
+ steps:
+ - uses: actions/checkout@v5
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: 'temurin'
+ architecture: x64
+ cache: maven
+ - name: Build Project
+ env:
+ SCALA_PROFILE: 'scala-2.12'
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+ FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+ run:
+ mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl
hudi-examples/hudi-examples-flink -am -Davro.version="$FLINK_AVRO_VERSION"
-Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
+ - name: Integration Test 2
+ env:
+ SCALA_PROFILE: 'scala-2.12'
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+ FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+ if: ${{ endsWith(env.FLINK_PROFILE, '2.2') }}
+ run: |
+ mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-DskipTests=true $MVN_ARGS
+ mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
$FLINK_IT_FILTER2 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
-Djacoco.skip=false
+ - name: Generate merged coverage report
+ if: always() && endsWith(matrix.flinkProfile, '2.2')
+ run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
+ - name: Upload coverage to Codecov
+ if: always() && endsWith(matrix.flinkProfile, '2.2')
+ uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ with:
+ files: ./jacoco-report.xml
+ disable_search: true
+ flags: flink-integration-tests
+ token: ${{ secrets.CODECOV_TOKEN }}
+
+ # [CI-TRIM] to revisit for CI improvement
+# docker-java17-test:
+# runs-on: ubuntu-latest
+# strategy:
+# matrix:
+# include:
+# - scalaProfile: 'scala-2.13'
+# flinkProfile: 'flink2.1'
+# sparkProfile: 'spark3.5'
+# sparkRuntime: 'spark3.5.0'
+# - scalaProfile: 'scala-2.12'
+# flinkProfile: 'flink2.1'
+# sparkProfile: 'spark3.5'
+# sparkRuntime: 'spark3.5.0'
+# - scalaProfile: 'scala-2.13'
+# flinkProfile: 'flink2.1'
+# sparkProfile: 'spark4.0'
+# sparkRuntime: 'spark4.0.0'
+# - scalaProfile: 'scala-2.13'
+# flinkProfile: 'flink2.1'
+# sparkProfile: 'spark4.1'
+# sparkRuntime: 'spark4.1.1'
+# - scalaProfile: 'scala-2.13'
+# flinkProfile: 'flink1.20'
+# sparkProfile: 'spark4.2'
+# sparkRuntime: 'spark4.2.0'
+
+# steps:
+# - uses: actions/checkout@v5
+# - name: Set up JDK 17
+# uses: actions/setup-java@v5
+# with:
+# java-version: '17'
+# distribution: 'temurin'
+# architecture: x64
+# # No `cache: maven` on purpose. This job runs no Maven on the host:
run_docker_java17.sh
+# # builds inside the container and never mounts $HOME/.m2. Caching
here seals a near-empty
+# # repo under the key all jobs share, which every later job then
discards and re-downloads.
+# - name: UT/FT - Docker Test - OpenJDK 17
+# env:
+# FLINK_PROFILE: ${{ matrix.flinkProfile }}
+# SPARK_PROFILE: ${{ matrix.sparkProfile }}
+# SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
+# SCALA_PROFILE: ${{ matrix.scalaProfile }}
+# # Only support Spark 3.4 for now
+# if: ${{ env.SPARK_PROFILE >= 'spark3.4' }}
+# run: |
+# ./packaging/bundle-validation/run_docker_java17.sh
+
+ validate-bundles:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ include:
+ - scalaProfile: 'scala-2.12'
+ flinkProfile: 'flink2.2'
+ flinkAvroVersion: '1.11.4'
+ flinkParquetVersion: '1.15.2'
+ sparkProfile: 'spark3.5'
+ sparkRuntime: 'spark3.5.1'
+ # [CI-TRIM] to revisit for CI improvement
+# - scalaProfile: 'scala-2.13'
+# flinkProfile: 'flink1.19'
+# flinkAvroVersion: '1.11.4'
+# flinkParquetVersion: '1.13.1'
+# sparkProfile: 'spark3.5'
+# sparkRuntime: 'spark3.5.1'
+# - scalaProfile: 'scala-2.12'
+# flinkProfile: 'flink1.18'
+# flinkAvroVersion: '1.11.4'
+# flinkParquetVersion: '1.13.1'
+# sparkProfile: 'spark3.4'
+# sparkRuntime: 'spark3.4.3'
+# - scalaProfile: 'scala-2.12'
+# flinkProfile: 'flink1.18'
+# flinkAvroVersion: '1.11.4'
+# flinkParquetVersion: '1.13.1'
+# sparkProfile: 'spark3.3'
+# sparkRuntime: 'spark3.3.4'
+
+ steps:
+ - uses: actions/checkout@v5
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: 'temurin'
+ architecture: x64
+ cache: maven
+ - name: Build Project
+ env:
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+ FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+ run: |
+ if [ "$SCALA_PROFILE" == "scala-2.13" ]; then
+ mvn clean package -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-DdeployArtifacts=true -DskipTests=true $MVN_ARGS -pl
packaging/hudi-hadoop-mr-bundle,packaging/hudi-spark-bundle,packaging/hudi-utilities-bundle,packaging/hudi-utilities-slim-bundle,packaging/hudi-cli-bundle
-am
+ else
+ mvn clean package -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -DdeployArtifacts=true -DskipTests=true $MVN_ARGS
+ # TODO remove the sudo below. It's a needed workaround as detailed
in HUDI-5708.
+ sudo chown -R "$USER:$(id -g -n)"
hudi-platform-service/hudi-metaserver/target/generated-sources
+ mvn package -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-DdeployArtifacts=true -DskipTests=true $MVN_ARGS -pl
packaging/hudi-flink-bundle -am -Davro.version="$FLINK_AVRO_VERSION"
-Dparquet.version="$FLINK_PARQUET_VERSION"
+ fi
+ - name: Validate Bundle UI Assets
+ run: ./scripts/release/validate_bundle_ui_assets.sh
+ - name: IT - Bundle Validation - OpenJDK 11
+ env:
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ run: |
+ HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
+ ./packaging/bundle-validation/ci_run.sh hudi_docker_java11
$HUDI_VERSION openjdk11
+ - name: IT - Bundle Validation - OpenJDK 17
+ env:
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ run: |
+ HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
+ ./packaging/bundle-validation/ci_run.sh hudi_docker_java17
$HUDI_VERSION openjdk17
+
+ validate-bundle-spark4:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ include:
+ # [CI-TRIM] to revisit for CI improvement
+# - scalaProfile: 'scala-2.13'
+# flinkProfile: 'flink1.20'
+# flinkAvroVersion: '1.11.4'
+# flinkParquetVersion: '1.13.1'
+# sparkProfile: 'spark4.0'
+# sparkRuntime: 'spark4.0.0'
+# - scalaProfile: 'scala-2.13'
+# flinkProfile: 'flink1.20'
+# flinkAvroVersion: '1.11.4'
+# flinkParquetVersion: '1.13.1'
+# sparkProfile: 'spark4.1'
+# sparkRuntime: 'spark4.1.1'
+ - scalaProfile: 'scala-2.13'
+ flinkProfile: 'flink1.20'
+ flinkAvroVersion: '1.11.4'
+ flinkParquetVersion: '1.13.1'
+ sparkProfile: 'spark4.2'
+ sparkRuntime: 'spark4.2.0'
+ steps:
+ - uses: actions/checkout@v5
+ - name: Set up JDK 17
+ uses: actions/setup-java@v5
+ with:
+ java-version: '17'
+ distribution: 'temurin'
+ architecture: x64
+ - name: Build Project
+ env:
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+ FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+ run: |
+ mvn clean package -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DdeployArtifacts=true -DskipTests=true $MVN_ARGS -pl
packaging/hudi-hadoop-mr-bundle,packaging/hudi-spark-bundle,packaging/hudi-utilities-bundle,packaging/hudi-utilities-slim-bundle,packaging/hudi-cli-bundle
-am
+ - name: Validate Bundle UI Assets
+ run: ./scripts/release/validate_bundle_ui_assets.sh
+ - name: IT - Spark4 Bundle Validation - OpenJDK 17
+ env:
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ run: |
+ HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
+ ./packaging/bundle-validation/ci_run.sh hudi_docker_java17
$HUDI_VERSION openjdk17
+
+ # [CI-TRIM] to revisit for CI improvement
+# # flink 2.0/2.1 only support Java 11 and above version
+# validate-bundles-java11:
+# runs-on: ubuntu-latest
+# strategy:
+# matrix:
+# include:
+# - scalaProfile: 'scala-2.12'
+# flinkProfile: 'flink2.0'
+# flinkAvroVersion: '1.11.4'
+# flinkParquetVersion: '1.14.4'
+# sparkProfile: 'spark3.5'
+# sparkRuntime: 'spark3.5.1'
+# - scalaProfile: 'scala-2.12'
+# flinkProfile: 'flink2.1'
+# flinkAvroVersion: '1.11.4'
+# flinkParquetVersion: '1.15.2'
+# sparkProfile: 'spark3.5'
+# sparkRuntime: 'spark3.5.1'
+
+# steps:
+# - uses: actions/checkout@v5
+# - name: Set up JDK 11
+# uses: actions/setup-java@v5
+# with:
+# java-version: '11'
+# distribution: 'temurin'
+# architecture: x64
+# cache: maven
+# - name: Build Project
+# env:
+# FLINK_PROFILE: ${{ matrix.flinkProfile }}
+# SPARK_PROFILE: ${{ matrix.sparkProfile }}
+# SCALA_PROFILE: ${{ matrix.scalaProfile }}
+# FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+# FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+# run: |
+# mvn clean package -T 2 -D"$SCALA_PROFILE" -D"$FLINK_PROFILE"
-DskipTests=true $MVN_ARGS -pl packaging/hudi-flink-bundle -am
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
+# - name: Validate Bundle UI Assets
+# run: ./scripts/release/validate_bundle_ui_assets.sh
+# - name: IT - Bundle Validation - OpenJDK 11
+# env:
+# FLINK_PROFILE: ${{ matrix.flinkProfile }}
+# SPARK_PROFILE: ${{ matrix.sparkProfile }}
+# SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
+# SCALA_PROFILE: ${{ matrix.scalaProfile }}
+# run: |
+# HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
+# ./packaging/bundle-validation/ci_run.sh hudi_docker_java11
$HUDI_VERSION openjdk11
+# - name: IT - Bundle Validation - OpenJDK 17
+# env:
+# FLINK_PROFILE: ${{ matrix.flinkProfile }}
+# SPARK_PROFILE: ${{ matrix.sparkProfile }}
+# SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
+# SCALA_PROFILE: ${{ matrix.scalaProfile }}
+# run: |
+# HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
+# ./packaging/bundle-validation/ci_run.sh hudi_docker_java17
$HUDI_VERSION openjdk17
+
+ integration-tests:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ include:
+ - sparkProfile: 'spark3.5'
+ flinkProfile: 'flink2.2'
+ sparkArchive: 'spark-3.5.9/spark-3.5.9-bin-hadoop3.tgz'
+ steps:
+ - uses: actions/checkout@v5
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: 'temurin'
+ architecture: x64
+ cache: maven
+ - name: Check disk space
+ run: df -h
+ - name: 'Free space'
+ run: |
+ sudo rm -rf /usr/share/dotnet
+ sudo rm -rf /usr/local/lib/android
+ sudo rm -rf /opt/ghc
+ sudo rm -rf /usr/local/share/boost
+ docker system prune --all --force --volumes
+ - name: Check disk space after cleanup
+ run: df -h
+ - name: Build Project
+ env:
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ FLINK_PROFILE: ${{ matrix.flinkProfile }}
+ SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ run:
+ mvn clean install -T 2 $SCALA_PROFILE -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -Pintegration-tests -DskipTests=true $MVN_ARGS
-Ddocker.compose.skip=true
+ - name: 'UT integ-test'
+ env:
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ run:
+ mvn test $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests
-DskipUTs=false -DskipITs=true -pl hudi-integ-test $MVN_ARGS -Djacoco.skip=false
+ - name: 'IT'
+ env:
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
+ SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
+ run: |
+ # dlcdn only carries the current release of each line; fall back to
+ # the archive for older pins (#19883). Plain --retry, not
+ # --retry-all-errors, so a 404 on the CDN falls through immediately.
+ # --speed-limit is a dead-connection detector, not a slowness one:
+ # --retry truncates the output back to byte 0, so a floor set near
+ # the archive's real throughput re-downloads 382MB per abort.
+ DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ downloaded=false
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 1000 --speed-time 120 \
+ "$base/$SPARK_ARCHIVE"; then
+ downloaded=true
+ break
+ fi
+ echo "$base did not serve $SPARK_ARCHIVE"
+ rm -f "$DEST"
+ done
+ if [ "$downloaded" != true ]; then
+ echo "ERROR: could not download $SPARK_ARCHIVE from any source"
+ exit 1
+ fi
+ tar -xf "$DEST" -C $GITHUB_WORKSPACE/
+ mkdir /tmp/spark-events/
+ SPARK_ARCHIVE_BASENAME=$(basename $SPARK_ARCHIVE)
+ export SPARK_HOME=$GITHUB_WORKSPACE/${SPARK_ARCHIVE_BASENAME%.*}
+ rm -f $GITHUB_WORKSPACE/$SPARK_ARCHIVE
+ mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests -pl
!hudi-flink-datasource/hudi-flink $MVN_ARGS -Djacoco.skip=false
+ - name: Generate merged coverage report
+ if: always()
+ run: ./scripts/jacoco/generate_merged_coverage_report.sh
$GITHUB_WORKSPACE
+ - name: Upload coverage to Codecov
+ if: always()
+ uses: codecov/codecov-action@75cd11691c0faa626561e295848008c8a7dddffe
# v5
+ with:
+ files: ./jacoco-report.xml
+ disable_search: true
+ flags: integration-tests
+ token: ${{ secrets.CODECOV_TOKEN }}
+
+ # [CI-TRIM] to revisit for CI improvement
+# integration-tests-hive-sync:
+# # Testcontainers-based E2E hive sync coverage for Hudi's custom logical
types
+# # (VECTOR, BLOB) and the Spark 4.0+ VARIANT type. Runs the integ2
testcontainers
+# # suite (ITTestCustomTypeHiveSync) against a real Hive metastore on Spark
3.5.9,
+# # 4.0.2, and 4.1.1 stacks.
+# runs-on: ubuntu-latest
+# strategy:
+# fail-fast: false
+# matrix:
+# include:
+# - sparkProfile: 'spark3.5'
+# scalaProfile: '-Dscala-2.12 -Dscala.binary.version=2.12'
+# flinkProfile: 'flink1.20'
+# jdkVersion: '11'
+# composePrefix: 'docker-compose_hadoop284_hive2310_spark359'
+# sparkAdhocImage:
'apachehudi/hudi-hadoop_2.8.4-hive_2.3.10-sparkadhoc_3.5.9:latest'
+# - sparkProfile: 'spark4.0'
+# scalaProfile: '-Dscala-2.13 -Dscala.binary.version=2.13'
+# flinkProfile: 'flink1.20'
+# jdkVersion: '17'
+# composePrefix: 'docker-compose_hadoop340_hive2310_spark402'
+# sparkAdhocImage:
'apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-sparkadhoc_4.0.2:latest'
+# - sparkProfile: 'spark4.1'
+# scalaProfile: '-Dscala-2.13 -Dscala.binary.version=2.13'
+# flinkProfile: 'flink1.20'
+# jdkVersion: '17'
+# composePrefix: 'docker-compose_hadoop340_hive2310_spark411'
+# sparkAdhocImage:
'apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-sparkadhoc_4.1.1:latest'
+# steps:
+# - uses: actions/checkout@v5
+# - name: Set up JDK ${{ matrix.jdkVersion }}
+# uses: actions/setup-java@v5
+# with:
+# java-version: ${{ matrix.jdkVersion }}
+# distribution: 'temurin'
+# architecture: x64
+# cache: maven
+# - name: Free disk space
+# run: |
+# sudo rm -rf /usr/share/dotnet
+# sudo rm -rf /usr/local/lib/android
+# sudo rm -rf /opt/ghc
+# sudo rm -rf /usr/local/share/boost
+# docker system prune --all --force --volumes
+# - name: Pre-pull compose images (fails fast if not published)
+# env:
+# SPARK_ADHOC_IMAGE: ${{ matrix.sparkAdhocImage }}
+# run: |
+# # Surface missing Spark 4.0.2 images before the 15-minute Maven
install.
+# # The remaining images in the compose stack are pulled by
docker-compose at
+# # test time.
+# docker pull "$SPARK_ADHOC_IMAGE"
+# - name: Build and install Hudi artifacts
+# env:
+# SPARK_PROFILE: ${{ matrix.sparkProfile }}
+# FLINK_PROFILE: ${{ matrix.flinkProfile }}
+# SCALA_PROFILE: ${{ matrix.scalaProfile }}
+# run:
+# mvn clean install -T 2 $SCALA_PROFILE -D"$SPARK_PROFILE"
-D"$FLINK_PROFILE" -Pintegration-tests -DskipTests=true
-Ddocker.compose.skip=true $MVN_ARGS
+# - name: Run integ2 testcontainers suite
+# env:
+# SPARK_PROFILE: ${{ matrix.sparkProfile }}
+# SCALA_PROFILE: ${{ matrix.scalaProfile }}
+# COMPOSE_PREFIX: ${{ matrix.composePrefix }}
+# run: |
+# # -DskipITs=false overrides the spark4.0 profile's skipITs=true
default
+# # (see root pom.xml). Without it, failsafe skips all ITs on the
spark4.0 matrix row.
+# mvn verify $SCALA_PROFILE -D"$SPARK_PROFILE" -Pintegration-tests \
+# -pl hudi-integ-test \
+# -DskipITs=false \
+# -Ddocker.compose.skip=true \
+# -Dit.test='ITTestCustomTypeHiveSync' \
+# -Dspark.docker.compose.prefix=$COMPOSE_PREFIX \
+# $MVN_ARGS
+
+ # [CI-TRIM] to revisit for CI improvement
+# build-flink-java17:
+# runs-on: ubuntu-latest
+# strategy:
+# matrix:
+# include:
+# - scalaProfile: "scala-2.12"
+# flinkProfile: "flink2.1"
+# flinkAvroVersion: '1.11.4'
+# flinkParquetVersion: '1.15.2'
+# steps:
+# - uses: actions/checkout@v5
+# - name: Set up JDK 17
+# uses: actions/setup-java@v5
+# with:
+# java-version: '17'
+# distribution: 'temurin'
+# architecture: x64
+# cache: maven
+# - name: Build Project
+# env:
+# SCALA_PROFILE: ${{ matrix.scalaProfile }}
+# FLINK_PROFILE: ${{ matrix.flinkProfile }}
+# FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
+# FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
+# run:
+# mvn clean install -T 2 -Djava17 -Djava.version=17
-D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-examples/hudi-examples-flink -am
-Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION"
-DskipTests=true $MVN_ARGS
+# - name: Quickstart Test
+# env:
+# SCALA_PROFILE: ${{ matrix.scalaProfile }}
+# FLINK_PROFILE: ${{ matrix.flinkProfile }}
+# run:
+# mvn test -Punit-tests -Djava17 -Djava.version=17 -D"$SCALA_PROFILE"
-D"$FLINK_PROFILE" -pl hudi-examples/hudi-examples-flink $MVN_ARGS
diff --git a/.github/workflows/validate_source.yml
b/.github/workflows/validate_source.yml
new file mode 100644
index 000000000000..cfd67347e75e
--- /dev/null
+++ b/.github/workflows/validate_source.yml
@@ -0,0 +1,72 @@
+name: Source Validation
+
+# Both jobs here are required status checks in .asf.yaml. A required check
must report on every PR,
+# including docs-only ones, and a path-filtered workflow is never instantiated
for PRs that miss the
+# filter, so these jobs cannot live in bot.yml once it is path-filtered.
+on:
+ push:
+ branches:
+ - master
+ - 'release-*'
+ - branch-0.x
+ pull_request:
+ branches:
+ - master
+ - 'release-*'
+ - branch-0.x
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: ${{ !contains(github.ref, 'master') &&
!contains(github.ref, 'branch-0.x') && !contains(github.ref, 'release-') }}
+
+jobs:
+ # [CI-TRIM] to revisit for CI improvement
+ # This only blocks PRs whose checks predate the matrix trim (#19514). Since
bot.yml and
+ # java_ci_engines.yml are path-filtered, a green result here no longer means
any Java CI
+ # job ran; reviewers gate on the checks the PR shows. Delete under #19535.
+ validate-ci-baseline:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Confirm CI ran against the reduced matrix
+ run: echo "Ran against the reduced CI matrix."
+
+ validate-source:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v5
+ - name: Set up JDK 11
+ uses: actions/setup-java@v5
+ with:
+ java-version: '11'
+ distribution: 'temurin'
+ architecture: x64
+ # No `cache: maven` on purpose. All jobs share one cache key and the
first job to finish
+ # wins the write. This one only runs `mvn apache-rat:check`,
finishes in well under a
+ # minute, and would seal a ~10 MB local repo that every build job
then has to discard and
+ # re-download from Maven Central.
+ # Both checks describe the source release, not the git checkout, so they
run inside the
+ # directory create_source_directory.sh produces - the same way
validate_staged_release.sh
+ # runs them inside an extracted source tarball. On the raw checkout they
would see the
+ # images under rfc/, docker/images/ and hudi-notebooks/, which the
source release excludes.
+ #
+ # Stage the copy outside the workspace. `mvn apache-rat:check` in the
RAT check step scans
+ # the filesystem from the repository root, so a copy of the tree left
inside the workspace
+ # gets scanned too, and every root-anchored exclude in the root pom
(such as `.github/**`)
+ # silently stops applying to it.
+ #
+ # hudi-trino needs no RAT exclude: its files carry AL headers that RAT's
default matchers
+ # accept. The Trino-style header check (airbase's license-maven-plugin)
runs only in the
+ # upstream trinodb/trino build; nothing in this repo runs it since
RFC-105 dropped the
+ # trino-root parent.
+ - name: Create source release directory
+ run: ./scripts/release/create_source_directory.sh "${{ runner.temp
}}/hudi-tmp-repo"
+ - name: Check Binary Files
+ run: |
+ cd "${{ runner.temp }}/hudi-tmp-repo"
+ ./scripts/release/validate_source_binary_files.sh
+ - name: Check Copyright
+ run: |
+ cd "${{ runner.temp }}/hudi-tmp-repo"
+ ./scripts/release/validate_source_copyright.sh
+ - name: RAT check
+ run: ./scripts/release/validate_source_rat.sh
diff --git a/README.md b/README.md
index ee41fcaa7704..1a29a466cc4e 100644
--- a/README.md
+++ b/README.md
@@ -26,6 +26,7 @@ to ingest, index, store, serve, transform and manage your
data across multiple c
<https://hudi.apache.org/>
[](https://github.com/apache/hudi/actions/workflows/bot.yml)
+[](https://github.com/apache/hudi/actions/workflows/java_ci_engines.yml)
[](https://dev.azure.com/apachehudi/hudi-oss-ci/_build/latest?definitionId=5&branchName=master)
[](https://www.apache.org/licenses/LICENSE-2.0.html)
[](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.hudi%22)
diff --git a/hudi-integ-test/pom.xml b/hudi-integ-test/pom.xml
index 95ccd2a96f9f..98c03007b690 100644
--- a/hudi-integ-test/pom.xml
+++ b/hudi-integ-test/pom.xml
@@ -540,7 +540,7 @@
its own docker-compose demo in the same job, which collides on host
ports
(e.g. zookeeper 2181). Exclude the integ2 package here so only the
dedicated `integration-tests-hive-sync` stage runs them (via
-Dit.test=).
- That stage is currently parked in .github/workflows/bot.yml to cut CI
+ That stage is currently parked in
.github/workflows/java_ci_engines.yml to cut CI
usage, so nothing runs the integ2 suite until it is restored.
-->
<build>
diff --git a/packaging/bundle-validation/README.md
b/packaging/bundle-validation/README.md
index b9812bc5d36b..6fac82d14061 100644
--- a/packaging/bundle-validation/README.md
+++ b/packaging/bundle-validation/README.md
@@ -20,7 +20,7 @@
# Bundle Validation for Hudi
This directory contains scripts for running bundle validation in GitHub
Actions (`validate-bundles`
-specified in `.github/workflows/bot.yml`) and build profile for Docker images
used.
+specified in `.github/workflows/java_ci_engines.yml`) and build profile for
Docker images used.
## Docker Image for Bundle Validation
diff --git a/scripts/jacoco/README.md b/scripts/jacoco/README.md
index a0863683c0f0..b31ce1e72625 100644
--- a/scripts/jacoco/README.md
+++ b/scripts/jacoco/README.md
@@ -115,7 +115,7 @@ In addition to the aggregated Azure report described above,
coverage is uploaded
[Codecov](https://app.codecov.io/gh/apache/hudi) on every pull request and
every commit to master.
This is the canonical per-PR view.
-- Each test job in `.github/workflows/bot.yml` runs with the JaCoCo agent,
builds a merged report
+- Each test job in `.github/workflows/bot.yml` and `java_ci_engines.yml` runs
with the JaCoCo agent, builds a merged report
via `scripts/jacoco/generate_merged_coverage_report.sh`, and uploads
`jacoco-report.xml` to
Codecov under a flag (`spark-java-tests`, `spark-scala-tests`, `utilities`,
`common-and-other-modules`, `spark-client-hadoop-common`,
`hadoop-mr-java-client`,
diff --git a/scripts/release/validate_bundle_ui_assets.sh
b/scripts/release/validate_bundle_ui_assets.sh
index 9243ed434526..b9a56e9145cb 100755
--- a/scripts/release/validate_bundle_ui_assets.sh
+++ b/scripts/release/validate_bundle_ui_assets.sh
@@ -32,7 +32,7 @@
# dir Optional directory to search for built bundle jars.
# Defaults to the repo "packaging" directory (scans
packaging/*/target/).
#
-# CI: invoked by the validate-bundles* jobs in .github/workflows/bot.yml right
after the bundle build.
+# CI: invoked by the validate-bundles* jobs in
.github/workflows/java_ci_engines.yml right after the bundle build.
#
set -u
diff --git a/scripts/release/validate_source_binary_files.sh
b/scripts/release/validate_source_binary_files.sh
index 80769c34d3b7..607015e0422f 100755
--- a/scripts/release/validate_source_binary_files.sh
+++ b/scripts/release/validate_source_binary_files.sh
@@ -20,7 +20,7 @@
#
# Run this from the root of a source release tree, i.e. the output of
-# create_source_directory.sh (see the validate-source job in bot.yml) or an
+# create_source_directory.sh (see the validate-source job in
validate_source.yml) or an
# extracted source tarball (see validate_staged_release.sh). Paths that never
# reach the source release are excluded by create_source_directory.sh, not
here.
