voonhous commented on code in PR #19642:
URL: https://github.com/apache/hudi/pull/19642#discussion_r3794803038
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -62,50 +64,139 @@ jobs:
# hudi-trino at HEAD, assembles the plugin dir via the in-repo shim
# (docker/trino/shim, standing in for the not-yet-released upstream
# trinodb/trino plugin/trino-hudi shim), bakes it into a local
- # apachehudi/hudi-trino_481 image, and runs ITTestTrino* against the
- # spark402 compose stack (the only pair with the trinocoordinator service).
+ # apachehudi/hudi-trino-e2e image on top of the released trino.e2e.version
+ # server, and runs ITTestTrino* against the spark402 compose stack (the
only
+ # pair with the trinocoordinator service).
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
+ - name: Read Trino pin
+ id: trino-pin
+ run: |
+ set -euo pipefail
+ TRINO_SHA=$(sed -n 's|.*<trino.sha>\(.*\)</trino.sha>.*|\1|p'
pom.xml)
+ TRINO_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xml)
+ E2E_VERSION=$(sed -n
's|.*<trino.e2e.version>\(.*\)</trino.e2e.version>.*|\1|p' pom.xml)
+ echo "Connector builds at $TRINO_VERSION ($TRINO_SHA); server image
is $E2E_VERSION"
+ echo "trino_sha=$TRINO_SHA" >> "$GITHUB_OUTPUT"
+ echo "trino_version=$TRINO_VERSION" >> "$GITHUB_OUTPUT"
+ echo "e2e_version=$E2E_VERSION" >> "$GITHUB_OUTPUT"
+ - name: SPI drift gate
+ id: spi-drift
+ # The plugin is built at the pin but loaded by the released
trino.e2e.version server, so
+ # any SPI / filesystem change between the two can make the image
unbootable. Skip the run
+ # instead of reporting a failure that no connector change caused.
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TRINO_SHA: ${{ steps.trino-pin.outputs.trino_sha }}
+ TRINO_VERSION: ${{ steps.trino-pin.outputs.trino_version }}
+ E2E_VERSION: ${{ steps.trino-pin.outputs.e2e_version }}
+ run: |
+ set -euo pipefail
+ # Per-path commit queries, NOT the compare API: compare caps its
file list at 300
+ # and a single Trino release cycle already exceeds that, so a capped
compare would
+ # flag every pin more than a release old as drifted. The commits API
is uncapped;
+ # any commit reachable from the pin that touched a boundary-crossing
path after the
+ # released tag's commit date (excluding the tag commit itself) is
drift. Existence
+ # is enough, so the first page settles it -- truncation cannot yield
a false pass.
+ # Only two surfaces cross the plugin/server boundary and are
therefore gated:
+ # core/trino-spi (the server provides it to the plugin classloader)
and the
+ # reflective HdfsFileSystemLoader contract (bundled
trino-filesystem-manager loads
+ # the server image's version-matched hdfs jar set; see
docker/trino/Dockerfile).
+ # lib/trino-filesystem ships inside the plugin dir, so it cannot
skew the boot.
+ TAG_SHA=$(gh api "repos/trinodb/trino/commits/${E2E_VERSION}" --jq
.sha)
+ TAG_DATE=$(gh api "repos/trinodb/trino/commits/${E2E_VERSION}" --jq
.commit.committer.date)
+ DRIFTED=false
+ for p in core/trino-spi lib/trino-filesystem-manager lib/trino-hdfs;
do
Review Comment:
Okay, `HdfsClassLoader` exact-package-delegates `io.trino.filesystem` (and
`io.trino.memory.context`) back to the plugin's bundled copies, so the
server-built hdfs jars bind against the pin's classes and bundling is exactly
why it crosses. Restored `lib/trino-filesystem` and added
`lib/trino-memory-context` in b92f718be3f5, with the comment rewritten around
the delegation list instead of the wrong bundled-therefore-safe claim.
##########
.github/workflows/hudi_trino_compat.yml:
##########
@@ -59,18 +82,27 @@ jobs:
echo "trino_version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Detected Trino version: $VERSION"
- name: Install Trino modules from master (JDK 25)
- working-directory: trino
- # hudi-trino compiles against these plus their transitive modules
(spi, cache, metastore,
- # hive-formats, memory-context). They must come from the master
checkout -- resolving from
- # Maven Central would defeat the point of the drift check.
- run: mvn $MVN_ARGS install -pl
:trino-hive,:trino-filesystem-manager,:trino-parquet,:trino-plugin-toolkit -am
-DskipTests -Dair.check.skip-all=true
+ # Same module set the pinned build uses, so a green compile here is a
promotable pin. The
+ # script purges ~/.m2/repository/io/trino itself and only warns when
master's version has
+ # rolled past the pinned trino.version.
+ env:
+ HEAD_SHA: ${{ steps.trino-head.outputs.head_sha }}
+ run: hudi/scripts/trino/bootstrap_trino.sh trino --skip-checkout --ref
"$HEAD_SHA"
Review Comment:
the Floci container migration and the BlobCache span rename were both
test-side drift that `compile` would have waved through. Added in b92f718be3f5:
the compat job now runs `-Phudi-trino,hudi-trino-tests -pl hudi-trino test`
against trino HEAD, the JDK 17 leg installs
`hudi-client-common`/`hudi-java-client` like the CI workflow, the drift-issue
handler covers test failures, and a test failure blocks both the cache save and
the pin advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
