voonhous commented on code in PR #19642:
URL: https://github.com/apache/hudi/pull/19642#discussion_r3794224423
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -62,50 +64,130 @@ jobs:
# hudi-trino at HEAD, assembles the plugin dir via the in-repo shim
# (docker/trino/shim, standing in for the not-yet-released upstream
# trinodb/trino plugin/trino-hudi shim), bakes it into a local
- # apachehudi/hudi-trino_481 image, and runs ITTestTrino* against the
- # spark402 compose stack (the only pair with the trinocoordinator service).
+ # apachehudi/hudi-trino-e2e image on top of the released trino.e2e.version
+ # server, and runs ITTestTrino* against the spark402 compose stack (the
only
+ # pair with the trinocoordinator service).
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
+ - name: Read Trino pin
+ id: trino-pin
+ run: |
+ set -euo pipefail
+ TRINO_SHA=$(sed -n 's|.*<trino.sha>\(.*\)</trino.sha>.*|\1|p'
pom.xml)
+ TRINO_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xml)
+ E2E_VERSION=$(sed -n
's|.*<trino.e2e.version>\(.*\)</trino.e2e.version>.*|\1|p' pom.xml)
+ echo "Connector builds at $TRINO_VERSION ($TRINO_SHA); server image
is $E2E_VERSION"
+ echo "trino_sha=$TRINO_SHA" >> "$GITHUB_OUTPUT"
+ echo "trino_version=$TRINO_VERSION" >> "$GITHUB_OUTPUT"
+ echo "e2e_version=$E2E_VERSION" >> "$GITHUB_OUTPUT"
+ - name: SPI drift gate
+ id: spi-drift
+ # The plugin is built at the pin but loaded by the released
trino.e2e.version server, so
+ # any SPI / filesystem change between the two can make the image
unbootable. Skip the run
+ # instead of reporting a failure that no connector change caused.
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TRINO_SHA: ${{ steps.trino-pin.outputs.trino_sha }}
+ TRINO_VERSION: ${{ steps.trino-pin.outputs.trino_version }}
+ E2E_VERSION: ${{ steps.trino-pin.outputs.e2e_version }}
+ run: |
+ set -euo pipefail
+ # Per-path commit queries, NOT the compare API: compare caps its
file list at 300
+ # and a single Trino release cycle already exceeds that, so a capped
compare would
+ # flag every pin more than a release old as drifted. The commits API
is uncapped;
+ # any commit reachable from the pin that touched an SPI-relevant
path after the
+ # released tag's commit date (excluding the tag commit itself) is
drift. Existence
+ # is enough, so the first page settles it -- truncation cannot yield
a false pass.
+ TAG_SHA=$(gh api "repos/trinodb/trino/commits/${E2E_VERSION}" --jq
.sha)
+ TAG_DATE=$(gh api "repos/trinodb/trino/commits/${E2E_VERSION}" --jq
.commit.committer.date)
+ DRIFTED=false
+ for p in core/trino-spi lib/trino-filesystem
lib/trino-filesystem-manager lib/trino-hdfs; do
Review Comment:
Half right, half deliberate. `lib/trino-filesystem` is indeed fully bundled
and cannot skew the boot -- dropped from the gate in 651552d59bd5. But SPI-only
would be too narrow: bundled `trino-filesystem-manager` reflectively loads the
SERVER image's version-matched hdfs jar set via `HdfsFileSystemLoader`
(`io.trino:trino-hdfs:zip` is not on Central; see the Dockerfile comment), so
that contract crosses the boundary from both sides. The gate now keys on
`core/trino-spi`, `lib/trino-filesystem-manager` and `lib/trino-hdfs`, with the
two contracts spelled out in the step comment.
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -62,50 +64,130 @@ jobs:
# hudi-trino at HEAD, assembles the plugin dir via the in-repo shim
# (docker/trino/shim, standing in for the not-yet-released upstream
# trinodb/trino plugin/trino-hudi shim), bakes it into a local
- # apachehudi/hudi-trino_481 image, and runs ITTestTrino* against the
- # spark402 compose stack (the only pair with the trinocoordinator service).
+ # apachehudi/hudi-trino-e2e image on top of the released trino.e2e.version
+ # server, and runs ITTestTrino* against the spark402 compose stack (the
only
+ # pair with the trinocoordinator service).
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
+ - name: Read Trino pin
+ id: trino-pin
+ run: |
+ set -euo pipefail
+ TRINO_SHA=$(sed -n 's|.*<trino.sha>\(.*\)</trino.sha>.*|\1|p'
pom.xml)
+ TRINO_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xml)
+ E2E_VERSION=$(sed -n
's|.*<trino.e2e.version>\(.*\)</trino.e2e.version>.*|\1|p' pom.xml)
+ echo "Connector builds at $TRINO_VERSION ($TRINO_SHA); server image
is $E2E_VERSION"
+ echo "trino_sha=$TRINO_SHA" >> "$GITHUB_OUTPUT"
+ echo "trino_version=$TRINO_VERSION" >> "$GITHUB_OUTPUT"
+ echo "e2e_version=$E2E_VERSION" >> "$GITHUB_OUTPUT"
+ - name: SPI drift gate
+ id: spi-drift
+ # The plugin is built at the pin but loaded by the released
trino.e2e.version server, so
+ # any SPI / filesystem change between the two can make the image
unbootable. Skip the run
+ # instead of reporting a failure that no connector change caused.
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TRINO_SHA: ${{ steps.trino-pin.outputs.trino_sha }}
+ TRINO_VERSION: ${{ steps.trino-pin.outputs.trino_version }}
+ E2E_VERSION: ${{ steps.trino-pin.outputs.e2e_version }}
+ run: |
+ set -euo pipefail
+ # Per-path commit queries, NOT the compare API: compare caps its
file list at 300
+ # and a single Trino release cycle already exceeds that, so a capped
compare would
+ # flag every pin more than a release old as drifted. The commits API
is uncapped;
+ # any commit reachable from the pin that touched an SPI-relevant
path after the
+ # released tag's commit date (excluding the tag commit itself) is
drift. Existence
+ # is enough, so the first page settles it -- truncation cannot yield
a false pass.
+ TAG_SHA=$(gh api "repos/trinodb/trino/commits/${E2E_VERSION}" --jq
.sha)
+ TAG_DATE=$(gh api "repos/trinodb/trino/commits/${E2E_VERSION}" --jq
.commit.committer.date)
+ DRIFTED=false
+ for p in core/trino-spi lib/trino-filesystem
lib/trino-filesystem-manager lib/trino-hdfs; do
+ for c in $(gh api
"repos/trinodb/trino/commits?sha=${TRINO_SHA}&path=${p}&since=${TAG_DATE}" --jq
'.[].sha'); do
Review Comment:
Confirmed -- `gh api` writes the error body to stdout and the substitution
failure in the for-list slips past `set -e`, so a 502 would have skipped the
suite green. Fixed in 651552d59bd5: each response is assigned to a variable
first, so the failing assignment aborts the step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
