This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 5ba8d7ca1915 test(trino): build the E2E server image from the pin
(#19953)
5ba8d7ca1915 is described below
commit 5ba8d7ca1915b8734a44ea4bdd822463140a3ff4
Author: voonhous <[email protected]>
AuthorDate: Wed Sep 16 17:22:52 2026 +0800
test(trino): build the E2E server image from the pin (#19953)
The Trino E2E job built the connector against the trinodb/trino commit
in trino.sha but ran it on the released trinodb/trino image. Master
tracks Trino master, so the two always differed, and an SPI drift gate
skipped every test step while the job still reported success. The
suite had not run on master since the pin started tracking Trino
master.
Build the server image from the pinned commit instead, so the server
and the connector always come from the same commit, and drop the gate:
- build_trino_server_image.sh packages trinodb/trino at trino.sha and
builds the image with Trino's own core/docker/build.sh. It leaves
the heap to trino's .mvn/jvm.config.
- The E2E job builds that image on every run and bakes the plugin on
top through build_image.sh --base-image. Neither the image nor the
io.trino artifacts are cached: the scheduled delete-all-caches job
empties both before a later run can restore them.
- The Dockerfile takes TRINO_BASE_IMAGE, with the released image as
the default for quick local runs.
- overlay-entrypoint.sh runs bin/launcher directly with bash builtins
only: Trino master removed run-trino and moved to a base image
without grep or find.
Closes #19954.
---
.github/workflows/hudi_trino_e2e.yml | 116 ++++++-------------------
docker/README.md | 6 +-
docker/trino/Dockerfile | 6 +-
docker/trino/build_image.sh | 23 +++--
docker/trino/build_trino_server_image.sh | 145 +++++++++++++++++++++++++++++++
docker/trino/overlay-entrypoint.sh | 32 ++++++-
hudi-trino/README.md | 19 ++--
pom.xml | 3 +-
release/release_guide.md | 7 +-
9 files changed, 247 insertions(+), 110 deletions(-)
diff --git a/.github/workflows/hudi_trino_e2e.yml
b/.github/workflows/hudi_trino_e2e.yml
index e6269c94bd47..f6ebacadba90 100644
--- a/.github/workflows/hudi_trino_e2e.yml
+++ b/.github/workflows/hudi_trino_e2e.yml
@@ -16,9 +16,10 @@ on:
# which compiles the whole module, so a break
# anywhere in it fails this pipeline -- not just
# under integ2/.
- # pom.xml owns trino.version / trino.sha /
trino.e2e.version:
- # the shim pom's parent, the plugin dir path and
the
- # server image version below are all derived
from them.
+ # pom.xml owns trino.version / trino.sha (the shim pom's
parent,
+ # the plugin dir path, and the commit both the
server
+ # image and the connector are built from) and
+ # trino.e2e.version (trino-jdbc for the ITs).
- 'hudi-trino/**'
- 'docker/trino/**'
- 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
@@ -64,8 +65,9 @@ jobs:
# hudi-trino at HEAD, assembles the plugin dir via the in-repo shim
# (docker/trino/shim, standing in for the not-yet-released upstream
# trinodb/trino plugin/trino-hudi shim), bakes it into a local
- # apachehudi/hudi-trino-e2e image on top of the released trino.e2e.version
- # server, and runs ITTestTrino* against the spark402 compose stack (the
only
+ # apachehudi/hudi-trino-e2e image on top of a Trino server image built from
+ # the same pinned trinodb/trino commit (so server and connector always
+ # match), and runs ITTestTrino* against the spark402 compose stack (the
only
# pair with the trinocoordinator service).
runs-on: ubuntu-latest
steps:
@@ -76,69 +78,16 @@ jobs:
set -euo pipefail
TRINO_SHA=$(sed -n 's|.*<trino.sha>\(.*\)</trino.sha>.*|\1|p'
pom.xml)
TRINO_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xml)
- E2E_VERSION=$(sed -n
's|.*<trino.e2e.version>\(.*\)</trino.e2e.version>.*|\1|p' pom.xml)
- # sed -n ...p exits 0 on no match; an empty sha would make the
commits API answer
- # for the default branch and silently skip the suite, so fail loudly
instead.
- if [ -z "$TRINO_SHA" ] || [ -z "$TRINO_VERSION" ] || [ -z
"$E2E_VERSION" ]; then
- echo "ERROR: could not read
trino.sha/trino.version/trino.e2e.version from pom.xml" >&2
+ # sed -n ...p exits 0 on no match; an empty sha would check out
trinodb/trino's default
+ # branch and key the caches on nothing, so fail loudly instead.
+ if [ -z "$TRINO_SHA" ] || [ -z "$TRINO_VERSION" ]; then
+ echo "ERROR: could not read trino.sha/trino.version from pom.xml"
>&2
exit 1
fi
- echo "Connector builds at $TRINO_VERSION ($TRINO_SHA); server image
is $E2E_VERSION"
+ echo "Server image and connector both build at $TRINO_VERSION
($TRINO_SHA)"
echo "trino_sha=$TRINO_SHA" >> "$GITHUB_OUTPUT"
echo "trino_version=$TRINO_VERSION" >> "$GITHUB_OUTPUT"
- echo "e2e_version=$E2E_VERSION" >> "$GITHUB_OUTPUT"
- - name: SPI drift gate
- id: spi-drift
- # The plugin is built at the pin but loaded by the released
trino.e2e.version server, so
- # any SPI / filesystem change between the two can make the image
unbootable. Skip the run
- # instead of reporting a failure that no connector change caused.
- env:
- GH_TOKEN: ${{ github.token }}
- TRINO_SHA: ${{ steps.trino-pin.outputs.trino_sha }}
- TRINO_VERSION: ${{ steps.trino-pin.outputs.trino_version }}
- E2E_VERSION: ${{ steps.trino-pin.outputs.e2e_version }}
- run: |
- set -euo pipefail
- # Per-path commit queries, NOT the compare API: compare caps its
file list at 300
- # and a single Trino release cycle already exceeds that, so a capped
compare would
- # flag every pin more than a release old as drifted. The commits API
is uncapped;
- # any commit reachable from the pin that touched a boundary-crossing
path after the
- # released tag's commit date (excluding the tag commit itself) is
drift. Existence
- # is enough, so the first page settles it -- truncation cannot yield
a false pass.
- # Gated paths are the surfaces where server-built and pin-built
classes meet:
- # core/trino-spi (the server provides it to the plugin classloader),
and the
- # HdfsFileSystemLoader contract, where bundled
trino-filesystem-manager loads the
- # server image's version-matched hdfs jar set (see
docker/trino/Dockerfile) whose
- # HdfsClassLoader then delegates the exact packages
io.trino.filesystem and
- # io.trino.memory.context back to the plugin's bundled copies -- so
those two libs
- # cross the boundary precisely BECAUSE they are bundled.
- TAG_SHA=$(gh api "repos/trinodb/trino/commits/${E2E_VERSION}" --jq
.sha)
- TAG_DATE=$(gh api "repos/trinodb/trino/commits/${E2E_VERSION}" --jq
.commit.committer.date)
- DRIFTED=false
- for p in core/trino-spi lib/trino-filesystem
lib/trino-filesystem-manager lib/trino-hdfs lib/trino-memory-context; do
- # Assign before iterating: a failing substitution in the for-list
would not trip
- # set -e, and gh api prints the error body to stdout, so a
transient API error
- # would otherwise iterate over error JSON and silently skip the
suite.
- SHAS=$(gh api
"repos/trinodb/trino/commits?sha=${TRINO_SHA}&path=${p}&since=${TAG_DATE}" --jq
'.[].sha')
- for c in $SHAS; do
- if [ "$c" != "$TAG_SHA" ]; then
- echo "Boundary-crossing change under ${p}: ${c}"
- DRIFTED=true
- fi
- done
- done
- echo "drifted=$DRIFTED" >> "$GITHUB_OUTPUT"
- if [ "$DRIFTED" = "true" ]; then
- {
- echo "## Trino E2E skipped: SPI drift window"
- echo ""
- echo "The connector is built against trinodb/trino
\`${TRINO_VERSION}\` (\`${TRINO_SHA}\`), while the"
- echo "e2e server image is released Trino \`${E2E_VERSION}\`. SPI
/ filesystem paths changed between"
- echo "the two, so this run is skipped until the pin and the
released version re-align."
- } >> "$GITHUB_STEP_SUMMARY"
- fi
- name: Free disk space
- if: steps.spi-drift.outputs.drifted != 'true'
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
@@ -146,14 +95,12 @@ jobs:
sudo rm -rf /usr/local/share/boost
docker system prune --all --force --volumes
- name: Pre-pull compose images (fails fast if not published)
- if: steps.spi-drift.outputs.drifted != 'true'
run: |
# Surface a missing sparkadhoc image before the long Maven install.
The
# remaining stack images are pulled by docker-compose at test time;
the
# trino image is built locally below, never pulled.
docker pull
apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-sparkadhoc_4.0.2:latest
- name: Set up JDK 17
- if: steps.spi-drift.outputs.drifted != 'true'
uses: actions/setup-java@v5
with:
java-version: '17'
@@ -161,50 +108,45 @@ jobs:
architecture: x64
cache: maven
- name: Build and install Hudi artifacts (JDK 17)
- if: steps.spi-drift.outputs.drifted != 'true'
# Full reactor: the compose containers mount the workspace and the
tests
# use bundles staged by the -Pintegration-tests build (e.g.
# docker/hoodie/hadoop/hive_base/target/hoodie-spark-bundle.jar).
run:
mvn clean install -T 2 $SCALA_PROFILE -Dspark4.0 -Dflink1.20
-Pintegration-tests -DskipTests=true -Ddocker.compose.skip=true $MVN_ARGS
- name: Set up JDK 25
- if: steps.spi-drift.outputs.drifted != 'true'
uses: actions/setup-java@v5
with:
java-version: '25'
distribution: 'temurin'
cache: maven
- name: Purge Trino artifacts from the local m2
- if: steps.spi-drift.outputs.drifted != 'true'
# Artifacts an older pin left behind carry the same SNAPSHOT
coordinates as the current ones.
run: rm -rf ~/.m2/repository/io/trino
# The connector and the shim assembly resolve io.trino from the pinned
trinodb/trino
# commit: Trino publishes no SNAPSHOT artifacts, so nothing here comes
from Central.
- - name: Restore Trino artifacts for the pinned commit
- id: trino-m2
- if: steps.spi-drift.outputs.drifted != 'true'
- uses: actions/cache@v4
- with:
- path: ~/.m2/repository/io/trino
- key: trino-m2-v2-${{ hashFiles('scripts/trino/bootstrap_trino.sh')
}}-${{ steps.trino-pin.outputs.trino_sha }}
- # No restore-keys on purpose: a partial restore from another pin
collides on the
- # same SNAPSHOT coordinates and poisons the build.
+ # Not cached, for the same reason as the server image below.
- name: Checkout trinodb/trino at the pinned commit
- if: steps.spi-drift.outputs.drifted != 'true' &&
steps.trino-m2.outputs.cache-hit != 'true'
uses: actions/checkout@v5
with:
repository: trinodb/trino
ref: ${{ steps.trino-pin.outputs.trino_sha }}
path: trino-src
- name: Build Trino artifacts from source (JDK 25)
- if: steps.spi-drift.outputs.drifted != 'true' &&
steps.trino-m2.outputs.cache-hit != 'true'
run: scripts/trino/bootstrap_trino.sh trino-src --skip-checkout
+ # Built on every run, from the same commit as the connector. Not cached:
the scheduled
+ # workflow's delete-all-caches job removes Actions caches before a later
run can restore them.
+ - name: Build the Trino server image from the pinned commit (JDK 25)
+ env:
+ TRINO_SHA: ${{ steps.trino-pin.outputs.trino_sha }}
+ run: docker/trino/build_trino_server_image.sh trino-src --image
"hudi-trino-server:${TRINO_SHA}" --arch amd64
+ - name: Free the Trino build tree
+ # The server image is in the local docker store and io.trino is in the
m2 by now;
+ # reclaim the disk before the connector build and the compose stack.
+ run: rm -rf trino-src
- name: Build hudi-trino connector (JDK 25)
- if: steps.spi-drift.outputs.drifted != 'true'
run:
mvn $MVN_ARGS -Phudi-trino -pl hudi-trino install
-Dmaven.test.skip=true
- name: Assemble trino-hudi plugin dir via in-repo shim (JDK 25)
- if: steps.spi-drift.outputs.drifted != 'true'
# package, NOT install: installing would shadow the real
# io.trino:trino-hudi release coordinates in the local m2 (the shim pom
# also hard-disables install via maven.install.skip).
@@ -217,23 +159,21 @@ jobs:
echo "Building shim against hudi version: $HUDI_VERSION"
mvn $MVN_ARGS -f docker/trino/shim/pom.xml clean package -DskipTests
-Ddep.hudi.version="$HUDI_VERSION"
- name: Build apachehudi/hudi-trino-e2e image
- if: steps.spi-drift.outputs.drifted != 'true'
- # The plugin dir is named after the version it was built at; the
server underneath is
- # the released trino.e2e.version.
+ # The plugin dir is named after trino.version; the server underneath
is the image built
+ # from the same pinned commit above.
env:
TRINO_VERSION: ${{ steps.trino-pin.outputs.trino_version }}
- E2E_VERSION: ${{ steps.trino-pin.outputs.e2e_version }}
+ TRINO_SHA: ${{ steps.trino-pin.outputs.trino_sha }}
run: |
PLUGIN_DIR="docker/trino/shim/target/trino-hudi-${TRINO_VERSION}"
# trino-maven-plugin 24 emits only the zip; explode it into the
plugin dir layout.
unzip -o -q "${PLUGIN_DIR}.zip" -d docker/trino/shim/target
- docker/trino/build_image.sh --plugin-dir "$PLUGIN_DIR"
--trino-version "${E2E_VERSION}"
+ docker/trino/build_image.sh --plugin-dir "$PLUGIN_DIR" --base-image
"hudi-trino-server:${TRINO_SHA}"
# Sanity: the shim must have produced a populated plugin dir with a
# service descriptor jar, or Trino cannot load the plugin at boot.
echo "plugin dir jar count: $(ls "$PLUGIN_DIR" | wc -l)"
ls "$PLUGIN_DIR"/*services*.jar
- name: Smoke-boot the Trino image standalone
- if: steps.spi-drift.outputs.drifted != 'true'
# Catches image-level boot failures (plugin load errors, bad etc/
config)
# ~30 min before the IT step would, with the full boot log on screen.
# --hostname trinocoordinator makes the baked discovery.uri
self-resolve.
@@ -260,7 +200,6 @@ jobs:
fi
docker rm -f trino-smoke
- name: Set up JDK 17 (restore for the IT run)
- if: steps.spi-drift.outputs.drifted != 'true'
# setup-java resets JAVA_HOME on each call; hudi-integ-test needs 17.
uses: actions/setup-java@v5
with:
@@ -268,7 +207,6 @@ jobs:
distribution: 'temurin'
architecture: x64
- name: Run Trino E2E ITs (JDK 17)
- if: steps.spi-drift.outputs.drifted != 'true'
run: |
# -DskipITs=false overrides the spark4.0 profile's skipITs=true
default
# (see root pom.xml). -Dcompose.profiles=trino starts the
profile-gated
diff --git a/docker/README.md b/docker/README.md
index a3e6aa1fc8ed..bcf8949c51d0 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -207,8 +207,10 @@ changes are needed for the current amd64 plus arm64 image
set in this repository
## Trino E2E image - `/trino`
The Trino E2E stack does not use the `hoodie/hadoop` image tree.
`docker/trino/` builds
-`apachehudi/hudi-trino-e2e` directly on top of the official `trinodb/trino`
image at the
-root pom's `trino.e2e.version`, baking in a locally-assembled native
`trino-hudi` plugin
+`apachehudi/hudi-trino-e2e` on top of a Trino server image built from the
pinned
+`trinodb/trino` commit (`trino.sha`, via
`docker/trino/build_trino_server_image.sh`; this is
+what CI does) or, for quick local runs, the released
`trinodb/trino:<trino.e2e.version>`,
+baking in a locally-assembled native `trino-hudi` plugin
directory and the E2E catalog config (`connector.name=hudi`, metastore at
`thrift://hivemetastore:9083`).
diff --git a/docker/trino/Dockerfile b/docker/trino/Dockerfile
index 891a41aa1ec7..752dfc93308d 100644
--- a/docker/trino/Dockerfile
+++ b/docker/trino/Dockerfile
@@ -15,8 +15,12 @@
# specific language governing permissions and limitations
# under the License.
+# CI passes TRINO_BASE_IMAGE: a server image built from the pinned trino.sha
+# (build_trino_server_image.sh). The released-image default is for quick local
runs and
+# only matches the connector when the pin is a release.
ARG TRINO_VERSION=483
-FROM trinodb/trino:${TRINO_VERSION}
+ARG TRINO_BASE_IMAGE=trinodb/trino:${TRINO_VERSION}
+FROM ${TRINO_BASE_IMAGE}
USER root
diff --git a/docker/trino/build_image.sh b/docker/trino/build_image.sh
index 9a2303ff7117..fc9bddbd7df5 100755
--- a/docker/trino/build_image.sh
+++ b/docker/trino/build_image.sh
@@ -20,10 +20,13 @@
# docker/trino/shim/target/trino-hudi-<v>, see docker/trino/shim/pom.xml) is
# staged into the build context at docker/trino/plugin/ (gitignored), then
# baked into the image.
-# Usage: ./build_image.sh --plugin-dir <path> [--trino-version <v>]
[--image-tag <t>]
-# Typical: ./build_image.sh --plugin-dir "$(dirname
"$0")/shim/target/trino-hudi-<trino.version>"
-# Note: --trino-version is the released Trino server image to build on top of
-# (the root pom's trino.e2e.version), not the version the plugin was built at.
+# Usage: ./build_image.sh --plugin-dir <path> [--base-image <image>]
[--trino-version <v>] [--image-tag <t>]
+# Typical: ./build_image.sh --plugin-dir "$(dirname
"$0")/shim/target/trino-hudi-<trino.version>" \
+# --base-image hudi-trino-server:<trino.sha>
+# Note: --base-image (e.g. the output of build_trino_server_image.sh, built
from the pinned
+# trino.sha) takes precedence over --trino-version. --trino-version is the
released-image
+# fallback, trinodb/trino:<v> (default: the root pom's trino.e2e.version), not
the version
+# the plugin was built at; it only boots when the pin's SPI matches that
release.
set -e
@@ -33,6 +36,7 @@ SCRIPT_DIR=$(cd $(dirname "$0") && pwd)
# Default values. The server version defaults to the root pom's
trino.e2e.version (the
# nightly pin-advance job keeps that current; a literal default here would
rot).
PLUGIN_DIR=""
+BASE_IMAGE=""
TRINO_VERSION=$(sed -n
's|.*<trino.e2e.version>\(.*\)</trino.e2e.version>.*|\1|p'
"$SCRIPT_DIR/../../pom.xml")
IMAGE_TAG="latest"
@@ -40,6 +44,7 @@ IMAGE_TAG="latest"
while [[ "$#" -gt 0 ]]; do
case $1 in
--plugin-dir) PLUGIN_DIR="$2"; shift ;;
+ --base-image) BASE_IMAGE="$2"; shift ;;
--trino-version) TRINO_VERSION="$2"; shift ;;
--image-tag) IMAGE_TAG="$2"; shift ;;
*) echo "Unknown parameter passed: $1"; exit 1 ;;
@@ -73,8 +78,14 @@ rm -rf "$STAGE_DIR"
cp -r "$PLUGIN_DIR" "$STAGE_DIR"
IMAGE="apachehudi/hudi-trino-e2e:${IMAGE_TAG}"
-echo "Building $IMAGE (TRINO_VERSION=${TRINO_VERSION})"
-docker build --build-arg TRINO_VERSION="${TRINO_VERSION}" -t "$IMAGE"
"$SCRIPT_DIR"
+BUILD_ARGS=(--build-arg TRINO_VERSION="${TRINO_VERSION}")
+if [ -n "$BASE_IMAGE" ]; then
+ echo "Building $IMAGE on base image ${BASE_IMAGE}"
+ BUILD_ARGS+=(--build-arg TRINO_BASE_IMAGE="${BASE_IMAGE}")
+else
+ echo "Building $IMAGE on released base image trinodb/trino:${TRINO_VERSION}"
+fi
+docker build "${BUILD_ARGS[@]}" -t "$IMAGE" "$SCRIPT_DIR"
# Clean up the staged plugin dir
echo "Cleaning up staged plugin dir '$STAGE_DIR'"
diff --git a/docker/trino/build_trino_server_image.sh
b/docker/trino/build_trino_server_image.sh
new file mode 100755
index 000000000000..8be4d5494967
--- /dev/null
+++ b/docker/trino/build_trino_server_image.sh
@@ -0,0 +1,145 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+##
+## Builds a local Trino server image from the trinodb/trino commit pinned by
+## <trino.sha> in the root pom, so the server and the hudi-trino connector
baked
+## on top of it (docker/trino/build_image.sh --base-image) come from the same
+## commit. The image is local-only and never published.
+##
+## Usage: build_trino_server_image.sh <path-to-trino-checkout> [--image
<name:tag>] [--arch <amd64|arm64>]
+##
+## <path-to-trino-checkout> a trinodb/trino git checkout at exactly
<trino.sha>
+## --image <name:tag> tag for the resulting image
+## (default: hudi-trino-server:<trino.sha>)
+## --arch <amd64|arm64> image architecture (default: the Docker
daemon's arch)
+##
+## Requires JDK 25, jq and docker on PATH.
+##
+
+set -euo pipefail
+
+HUDI_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+
+TRINO_REPO=""
+IMAGE=""
+ARCH=""
+
+while [[ $# -gt 0 ]]; do
+ case "$1" in
+ --image)
+ IMAGE="$2"
+ shift 2
+ ;;
+ --arch)
+ ARCH="$2"
+ shift 2
+ ;;
+ -h|--help)
+ grep '^##' "$0" | sed 's/^## \{0,1\}//'
+ exit 0
+ ;;
+ *)
+ if [[ -z "$TRINO_REPO" ]]; then
+ TRINO_REPO="$1"
+ shift
+ else
+ echo "ERROR: unexpected argument: $1" >&2
+ exit 1
+ fi
+ ;;
+ esac
+done
+
+if [[ -z "$TRINO_REPO" || ! -d "$TRINO_REPO/.git" && ! -f "$TRINO_REPO/.git"
]]; then
+ echo "ERROR: first argument must be a trinodb/trino git checkout" >&2
+ echo "Usage: $0 <path-to-trino-checkout> [--image <name:tag>] [--arch
<amd64|arm64>]" >&2
+ exit 1
+fi
+TRINO_REPO="$(cd "$TRINO_REPO" && pwd)"
+
+# Keep each property on one line in the root pom; this sed depends on it.
+TRINO_SHA=$(sed -n 's|.*<trino.sha>\(.*\)</trino.sha>.*|\1|p'
"$HUDI_ROOT/pom.xml")
+TRINO_VERSION=$(sed -n 's|.*<trino.version>\(.*\)</trino.version>.*|\1|p'
"$HUDI_ROOT/pom.xml")
+if [[ -z "$TRINO_SHA" || -z "$TRINO_VERSION" ]]; then
+ echo "ERROR: could not read <trino.sha>/<trino.version> from
$HUDI_ROOT/pom.xml" >&2
+ exit 1
+fi
+
+# JDK gate: trino at head enforces JDK 25.
+JAVA_MAJOR=$(java -version 2>&1 | awk -F[\".] '/version/ {print $2}')
+if [[ "$JAVA_MAJOR" != "25" ]]; then
+ echo "ERROR: JDK 25 required to build trino (found major version:
${JAVA_MAJOR:-unknown})." >&2
+ echo "Hint: export JAVA_HOME=\$(/usr/libexec/java_home -v 25)" >&2
+ exit 1
+fi
+
+for tool in jq docker; do
+ if ! command -v "$tool" > /dev/null 2>&1; then
+ echo "ERROR: $tool is required on PATH (trino's core/docker/build.sh needs
jq and docker)" >&2
+ exit 1
+ fi
+done
+
+ACTUAL_SHA=$(git -C "$TRINO_REPO" rev-parse HEAD)
+if [[ "$ACTUAL_SHA" != "$TRINO_SHA" ]]; then
+ echo "ERROR: $TRINO_REPO is at $ACTUAL_SHA, but the root pom pins
<trino.sha>$TRINO_SHA</trino.sha>." >&2
+ echo "Hint: git -C $TRINO_REPO -c advice.detachedHead=false checkout
--detach $TRINO_SHA" >&2
+ exit 1
+fi
+
+ACTUAL_VERSION=$("$TRINO_REPO/mvnw" -q -N -f "$TRINO_REPO/pom.xml"
help:evaluate -Dexpression=project.version -DforceStdout)
+if [[ "$ACTUAL_VERSION" != "$TRINO_VERSION" ]]; then
+ echo "ERROR: trino at pinned sha $TRINO_SHA has version $ACTUAL_VERSION, but
the root pom says <trino.version>$TRINO_VERSION</trino.version>." >&2
+ echo "The pin properties must advance together; fix the pom or your
checkout." >&2
+ exit 1
+fi
+
+if [[ -z "$ARCH" ]]; then
+ ARCH=$(docker version --format '{{.Server.Arch}}')
+fi
+if [[ "$ARCH" != "amd64" && "$ARCH" != "arm64" ]]; then
+ echo "ERROR: --arch must be amd64 or arm64 (got: ${ARCH:-empty})" >&2
+ exit 1
+fi
+
+if [[ -z "$IMAGE" ]]; then
+ IMAGE="hudi-trino-server:${TRINO_SHA}"
+fi
+
+# No MAVEN_OPTS default: trino's .mvn/jvm.config asks for -Xmx8192m and
Maven's launcher
+# appends MAVEN_OPTS after it, so a default here would halve the heap for the
build below.
+
+# Why the whole repo (minus docs) instead of -pl :trino-server -am: the
server's provisio
+# descriptors pull in plugin zips that are not Maven dependencies, so -am
would not build
+# them. Why the full trino-server tarball instead of trino-server-core: only
the full server
+# packages plugin/hudi, whose hdfs/ jar set docker/trino/Dockerfile carries
over.
+# package, NOT install: ~/.m2/repository/io/trino belongs to
scripts/trino/bootstrap_trino.sh.
+echo "Building the trino server at $TRINO_SHA (version $TRINO_VERSION)"
+(cd "$TRINO_REPO" && ./mvnw package -B -DskipTests -Dair.check.skip-all=true
-Dmaven.source.skip=true -T1C \
+ -pl '!:trino-docs')
+
+# Without -r, trino's build.sh builds from the locally built server tarball
and CLI jar and
+# tags <prefix>:<project.version>-<arch>; -x skips its image tests.
+BUILD_PREFIX="hudi-trino-server-build"
+echo "Building the trino server image for $ARCH"
+(cd "$TRINO_REPO/core/docker" && ./build.sh -a "$ARCH" -t "$BUILD_PREFIX" -x)
+
+docker tag "${BUILD_PREFIX}:${TRINO_VERSION}-${ARCH}" "$IMAGE"
+docker rmi "${BUILD_PREFIX}:${TRINO_VERSION}-${ARCH}" > /dev/null
+
+echo "Done: $IMAGE"
diff --git a/docker/trino/overlay-entrypoint.sh
b/docker/trino/overlay-entrypoint.sh
index 2ba9b517e26a..19e07ffcaf48 100755
--- a/docker/trino/overlay-entrypoint.sh
+++ b/docker/trino/overlay-entrypoint.sh
@@ -24,6 +24,11 @@
# image's baked-in trino-hudi plugin with it (rm -rf then copy), so plugin
# iterations need only a rebuild of that dir plus a container restart, not a
# docker image rebuild. Otherwise the image-baked plugin is used as-is.
+#
+# This runs on released images (<= 483, which still shipped bin/run-trino) and
on
+# images built from the pinned trino.sha (after trinodb/trino f0d1f3c982e,
which
+# removed run-trino in favor of launcher in the CMD, on a hardened base image
+# without grep or find). Keep it bash builtins only, apart from cp/rm/mkdir.
set -euo pipefail
OVERLAY=/opt/hudi-plugin-overlay
@@ -32,7 +37,14 @@ PLUGIN_DIR=/usr/lib/trino/plugin/hudi
# The overlay counts as present only if it holds at least one jar: the compose
# default mount is docker/trino/empty-overlay, whose .gitkeep must not trigger
# a wipe of the baked-in plugin.
-if [ -d "$OVERLAY" ] && [ -n "$(find "$OVERLAY" -name '*.jar' -print -quit
2>/dev/null)" ]; then
+# dotglob mirrors find, which also descends into hidden dirs.
+OVERLAY_JARS=()
+if [ -d "$OVERLAY" ]; then
+ shopt -s globstar nullglob dotglob
+ OVERLAY_JARS=("$OVERLAY"/**/*.jar)
+ shopt -u globstar nullglob dotglob
+fi
+if [ "${#OVERLAY_JARS[@]}" -gt 0 ]; then
echo "Applying trino-hudi plugin overlay from $OVERLAY (fully replacing
$PLUGIN_DIR)"
rm -rf "$PLUGIN_DIR"
mkdir -p "$PLUGIN_DIR"
@@ -49,4 +61,20 @@ if [ ! -d "$PLUGIN_DIR/hdfs" ] && [ -d /opt/hudi-hdfs-lib ];
then
cp -r /opt/hudi-hdfs-lib "$PLUGIN_DIR/hdfs"
fi
-exec /usr/lib/trino/bin/run-trino
+# Inlined from the removed bin/run-trino wrapper. The node.id check mirrors its
+# `grep -s -q 'node.id' /etc/trino/node.properties`: a missing or unreadable
file
+# means not set.
+launcher_opts=(--etc-dir /etc/trino)
+NODE_ID_SET=false
+if [ -r /etc/trino/node.properties ]; then
+ while IFS= read -r line || [ -n "$line" ]; do
+ if [[ "$line" == *node.id* ]]; then
+ NODE_ID_SET=true
+ break
+ fi
+ done < /etc/trino/node.properties
+fi
+if [ "$NODE_ID_SET" != "true" ]; then
+ launcher_opts+=("-Dnode.id=${HOSTNAME}")
+fi
+exec /usr/lib/trino/bin/launcher run "${launcher_opts[@]}" "$@"
diff --git a/hudi-trino/README.md b/hudi-trino/README.md
index 763eade3cc70..d9bd21680f9d 100644
--- a/hudi-trino/README.md
+++ b/hudi-trino/README.md
@@ -66,9 +66,9 @@ project mirroring the upstream `trinodb/trino`
`plugin/trino-hudi` shim planned
RFC-105 (not yet released upstream). CI runs the same flow via
`.github/workflows/hudi_trino_e2e.yml`.
-The plugin is built at the pinned `trino.version` while the server image is
the released
-`trino.e2e.version`; CI auto-skips the suite during SPI drift windows (SPI or
filesystem changes
-between the two).
+CI builds the Trino server image from the pinned `trino.sha`
+(`docker/trino/build_trino_server_image.sh`), so the plugin and the server
always come from
+the same commit.
Local flow (after bootstrap):
@@ -91,9 +91,16 @@ TRINO_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xm
mvn -f docker/trino/shim/pom.xml clean package -DskipTests
-Ddep.hudi.version="$HUDI_VERSION"
unzip -o -q "docker/trino/shim/target/trino-hudi-$TRINO_VERSION.zip" -d
docker/trino/shim/target # trino-maven-plugin 24 emits only the zip
-# 4. Build the Trino image (locally tagged; never published). The base server
defaults to
-# trino.e2e.version; pass --trino-version to override it.
-docker/trino/build_image.sh --plugin-dir
"docker/trino/shim/target/trino-hudi-$TRINO_VERSION"
+# 3a. Optional, JDK 25: build the server image from a trinodb/trino checkout
at trino.sha
+# (the same one bootstrap used). Builds the whole trino repo, so it takes
a while.
+docker/trino/build_trino_server_image.sh /path/to/trino
+TRINO_SHA=$(sed -n 's|.*<trino.sha>\(.*\)</trino.sha>.*|\1|p' pom.xml)
+
+# 4. Build the Trino image (locally tagged; never published) on the server
from 3a.
+# Without --base-image it falls back to the released
trinodb/trino:<trino.e2e.version>
+# (or --trino-version), which only boots when the pin's SPI matches that
release.
+docker/trino/build_image.sh --plugin-dir
"docker/trino/shim/target/trino-hudi-$TRINO_VERSION" \
+ --base-image "hudi-trino-server:$TRINO_SHA"
# 5. JDK 17: run the suite (only the spark402 compose pair has the trino
service)
mvn verify -pl hudi-integ-test -Dscala-2.13 -Dscala.binary.version=2.13
-Dspark4.0 \
diff --git a/pom.xml b/pom.xml
index 11dc55c3355f..787a7b445eb6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -140,7 +140,8 @@
<presto.version>0.273</presto.version>
<!-- trino.sha is the exact trinodb/trino commit hudi-trino builds
against; its project
version must equal trino.version and the two advance only together.
trino.e2e.version is
- the latest released Trino (e2e server image, trino-jdbc for the integ
tests); the nightly
+ the latest released Trino (trino-jdbc for the integ tests, and the
fallback server image
+ for local E2E runs; CI's E2E server image is built from trino.sha);
the nightly
pin-advance job refreshes all three, humans merge its bot/trino-pin
branch. Keep each
property on one line: the workflows and
scripts/trino/bootstrap_trino.sh read them with sed. -->
<trino.version>484-SNAPSHOT</trino.version>
diff --git a/release/release_guide.md b/release/release_guide.md
index ae49b2d281a7..e48599cd01b9 100644
--- a/release/release_guide.md
+++ b/release/release_guide.md
@@ -306,7 +306,8 @@ Source Release step) -- otherwise the voted tarball ships a
`-SNAPSHOT` Trino pi
and revert them forward on the release branch only, never on master.
4. On the release branch set `trino.version=NNN`, `trino.sha=TAG_SHA` and
`trino.e2e.version=NNN` in the root
pom, the `<parent>` version in `docker/trino/shim/pom.xml`, and the `ARG
TRINO_VERSION` default in
- `docker/trino/Dockerfile` (`build_image.sh` reads `trino.e2e.version` from
the root pom). Re-check SPI-surface-coupled
+ `docker/trino/Dockerfile` (`trino.e2e.version` is the trino-jdbc version
and the released fallback base image that
+ `build_image.sh` uses without `--base-image`). Re-check SPI-surface-coupled
dependency scopes against `NNN` (e.g. `jts-core` is `provided` because it
joined the Trino SPI surface in 482;
the shim's SpiDependencyChecker fails the build loudly if a scope no longer
matches the target release).
5. Verify the released Trino resolves from Central against an empty local
repository
@@ -337,8 +338,8 @@ Source Release step) -- otherwise the voted tarball ships a
`-SNAPSHOT` Trino pi
Locally the script exits 1 on drift and prints the same table.
If it reports drift, bump the matching version properties in
`hudi-trino/pom.xml` on the release branch and rerun
until it is clean.
-7. CI and the E2E workflow then run with zero SPI drift; the staging deploy
flow in "Build a release candidate"
- is unchanged.
+7. The E2E workflow builds its Trino server image from the pinned tag commit,
as on master; the staging deploy flow
+ in "Build a release candidate" is unchanged.
## Verify that a Release Build Works
