wombatu-kun commented on code in PR #19217:
URL: https://github.com/apache/hudi/pull/19217#discussion_r3695303467
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -0,0 +1,157 @@
+name: Hudi Trino E2E
+
+on:
+ push:
+ branches:
+ - master
+ - 'release-*'
+ # TEMPORARY: validate the workflow on the dev branch; drop this line
+ # (and the commit that added it) before merging upstream.
+ - trino-e2e-testcontainers
Review Comment:
Only the squash merge button is enabled in `.asf.yaml`, so there is no
separate commit to drop at merge time and this line ships to master unless it
comes off the branch head first. Please strip it before this goes in.
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -0,0 +1,157 @@
+name: Hudi Trino E2E
+
+on:
+ push:
+ branches:
+ - master
+ - 'release-*'
+ # TEMPORARY: validate the workflow on the dev branch; drop this line
+ # (and the commit that added it) before merging upstream.
+ - trino-e2e-testcontainers
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ pull_request:
+ branches:
+ - master
+ - 'release-*'
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ workflow_dispatch:
+
+concurrency:
+ group: hudi-trino-e2e-${{ github.ref }}
+ cancel-in-progress: ${{ !contains(github.ref, 'master') &&
!contains(github.ref, 'release-') }}
+
+env:
+ MVN_ARGS: -e -ntp -B -V -Dgpg.skip -Djacoco.skip -Pwarn-log
Review Comment:
This drops the `aether.connector.http.retryHandler.*` flags that `bot.yml`
carries, and the JDK 17 step here is exactly the cold-cache full-reactor build
those were added for. Copy the retry block over from `bot.yml`'s `MVN_ARGS`.
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -0,0 +1,157 @@
+name: Hudi Trino E2E
+
+on:
+ push:
+ branches:
+ - master
+ - 'release-*'
+ # TEMPORARY: validate the workflow on the dev branch; drop this line
+ # (and the commit that added it) before merging upstream.
+ - trino-e2e-testcontainers
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ pull_request:
+ branches:
+ - master
+ - 'release-*'
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
Review Comment:
`ITTestTrinoCustomType` also drives `sparksql-blob-type-sql`,
`sparksql-blob-type-df`, `sparksql-vector-type-sql`,
`sparksql-variant-type-sql` and `setup_demo_container.sh`, so editing any of
those cannot trigger this workflow and the break only surfaces on the next
unrelated hudi-trino PR. Widen this entry to `docker/demo/**`.
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -0,0 +1,157 @@
+name: Hudi Trino E2E
+
+on:
+ push:
+ branches:
+ - master
+ - 'release-*'
+ # TEMPORARY: validate the workflow on the dev branch; drop this line
+ # (and the commit that added it) before merging upstream.
+ - trino-e2e-testcontainers
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ pull_request:
+ branches:
+ - master
+ - 'release-*'
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ workflow_dispatch:
+
+concurrency:
+ group: hudi-trino-e2e-${{ github.ref }}
+ cancel-in-progress: ${{ !contains(github.ref, 'master') &&
!contains(github.ref, 'release-') }}
+
+env:
+ MVN_ARGS: -e -ntp -B -V -Dgpg.skip -Djacoco.skip -Pwarn-log
+ SCALA_PROFILE: -Dscala-2.13 -Dscala.binary.version=2.13
+ COMPOSE_PREFIX: docker-compose_hadoop340_hive2310_spark402
+
+jobs:
+ trino-e2e:
+ # Testcontainers E2E for the RFC-105 native trino-hudi connector: builds
+ # hudi-trino at HEAD, assembles the plugin dir via the in-repo shim
+ # (docker/trino/shim, standing in for the not-yet-released upstream
+ # trinodb/trino plugin/trino-hudi shim), bakes it into a local
+ # apachehudi/hudi-trino_481 image, and runs ITTestTrino* against the
+ # spark402 compose stack (the only pair with the trinocoordinator service).
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v5
+ - name: Free disk space
+ run: |
+ sudo rm -rf /usr/share/dotnet
+ sudo rm -rf /usr/local/lib/android
+ sudo rm -rf /opt/ghc
+ sudo rm -rf /usr/local/share/boost
+ docker system prune --all --force --volumes
+ - name: Pre-pull compose images (fails fast if not published)
+ run: |
+ # Surface a missing sparkadhoc image before the long Maven install.
The
+ # remaining stack images are pulled by docker-compose at test time;
the
+ # trino image is built locally below, never pulled.
+ docker pull
apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-sparkadhoc_4.0.2:latest
+ - name: Set up JDK 17
+ uses: actions/setup-java@v5
+ with:
+ java-version: '17'
+ distribution: 'temurin'
+ architecture: x64
+ cache: maven
+ - name: Build and install Hudi artifacts (JDK 17)
+ # Full reactor: the compose containers mount the workspace and the
tests
+ # use bundles staged by the -Pintegration-tests build (e.g.
+ # docker/hoodie/hadoop/hive_base/target/hoodie-spark-bundle.jar).
+ run:
+ mvn clean install -T 2 $SCALA_PROFILE -Dspark4.0 -Dflink1.20
-Pintegration-tests -DskipTests=true -Ddocker.compose.skip=true $MVN_ARGS
+ - name: Set up JDK 25
+ uses: actions/setup-java@v5
+ with:
+ java-version: '25'
+ distribution: 'temurin'
+ cache: maven
+ - name: Build hudi-trino connector (JDK 25)
+ # No trinodb/trino checkout needed: the unpublished Trino test-jars sit
+ # behind the off-by-default hudi-trino-tests profile and packaging
+ # resolves entirely from Maven Central.
+ run:
+ mvn $MVN_ARGS -Phudi-trino -pl hudi-trino install
-Dmaven.test.skip=true
+ - name: Assemble trino-hudi plugin dir via in-repo shim (JDK 25)
+ # package, NOT install: installing would shadow the real
+ # io.trino:trino-hudi release coordinates in the local m2.
+ run:
+ mvn $MVN_ARGS -f docker/trino/shim/pom.xml clean package -DskipTests
+ - name: Build apachehudi/hudi-trino_481 image
+ run: |
+ docker/trino/build_image.sh --plugin-dir
docker/trino/shim/target/trino-hudi-481
+ # Sanity: the shim must have produced a populated plugin dir with a
+ # service descriptor jar, or Trino cannot load the plugin at boot.
+ echo "plugin dir jar count: $(ls
docker/trino/shim/target/trino-hudi-481 | wc -l)"
+ ls docker/trino/shim/target/trino-hudi-481/*services*.jar
+ - name: Smoke-boot the Trino image standalone
+ # Catches image-level boot failures (plugin load errors, bad etc/
config)
+ # ~30 min before the IT step would, with the full boot log on screen.
+ # --hostname trinocoordinator makes the baked discovery.uri
self-resolve.
+ run: |
+ docker run -d --name trino-smoke --hostname trinocoordinator \
+ apachehudi/hudi-trino_481:latest
+ ok=""
+ for i in $(seq 1 18); do
+ if [ "$(docker inspect -f '{{.State.Running}}' trino-smoke)" !=
"true" ]; then
+ echo "trino-smoke container died during startup" >&2
+ break
+ fi
+ if docker exec trino-smoke trino --server localhost:8080 \
+ --execute "SELECT 1" >/dev/null 2>&1; then
+ ok=1; echo "Trino answered SELECT 1 (attempt $i)"; break
+ fi
+ sleep 10
+ done
+ if [ -z "$ok" ]; then
+ echo "==== trino-smoke boot log ===="
+ docker logs trino-smoke 2>&1 | tail -200
+ docker rm -f trino-smoke >/dev/null 2>&1 || true
+ exit 1
+ fi
+ docker rm -f trino-smoke
+ - name: Set up JDK 17 (restore for the IT run)
+ # setup-java resets JAVA_HOME on each call; hudi-integ-test needs 17.
+ uses: actions/setup-java@v5
+ with:
+ java-version: '17'
+ distribution: 'temurin'
+ architecture: x64
+ - name: Run Trino E2E ITs (JDK 17)
+ run: |
+ # -DskipITs=false overrides the spark4.0 profile's skipITs=true
default
+ # (see root pom.xml). -Dcompose.profiles=trino starts the
profile-gated
+ # trinocoordinator service; without it every ITTestTrino* class
skips.
+ mvn verify $SCALA_PROFILE -Dspark4.0 -Pintegration-tests \
+ -pl hudi-integ-test \
+ -DskipITs=false \
+ -Ddocker.compose.skip=true \
+ -Dit.test='ITTestTrino*' \
+ -Dcompose.profiles=trino \
+ -Dspark.docker.compose.prefix=$COMPOSE_PREFIX \
+ $MVN_ARGS
+ - name: Dump failsafe test outputs on failure
+ # Surefire redirects test stdout (incl. the streamed trinocoordinator
+ # boot/query logs) into per-class output files; print their tails so
+ # server-side failures are readable straight from the workflow log.
+ if: failure()
+ run: |
+ for f in hudi-integ-test/target/failsafe-reports/*-output.txt; do
Review Comment:
`redirectTestOutputToFile` is not set anywhere in this repo and failsafe
defaults it to false, so `failsafe-reports/*-output.txt` never exists and this
loop always no-ops. Either add `-Dsurefire.redirectTestOutputToFile=true` to
the IT step or drop the step, since that output already reaches the workflow
log.
##########
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/trino/ITTestTrinoCustomType.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ2.testcontainers.trino;
+
+import org.apache.hudi.integ2.testcontainers.ITTestBaseTestcontainers;
+import org.apache.hudi.integ2.testcontainers.ITTestCustomTypeHiveSync;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInstance;
+
+import static org.apache.hudi.integ2.testcontainers.TestcontainersConfig.Paths;
+
+/**
+ * Trino read coverage for Hudi's custom logical types (BLOB struct, VECTOR
+ * fixed_len_byte_array, VARIANT), complementing {@link
ITTestCustomTypeHiveSync}
+ * which asserts the same fixtures round-trip through the Hive serde. A flip in
+ * either direction (e.g. VECTOR decoded as array<float> instead of
+ * binary, BLOB struct field projection broken, VARIANT row count off) shows up
+ * here.
+ *
+ * <p>This test reuses the same {@code sparksql-*-sql.commands} fixtures that
+ * {@code ITTestCustomTypeHiveSync} drives, so the two tests can run in either
+ * order without cross-contamination (each has its own {@code @BeforeAll} that
+ * re-seeds, and an {@code @AfterAll} that cleans up).
+ *
+ * <p>VARIANT seeding and tests only fire on a Spark 4.x compose; on Spark 3.5
+ * the BLOB and VECTOR coverage still runs.
+ */
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class ITTestTrinoCustomType extends ITTestBaseTestcontainers {
+
+ private static final String BLOB_TEST_PATH =
"/user/hive/warehouse/blob_test";
+ private static final String BLOB_TEST_DF_PATH =
"/user/hive/warehouse/blob_test_df";
+ private static final String VECTOR_TEST_PATH =
"/user/hive/warehouse/vector_test";
+ private static final String VARIANT_TEST_PATH =
"/user/hive/warehouse/variant_test";
+ private static final String SPARKSQL_BLOB_TYPE_SQL_COMMANDS =
+ Paths.DEMO_DIR + "/sparksql-blob-type-sql.commands";
+ private static final String SPARKSQL_BLOB_TYPE_DF_COMMANDS =
+ Paths.DEMO_DIR + "/sparksql-blob-type-df.commands";
+ private static final String SPARKSQL_VECTOR_TYPE_SQL_COMMANDS =
+ Paths.DEMO_DIR + "/sparksql-vector-type-sql.commands";
+ private static final String SPARKSQL_VARIANT_TYPE_SQL_COMMANDS =
+ Paths.DEMO_DIR + "/sparksql-variant-type-sql.commands";
+
+ @BeforeAll
+ public void setupOnce() throws Exception {
+ assumeTrinoProfile();
+ initializeServices();
+ waitForHdfs();
+ sparkAdhoc1.executeShellCommand("/bin/bash " +
Paths.DEMO_SETUP).expectToSucceed();
+ sparkAdhoc1.executeSQLFile(SPARKSQL_BLOB_TYPE_SQL_COMMANDS)
+ .expectToSucceed()
+ .assertStdOutContainsLine("BLOB_SQL_TEST_SUCCESS");
+ // The DF fixture writes blob_test_df with the INLINE branch of the BLOB
struct
+ // (data field non-null, reference null). The SQL fixture exercises only
the
+ // OUT_OF_LINE branch, so seeding both gives Trino read coverage of both
shapes.
+ sparkAdhoc1.executeSQLFile(SPARKSQL_BLOB_TYPE_DF_COMMANDS)
+ .expectToSucceed()
+ .assertStdOutContainsLine("BLOB_DF_TEST_SUCCESS");
+ sparkAdhoc1.executeSQLFile(SPARKSQL_VECTOR_TYPE_SQL_COMMANDS)
+ .expectToSucceed()
+ .assertStdOutContainsLine("VECTOR_SQL_TEST_SUCCESS");
+ if (isSpark4Compose()) {
+ // VARIANT type is Spark 4.x only - guard the seed so the BLOB/VECTOR
+ // coverage still runs on a Spark 3.5 stack.
+ sparkAdhoc1.executeSQLFile(SPARKSQL_VARIANT_TYPE_SQL_COMMANDS)
+ .expectToSucceed()
+ .assertStdOutContainsLine("VARIANT_SQL_TEST_SUCCESS");
+ }
+ trino.waitUntilReady();
+ }
+
+ @AfterAll
+ public void clean() throws Exception {
+ // -f silently skips non-existent paths so the variant_test cleanup is safe
+ // even on Spark 3.5 runs where the table was never created.
+ sparkAdhoc1.executeShellCommand("hdfs dfs -rm -R -f "
Review Comment:
`clean` runs even when `assumeTrinoProfile` aborted `setupOnce` - JUnit
invokes `@AfterAll` regardless - so `sparkAdhoc1` is still null and the
intended skip surfaces as an NPE failure, same in `ITTestTrinoStockTicks`.
Early-return from `clean` when `sparkAdhoc1` is null.
##########
docker/demo/sparksql-stock-ticks-trino.commands:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Self-contained COW + MOR stock-ticks seed for the Trino E2E tests
+// (ITTestTrinoStockTicks). Mirrors the docker/demo/trino-batch1.commands data
+// shape without the Kafka/streaming pipeline, which integ2 does not exercise.
+// ts stays STRING so Trino's CSV_UNQUOTED output matches the test's exact
+// row assertion: GOOG,2018-08-31 10:29:00,6330,1230.5,1230.5
+spark.sql("""
+ CREATE TABLE stock_ticks_cow (
+ symbol STRING,
+ ts STRING,
+ volume LONG,
+ open DOUBLE,
+ close DOUBLE,
+ dt STRING
+ ) USING hudi
+ PARTITIONED BY (dt)
+ LOCATION '/user/hive/warehouse/stock_ticks_cow'
+ TBLPROPERTIES (
+ 'primaryKey' = 'symbol',
+ 'preCombineField' = 'ts',
+ 'hoodie.datasource.hive_sync.enable' = 'true',
+ 'hoodie.datasource.hive_sync.database' = 'default',
+ 'hoodie.datasource.hive_sync.table' = 'stock_ticks_cow',
+ 'hoodie.datasource.hive_sync.jdbcurl' = 'jdbc:hive2://hiveserver:10000/',
+ 'hoodie.datasource.hive_sync.mode' = 'jdbc',
+ 'hoodie.datasource.hive_sync.partition_fields' = 'dt',
+ 'hoodie.datasource.hive_sync.partition_extractor_class' =
'org.apache.hudi.hive.MultiPartKeysValueExtractor',
+ 'hoodie.datasource.hive_sync.username' = 'hive',
+ 'hoodie.datasource.hive_sync.password' = 'hive'
+ )
+""")
+
+spark.sql("INSERT INTO stock_ticks_cow VALUES ('GOOG', '2018-08-31 10:29:00',
6330, 1230.5, 1230.5, '2018-08-31')")
+spark.sql("select symbol, ts, volume, open, close from
stock_ticks_cow").show(10, false)
+println("STOCK_TICKS_COW_SETUP_SUCCESS")
+
+// MOR variant: identical schema and seed row. 'type' = 'mor' makes hive sync
+// register stock_ticks_mor_ro / stock_ticks_mor_rt; the initial insert writes
+// parquet base files, so the _ro view already sees the row.
+spark.sql("""
+ CREATE TABLE stock_ticks_mor (
+ symbol STRING,
+ ts STRING,
+ volume LONG,
+ open DOUBLE,
+ close DOUBLE,
+ dt STRING
+ ) USING hudi
+ PARTITIONED BY (dt)
+ LOCATION '/user/hive/warehouse/stock_ticks_mor'
+ TBLPROPERTIES (
+ 'type' = 'mor',
+ 'primaryKey' = 'symbol',
+ 'preCombineField' = 'ts',
+ 'hoodie.datasource.hive_sync.enable' = 'true',
+ 'hoodie.datasource.hive_sync.database' = 'default',
+ 'hoodie.datasource.hive_sync.table' = 'stock_ticks_mor',
+ 'hoodie.datasource.hive_sync.jdbcurl' = 'jdbc:hive2://hiveserver:10000/',
+ 'hoodie.datasource.hive_sync.mode' = 'jdbc',
+ 'hoodie.datasource.hive_sync.partition_fields' = 'dt',
+ 'hoodie.datasource.hive_sync.partition_extractor_class' =
'org.apache.hudi.hive.MultiPartKeysValueExtractor',
+ 'hoodie.datasource.hive_sync.username' = 'hive',
+ 'hoodie.datasource.hive_sync.password' = 'hive'
+ )
+""")
+
+spark.sql("INSERT INTO stock_ticks_mor VALUES ('GOOG', '2018-08-31 10:29:00',
6330, 1230.5, 1230.5, '2018-08-31')")
Review Comment:
`stock_ticks_mor` gets a single INSERT, so it has base files only and the
`_ro` tests traverse the same read path as the COW ones - no MOR-specific
signal, and `stock_ticks_mor_rt` is never queried. One follow-up upsert on the
same key plus an `_rt` assertion would make the MOR row earn its runtime;
follow-up, not a blocker.
##########
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/trino/ITTestTrinoStockTicks.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ2.testcontainers.trino;
+
+import org.apache.hudi.integ2.testcontainers.ITTestBaseTestcontainers;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInstance;
+
+import static org.apache.hudi.integ2.testcontainers.TestcontainersConfig.Paths;
+
+/**
+ * End-to-end coverage that the native trino-hudi connector can read both COW
and
+ * MOR tables that came from Spark + Hive sync. Mirrors the historical
+ * {@code docker/demo/trino-batch1.commands} demo flow but uses a
self-contained
Review Comment:
With this in place the legacy path is fully dead: `ITTestBase` still defines
`TRINO_COORDINATOR_URL` at `trino-coordinator-1:8091`, a service no compose
file has any more, and `ITTestHoodieDemo` still copies the three
`docker/demo/trino-*.commands`. Retire them here, or is that still parked on
HUDI-8269/8270?
##########
docker/trino/build_image.sh:
##########
@@ -0,0 +1,77 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Builds the apachehudi/hudi-trino_<version> image with a locally-built
+# trino-hudi plugin baked in. The plugin dir (typically the in-repo shim's
+# docker/trino/shim/target/trino-hudi-<v>, see docker/trino/shim/pom.xml) is
+# staged into the build context at docker/trino/plugin/ (gitignored), then
+# baked into the image.
+# Usage: ./build_image.sh --plugin-dir <path> [--trino-version <v>]
[--image-tag <t>]
+# Typical: ./build_image.sh --plugin-dir "$(dirname
"$0")/shim/target/trino-hudi-481"
+# Note: --trino-version must match the shim pom's parent version and the root
+# pom's trino.version property.
+
+set -e
+
+# Default values
+PLUGIN_DIR=""
+TRINO_VERSION="481"
+IMAGE_TAG="latest"
+
+# Parse command-line arguments
+while [[ "$#" -gt 0 ]]; do
+ case $1 in
+ --plugin-dir) PLUGIN_DIR="$2"; shift ;;
+ --trino-version) TRINO_VERSION="$2"; shift ;;
+ --image-tag) IMAGE_TAG="$2"; shift ;;
+ *) echo "Unknown parameter passed: $1"; exit 1 ;;
+ esac
+ shift
+done
+
+# Directory of this script, so the build context path is stable regardless of
cwd
+SCRIPT_DIR=$(cd $(dirname "$0") && pwd)
+
+# Validate --plugin-dir: required, must exist and be non-empty
+if [ -z "$PLUGIN_DIR" ]; then
+ echo "Error: --plugin-dir <path> is required (the locally-built trino-hudi
plugin directory)." >&2
+ exit 1
+fi
+if [ ! -d "$PLUGIN_DIR" ]; then
+ echo "Error: plugin dir '$PLUGIN_DIR' does not exist." >&2
+ exit 1
+fi
+if [ -z "$(ls -A "$PLUGIN_DIR" 2>/dev/null)" ]; then
+ echo "Error: plugin dir '$PLUGIN_DIR' is empty." >&2
+ exit 1
+fi
+
+# Stage the plugin into the build context (plugin/ must be IN the context to
be COPY-able)
+STAGE_DIR="$SCRIPT_DIR/plugin"
+echo "Staging plugin from '$PLUGIN_DIR' into '$STAGE_DIR'"
+rm -rf "$STAGE_DIR"
+cp -r "$PLUGIN_DIR" "$STAGE_DIR"
+
+IMAGE="apachehudi/hudi-trino_${TRINO_VERSION}:${IMAGE_TAG}"
+echo "Building $IMAGE (TRINO_VERSION=${TRINO_VERSION})"
+docker build --build-arg TRINO_VERSION="${TRINO_VERSION}" -t "$IMAGE"
"$SCRIPT_DIR"
Review Comment:
`docker/trino/` has no `.dockerignore`, so the build context ships
`shim/target/` - which holds the very plugin dir just copied into `plugin/` -
on every build. Add `docker/trino/.dockerignore` with `shim/`; follow-up, not a
blocker.
##########
docker/trino/shim/pom.xml:
##########
@@ -0,0 +1,157 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<!--
+ In-repo mirror of the RFC-105 Trino-side shim (trinodb/trino
plugin/trino-hudi),
+ which is not yet released upstream. It exists so this repo can assemble the
+ deployable Trino plugin directory (hudi-trino jar + runtime deps + service
+ descriptor) for the E2E docker image without a trinodb/trino checkout, and it
+ stays useful afterwards: CI must test hudi-trino at HEAD while the upstream
shim
+ pins released versions.
+
+ Standalone project - NOT part of the Hudi reactor and never deployed.
+
+ Build (JDK 25, after installing hudi-trino into the local m2):
+ mvn -f docker/trino/shim/pom.xml clean package -DskipTests
+ Output plugin dir: target/trino-hudi-<trino.version>/
+ Use `package`, never `install`: installing would shadow the real
+ io.trino:trino-hudi release coordinates in the local repository.
+
+ The parent version below must stay in sync with trino.version in the root
pom,
+ the TRINO_VERSION ARG in docker/trino/Dockerfile, and the compose image tag.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>io.trino</groupId>
+ <artifactId>trino-root</artifactId>
+ <version>481</version>
+ <!-- Resolve the parent from Maven Central, never the file system. -->
+ <relativePath />
+ </parent>
+
+ <artifactId>trino-hudi</artifactId>
+ <packaging>trino-plugin</packaging>
+ <description>Trino - Hudi connector plugin assembly (in-repo E2E shim
mirroring the upstream plugin/trino-hudi shim planned by RFC-105; never
deployed)</description>
+
+ <properties>
+ <!-- Hudi connector under test; override with -Ddep.hudi.version=...
-->
+ <dep.hudi.version>1.3.0-SNAPSHOT</dep.hudi.version>
+ <!-- Assembly-only project: skip airbase/basepom checks (license header
+ style, checkstyle, enforcer, modernizer, duplicate-finder,
dependency
+ analysis). ASF headers are enforced separately by
+ scripts/release/validate_source_copyright.sh. -->
+ <air.check.skip-all>true</air.check.skip-all>
+ <!-- Test infra only: never deploy/release this artifact. -->
+ <maven.deploy.skip>true</maven.deploy.skip>
Review Comment:
Nothing enforces the package-never-install rule here, so one stray `mvn -f
docker/trino/shim/pom.xml install` publishes a stub at the real
`io.trino:trino-hudi:481` coordinates and poisons the local m2. Add
`<maven.install.skip>true</maven.install.skip>` next to `maven.deploy.skip`.
##########
docker/trino/shim/pom.xml:
##########
@@ -0,0 +1,157 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<!--
+ In-repo mirror of the RFC-105 Trino-side shim (trinodb/trino
plugin/trino-hudi),
+ which is not yet released upstream. It exists so this repo can assemble the
+ deployable Trino plugin directory (hudi-trino jar + runtime deps + service
+ descriptor) for the E2E docker image without a trinodb/trino checkout, and it
+ stays useful afterwards: CI must test hudi-trino at HEAD while the upstream
shim
+ pins released versions.
+
+ Standalone project - NOT part of the Hudi reactor and never deployed.
+
+ Build (JDK 25, after installing hudi-trino into the local m2):
+ mvn -f docker/trino/shim/pom.xml clean package -DskipTests
+ Output plugin dir: target/trino-hudi-<trino.version>/
+ Use `package`, never `install`: installing would shadow the real
+ io.trino:trino-hudi release coordinates in the local repository.
+
+ The parent version below must stay in sync with trino.version in the root
pom,
+ the TRINO_VERSION ARG in docker/trino/Dockerfile, and the compose image tag.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>io.trino</groupId>
+ <artifactId>trino-root</artifactId>
+ <version>481</version>
+ <!-- Resolve the parent from Maven Central, never the file system. -->
+ <relativePath />
+ </parent>
+
+ <artifactId>trino-hudi</artifactId>
+ <packaging>trino-plugin</packaging>
+ <description>Trino - Hudi connector plugin assembly (in-repo E2E shim
mirroring the upstream plugin/trino-hudi shim planned by RFC-105; never
deployed)</description>
+
+ <properties>
+ <!-- Hudi connector under test; override with -Ddep.hudi.version=...
-->
+ <dep.hudi.version>1.3.0-SNAPSHOT</dep.hudi.version>
Review Comment:
This pom is outside the Hudi reactor (parent is `io.trino:trino-root`), so
the `mvn versions:set` in `cut_release_branch.sh` cannot bump it and the next
version cut leaves the shim resolving a `hudi-trino` the reactor no longer
builds - and with `cache: maven` able to supply the stale jar, that fails
silently rather than loudly. Have the workflow pass `-Ddep.hudi.version=` from
`help:evaluate -Dexpression=project.version` instead of the literal.
##########
hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/trino/ITTestTrinoCustomType.java:
##########
@@ -0,0 +1,369 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ2.testcontainers.trino;
+
+import org.apache.hudi.integ2.testcontainers.ITTestBaseTestcontainers;
+import org.apache.hudi.integ2.testcontainers.ITTestCustomTypeHiveSync;
+
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInstance;
+
+import static org.apache.hudi.integ2.testcontainers.TestcontainersConfig.Paths;
+
+/**
+ * Trino read coverage for Hudi's custom logical types (BLOB struct, VECTOR
+ * fixed_len_byte_array, VARIANT), complementing {@link
ITTestCustomTypeHiveSync}
+ * which asserts the same fixtures round-trip through the Hive serde. A flip in
+ * either direction (e.g. VECTOR decoded as array<float> instead of
+ * binary, BLOB struct field projection broken, VARIANT row count off) shows up
+ * here.
+ *
+ * <p>This test reuses the same {@code sparksql-*-sql.commands} fixtures that
+ * {@code ITTestCustomTypeHiveSync} drives, so the two tests can run in either
+ * order without cross-contamination (each has its own {@code @BeforeAll} that
+ * re-seeds, and an {@code @AfterAll} that cleans up).
+ *
+ * <p>VARIANT seeding and tests only fire on a Spark 4.x compose; on Spark 3.5
+ * the BLOB and VECTOR coverage still runs.
+ */
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class ITTestTrinoCustomType extends ITTestBaseTestcontainers {
+
+ private static final String BLOB_TEST_PATH =
"/user/hive/warehouse/blob_test";
+ private static final String BLOB_TEST_DF_PATH =
"/user/hive/warehouse/blob_test_df";
+ private static final String VECTOR_TEST_PATH =
"/user/hive/warehouse/vector_test";
+ private static final String VARIANT_TEST_PATH =
"/user/hive/warehouse/variant_test";
+ private static final String SPARKSQL_BLOB_TYPE_SQL_COMMANDS =
+ Paths.DEMO_DIR + "/sparksql-blob-type-sql.commands";
+ private static final String SPARKSQL_BLOB_TYPE_DF_COMMANDS =
+ Paths.DEMO_DIR + "/sparksql-blob-type-df.commands";
+ private static final String SPARKSQL_VECTOR_TYPE_SQL_COMMANDS =
+ Paths.DEMO_DIR + "/sparksql-vector-type-sql.commands";
+ private static final String SPARKSQL_VARIANT_TYPE_SQL_COMMANDS =
+ Paths.DEMO_DIR + "/sparksql-variant-type-sql.commands";
+
+ @BeforeAll
+ public void setupOnce() throws Exception {
+ assumeTrinoProfile();
+ initializeServices();
+ waitForHdfs();
+ sparkAdhoc1.executeShellCommand("/bin/bash " +
Paths.DEMO_SETUP).expectToSucceed();
+ sparkAdhoc1.executeSQLFile(SPARKSQL_BLOB_TYPE_SQL_COMMANDS)
+ .expectToSucceed()
+ .assertStdOutContainsLine("BLOB_SQL_TEST_SUCCESS");
+ // The DF fixture writes blob_test_df with the INLINE branch of the BLOB
struct
+ // (data field non-null, reference null). The SQL fixture exercises only
the
+ // OUT_OF_LINE branch, so seeding both gives Trino read coverage of both
shapes.
+ sparkAdhoc1.executeSQLFile(SPARKSQL_BLOB_TYPE_DF_COMMANDS)
+ .expectToSucceed()
+ .assertStdOutContainsLine("BLOB_DF_TEST_SUCCESS");
+ sparkAdhoc1.executeSQLFile(SPARKSQL_VECTOR_TYPE_SQL_COMMANDS)
+ .expectToSucceed()
+ .assertStdOutContainsLine("VECTOR_SQL_TEST_SUCCESS");
+ if (isSpark4Compose()) {
+ // VARIANT type is Spark 4.x only - guard the seed so the BLOB/VECTOR
+ // coverage still runs on a Spark 3.5 stack.
+ sparkAdhoc1.executeSQLFile(SPARKSQL_VARIANT_TYPE_SQL_COMMANDS)
+ .expectToSucceed()
+ .assertStdOutContainsLine("VARIANT_SQL_TEST_SUCCESS");
+ }
+ trino.waitUntilReady();
+ }
+
+ @AfterAll
+ public void clean() throws Exception {
+ // -f silently skips non-existent paths so the variant_test cleanup is safe
+ // even on Spark 3.5 runs where the table was never created.
+ sparkAdhoc1.executeShellCommand("hdfs dfs -rm -R -f "
+ + BLOB_TEST_PATH + " " + BLOB_TEST_DF_PATH + " "
+ + VECTOR_TEST_PATH + " " + VARIANT_TEST_PATH).expectToSucceed();
+ }
+
+ // ---------- BLOB OUT_OF_LINE (blob_test) ----------
+
+ @Test
+ public void testTrinoCountBlob() throws Exception {
+ // Post-DELETE state of sparksql-blob-type-sql.commands is 2 rows (id=1
updated,
+ // id=2 merged, id=3 inserted then deleted) - parity with the Hive count
assertion
+ // in ITTestCustomTypeHiveSync#testBlobTypeWithHiveSyncSQL.
+ trino.execute("SELECT count(*) FROM blob_test")
+ .expectToSucceed()
+ .assertStdOutContains("2");
+ }
+
+ @Test
+ public void testTrinoProjectsBlobUpdatedRow() throws Exception {
+ // Full per-row shape for id=1 (post-UPDATE state): type discriminator +
+ // every reference subfield + the OUT_OF_LINE invariant that data IS NULL.
+ // One query, one substring assertion - catches column-order shifts,
+ // nested-struct field renames, and per-field decoding bugs.
+ trino.execute("SELECT blob_data.type, blob_data.data IS NULL, "
+ + "blob_data.reference.external_path, blob_data.reference.offset, "
+ + "blob_data.reference.length, blob_data.reference.managed "
+ + "FROM blob_test WHERE id = 1")
+ .expectToSucceed()
+ .assertStdOutContains("OUT_OF_LINE,true,blobs/updated-1,10,100,true");
+ }
+
+ @Test
+ public void testTrinoProjectsBlobMergedRow() throws Exception {
+ // id=2 was MATCHED by the MERGE clause and rewritten to 'blobs/merged-2'.
+ // Same full-shape assertion as id=1 - confirms both UPDATE and MERGE write
+ // paths land at an identical on-disk OUT_OF_LINE shape.
+ trino.execute("SELECT blob_data.type, blob_data.data IS NULL, "
+ + "blob_data.reference.external_path, blob_data.reference.offset, "
+ + "blob_data.reference.length, blob_data.reference.managed "
+ + "FROM blob_test WHERE id = 2")
+ .expectToSucceed()
+ .assertStdOutContains("OUT_OF_LINE,true,blobs/merged-2,20,200,true");
+ }
+
+ @Test
+ public void testTrinoBlobDeletedRowAbsent() throws Exception {
+ // id=3 was MERGE-inserted into dt=2024-01-02 then DELETEd. A DELETE that
+ // leaves the row visible (e.g. tombstone not honored on read) shows up as
+ // count = 1 here. Pairs with testTrinoCountBlob = 2 (total post-delete)
+ // to catch the case where DELETE silently no-ops.
+ trino.execute("SELECT count(*) FROM blob_test WHERE id = 3")
+ .expectToSucceed()
+ .assertStdOutContains("0");
+ }
+
+ @Test
+ public void testTrinoBlobBothPartitionsVisible() throws Exception {
Review Comment:
`testTrinoBlobBothPartitionsVisible` asserts only that dt=2024-01-01 holds 2
rows, and its own comment says the second partition is invisible to Trino, so
the name states the opposite of what is checked. Rename it and
`testTrinoVariantBothPartitionsVisible`, or pin the claim with
`assertStdOutContains("2024-01-02", 0)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
