wombatu-kun commented on code in PR #19217: URL: https://github.com/apache/hudi/pull/19217#discussion_r3717469573
########## .github/workflows/hudi_trino_e2e.yml: ########## @@ -0,0 +1,171 @@ +name: Hudi Trino E2E + +on: + push: + branches: + - master + - 'release-*' + paths: + # docker/demo/** is deliberately broad: the ITs drive several demo fixture + # scripts (sparksql-*.commands, setup_demo_container.sh), so any demo edit + # must re-run this pipeline. + - 'hudi-trino/**' + - 'docker/trino/**' + - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*' + - 'docker/demo/**' + - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**' + - '.github/workflows/hudi_trino_e2e.yml' + pull_request: + branches: + - master + - 'release-*' + paths: + - 'hudi-trino/**' + - 'docker/trino/**' + - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*' Review Comment: `ITTestBaseTestcontainers` copies `docker/compose/hadoop.env` into the generated compose dir and every service in the stack loads it, but this filter matches only the spark402 YAMLs, so an edit to `hadoop.env` never runs this pipeline. Add `docker/compose/hadoop.env` to both path lists. ########## docker/trino/etc/catalog/hudi.properties: ########## @@ -0,0 +1,30 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Native trino-hudi connector (assembled from org.apache.hudi:hudi-trino by the +# docker/trino/shim project). +# Pre-Trino 472 the only way to read Hudi was the hive-connector + hudi-trino-bundle shim; Review Comment: `plugin/trino-hudi` first appears in trinodb/trino at tag 398, not 472, so this history is wrong. Change the version to 398 or drop the historical clause and keep just the `connector.name=hudi` rationale. ########## hudi-trino/README.md: ########## @@ -45,6 +45,45 @@ mvn -Phudi-trino,hudi-trino-tests -pl hudi-trino test CI follows the same two steps: `.github/workflows/hudi_trino_ci.yml` installs the test-jars from a source checkout of the pinned Trino tag, then runs with both profiles enabled. +## End-to-end tests (docker) + +The testcontainers E2E suite (`hudi-integ-test`, classes `ITTestTrino*` under +`org.apache.hudi.integ2.testcontainers.trino`) runs Trino queries against a real +HDFS + Hive metastore + Spark stack. The Trino container image bakes in a plugin +directory assembled by the in-repo shim at `docker/trino/shim/` -- a standalone Maven +project mirroring the upstream `trinodb/trino` `plugin/trino-hudi` shim planned by +RFC-105 (not yet released upstream). CI runs the same flow via +`.github/workflows/hudi_trino_e2e.yml`. + +Local flow: + +``` +# 1. JDK 17: full reactor incl. the integ-test bundles the containers mount +mvn clean install -T 2 -Dscala-2.13 -Dscala.binary.version=2.13 -Dspark4.0 -Dflink1.20 \ + -Pintegration-tests -DskipTests=true -Ddocker.compose.skip=true + +# 2. JDK 25: the connector +mvn -Phudi-trino -pl hudi-trino install -Dmaven.test.skip=true + +# 3. JDK 25: assemble the plugin dir (package, NOT install -- installing would +# shadow the real io.trino:trino-hudi release coordinates in the local m2) +mvn -f docker/trino/shim/pom.xml clean package -DskipTests Review Comment: This step and the one in `docker/README.md` build the shim against the pom's literal `dep.hudi.version`, which the workflow comment says goes stale because `cut_release_branch.sh` cannot bump a pom outside the reactor. Add `-Ddep.hudi.version=` derived the same way the workflow does it; follow-up, not a blocker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
