wombatu-kun commented on code in PR #19265: URL: https://github.com/apache/hudi/pull/19265#discussion_r3567769684
########## hudi-lakehouse/local-dev/example/spark-app.yaml: ########## @@ -0,0 +1,61 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# SparkApplication for the Apache Spark Kubernetes Operator +# (https://github.com/apache/spark-kubernetes-operator), submitted by +# local-dev/scripts/run-example.sh after uploading hudi_table_writer.py to +# s3a://warehouse/jobs/. +# +# To run your own job: copy this file + your script, adjust pyFiles (or +# mainClass/jars for JVM jobs), the env block, and any spark conf. +apiVersion: spark.apache.org/v1alpha1 +kind: SparkApplication +metadata: + name: hudi-table-writer + namespace: hudi-lakehouse +spec: + pyFiles: "s3a://warehouse/jobs/hudi_table_writer.py" + runtimeVersions: + sparkVersion: "3.5.7" + scalaVersion: "2.12" + sparkConf: + # Service account created by the spark-kubernetes-operator helm chart + # (the namespace default SA cannot manage executor pods). + spark.kubernetes.authenticate.driver.serviceAccountName: "spark" + spark.kubernetes.container.image: "hudi-lakehouse-spark:3.5" Review Comment: Agreed that one Spark version is enough for a quickstart. What is broken today: `quickstart.sh --spark-version 4.1` builds the Scala 2.13 bundle and has build-images.sh tag the image `hudi-lakehouse-spark:4.1`, but this manifest pins `hudi-lakehouse-spark:3.5` with `sparkVersion: 3.5.7` / `scalaVersion: 2.12` and nothing templates them. So `--spark-version 4.1 --run-example` submits a SparkApplication pointing at an image that was never built, and run-example.sh then burns its full poll timeout on ErrImagePull. Fix: drop `--spark-version` from quickstart.sh, build-jars.sh and build-images.sh, and keep 3.5 hardcoded. ########## hudi-lakehouse/scripts/build-jars.sh: ########## @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Builds the two Maven artifacts the hudi-lakehouse images need: +# 1. the Hudi Spark bundle (packaging/hudi-spark-bundle) -- JDK 11/17 +# 2. the Hudi Trino plugin (hudi-trino-plugin) -- JDK 23+ +# +# Usage: build-jars.sh [--spark-version 3.5|4.1] [--skip-bundle] [--skip-plugin] + +set -euo pipefail +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SPARK_VERSION="3.5" +BUILD_BUNDLE=1 +BUILD_PLUGIN=1 + +while [[ $# -gt 0 ]]; do + case "$1" in + --spark-version) SPARK_VERSION="$2"; shift 2 ;; + --skip-bundle) BUILD_BUNDLE=0; shift ;; + --skip-plugin) BUILD_PLUGIN=0; shift ;; + *) echo "unknown arg: $1" >&2; exit 1 ;; + esac +done + +MVN_ARGS=(-DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Drat.skip=true -Dmaven.javadoc.skip=true -Dgpg.skip=true) + +if [[ "$BUILD_BUNDLE" == 1 ]]; then + # Profiles must be activated explicitly: activating any -D profile property + # disables Maven's activeByDefault profiles, so relying on defaults is fragile. + echo ">>> Building Hudi Spark bundle (spark${SPARK_VERSION})" + (cd "$REPO_ROOT" && mvn clean package -pl packaging/hudi-spark-bundle -am \ + "-Dspark${SPARK_VERSION}" -Dflink2.1 "${MVN_ARGS[@]}") +fi + +if [[ "$BUILD_PLUGIN" == 1 ]]; then + JAVA_MAJOR=$(java -version 2>&1 | sed -n 's/.*version "\([0-9]*\).*/\1/p' | head -1) Review Comment: This guard probes `java` from `PATH`, but quickstart.sh invokes the script as `JAVA_HOME="$JDK23" build-jars.sh --skip-bundle`. Maven honours `JAVA_HOME`, this check does not, so on any machine whose default JDK is 11/17/21, exactly the machine `find_jdk23` exists to serve, it exits 1 and the one-command quickstart always dies at the Trino-plugin step. Fix: probe the JDK Maven will actually use. ```bash JAVA_MAJOR=$("${JAVA_HOME:+$JAVA_HOME/bin/}java" -version 2>&1 | sed -n 's/.*version "\([0-9]*\).*/\1/p' | head -1) ``` ########## scripts/release/validate_source_copyright.sh: ########## @@ -50,7 +50,7 @@ echo "Performing custom Licensing Check " # Exclude the 'hudi-trino-plugin' directory. Its license checks are handled by airlift: # https://github.com/airlift/airbase/blob/823101482dbc60600d7862f0f5c93aded6190996/airbase/pom.xml#L1239 # --- -numfilesWithNoLicense=$(find . -path './hudi-trino-plugin' -prune -o -type f -iname '*' | grep -v './hudi-trino-plugin' | grep -v NOTICE | grep -v LICENSE | grep -v '.jpg' | grep -v '.json' | grep -v '.zip' | grep -v '.hfile' | grep -v '.data' | grep -v '.commit' | grep -v emptyFile | grep -v DISCLAIMER | grep -v '.sqltemplate' | grep -v KEYS | grep -v '.mailmap' | grep -v 'banner.txt' | grep -v '.txt' | grep -v "fixtures" | xargs grep -L "Licensed to the Apache Software Foundation (ASF)") +numfilesWithNoLicense=$(find . -path './hudi-trino-plugin' -prune -o -type f -iname '*' | grep -v './hudi-trino-plugin' | grep -v NOTICE | grep -v LICENSE | grep -v '.jpg' | grep -v '.png' | grep -v '.json' | grep -v '.zip' | grep -v '.hfile' | grep -v '.data' | grep -v '.commit' | grep -v emptyFile | grep -v DISCLAIMER | grep -v '.sqltemplate' | grep -v KEYS | grep -v '.mailmap' | grep -v 'banner.txt' | grep -v '.txt' | grep -v "fixtures" | xargs grep -L "Licensed to the Apache Software Foundation (ASF)") Review Comment: This exempts every `*.png` in the repo, present and future, from the source-release ASF header check, to accommodate one 9.5 KB logo. The project's usual remedy (keep the image out of the source tarball) does not work here, since the UI serves this one at runtime. Fix, as you suggested: inline the logo as SVG or a data-URI in style.css, and revert this line. Also worth correcting in the PR body: it says the only edit to an existing file is `**/py.typed` in the root pom rat excludes, but the pom is unchanged and this line is in fact the PR's only edit to a pre-existing file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
