voonhous commented on code in PR #19674:
URL: https://github.com/apache/hudi/pull/19674#discussion_r3818397612
##########
.github/workflows/bot.yml:
##########
@@ -1354,7 +1390,7 @@ jobs:
FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
run: |
- mvn clean package -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DdeployArtifacts=true -DskipTests=true
-Dmaven.javadoc.skip=true $MVN_ARGS -pl
packaging/hudi-hadoop-mr-bundle,packaging/hudi-spark-bundle,packaging/hudi-utilities-bundle,packaging/hudi-utilities-slim-bundle,packaging/hudi-cli-bundle
-am
+ mvn clean package -T 2 -Pjava17 -D"$SCALA_PROFILE"
-D"$SPARK_PROFILE" -DdeployArtifacts=true -DskipTests=true
-Dmaven.javadoc.skip=true $MVN_ARGS -pl
packaging/hudi-hadoop-mr-bundle,packaging/hudi-spark-bundle,packaging/hudi-native-spark-bundle,packaging/hudi-utilities-bundle,packaging/hudi-utilities-slim-bundle,packaging/hudi-cli-bundle
-am
Review Comment:
This addition is a no-op as the matrix stands: the only live entry in
`validate-bundle-spark4` is `spark4.2`, where the native module has no Comet
profile and skips. Net effect: `hudi-native-spark4.0-bundle_2.13`, `4.1_2.13`
(and `3.4_2.12`, which no lane builds either) are first compiled and shaded on
release day -- the miss class of HUDI-7766 (#11234). Side note: with
`-DdeployArtifacts=true` the skipped module still emits a lone `-sources.jar`
and no main jar, contrary to the pom comment that it yields no bundle at all.
Please drop the module from this `-pl` list and instead add build-only
matrix entries (no docker run needed) to `validate-native-bundle` for spark3.4
(JDK 11) and spark4.0/spark4.1 (JDK 17), so every published coordinate is at
least shaded in CI.
##########
scripts/release/validate_staged_bundles.sh:
##########
@@ -87,7 +87,10 @@ declare -a extensions=("-javadoc.jar" "-javadoc.jar.asc"
"-javadoc.jar.md5" "-ja
declare -a bundles=("hudi-aws-bundle" "hudi-azure-bundle"
"hudi-cli-bundle_2.12" "hudi-cli-bundle_2.13" "hudi-datahub-sync-bundle"
"hudi-flink1.18-bundle" "hudi-flink1.19-bundle" "hudi-flink1.20-bundle"
"hudi-flink2.0-bundle" "hudi-flink2.1-bundle" "hudi-gcp-bundle"
"hudi-hadoop-mr-bundle" "hudi-hive-sync-bundle" "hudi-integ-test-bundle"
-"hudi-kafka-connect-bundle" "hudi-metaserver-server-bundle"
"hudi-presto-bundle"
+"hudi-kafka-connect-bundle" "hudi-metaserver-server-bundle"
+"hudi-native-spark3.4-bundle_2.12" "hudi-native-spark3.5-bundle_2.12"
"hudi-native-spark3.5-bundle_2.13"
Review Comment:
These five coordinates are now existence-checked at staging, but nothing
ever runs them from staging: `native_spark/ci_run.sh` implements only the
local-jar leg, and no release workflow calls it
(`release_candidate_validation.yml` drives
`packaging/bundle-validation/ci_run.sh`, which has no native entry). Combined
with the CI build gap on 3.4/4.0/4.1, four of the five published bundles are
never executed anywhere.
Please either add staging-repo download handling (mirroring
`packaging/bundle-validation/ci_run.sh`'s `STAGING_REPO_NUM` leg) to
`native_spark/ci_run.sh` plus a step in `release_candidate_validation.yml`, or
add a comment here stating the native bundles are existence-checked only and
file a follow-up.
##########
packaging/hudi-native-spark-bundle/pom.xml:
##########
@@ -0,0 +1,493 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.3.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
<artifactId>hudi-native-spark${sparkbundle.version}-bundle_${scala.binary.version}</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <checkstyle.skip>true</checkstyle.skip>
+ <main.basedir>${project.parent.basedir}</main.basedir>
+ <skipTests>true</skipTests>
+ <javax.servlet.version>3.1.0</javax.servlet.version>
+ <!-- Apache DataFusion Comet releases one artifact per Spark minor
version, and does not
+ cover every Spark version Hudi builds against. Default to producing
nothing, so that a
+ build for an uncovered Spark version yields no bundle at all rather
than one named
+ "native" that carries no native code. The comet-spark<version>
profiles below opt in. -->
+ <hudi.native.bundle.skip>true</hudi.native.bundle.skip>
+ <hudi.native.bundle.jar.phase>none</hudi.native.bundle.jar.phase>
+ <maven.install.skip>${hudi.native.bundle.skip}</maven.install.skip>
+ <maven.deploy.skip>${hudi.native.bundle.skip}</maven.deploy.skip>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>default-jar</id>
+ <!-- none when this module produces no bundle. Without it
maven-jar-plugin still emits
+ a jar named as if it were a bundle, holding nothing but the
generated META-INF.
+ skipIfEmpty does not help: remote-resources populates
target/classes regardless. -->
+ <phase>${hudi.native.bundle.jar.phase}</phase>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>${maven-shade-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <skip>${hudi.native.bundle.skip}</skip>
+ <createSourcesJar>${shadeSources}</createSourcesJar>
+
<dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml
+ </dependencyReducedPomLocation>
+ <transformers>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer"
/>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer">
+ <addHeader>true</addHeader>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
+ <resource>META-INF/LICENSE</resource>
+ <file>target/classes/META-INF/LICENSE</file>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+
<resource>META-INF/services/org.apache.spark.sql.sources.DataSourceRegister</resource>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ </transformers>
+ <artifactSet>
+ <includes combine.children="append">
+ <include>org.apache.hudi:hudi-hadoop-common</include>
+ <include>org.apache.hudi:hudi-common</include>
+ <include>org.apache.hudi:hudi-client-common</include>
+ <include>org.apache.hudi:hudi-spark-client</include>
+
<include>org.apache.hudi:hudi-spark-common_${scala.binary.version}</include>
+
<include>org.apache.hudi:hudi-spark_${scala.binary.version}</include>
+
<include>org.apache.hudi:${hudi.spark.module}_${scala.binary.version}</include>
+
<include>org.apache.hudi:${hudi.spark.common.module}</include>
+ <include>org.apache.hudi:hudi-hive-sync</include>
+ <include>org.apache.hudi:hudi-sync-common</include>
+ <include>org.apache.hudi:hudi-hadoop-mr</include>
+ <include>org.apache.hudi:hudi-timeline-service</include>
+
+ <!-- NOTE: Comet must never be relocated. Its native library
is bound through JNI
+ symbols named Java_org_apache_comet_Native_*, and
the shared objects are
+ resolved as classpath resources under
org/apache/comet/<os>/<arch>/, so a
+ relocation of org.apache.comet builds cleanly and
then fails on the first
+ native call. The same applies to
org.apache.arrow.c, which Comet carries
+ unshaded for the Arrow C data interface JNI
bindings. -->
+ <include>org.apache.datafusion:*</include>
+
+ <include>javax.servlet:javax.servlet-api</include>
+ <include>com.beust:jcommander</include>
+ <include>io.javalin:javalin</include>
+ <!-- Spark only has mortbay jetty -->
+ <include>org.eclipse.jetty:*</include>
+ <include>org.eclipse.jetty.websocket:*</include>
+ <include>org.jetbrains.kotlin:*</include>
+ <include>org.rocksdb:rocksdbjni</include>
+ <!-- Bundle Jackson JSR310 library since it is not present
in spark 2.x. For spark 3.x this will
+ bundle the same JSR310 version that is included in
spark runtime -->
+
<include>com.fasterxml.jackson.datatype:jackson-datatype-jsr310</include>
+ <include>com.lmax:disruptor</include>
+ <include>com.github.davidmoten:guava-mini</include>
+ <include>com.github.davidmoten:hilbert-curve</include>
+ <include>com.github.ben-manes.caffeine:caffeine</include>
+ <include>org.apache.parquet:parquet-avro</include>
+ <include>com.twitter:chill-protobuf</include>
+
+ <include>io.dropwizard.metrics:metrics-core</include>
+ <include>io.dropwizard.metrics:metrics-graphite</include>
+ <include>io.dropwizard.metrics:metrics-jmx</include>
+ <include>io.prometheus:simpleclient</include>
+ <include>io.prometheus:simpleclient_httpserver</include>
+ <include>io.prometheus:simpleclient_dropwizard</include>
+ <include>io.prometheus:simpleclient_pushgateway</include>
+ <include>io.prometheus:simpleclient_common</include>
+ <include>com.uber.m3:tally-m3</include>
+ <include>com.uber.m3:tally-core</include>
+
+ <include>org.apache.hive:hive-common</include>
+ <include>org.apache.hive:hive-service</include>
+ <include>org.apache.hive:hive-service-rpc</include>
+ <include>org.apache.hive:hive-metastore</include>
+ <include>org.apache.hive:hive-jdbc</include>
+
+ <include>org.apache.curator:curator-framework</include>
+ <include>org.apache.curator:curator-client</include>
+ <include>org.apache.curator:curator-recipes</include>
+ <include>commons-codec:commons-codec</include>
+ <include>commons-io:commons-io</include>
+ <include>org.openjdk.jol:jol-core</include>
+ </includes>
+ </artifactSet>
+ <relocations combine.children="append">
+ <!-- NOTE: We have to relocate all classes w/in
org.apache.spark.sql.avro to avoid
+ potential classpath collisions in case users would
like to also use "spark-avro" w/in
+ their runtime, since Hudi carries some of the same
classes as "spark-avro" -->
+ <relocation>
+ <pattern>org.apache.spark.sql.avro.</pattern>
+
<shadedPattern>org.apache.hudi.org.apache.spark.sql.avro.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>javax.servlet.</pattern>
+ <shadedPattern>org.apache.hudi.javax.servlet.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.beust.jcommander.</pattern>
+
<shadedPattern>org.apache.hudi.com.beust.jcommander.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.commons.io.</pattern>
+
<shadedPattern>org.apache.hudi.org.apache.commons.io.</shadedPattern>
+ </relocation>
+ <!-- TODO: clean up hive dep - Revisit GH ISSUE #533 &
PR#633-->
+ <relocation>
+ <pattern>org.apache.hive.jdbc.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hive.jdbc.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.hive.metastore.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.metastore.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hive.common.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hive.common.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.hive.common.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.common.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.hive.conf.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.conf.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hive.service.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hive.service.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.hive.service.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.service.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.codahale.metrics.</pattern>
Review Comment:
**blocker** Including `org.apache.datafusion:*` under this inherited
relocation breaks Comet's Spark metrics integration. Shade rewrites Comet's
`org.apache.spark.CometSource$` (which implements Spark's
`org.apache.spark.metrics.source.Source`), so its `metricRegistry()` returns
`org.apache.hudi.com.codahale.metrics.MetricRegistry` and no longer matches the
interface descriptor. With `spark.comet.metrics.enabled=true`,
`MetricsSystem.registerSource` then throws `AbstractMethodError` at
SparkContext init (its catch handles only IllegalArgumentException). Off by
default, which is why validation stays green.
Evidence: javap on `comet-spark-spark3.5_2.12-1.0.0` shows `CometSource$
implements org.apache.spark.metrics.source.Source` with
`com.codahale.metrics.*` members, and the released
`hudi-spark3.5-bundle_2.12-1.0.2` demonstrably rewrites third-party classes
under this same pattern (`io.prometheus.client.dropwizard.DropwizardExports`
takes the relocated MetricRegistry). Same bug class as the flink dropwizard
relocation fix, #18982.
Since Comet itself must not be relocated, please either (a) drop
`io.dropwizard.metrics:*` from the artifactSet and this `com.codahale.metrics.`
relocation in this module only, or (b) document under the README "Shading"
section that `spark.comet.metrics.enabled` is unsupported in this bundle, and
add that config to validate.sh so the failure mode is pinned.
##########
packaging/bundle-validation/native_spark/validate.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.spark.sql.SaveMode._
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.config.HoodieWriteConfig._
+
+val outputDir = "/tmp/native-spark-bundle"
+
+// Force a real join rather than a broadcast, so the plan exercises Comet's
join, shuffle and sort.
+spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
+
+// Deterministic input, so the query result can be asserted exactly. 300 rows
spread evenly over
+// three partitions, fare equal to the row id.
+def writeTable(name: String): String = {
Review Comment:
Validation covers copy-on-write only, but the module README documents
merge-on-read-specific behavior (`Batched: false`, `CometSparkRowToColumnar`
per-cell bridging) that nothing exercises. The existing spark-bundle lane does
validate MOR (`packaging/bundle-validation/validate.sh:158`), so the bar exists.
Please write one of the two tables as MERGE_ON_READ (add the table type
option plus a second commit so a log file exists), keep the exact-row
assertion, and assert `CometSparkRowToColumnar` appears in the MOR plan the
same way `CometSortMergeJoin` is asserted today.
##########
.github/workflows/bot.yml:
##########
@@ -1310,6 +1310,42 @@ jobs:
HUDI_VERSION=$(mvn help:evaluate -Dexpression=project.version -q
-DforceStdout)
./packaging/bundle-validation/ci_run.sh hudi_docker_java17
$HUDI_VERSION openjdk17
+ validate-native-bundle:
+ runs-on: ubuntu-latest
+ needs: changes
+ strategy:
+ matrix:
+ include:
+ - scalaProfile: 'scala-2.12'
+ sparkProfile: 'spark3.5'
+ sparkRuntime: '3.5.5'
Review Comment:
nit, feel free to ignore: every other lane's `sparkRuntime` uses the
`sparkX.Y.Z` grammar (e.g. `'spark3.5.1'`) that the shared `ci_run.sh`
string-matches on; this one is bare `'3.5.5'`, and the same version is also
hardcoded as the Dockerfile `ARG` default and in the `ci_run.sh` doc comment.
Consider naming it `sparkVersion` and keeping the value only here.
##########
packaging/bundle-validation/native_spark/ci_run.sh:
##########
@@ -0,0 +1,56 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+##
+# Builds a glibc Java 17 image and validates the native spark bundle inside it.
+#
+# env vars:
+# SPARK_RUNTIME: spark version to validate against, e.g. 3.5.5
+# SPARK_HADOOP_VERSION: hadoop version suffix of the spark distribution,
defaults to 3
+# GITHUB_WORKSPACE: repository root
+##
+set -o errexit
+set -o nounset
+
+SPARK_HADOOP_VERSION=${SPARK_HADOOP_VERSION:-3}
+WORKDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+CONTAINER_NAME=hudi-native-bundle-validation
Review Comment:
nit, feel free to ignore: this variable holds an image name, not a container
name (it is only used as `-t $CONTAINER_NAME:$SPARK_RUNTIME`, and `docker run
--rm` never names a container). Rename to `IMAGE_NAME`.
##########
packaging/hudi-native-spark-bundle/pom.xml:
##########
@@ -0,0 +1,493 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.3.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
<artifactId>hudi-native-spark${sparkbundle.version}-bundle_${scala.binary.version}</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <checkstyle.skip>true</checkstyle.skip>
+ <main.basedir>${project.parent.basedir}</main.basedir>
+ <skipTests>true</skipTests>
+ <javax.servlet.version>3.1.0</javax.servlet.version>
+ <!-- Apache DataFusion Comet releases one artifact per Spark minor
version, and does not
+ cover every Spark version Hudi builds against. Default to producing
nothing, so that a
+ build for an uncovered Spark version yields no bundle at all rather
than one named
+ "native" that carries no native code. The comet-spark<version>
profiles below opt in. -->
+ <hudi.native.bundle.skip>true</hudi.native.bundle.skip>
+ <hudi.native.bundle.jar.phase>none</hudi.native.bundle.jar.phase>
+ <maven.install.skip>${hudi.native.bundle.skip}</maven.install.skip>
+ <maven.deploy.skip>${hudi.native.bundle.skip}</maven.deploy.skip>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>default-jar</id>
+ <!-- none when this module produces no bundle. Without it
maven-jar-plugin still emits
+ a jar named as if it were a bundle, holding nothing but the
generated META-INF.
+ skipIfEmpty does not help: remote-resources populates
target/classes regardless. -->
+ <phase>${hudi.native.bundle.jar.phase}</phase>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>${maven-shade-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <skip>${hudi.native.bundle.skip}</skip>
+ <createSourcesJar>${shadeSources}</createSourcesJar>
+
<dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml
+ </dependencyReducedPomLocation>
+ <transformers>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer"
/>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer">
+ <addHeader>true</addHeader>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
+ <resource>META-INF/LICENSE</resource>
+ <file>target/classes/META-INF/LICENSE</file>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+
<resource>META-INF/services/org.apache.spark.sql.sources.DataSourceRegister</resource>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ </transformers>
+ <artifactSet>
+ <includes combine.children="append">
+ <include>org.apache.hudi:hudi-hadoop-common</include>
+ <include>org.apache.hudi:hudi-common</include>
+ <include>org.apache.hudi:hudi-client-common</include>
+ <include>org.apache.hudi:hudi-spark-client</include>
+
<include>org.apache.hudi:hudi-spark-common_${scala.binary.version}</include>
+
<include>org.apache.hudi:hudi-spark_${scala.binary.version}</include>
+
<include>org.apache.hudi:${hudi.spark.module}_${scala.binary.version}</include>
+
<include>org.apache.hudi:${hudi.spark.common.module}</include>
+ <include>org.apache.hudi:hudi-hive-sync</include>
+ <include>org.apache.hudi:hudi-sync-common</include>
+ <include>org.apache.hudi:hudi-hadoop-mr</include>
+ <include>org.apache.hudi:hudi-timeline-service</include>
+
+ <!-- NOTE: Comet must never be relocated. Its native library
is bound through JNI
+ symbols named Java_org_apache_comet_Native_*, and
the shared objects are
+ resolved as classpath resources under
org/apache/comet/<os>/<arch>/, so a
+ relocation of org.apache.comet builds cleanly and
then fails on the first
+ native call. The same applies to
org.apache.arrow.c, which Comet carries
+ unshaded for the Arrow C data interface JNI
bindings. -->
+ <include>org.apache.datafusion:*</include>
+
+ <include>javax.servlet:javax.servlet-api</include>
+ <include>com.beust:jcommander</include>
+ <include>io.javalin:javalin</include>
+ <!-- Spark only has mortbay jetty -->
+ <include>org.eclipse.jetty:*</include>
+ <include>org.eclipse.jetty.websocket:*</include>
+ <include>org.jetbrains.kotlin:*</include>
+ <include>org.rocksdb:rocksdbjni</include>
+ <!-- Bundle Jackson JSR310 library since it is not present
in spark 2.x. For spark 3.x this will
+ bundle the same JSR310 version that is included in
spark runtime -->
+
<include>com.fasterxml.jackson.datatype:jackson-datatype-jsr310</include>
+ <include>com.lmax:disruptor</include>
+ <include>com.github.davidmoten:guava-mini</include>
+ <include>com.github.davidmoten:hilbert-curve</include>
+ <include>com.github.ben-manes.caffeine:caffeine</include>
+ <include>org.apache.parquet:parquet-avro</include>
+ <include>com.twitter:chill-protobuf</include>
+
+ <include>io.dropwizard.metrics:metrics-core</include>
+ <include>io.dropwizard.metrics:metrics-graphite</include>
+ <include>io.dropwizard.metrics:metrics-jmx</include>
+ <include>io.prometheus:simpleclient</include>
+ <include>io.prometheus:simpleclient_httpserver</include>
+ <include>io.prometheus:simpleclient_dropwizard</include>
+ <include>io.prometheus:simpleclient_pushgateway</include>
+ <include>io.prometheus:simpleclient_common</include>
+ <include>com.uber.m3:tally-m3</include>
+ <include>com.uber.m3:tally-core</include>
+
+ <include>org.apache.hive:hive-common</include>
+ <include>org.apache.hive:hive-service</include>
+ <include>org.apache.hive:hive-service-rpc</include>
+ <include>org.apache.hive:hive-metastore</include>
+ <include>org.apache.hive:hive-jdbc</include>
+
+ <include>org.apache.curator:curator-framework</include>
+ <include>org.apache.curator:curator-client</include>
+ <include>org.apache.curator:curator-recipes</include>
+ <include>commons-codec:commons-codec</include>
+ <include>commons-io:commons-io</include>
+ <include>org.openjdk.jol:jol-core</include>
+ </includes>
+ </artifactSet>
+ <relocations combine.children="append">
+ <!-- NOTE: We have to relocate all classes w/in
org.apache.spark.sql.avro to avoid
+ potential classpath collisions in case users would
like to also use "spark-avro" w/in
+ their runtime, since Hudi carries some of the same
classes as "spark-avro" -->
+ <relocation>
+ <pattern>org.apache.spark.sql.avro.</pattern>
+
<shadedPattern>org.apache.hudi.org.apache.spark.sql.avro.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>javax.servlet.</pattern>
+ <shadedPattern>org.apache.hudi.javax.servlet.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.beust.jcommander.</pattern>
+
<shadedPattern>org.apache.hudi.com.beust.jcommander.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.commons.io.</pattern>
+
<shadedPattern>org.apache.hudi.org.apache.commons.io.</shadedPattern>
+ </relocation>
+ <!-- TODO: clean up hive dep - Revisit GH ISSUE #533 &
PR#633-->
+ <relocation>
+ <pattern>org.apache.hive.jdbc.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hive.jdbc.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.hive.metastore.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.metastore.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hive.common.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hive.common.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.hive.common.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.common.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.hive.conf.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.conf.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hive.service.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hive.service.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.hive.service.</pattern>
+
<shadedPattern>${spark.bundle.hive.shade.prefix}org.apache.hadoop.hive.service.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.codahale.metrics.</pattern>
+
<shadedPattern>org.apache.hudi.com.codahale.metrics.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.commons.codec.</pattern>
+
<shadedPattern>org.apache.hudi.org.apache.commons.codec.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.eclipse.jetty.</pattern>
+
<shadedPattern>org.apache.hudi.org.apache.jetty.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.openjdk.jol.</pattern>
+
<shadedPattern>org.apache.hudi.org.openjdk.jol.</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.uber.m3.</pattern>
+ <shadedPattern>org.apache.hudi.com.uber.m3.</shadedPattern>
+ </relocation>
+ </relocations>
+ <filters>
Review Comment:
nit, feel free to ignore: the Comet jar ships
`META-INF/maven/org.apache.arrow/**/pom.properties` (declaring 18.3.0) and
`META-INF/maven/com.google.guava/guava/pom.properties` (33.2.1-jre) while the
corresponding classes are relocated under `org/apache/comet/shaded/**` -- the
version-probe trap `HoodieAvroUtils` was hardened against. Consider adding
those two `META-INF/maven` paths to the excludes for `org.apache.datafusion:*`.
##########
pom.xml:
##########
@@ -55,6 +55,7 @@
<module>packaging/hudi-gcp-bundle</module>
<module>packaging/hudi-azure-bundle</module>
<module>packaging/hudi-spark-bundle</module>
+ <module>packaging/hudi-native-spark-bundle</module>
Review Comment:
With the module in the default `<modules>`, every full-reactor CI lane
already passes `-Dspark3.5`, which activates the Comet profile: an 88 MB Comet
download plus a ~200 MB uber jar shaded on every PR run (the full-reactor build
in `test-spark-java-tests`-style jobs, and the Azure jacoco job --
`azure-pipelines-20230430.yml:98` excludes `!packaging/hudi-spark-bundle` but
not this module), with the artifacts also landing in the shared maven caches.
Please gate the module behind a profile so it only enters the reactor when
explicitly requested; at minimum add `- '!packaging/hudi-native-spark-bundle'`
to the Azure `jacocoModules` list and exclude it from the full-reactor PR
builds.
##########
packaging/bundle-validation/native_spark/ci_run.sh:
##########
@@ -0,0 +1,56 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+##
+# Builds a glibc Java 17 image and validates the native spark bundle inside it.
+#
+# env vars:
+# SPARK_RUNTIME: spark version to validate against, e.g. 3.5.5
+# SPARK_HADOOP_VERSION: hadoop version suffix of the spark distribution,
defaults to 3
+# GITHUB_WORKSPACE: repository root
+##
+set -o errexit
+set -o nounset
+
+SPARK_HADOOP_VERSION=${SPARK_HADOOP_VERSION:-3}
+WORKDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+CONTAINER_NAME=hudi-native-bundle-validation
+
+NATIVE_BUNDLE_JAR=$(ls
${GITHUB_WORKSPACE}/packaging/hudi-native-spark-bundle/target/hudi-native-spark*-bundle_*.jar
\
Review Comment:
This glob has no scala-binary filter, and the Dockerfile has no scala-2.13
tarball branch (the 2.13 Spark distro filename differs; `base/Dockerfile:55-64`
handles it). `hudi-native-spark3.5-bundle_2.13` is a released coordinate, so a
future scala-2.13 matrix entry would pick an arbitrary jar here and validate it
against 2.12 Spark.
Please plumb a `SCALA_VERSION` arg through the glob and the Dockerfile, or
fail fast when the selected jar name contains `_2.13`.
##########
packaging/bundle-validation/native_spark/Dockerfile:
##########
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# A glibc image with Java 17, which the native bundle needs and the shared
bundle-validation base
+# image cannot provide: that one is Alpine based, so its JVMs are musl linked
and Comet's
+# glibc-linked libcomet.so cannot load into them. Kept separate from the
shared base image so this
+# does not change how any other bundle is validated.
+FROM eclipse-temurin:17-jdk
+
+ARG SPARK_VERSION=3.5.5
+ARG SPARK_HADOOP_VERSION=3
+
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends wget \
+ && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir /opt/native-bundle-validation
+ENV WORKDIR=/opt/native-bundle-validation
+
+RUN wget -q
https://archive.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$SPARK_HADOOP_VERSION.tgz
-P "$WORKDIR" \
Review Comment:
This pulls a 400 MB Spark tarball from `archive.apache.org` inside a docker
build that runs on every PR. Every other bundle-validation lane consumes the
pre-published `apachehudi/hudi-ci-bundle-validation-base` image precisely to
avoid repeating these downloads, and CI was deliberately moved off the archive
host in #17723 (it is the ASF cold archive, discouraged for CI).
Also worth noting: `dlcdn.apache.org` no longer carries 3.5.5 (404) but has
3.5.9, and 3.5.9 is the exact version `comet-spark-spark3.5_2.12:1.0.0` is
compiled against (its pom declares `spark-sql_2.12:3.5.9` provided), while this
lane validates 3.5.5.
Please publish a glibc/JDK17 base image alongside
`packaging/bundle-validation/base` and `FROM` it here; at minimum switch to
`dlcdn.apache.org` and bump the runtime to Spark 3.5.9.
##########
packaging/hudi-native-spark-bundle/README.md:
##########
@@ -0,0 +1,118 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# hudi-native-spark-bundle
+
+Everything in `hudi-spark-bundle`, plus [Apache DataFusion
Comet](https://datafusion.apache.org/comet/)
+for native vectorized execution. Use it in place of `hudi-spark-bundle`, not
alongside it.
Review Comment:
Worth adding the sharper co-existence hazard here: since Comet cannot be
relocated, a cluster- or user-provided `comet-spark-*.jar` at a different
version gives per-class first-jar-wins resolution across two Comet versions,
and `libcomet.so` is resolved as a classpath resource
(`org/apache/comet/<os>/<arch>/`) independently of which classes won, ending in
`UnsatisfiedLinkError`/`NoSuchMethodError`. This is Hudi's oldest bundle bug
class (HUDI-1787, HUDI-1797, HUDI-2136, HUDI-7016).
Please add one line: remove any Comet jar from `--jars`/`spark.jars` and the
cluster's `jars/` directory before using this bundle.
##########
README.md:
##########
@@ -139,6 +139,24 @@ Refer to the table below for building with different Spark
and Scala versions.
| `-Dspark4.2` | hudi-spark4.2-bundle_2.13 |
For Spark 4.2 and Scala 2.13 (Needs java 17) |
| `-Dspark3` | hudi-spark3-bundle_2.12 (legacy bundle name) |
For Spark 3.5.x and Scala 2.12 |
+### Native Spark bundle
+
+`hudi-native-spark-bundle` is `hudi-spark-bundle` plus [Apache DataFusion
Comet](https://datafusion.apache.org/comet/)
+for native vectorized execution. It is built alongside the Spark bundle for
the Spark versions Comet
+publishes artifacts for, and produces nothing for the others.
+
+| Maven build options | Expected native bundle jar name | Notes
|
+|:--------------------------|:--------------------------------|:-----------------------------------------------|
+| `-Dspark3.4` | hudi-native-spark3.4-bundle_2.12 | For Spark
3.4.x and Scala 2.12 |
+| `-Dspark3.5 -Dscala-2.12` | hudi-native-spark3.5-bundle_2.12 | For Spark
3.5.x and Scala 2.12 |
+| `-Dspark3.5 -Dscala-2.13` | hudi-native-spark3.5-bundle_2.13 | For Spark
3.5.x and Scala 2.13 |
+| `-Dspark4.0` | hudi-native-spark4.0-bundle_2.13 | For Spark 4.0
and Scala 2.13 |
+| `-Dspark4.1` | hudi-native-spark4.1-bundle_2.13 | For Spark 4.1
and Scala 2.13 |
+| `-Dspark3.3`, `-Dspark4.2` | none | Comet
publishes no artifact for these versions |
Review Comment:
`-Dspark3` (the legacy alias listed in the table above, and a live release
pass in `deploy_staging_jars.sh`) maps to Spark 3.5/Scala 2.12 but does not
activate `comet-spark3.5` -- a Maven profile cannot OR two property activators
-- so it produces no native bundle even though Comet publishes for 3.5. A
reader of "built alongside the Spark bundle for the Spark versions Comet
publishes artifacts for" would expect one.
```suggestion
| `-Dspark3.3`, `-Dspark4.2` | none | Comet
publishes no artifact for these versions |
| `-Dspark3` | none | Legacy
alias; does not activate the Comet profile for 3.5 |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
