yihua commented on code in PR #19674:
URL: https://github.com/apache/hudi/pull/19674#discussion_r4344140420
##########
packaging/bundle-validation/validate.sh:
##########
@@ -432,3 +524,15 @@ if [[ ${SCALA_PROFILE} != 'scala-2.13' ]]; then
fi
echo "::warning::validate.sh done validating metaserver bundle"
fi
+
+# Runs last, so a failure here cannot skip the validation of another bundle.
Review Comment:
Done, the native check now runs right after the Spark and Hadoop MR check,
and the Flink 1.20 + Spark 3.5 `validate-bundles` entry that #19655 replaced is
restored next to the Flink 2.2 one, so the Spark, utilities, CLI and native
bundles are validated in PR CI again.
##########
packaging/hudi-spark-native-bundle/pom.xml:
##########
@@ -0,0 +1,505 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.3.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
<artifactId>hudi-spark${sparkbundle.version}-native-bundle_${scala.binary.version}</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <checkstyle.skip>true</checkstyle.skip>
+ <main.basedir>${project.parent.basedir}</main.basedir>
+ <skipTests>true</skipTests>
+ <javax.servlet.version>3.1.0</javax.servlet.version>
+ <!-- Apache DataFusion Comet releases one artifact per Spark minor
version, and does not
+ cover every Spark version Hudi builds against. Default to producing
nothing, so that a
+ build for an uncovered Spark version yields no bundle rather than one
named "native" that
+ carries no native code. A sources jar is still produced, but nothing
is installed or
+ deployed. The comet-spark<version> profiles below opt in. -->
+ <hudi.spark.native.bundle.skip>true</hudi.spark.native.bundle.skip>
+
<hudi.spark.native.bundle.jar.phase>none</hudi.spark.native.bundle.jar.phase>
+ <maven.install.skip>${hudi.spark.native.bundle.skip}</maven.install.skip>
+ <maven.deploy.skip>${hudi.spark.native.bundle.skip}</maven.deploy.skip>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>default-jar</id>
+ <!-- none when this module produces no bundle. Without it
maven-jar-plugin still emits
+ a jar named as if it were a bundle, holding nothing but the
generated META-INF.
+ skipIfEmpty does not help: remote-resources populates
target/classes regardless. -->
+ <phase>${hudi.spark.native.bundle.jar.phase}</phase>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>${maven-shade-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <skip>${hudi.spark.native.bundle.skip}</skip>
+ <createSourcesJar>${shadeSources}</createSourcesJar>
+
<dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml
+ </dependencyReducedPomLocation>
+ <transformers>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer"
/>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer">
+ <addHeader>true</addHeader>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
+ <resource>META-INF/LICENSE</resource>
+ <file>target/classes/META-INF/LICENSE</file>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
+
<resource>META-INF/services/org.apache.spark.sql.sources.DataSourceRegister</resource>
+ </transformer>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ </transformers>
+ <artifactSet>
+ <includes combine.children="append">
+ <include>org.apache.hudi:hudi-hadoop-common</include>
+ <include>org.apache.hudi:hudi-common</include>
+ <include>org.apache.hudi:hudi-client-common</include>
+ <include>org.apache.hudi:hudi-spark-client</include>
+
<include>org.apache.hudi:hudi-spark-common_${scala.binary.version}</include>
+
<include>org.apache.hudi:hudi-spark_${scala.binary.version}</include>
+
<include>org.apache.hudi:${hudi.spark.module}_${scala.binary.version}</include>
+
<include>org.apache.hudi:${hudi.spark.common.module}</include>
+ <include>org.apache.hudi:hudi-hive-sync</include>
+ <include>org.apache.hudi:hudi-sync-common</include>
+ <include>org.apache.hudi:hudi-hadoop-mr</include>
+ <include>org.apache.hudi:hudi-timeline-service</include>
+
+ <!-- NOTE: Comet must never be relocated. Its native library
is bound through JNI
+ symbols named Java_org_apache_comet_Native_*, and
the shared objects are
+ resolved as classpath resources under
org/apache/comet/<os>/<arch>/, so a
+ relocation of org.apache.comet builds cleanly and
then fails on the first
+ native call. The same applies to
org.apache.arrow.c, which Comet carries
+ unshaded for the Arrow C data interface JNI
bindings. -->
+ <include>org.apache.datafusion:*</include>
+
+ <include>javax.servlet:javax.servlet-api</include>
+ <include>com.beust:jcommander</include>
+ <include>io.javalin:javalin</include>
+ <!-- Spark only has mortbay jetty -->
+ <include>org.eclipse.jetty:*</include>
+ <include>org.eclipse.jetty.websocket:*</include>
+ <include>org.jetbrains.kotlin:*</include>
+ <include>org.rocksdb:rocksdbjni</include>
+ <!-- Bundle Jackson JSR310 library since it is not present
in spark 2.x. For spark 3.x this will
+ bundle the same JSR310 version that is included in
spark runtime -->
+
<include>com.fasterxml.jackson.datatype:jackson-datatype-jsr310</include>
+ <include>com.lmax:disruptor</include>
+ <include>com.github.davidmoten:guava-mini</include>
+ <include>com.github.davidmoten:hilbert-curve</include>
+ <include>com.github.ben-manes.caffeine:caffeine</include>
+ <include>org.apache.parquet:parquet-avro</include>
+ <include>com.twitter:chill-protobuf</include>
+
+ <!-- NOTE: unlike hudi-spark-bundle, dropwizard metrics are
neither bundled nor
Review Comment:
Done, the module now shades the `hudi-spark-bundle` jar plus Comet with no
relocation of its own, so Hudi keeps its relocated dropwizard and `CometSource`
keeps Spark's names; the slim utilities bundle paired with the previous jar
failed the Streamer with `NoSuchMethodError: Metrics.getRegistry()` and passes
with this one, and that pairing is now a bundle validation step.
##########
packaging/bundle-validation/validate.sh:
##########
@@ -69,6 +76,91 @@ use_default_java_runtime () {
export JAVA_HOME=${DEFAULT_JAVA_HOME}
}
+##
+# Function to test the spark native bundle, which carries Apache DataFusion
Comet.
+#
+# Comet ships class file version 61 bytecode and a glibc linked libcomet.so,
so this only runs on
+# the Java 17 pass.
+#
+# A Hudi path read looks its table up in the session catalog, and the Hive
metastore configured in
+# hive-site.xml only runs inside test_spark_hadoop_mr_bundles, so the session
uses the in-memory
+# catalog.
+#
+# env vars (defined in container):
+# SPARK_HOME: path to the spark directory
+##
+test_spark_native_bundle () {
+ local outputDir=/tmp/spark-native-bundle
+ rm -rf $outputDir
+ change_java_runtime_version
+ echo "::warning::validate.sh Writing and querying Hudi tables with Comet
enabled"
+ $SPARK_HOME/bin/spark-shell --jars $JARS_DIR/spark-native.jar \
+ --conf 'spark.plugins=org.apache.spark.CometPlugin' \
+ --conf
'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension,org.apache.comet.CometSparkSessionExtensions'
\
+ --conf
'spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager'
\
+ --conf 'spark.comet.enabled=true' \
+ --conf 'spark.comet.exec.enabled=true' \
+ --conf 'spark.comet.convert.parquet.enabled=true' \
+ --conf 'spark.comet.explain.fallback.enabled=true' \
+ --conf 'spark.comet.metrics.enabled=true' \
Review Comment:
Done, the session and the README now set
`spark.memory.offHeap.enabled=true`, `spark.memory.offHeap.size=1g` and
`spark.comet.nativeLoadRequired=true`; `CometSource` is registered in the
validation run and it still passes.
##########
packaging/bundle-validation/validate.sh:
##########
@@ -69,6 +76,91 @@ use_default_java_runtime () {
export JAVA_HOME=${DEFAULT_JAVA_HOME}
}
+##
+# Function to test the spark native bundle, which carries Apache DataFusion
Comet.
+#
+# Comet ships class file version 61 bytecode and a glibc linked libcomet.so,
so this only runs on
Review Comment:
Done, the Comet check skips with a warning when `/lib/ld-musl-*.so.1` exists
(verified on the current base image, where the Java 17 JDK is Alpine's too),
and the bundle-validation README now says so.
##########
README.md:
##########
@@ -145,6 +145,25 @@ off-heap memory when reading ZSTD files
([PARQUET-2160](https://issues.apache.or
Upgrade to Spark 3.5 or newer before using ZSTD. Keeping GZIP as the write
default does not remove the
risk when an older Spark runtime reads ZSTD files produced by another engine.
+### Spark native bundle
+
+`hudi-spark-native-bundle` is `hudi-spark-bundle` plus [Apache DataFusion
Comet](https://datafusion.apache.org/comet/)
+for native vectorized execution. It is built alongside the Spark bundle for
the Spark versions Comet
+publishes artifacts for, and produces nothing for the others.
+
+| Maven build options | Expected native bundle jar name | Notes
|
+|:--------------------------|:--------------------------------|:-----------------------------------------------|
+| `-Dspark3.4` | hudi-spark3.4-native-bundle_2.12 | For Spark
3.4.x and Scala 2.12 |
Review Comment:
Done, the native bundle is a column of the Spark bundle table with one
paragraph after it.
##########
packaging/bundle-validation/README.md:
##########
@@ -22,6 +22,14 @@
This directory contains scripts for running bundle validation in GitHub
Actions (`validate-bundles`
specified in `.github/workflows/bot.yml`) and build profile for Docker images
used.
+## Spark native bundle
Review Comment:
Done, moved to the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
