This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new dd3f81c3d610 [SPARK-47152][SQL][BUILD] Provide `CodeHaus Jackson`
dependencies via a new optional directory
dd3f81c3d610 is described below
commit dd3f81c3d6102fe1427702e97f7f42aa64b0bf5e
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sat Feb 24 11:05:41 2024 -0800
[SPARK-47152][SQL][BUILD] Provide `CodeHaus Jackson` dependencies via a new
optional directory
### What changes were proposed in this pull request?
This PR aims to provide `Apache Hive`'s `CodeHaus Jackson` dependencies via
a new optional directory, `hive-jackson`, instead of the standard `jars`
directory of Apache Spark binary distribution. Additionally, two internal
configurations are added whose default values are `hive-jackson/*`.
- `spark.driver.defaultExtraClassPath`
- `spark.executor.defaultExtraClassPath`
For example, Apache Spark distributions have been providing
`spark-*-yarn-shuffle.jar` file under `yarn` directory instead of `jars`.
**YARN SHUFFLE EXAMPLE**
```
$ ls -al yarn/*jar
-rw-r--r-- 1 dongjoon staff 77352048 Sep 8 19:08
yarn/spark-3.5.0-yarn-shuffle.jar
```
This PR changes `Apache Hive`'s `CodeHaus Jackson` dependencies in a
similar way.
**BEFORE**
```
$ ls -al jars/*asl*
-rw-r--r-- 1 dongjoon staff 232248 Sep 8 19:08
jars/jackson-core-asl-1.9.13.jar
-rw-r--r-- 1 dongjoon staff 780664 Sep 8 19:08
jars/jackson-mapper-asl-1.9.13.jar
```
**AFTER**
```
$ ls -al jars/*asl*
zsh: no matches found: jars/*asl*
$ ls -al hive-jackson
total 1984
drwxr-xr-x 4 dongjoon staff 128 Feb 23 15:37 .
drwxr-xr-x 16 dongjoon staff 512 Feb 23 16:34 ..
-rw-r--r-- 1 dongjoon staff 232248 Feb 23 15:37
jackson-core-asl-1.9.13.jar
-rw-r--r-- 1 dongjoon staff 780664 Feb 23 15:37
jackson-mapper-asl-1.9.13.jar
```
### Why are the changes needed?
Since Apache Hadoop 3.3.5, only Apache Hive requires old CodeHaus Jackson
dependencies.
Apache Spark 3.5.0 tried to eliminate them completely but it's reverted due
to Hive UDF support.
- https://github.com/apache/spark/pull/40893
- https://github.com/apache/spark/pull/42446
SPARK-47119 added a way to exclude Apache Hive Jackson dependencies at the
distribution building stage for Apache Spark 4.0.0.
- #45201
This PR provides a way to exclude Apache Hive Jackson dependencies at
runtime for Apache Spark 4.0.0.
- Spark Shell without Apache Hive Jackson dependencies.
```
$ bin/spark-shell --driver-default-class-path ""
```
- Spark SQL Shell without Apache Hive Jackson dependencies.
```
$ bin/spark-sql --driver-default-class-path ""
```
- Spark Thrift Server without Apache Hive Jackson dependencies.
```
$ sbin/start-thriftserver.sh --driver-default-class-path ""
```
In addition, last but not least, this PR eliminates `CodeHaus Jackson`
dependencies from the following Apache Spark deamons (using `spark-daemon.sh
start`) because they don't require Hive `CodeHaus Jackson` dependencies
- Spark Master
- Spark Worker
- Spark History Server
```
$ grep 'spark-daemon.sh start' *
start-history-server.sh:exec "${SPARK_HOME}/sbin"/spark-daemon.sh start
$CLASS 1 "$"
start-master.sh:"${SPARK_HOME}/sbin"/spark-daemon.sh start $CLASS 1 \
start-worker.sh: "${SPARK_HOME}/sbin"/spark-daemon.sh start $CLASS
$WORKER_NUM \
```
### Does this PR introduce _any_ user-facing change?
No. There is no user-facing change by default.
- For the distributions with `hive-jackson-provided` profile, the `scope`
of Apache Hive Jackson dependencies is `provided` and `hive-jackson` directory
is not created at all.
- For the distributions with default setting, the `scope` of Apache Hive
Jackson dependencies is still `compile`. In addition, they are in the Apache
Spark's built-in class path like the following.
- The following Spark Deamon don't use `CodeHaus Jackson` dependencies.
- Spark Master
- Spark Worker
- Spark History Server
### How was this patch tested?
Pass the CIs and manually build a distribution and check the class paths in
the `Environment` Tab.
```
$ dev/make-distribution.sh -Phive,hive-thriftserver
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45237 from dongjoon-hyun/SPARK-47152.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../org/apache/spark/internal/config/package.scala | 17 +++++++++++++++++
dev/make-distribution.sh | 6 ++++++
.../apache/spark/launcher/AbstractCommandBuilder.java | 2 ++
.../java/org/apache/spark/launcher/SparkLauncher.java | 8 ++++++++
.../spark/launcher/SparkSubmitCommandBuilder.java | 8 ++++++++
.../apache/spark/launcher/SparkSubmitOptionParser.java | 2 ++
6 files changed, 43 insertions(+)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 0b026a888e88..7caac5884c74 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -17,6 +17,7 @@
package org.apache.spark.internal
+import java.io.File
import java.util.Locale
import java.util.concurrent.TimeUnit
@@ -64,8 +65,16 @@ package object config {
.stringConf
.createOptional
+ private[spark] val DRIVER_DEFAULT_EXTRA_CLASS_PATH =
+ ConfigBuilder(SparkLauncher.DRIVER_DEFAULT_EXTRA_CLASS_PATH)
+ .internal()
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(SparkLauncher.DRIVER_DEFAULT_EXTRA_CLASS_PATH_VALUE)
+
private[spark] val DRIVER_CLASS_PATH =
ConfigBuilder(SparkLauncher.DRIVER_EXTRA_CLASSPATH)
+ .withPrepended(DRIVER_DEFAULT_EXTRA_CLASS_PATH.key, File.pathSeparator)
.version("1.0.0")
.stringConf
.createOptional
@@ -254,8 +263,16 @@ package object config {
private[spark] val EXECUTOR_ID =
ConfigBuilder("spark.executor.id").version("1.2.0").stringConf.createOptional
+ private[spark] val EXECUTOR_DEFAULT_EXTRA_CLASS_PATH =
+ ConfigBuilder(SparkLauncher.EXECUTOR_DEFAULT_EXTRA_CLASS_PATH)
+ .internal()
+ .version("4.0.0")
+ .stringConf
+ .createWithDefault(SparkLauncher.EXECUTOR_DEFAULT_EXTRA_CLASS_PATH_VALUE)
+
private[spark] val EXECUTOR_CLASS_PATH =
ConfigBuilder(SparkLauncher.EXECUTOR_EXTRA_CLASSPATH)
+ .withPrepended(EXECUTOR_DEFAULT_EXTRA_CLASS_PATH.key, File.pathSeparator)
.version("1.0.0")
.stringConf
.createOptional
diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index ce5c94197d4a..5c4c36df37a6 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -189,6 +189,12 @@ echo "Build flags: $@" >> "$DISTDIR/RELEASE"
# Copy jars
cp "$SPARK_HOME"/assembly/target/scala*/jars/* "$DISTDIR/jars/"
+# Only create the hive-jackson directory if they exist.
+for f in "$DISTDIR"/jars/jackson-*-asl-*.jar; do
+ mkdir -p "$DISTDIR"/hive-jackson
+ mv $f "$DISTDIR"/hive-jackson/
+done
+
# Only create the yarn directory if the yarn artifacts were built.
if [ -f
"$SPARK_HOME"/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar ]; then
mkdir "$DISTDIR/yarn"
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
index 21861bdcb55e..914f4e4d4570 100644
---
a/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
+++
b/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java
@@ -271,6 +271,8 @@ abstract class AbstractCommandBuilder {
Properties p = loadPropertiesFile();
p.stringPropertyNames().forEach(key ->
effectiveConfig.computeIfAbsent(key, p::getProperty));
+
effectiveConfig.putIfAbsent(SparkLauncher.DRIVER_DEFAULT_EXTRA_CLASS_PATH,
+ SparkLauncher.DRIVER_DEFAULT_EXTRA_CLASS_PATH_VALUE);
}
return effectiveConfig;
}
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java
b/launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java
index 5d36ef56d2cf..f41474e12df9 100644
--- a/launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java
+++ b/launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java
@@ -54,6 +54,10 @@ public class SparkLauncher extends
AbstractLauncher<SparkLauncher> {
/** Configuration key for the driver memory. */
public static final String DRIVER_MEMORY = "spark.driver.memory";
+ /** Configuration key for the driver default extra class path. */
+ public static final String DRIVER_DEFAULT_EXTRA_CLASS_PATH =
+ "spark.driver.defaultExtraClassPath";
+ public static final String DRIVER_DEFAULT_EXTRA_CLASS_PATH_VALUE =
"hive-jackson/*";
/** Configuration key for the driver class path. */
public static final String DRIVER_EXTRA_CLASSPATH =
"spark.driver.extraClassPath";
/** Configuration key for the default driver VM options. */
@@ -65,6 +69,10 @@ public class SparkLauncher extends
AbstractLauncher<SparkLauncher> {
/** Configuration key for the executor memory. */
public static final String EXECUTOR_MEMORY = "spark.executor.memory";
+ /** Configuration key for the executor default extra class path. */
+ public static final String EXECUTOR_DEFAULT_EXTRA_CLASS_PATH =
+ "spark.executor.defaultExtraClassPath";
+ public static final String EXECUTOR_DEFAULT_EXTRA_CLASS_PATH_VALUE =
"hive-jackson/*";
/** Configuration key for the executor class path. */
public static final String EXECUTOR_EXTRA_CLASSPATH =
"spark.executor.extraClassPath";
/** Configuration key for the default executor VM options. */
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
index 5469b36cf961..d884f7e474c0 100644
---
a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
+++
b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
@@ -267,6 +267,12 @@ class SparkSubmitCommandBuilder extends
AbstractCommandBuilder {
Map<String, String> config = getEffectiveConfig();
boolean isClientMode = isClientMode(config);
String extraClassPath = isClientMode ?
config.get(SparkLauncher.DRIVER_EXTRA_CLASSPATH) : null;
+ String defaultExtraClassPath =
config.get(SparkLauncher.DRIVER_DEFAULT_EXTRA_CLASS_PATH);
+ if (extraClassPath == null || extraClassPath.trim().isEmpty()) {
+ extraClassPath = defaultExtraClassPath;
+ } else {
+ extraClassPath += File.pathSeparator + defaultExtraClassPath;
+ }
List<String> cmd = buildJavaCommand(extraClassPath);
// Take Thrift/Connect Server as daemon
@@ -498,6 +504,8 @@ class SparkSubmitCommandBuilder extends
AbstractCommandBuilder {
case DRIVER_MEMORY -> conf.put(SparkLauncher.DRIVER_MEMORY, value);
case DRIVER_JAVA_OPTIONS ->
conf.put(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, value);
case DRIVER_LIBRARY_PATH ->
conf.put(SparkLauncher.DRIVER_EXTRA_LIBRARY_PATH, value);
+ case DRIVER_DEFAULT_CLASS_PATH ->
+ conf.put(SparkLauncher.DRIVER_DEFAULT_EXTRA_CLASS_PATH, value);
case DRIVER_CLASS_PATH ->
conf.put(SparkLauncher.DRIVER_EXTRA_CLASSPATH, value);
case CONF -> {
checkArgument(value != null, "Missing argument to %s", CONF);
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitOptionParser.java
b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitOptionParser.java
index ea54986daab7..df4fccd0f01e 100644
---
a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitOptionParser.java
+++
b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitOptionParser.java
@@ -40,6 +40,7 @@ class SparkSubmitOptionParser {
protected final String CONF = "--conf";
protected final String DEPLOY_MODE = "--deploy-mode";
protected final String DRIVER_CLASS_PATH = "--driver-class-path";
+ protected final String DRIVER_DEFAULT_CLASS_PATH =
"--driver-default-class-path";
protected final String DRIVER_CORES = "--driver-cores";
protected final String DRIVER_JAVA_OPTIONS = "--driver-java-options";
protected final String DRIVER_LIBRARY_PATH = "--driver-library-path";
@@ -94,6 +95,7 @@ class SparkSubmitOptionParser {
{ DEPLOY_MODE },
{ DRIVER_CLASS_PATH },
{ DRIVER_CORES },
+ { DRIVER_DEFAULT_CLASS_PATH },
{ DRIVER_JAVA_OPTIONS },
{ DRIVER_LIBRARY_PATH },
{ DRIVER_MEMORY },
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
