This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5e375c8ef323 [SPARK-47119][BUILD] Add `hive-jackson-provided` profile
5e375c8ef323 is described below
commit 5e375c8ef323818d33229ce0c4216ee26039b9f1
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Wed Feb 21 12:36:18 2024 -0800
[SPARK-47119][BUILD] Add `hive-jackson-provided` profile
### What changes were proposed in this pull request?
This PR aims to provide a new profile, `hive-jackson-provided`, for Apache
Spark 4.0.0.
### Why are the changes needed?
Since Apache Hadoop 3.3.5, only Apache Hive requires old CodeHaus Jackson
dependencies.
Apache Spark 3.5.0 tried to eliminate them completely but it's reverted due
to Hive UDF support.
- #40893
- #42446
To allow Apache Spark 4.0 users
- To provide their own CodeHaus Jackson libraries
- To exclude them completely if they don't use `Hive UDF`.
### Does this PR introduce _any_ user-facing change?
No, this is a new profile.
### How was this patch tested?
Pass the CIs and manual build.
**Without `hive-jackson-provided`**
```
$ dev/make-distribution.sh -Phive,hive-thriftserver
$ ls -al dist/jars/*asl*
-rw-r--r-- 1 dongjoon staff 232248 Feb 21 10:53
dist.org/jars/jackson-core-asl-1.9.13.jar
-rw-r--r-- 1 dongjoon staff 780664 Feb 21 10:53
dist.org/jars/jackson-mapper-asl-1.9.13.jar
```
**With `hive-jackson-provided`**
```
$ dev/make-distribution.sh -Phive,hive-thriftserver,hive-jackson-provided
$ ls -al dist/jars/*asl*
zsh: no matches found: dist/jars/*asl*
$ ls -al dist/jars/*hive*
-rw-r--r-- 1 dongjoon staff 183633 Feb 21 11:00
dist/jars/hive-beeline-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 44704 Feb 21 11:00
dist/jars/hive-cli-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 436169 Feb 21 11:00
dist/jars/hive-common-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 10840949 Feb 21 11:00
dist/jars/hive-exec-2.3.9-core.jar
-rw-r--r-- 1 dongjoon staff 116364 Feb 21 11:00
dist/jars/hive-jdbc-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 326585 Feb 21 11:00
dist/jars/hive-llap-common-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 8195966 Feb 21 11:00
dist/jars/hive-metastore-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 916630 Feb 21 11:00
dist/jars/hive-serde-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 1679366 Feb 21 11:00
dist/jars/hive-service-rpc-3.1.3.jar
-rw-r--r-- 1 dongjoon staff 53902 Feb 21 11:00
dist/jars/hive-shims-0.23-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 8786 Feb 21 11:00
dist/jars/hive-shims-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 120293 Feb 21 11:00
dist/jars/hive-shims-common-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 12923 Feb 21 11:00
dist/jars/hive-shims-scheduler-2.3.9.jar
-rw-r--r-- 1 dongjoon staff 258346 Feb 21 11:00
dist/jars/hive-storage-api-2.8.1.jar
-rw-r--r-- 1 dongjoon staff 581739 Feb 21 11:00
dist/jars/spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar
-rw-r--r-- 1 dongjoon staff 687446 Feb 21 11:00
dist/jars/spark-hive_2.13-4.0.0-SNAPSHOT.jar
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45201 from dongjoon-hyun/SPARK-47119.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
assembly/pom.xml | 7 +++++++
pom.xml | 5 +++--
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 2b0afafa04fc..6c31ec745b5b 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -266,6 +266,13 @@
<id>hive-provided</id>
<properties>
<hive.deps.scope>provided</hive.deps.scope>
+ <hive.jackson.scope>provided</hive.jackson.scope>
+ </properties>
+ </profile>
+ <profile>
+ <id>hive-jackson-provided</id>
+ <properties>
+ <hive.jackson.scope>provided</hive.jackson.scope>
</properties>
</profile>
<profile>
diff --git a/pom.xml b/pom.xml
index 2e1eb8f3bbb8..124552a42ff7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -271,6 +271,7 @@
<hive.deps.scope>compile</hive.deps.scope>
<hive.storage.version>2.8.1</hive.storage.version>
<hive.storage.scope>compile</hive.storage.scope>
+ <hive.jackson.scope>compile</hive.jackson.scope>
<hive.common.scope>compile</hive.common.scope>
<hive.llap.scope>compile</hive.llap.scope>
<hive.serde.scope>compile</hive.serde.scope>
@@ -1878,13 +1879,13 @@
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>${codehaus.jackson.version}</version>
- <scope>${hive.deps.scope}</scope>
+ <scope>${hive.jackson.scope}</scope>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>${codehaus.jackson.version}</version>
- <scope>${hive.deps.scope}</scope>
+ <scope>${hive.jackson.scope}</scope>
</dependency>
<dependency>
<groupId>${hive.group}</groupId>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
