This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new e12a79b6ebe [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when
using Hive UDF
e12a79b6ebe is described below
commit e12a79b6ebe0bd5ef9220e0e0979d6579638f3ae
Author: Yuming Wang <yumw...@ebay.com>
AuthorDate: Tue Aug 15 11:42:23 2023 +0800
[SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF
### What changes were proposed in this pull request?
This PR partially reverts SPARK-43225. Added jackson-core-asl and
jackson-mapper-asl back to pre-built distributions.
### Why are the changes needed?
Fix `NoClassDefFoundError` when using Hive UDF:
```
spark-sql (default)> add jar
/Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar;
Time taken: 0.413 seconds
spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as
'net.petrabarus.hiveudfs.LongToIP';
Time taken: 0.038 seconds
spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10);
23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT
long_to_ip(2130706433L) FROM range(10)]
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory
at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
...
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
manual test.
Closes #42447 from wangyum/SPARK-44719-branch-3.5.
Authored-by: Yuming Wang <yumw...@ebay.com>
Signed-off-by: Kent Yao <y...@apache.org>
---
core/pom.xml | 8 ++++++++
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 ++
pom.xml | 13 +++++++++++--
sql/hive-thriftserver/pom.xml | 9 ---------
4 files changed, 21 insertions(+), 11 deletions(-)
diff --git a/core/pom.xml b/core/pom.xml
index 674005c96ee..eb4a563f1f3 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -481,6 +481,14 @@
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-mapper-asl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ </exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index d182f39f882..14af8ea340f 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -100,11 +100,13 @@ ini4j/0.5.4//ini4j-0.5.4.jar
istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
ivy/2.5.1//ivy-2.5.1.jar
jackson-annotations/2.15.2//jackson-annotations-2.15.2.jar
+jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.15.2//jackson-core-2.15.2.jar
jackson-databind/2.15.2//jackson-databind-2.15.2.jar
jackson-dataformat-cbor/2.15.2//jackson-dataformat-cbor-2.15.2.jar
jackson-dataformat-yaml/2.15.2//jackson-dataformat-yaml-2.15.2.jar
jackson-datatype-jsr310/2.15.2//jackson-datatype-jsr310-2.15.2.jar
+jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
jackson-module-scala_2.12/2.15.2//jackson-module-scala_2.12-2.15.2.jar
jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
diff --git a/pom.xml b/pom.xml
index 9438a112c24..d0c3bfc361a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1321,6 +1321,10 @@
<groupId>asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-mapper-asl</artifactId>
+ </exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
@@ -1821,12 +1825,17 @@
</exclusion>
</exclusions>
</dependency>
- <!-- Hive 2.3 need this to init Hive's FunctionRegistry -->
+ <dependency>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ <version>${codehaus.jackson.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>${codehaus.jackson.version}</version>
- <scope>test</scope>
+ <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<groupId>${hive.group}</groupId>
diff --git a/sql/hive-thriftserver/pom.xml b/sql/hive-thriftserver/pom.xml
index 7bcfb8e1908..3659a0f846a 100644
--- a/sql/hive-thriftserver/pom.xml
+++ b/sql/hive-thriftserver/pom.xml
@@ -147,15 +147,6 @@
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
</dependency>
- <!--
- SPARK-44601: Add this test dependency to ensure that `hive-thriftserver`
module
- can be tested using Maven
- -->
- <dependency>
- <groupId>org.codehaus.jackson</groupId>
- <artifactId>jackson-mapper-asl</artifactId>
- <scope>test</scope>
- </dependency>
</dependencies>
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
