Zoltan Chovan has posted comments on this change. (
http://gerrit.cloudera.org:8080/23244 )
Change subject: Upgrade to Spark3.5.5
......................................................................
Patch Set 1:
> Patch Set 1:
>
> I don't see the below dependencies in the dependencies of spark 3.5.5
> joda-time
> threeten
> apache-arrow
> apache-orc
> Probably they are being brought in transitively?
>
> However I was able to read and write using kudu-spark jar built on 1.18.x
> branch using this patch.
I'm not sure how you checked the dependencies, but you can run the following
command for example to identify where certain package dependencies are
originating from:
./gradlew -q :kudu-spark:dependencyInsight --dependency joda-time:joda-time
--configuration testRuntimeClasspath
joda-time:joda-time:2.12.5
Variant runtime:
| Attribute Name | Provided | Requested |
|--------------------------------|--------------|--------------|
| org.gradle.status | release | |
| org.gradle.category | library | library |
| org.gradle.libraryelements | jar | jar |
| org.gradle.usage | java-runtime | java-runtime |
| org.gradle.dependency.bundling | | external |
| org.gradle.jvm.environment | | standard-jvm |
| org.gradle.jvm.version | | 8 |
joda-time:joda-time:2.12.5
\--- com.google.crypto.tink:tink:1.9.0
\--- org.apache.spark:spark-network-common_2.12:3.5.5
+--- org.apache.spark:spark-core_2.12:3.5.5
| +--- testRuntimeClasspath
| +--- org.apache.spark:spark-sql_2.12:3.5.5
| | \--- testRuntimeClasspath
| \--- org.apache.spark:spark-catalyst_2.12:3.5.5
| \--- org.apache.spark:spark-sql_2.12:3.5.5 (*)
\--- org.apache.spark:spark-network-shuffle_2.12:3.5.5
\--- org.apache.spark:spark-core_2.12:3.5.5 (*)
Accordingly in Spark's root pom.xml:
https://github.com/apache/spark/blob/7c29c664cdc9321205a98a14858aaf8daaa19db2/pom.xml#L2699C7-L2703C20
threeten seems to be a transitive dep of apache-orc/spark-sql:
org.threeten:threeten-extra:1.7.1
\--- org.apache.orc:orc-core:1.9.5
\--- org.apache.spark:spark-sql_2.12:3.5.5
\--- testRuntimeClasspath
apache-arrow is again a transitive dependency of spark-sql (too many packages
to include here)
I found that these were only present on the testRuntimeClasspath and not on the
runtimeClasspath.
--
To view, visit http://gerrit.cloudera.org:8080/23244
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.18.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I341d84f7a7b32a26b8d9699b929cce1252d0ebad
Gerrit-Change-Number: 23244
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Attila Bukor <[email protected]>
Gerrit-Reviewer: Gabriella Lotz <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Zoltan Chovan <[email protected]>
Gerrit-Comment-Date: Tue, 12 Aug 2025 06:26:22 +0000
Gerrit-HasComments: No
