This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f888741 [SPARK-30491][INFRA] Enable dependency audit files to tell
dependency classifier
f888741 is described below
commit f88874194ac8099a6fa8067a7d2b7a98fca6a849
Author: Xinrong Meng <meng.care...@gmail.com>
AuthorDate: Wed Jan 15 20:19:44 2020 -0800
[SPARK-30491][INFRA] Enable dependency audit files to tell dependency
classifier
### What changes were proposed in this pull request?
Enable dependency audit files to tell the value of artifact id, version,
and classifier of a dependency.
For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to
`avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred`
is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.
### Why are the changes needed?
Dependency audit files are expected to be consumed by automated tests or
downstream tools.
However, current dependency audit files under `dev/deps` only show jar
names. And there isn't a simple rule on how to parse the jar name to get the
values of different fields. For example, `hadoop2` is the classifier of
`avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of
`htrace-core-3.1.0-incubating.jar`.
Reference: There is a good example of the downstream tool that would be
enabled as yhuai suggested,
> Say we have a Spark application that depends on a third-party dependency
`foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo`
depends on a different version of `jackson` than Spark. So, in the pom of this
Spark application, we use the dependency management section to pin the version
of `jackson`. By doing this, we are lifting `jackson` to the top-level
dependency of my application and I want to have a way to keep tracking what
Spark uses. What we can do is [...]
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Code changes are verified by generated dependency audit files naturally.
Thus, there are no tests added.
Closes #27177 from mengCareers/depsOptimize.
Lead-authored-by: Xinrong Meng <meng.care...@gmail.com>
Co-authored-by: mengCareers <meng.care...@gmail.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 416 +++++++++++++--------------
dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 446 ++++++++++++++---------------
dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 478 ++++++++++++++++----------------
dev/test-dependencies.sh | 20 +-
4 files changed, 688 insertions(+), 672 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
index 04c6e5f..af702e8 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
@@ -1,208 +1,208 @@
-JLargeArrays-1.5.jar
-JTransforms-3.1.jar
-JavaEWAH-0.3.2.jar
-RoaringBitmap-0.7.45.jar
-ST4-4.0.4.jar
-activation-1.1.1.jar
-aircompressor-0.10.jar
-algebra_2.12-2.0.0-M2.jar
-antlr-2.7.7.jar
-antlr-runtime-3.4.jar
-antlr4-runtime-4.7.1.jar
-aopalliance-1.0.jar
-aopalliance-repackaged-2.6.1.jar
-apache-log4j-extras-1.2.17.jar
-apacheds-i18n-2.0.0-M15.jar
-apacheds-kerberos-codec-2.0.0-M15.jar
-api-asn1-api-1.0.0-M20.jar
-api-util-1.0.0-M20.jar
-arpack_combined_all-0.1.jar
-arrow-format-0.15.1.jar
-arrow-memory-0.15.1.jar
-arrow-vector-0.15.1.jar
-audience-annotations-0.5.0.jar
-automaton-1.11-8.jar
-avro-1.8.2.jar
-avro-ipc-1.8.2.jar
-avro-mapred-1.8.2-hadoop2.jar
-bonecp-0.8.0.RELEASE.jar
-breeze-macros_2.12-1.0.jar
-breeze_2.12-1.0.jar
-cats-kernel_2.12-2.0.0-M4.jar
-chill-java-0.9.3.jar
-chill_2.12-0.9.3.jar
-commons-beanutils-1.9.4.jar
-commons-cli-1.2.jar
-commons-codec-1.10.jar
-commons-collections-3.2.2.jar
-commons-compiler-3.0.15.jar
-commons-compress-1.8.1.jar
-commons-configuration-1.6.jar
-commons-crypto-1.0.0.jar
-commons-dbcp-1.4.jar
-commons-digester-1.8.jar
-commons-httpclient-3.1.jar
-commons-io-2.4.jar
-commons-lang-2.6.jar
-commons-lang3-3.9.jar
-commons-logging-1.1.3.jar
-commons-math3-3.4.1.jar
-commons-net-3.1.jar
-commons-pool-1.5.4.jar
-commons-text-1.6.jar
-compress-lzf-1.0.3.jar
-core-1.1.2.jar
-curator-client-2.7.1.jar
-curator-framework-2.7.1.jar
-curator-recipes-2.7.1.jar
-datanucleus-api-jdo-3.2.6.jar
-datanucleus-core-3.2.10.jar
-datanucleus-rdbms-3.2.9.jar
-derby-10.12.1.1.jar
-flatbuffers-java-1.9.0.jar
-generex-1.0.2.jar
-gson-2.2.4.jar
-guava-14.0.1.jar
-guice-3.0.jar
-guice-servlet-3.0.jar
-hadoop-annotations-2.7.4.jar
-hadoop-auth-2.7.4.jar
-hadoop-client-2.7.4.jar
-hadoop-common-2.7.4.jar
-hadoop-hdfs-2.7.4.jar
-hadoop-mapreduce-client-app-2.7.4.jar
-hadoop-mapreduce-client-common-2.7.4.jar
-hadoop-mapreduce-client-core-2.7.4.jar
-hadoop-mapreduce-client-jobclient-2.7.4.jar
-hadoop-mapreduce-client-shuffle-2.7.4.jar
-hadoop-yarn-api-2.7.4.jar
-hadoop-yarn-client-2.7.4.jar
-hadoop-yarn-common-2.7.4.jar
-hadoop-yarn-server-common-2.7.4.jar
-hadoop-yarn-server-web-proxy-2.7.4.jar
-hk2-api-2.6.1.jar
-hk2-locator-2.6.1.jar
-hk2-utils-2.6.1.jar
-htrace-core-3.1.0-incubating.jar
-httpclient-4.5.6.jar
-httpcore-4.4.12.jar
-istack-commons-runtime-3.0.8.jar
-ivy-2.4.0.jar
-jackson-annotations-2.10.0.jar
-jackson-core-2.10.0.jar
-jackson-core-asl-1.9.13.jar
-jackson-databind-2.10.0.jar
-jackson-dataformat-yaml-2.10.0.jar
-jackson-jaxrs-1.9.13.jar
-jackson-mapper-asl-1.9.13.jar
-jackson-module-jaxb-annotations-2.10.0.jar
-jackson-module-paranamer-2.10.0.jar
-jackson-module-scala_2.12-2.10.0.jar
-jackson-xc-1.9.13.jar
-jakarta.activation-api-1.2.1.jar
-jakarta.annotation-api-1.3.5.jar
-jakarta.inject-2.6.1.jar
-jakarta.validation-api-2.0.2.jar
-jakarta.ws.rs-api-2.1.6.jar
-jakarta.xml.bind-api-2.3.2.jar
-janino-3.0.15.jar
-javassist-3.22.0-CR2.jar
-javax.inject-1.jar
-javax.servlet-api-3.1.0.jar
-javolution-5.5.1.jar
-jaxb-api-2.2.2.jar
-jaxb-runtime-2.3.2.jar
-jcl-over-slf4j-1.7.16.jar
-jdo-api-3.0.1.jar
-jersey-client-2.29.1.jar
-jersey-common-2.29.1.jar
-jersey-container-servlet-2.29.1.jar
-jersey-container-servlet-core-2.29.1.jar
-jersey-hk2-2.29.1.jar
-jersey-media-jaxb-2.29.1.jar
-jersey-server-2.29.1.jar
-jetty-6.1.26.jar
-jetty-sslengine-6.1.26.jar
-jetty-util-6.1.26.jar
-jline-2.14.6.jar
-joda-time-2.10.5.jar
-jodd-core-3.5.2.jar
-jpam-1.1.jar
-json4s-ast_2.12-3.6.6.jar
-json4s-core_2.12-3.6.6.jar
-json4s-jackson_2.12-3.6.6.jar
-json4s-scalap_2.12-3.6.6.jar
-jsp-api-2.1.jar
-jsr305-3.0.0.jar
-jta-1.1.jar
-jul-to-slf4j-1.7.16.jar
-kryo-shaded-4.0.2.jar
-kubernetes-client-4.6.4.jar
-kubernetes-model-4.6.4.jar
-kubernetes-model-common-4.6.4.jar
-leveldbjni-all-1.8.jar
-libfb303-0.9.3.jar
-libthrift-0.12.0.jar
-log4j-1.2.17.jar
-logging-interceptor-3.12.6.jar
-lz4-java-1.7.0.jar
-machinist_2.12-0.6.8.jar
-macro-compat_2.12-1.1.1.jar
-mesos-1.4.0-shaded-protobuf.jar
-metrics-core-4.1.1.jar
-metrics-graphite-4.1.1.jar
-metrics-jmx-4.1.1.jar
-metrics-json-4.1.1.jar
-metrics-jvm-4.1.1.jar
-minlog-1.3.0.jar
-netty-all-4.1.42.Final.jar
-objenesis-2.5.1.jar
-okhttp-3.12.6.jar
-okio-1.15.0.jar
-opencsv-2.3.jar
-orc-core-1.5.8-nohive.jar
-orc-mapreduce-1.5.8-nohive.jar
-orc-shims-1.5.8.jar
-oro-2.0.8.jar
-osgi-resource-locator-1.0.3.jar
-paranamer-2.8.jar
-parquet-column-1.10.1.jar
-parquet-common-1.10.1.jar
-parquet-encoding-1.10.1.jar
-parquet-format-2.4.0.jar
-parquet-hadoop-1.10.1.jar
-parquet-hadoop-bundle-1.6.0.jar
-parquet-jackson-1.10.1.jar
-protobuf-java-2.5.0.jar
-py4j-0.10.8.1.jar
-pyrolite-4.30.jar
-scala-collection-compat_2.12-2.1.1.jar
-scala-compiler-2.12.10.jar
-scala-library-2.12.10.jar
-scala-parser-combinators_2.12-1.1.2.jar
-scala-reflect-2.12.10.jar
-scala-xml_2.12-1.2.0.jar
-shapeless_2.12-2.3.3.jar
-shims-0.7.45.jar
-slf4j-api-1.7.16.jar
-slf4j-log4j12-1.7.16.jar
-snakeyaml-1.24.jar
-snappy-0.2.jar
-snappy-java-1.1.7.3.jar
-spire-macros_2.12-0.17.0-M1.jar
-spire-platform_2.12-0.17.0-M1.jar
-spire-util_2.12-0.17.0-M1.jar
-spire_2.12-0.17.0-M1.jar
-stax-api-1.0-2.jar
-stax-api-1.0.1.jar
-stream-2.9.6.jar
-stringtemplate-3.2.1.jar
-super-csv-2.2.0.jar
-univocity-parsers-2.8.3.jar
-xbean-asm7-shaded-4.15.jar
-xercesImpl-2.9.1.jar
-xmlenc-0.52.jar
-xz-1.5.jar
-zjsonpatch-0.3.0.jar
-zookeeper-3.4.14.jar
-zstd-jni-1.4.4-3.jar
+JLargeArrays/1.5//JLargeArrays-1.5.jar
+JTransforms/3.1//JTransforms-3.1.jar
+JavaEWAH/0.3.2//JavaEWAH-0.3.2.jar
+RoaringBitmap/0.7.45//RoaringBitmap-0.7.45.jar
+ST4/4.0.4//ST4-4.0.4.jar
+activation/1.1.1//activation-1.1.1.jar
+aircompressor/0.10//aircompressor-0.10.jar
+algebra_2.12/2.0.0-M2//algebra_2.12-2.0.0-M2.jar
+antlr-runtime/3.4//antlr-runtime-3.4.jar
+antlr/2.7.7//antlr-2.7.7.jar
+antlr4-runtime/4.7.1//antlr4-runtime-4.7.1.jar
+aopalliance-repackaged/2.6.1//aopalliance-repackaged-2.6.1.jar
+aopalliance/1.0//aopalliance-1.0.jar
+apache-log4j-extras/1.2.17//apache-log4j-extras-1.2.17.jar
+apacheds-i18n/2.0.0-M15//apacheds-i18n-2.0.0-M15.jar
+apacheds-kerberos-codec/2.0.0-M15//apacheds-kerberos-codec-2.0.0-M15.jar
+api-asn1-api/1.0.0-M20//api-asn1-api-1.0.0-M20.jar
+api-util/1.0.0-M20//api-util-1.0.0-M20.jar
+arpack_combined_all/0.1//arpack_combined_all-0.1.jar
+arrow-format/0.15.1//arrow-format-0.15.1.jar
+arrow-memory/0.15.1//arrow-memory-0.15.1.jar
+arrow-vector/0.15.1//arrow-vector-0.15.1.jar
+audience-annotations/0.5.0//audience-annotations-0.5.0.jar
+automaton/1.11-8//automaton-1.11-8.jar
+avro-ipc/1.8.2//avro-ipc-1.8.2.jar
+avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar
+avro/1.8.2//avro-1.8.2.jar
+bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar
+breeze-macros_2.12/1.0//breeze-macros_2.12-1.0.jar
+breeze_2.12/1.0//breeze_2.12-1.0.jar
+cats-kernel_2.12/2.0.0-M4//cats-kernel_2.12-2.0.0-M4.jar
+chill-java/0.9.3//chill-java-0.9.3.jar
+chill_2.12/0.9.3//chill_2.12-0.9.3.jar
+commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar
+commons-cli/1.2//commons-cli-1.2.jar
+commons-codec/1.10//commons-codec-1.10.jar
+commons-collections/3.2.2//commons-collections-3.2.2.jar
+commons-compiler/3.0.15//commons-compiler-3.0.15.jar
+commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-configuration/1.6//commons-configuration-1.6.jar
+commons-crypto/1.0.0//commons-crypto-1.0.0.jar
+commons-dbcp/1.4//commons-dbcp-1.4.jar
+commons-digester/1.8//commons-digester-1.8.jar
+commons-httpclient/3.1//commons-httpclient-3.1.jar
+commons-io/2.4//commons-io-2.4.jar
+commons-lang/2.6//commons-lang-2.6.jar
+commons-lang3/3.9//commons-lang3-3.9.jar
+commons-logging/1.1.3//commons-logging-1.1.3.jar
+commons-math3/3.4.1//commons-math3-3.4.1.jar
+commons-net/3.1//commons-net-3.1.jar
+commons-pool/1.5.4//commons-pool-1.5.4.jar
+commons-text/1.6//commons-text-1.6.jar
+compress-lzf/1.0.3//compress-lzf-1.0.3.jar
+core/1.1.2//core-1.1.2.jar
+curator-client/2.7.1//curator-client-2.7.1.jar
+curator-framework/2.7.1//curator-framework-2.7.1.jar
+curator-recipes/2.7.1//curator-recipes-2.7.1.jar
+datanucleus-api-jdo/3.2.6//datanucleus-api-jdo-3.2.6.jar
+datanucleus-core/3.2.10//datanucleus-core-3.2.10.jar
+datanucleus-rdbms/3.2.9//datanucleus-rdbms-3.2.9.jar
+derby/10.12.1.1//derby-10.12.1.1.jar
+flatbuffers-java/1.9.0//flatbuffers-java-1.9.0.jar
+generex/1.0.2//generex-1.0.2.jar
+gson/2.2.4//gson-2.2.4.jar
+guava/14.0.1//guava-14.0.1.jar
+guice-servlet/3.0//guice-servlet-3.0.jar
+guice/3.0//guice-3.0.jar
+hadoop-annotations/2.7.4//hadoop-annotations-2.7.4.jar
+hadoop-auth/2.7.4//hadoop-auth-2.7.4.jar
+hadoop-client/2.7.4//hadoop-client-2.7.4.jar
+hadoop-common/2.7.4//hadoop-common-2.7.4.jar
+hadoop-hdfs/2.7.4//hadoop-hdfs-2.7.4.jar
+hadoop-mapreduce-client-app/2.7.4//hadoop-mapreduce-client-app-2.7.4.jar
+hadoop-mapreduce-client-common/2.7.4//hadoop-mapreduce-client-common-2.7.4.jar
+hadoop-mapreduce-client-core/2.7.4//hadoop-mapreduce-client-core-2.7.4.jar
+hadoop-mapreduce-client-jobclient/2.7.4//hadoop-mapreduce-client-jobclient-2.7.4.jar
+hadoop-mapreduce-client-shuffle/2.7.4//hadoop-mapreduce-client-shuffle-2.7.4.jar
+hadoop-yarn-api/2.7.4//hadoop-yarn-api-2.7.4.jar
+hadoop-yarn-client/2.7.4//hadoop-yarn-client-2.7.4.jar
+hadoop-yarn-common/2.7.4//hadoop-yarn-common-2.7.4.jar
+hadoop-yarn-server-common/2.7.4//hadoop-yarn-server-common-2.7.4.jar
+hadoop-yarn-server-web-proxy/2.7.4//hadoop-yarn-server-web-proxy-2.7.4.jar
+hk2-api/2.6.1//hk2-api-2.6.1.jar
+hk2-locator/2.6.1//hk2-locator-2.6.1.jar
+hk2-utils/2.6.1//hk2-utils-2.6.1.jar
+htrace-core/3.1.0-incubating//htrace-core-3.1.0-incubating.jar
+httpclient/4.5.6//httpclient-4.5.6.jar
+httpcore/4.4.12//httpcore-4.4.12.jar
+istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
+ivy/2.4.0//ivy-2.4.0.jar
+jackson-annotations/2.10.0//jackson-annotations-2.10.0.jar
+jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
+jackson-core/2.10.0//jackson-core-2.10.0.jar
+jackson-databind/2.10.0//jackson-databind-2.10.0.jar
+jackson-dataformat-yaml/2.10.0//jackson-dataformat-yaml-2.10.0.jar
+jackson-jaxrs/1.9.13//jackson-jaxrs-1.9.13.jar
+jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
+jackson-module-jaxb-annotations/2.10.0//jackson-module-jaxb-annotations-2.10.0.jar
+jackson-module-paranamer/2.10.0//jackson-module-paranamer-2.10.0.jar
+jackson-module-scala_2.12/2.10.0//jackson-module-scala_2.12-2.10.0.jar
+jackson-xc/1.9.13//jackson-xc-1.9.13.jar
+jakarta.activation-api/1.2.1//jakarta.activation-api-1.2.1.jar
+jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
+jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
+jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar
+jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar
+jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
+janino/3.0.15//janino-3.0.15.jar
+javassist/3.22.0-CR2//javassist-3.22.0-CR2.jar
+javax.inject/1//javax.inject-1.jar
+javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar
+javolution/5.5.1//javolution-5.5.1.jar
+jaxb-api/2.2.2//jaxb-api-2.2.2.jar
+jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
+jcl-over-slf4j/1.7.16//jcl-over-slf4j-1.7.16.jar
+jdo-api/3.0.1//jdo-api-3.0.1.jar
+jersey-client/2.29.1//jersey-client-2.29.1.jar
+jersey-common/2.29.1//jersey-common-2.29.1.jar
+jersey-container-servlet-core/2.29.1//jersey-container-servlet-core-2.29.1.jar
+jersey-container-servlet/2.29.1//jersey-container-servlet-2.29.1.jar
+jersey-hk2/2.29.1//jersey-hk2-2.29.1.jar
+jersey-media-jaxb/2.29.1//jersey-media-jaxb-2.29.1.jar
+jersey-server/2.29.1//jersey-server-2.29.1.jar
+jetty-sslengine/6.1.26//jetty-sslengine-6.1.26.jar
+jetty-util/6.1.26//jetty-util-6.1.26.jar
+jetty/6.1.26//jetty-6.1.26.jar
+jline/2.14.6//jline-2.14.6.jar
+joda-time/2.10.5//joda-time-2.10.5.jar
+jodd-core/3.5.2//jodd-core-3.5.2.jar
+jpam/1.1//jpam-1.1.jar
+json4s-ast_2.12/3.6.6//json4s-ast_2.12-3.6.6.jar
+json4s-core_2.12/3.6.6//json4s-core_2.12-3.6.6.jar
+json4s-jackson_2.12/3.6.6//json4s-jackson_2.12-3.6.6.jar
+json4s-scalap_2.12/3.6.6//json4s-scalap_2.12-3.6.6.jar
+jsp-api/2.1//jsp-api-2.1.jar
+jsr305/3.0.0//jsr305-3.0.0.jar
+jta/1.1//jta-1.1.jar
+jul-to-slf4j/1.7.16//jul-to-slf4j-1.7.16.jar
+kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
+kubernetes-client/4.6.4//kubernetes-client-4.6.4.jar
+kubernetes-model-common/4.6.4//kubernetes-model-common-4.6.4.jar
+kubernetes-model/4.6.4//kubernetes-model-4.6.4.jar
+leveldbjni-all/1.8//leveldbjni-all-1.8.jar
+libfb303/0.9.3//libfb303-0.9.3.jar
+libthrift/0.12.0//libthrift-0.12.0.jar
+log4j/1.2.17//log4j-1.2.17.jar
+logging-interceptor/3.12.6//logging-interceptor-3.12.6.jar
+lz4-java/1.7.0//lz4-java-1.7.0.jar
+machinist_2.12/0.6.8//machinist_2.12-0.6.8.jar
+macro-compat_2.12/1.1.1//macro-compat_2.12-1.1.1.jar
+mesos/1.4.0/shaded-protobuf/mesos-1.4.0-shaded-protobuf.jar
+metrics-core/4.1.1//metrics-core-4.1.1.jar
+metrics-graphite/4.1.1//metrics-graphite-4.1.1.jar
+metrics-jmx/4.1.1//metrics-jmx-4.1.1.jar
+metrics-json/4.1.1//metrics-json-4.1.1.jar
+metrics-jvm/4.1.1//metrics-jvm-4.1.1.jar
+minlog/1.3.0//minlog-1.3.0.jar
+netty-all/4.1.42.Final//netty-all-4.1.42.Final.jar
+objenesis/2.5.1//objenesis-2.5.1.jar
+okhttp/3.12.6//okhttp-3.12.6.jar
+okio/1.15.0//okio-1.15.0.jar
+opencsv/2.3//opencsv-2.3.jar
+orc-core/1.5.8/nohive/orc-core-1.5.8-nohive.jar
+orc-mapreduce/1.5.8/nohive/orc-mapreduce-1.5.8-nohive.jar
+orc-shims/1.5.8//orc-shims-1.5.8.jar
+oro/2.0.8//oro-2.0.8.jar
+osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
+paranamer/2.8//paranamer-2.8.jar
+parquet-column/1.10.1//parquet-column-1.10.1.jar
+parquet-common/1.10.1//parquet-common-1.10.1.jar
+parquet-encoding/1.10.1//parquet-encoding-1.10.1.jar
+parquet-format/2.4.0//parquet-format-2.4.0.jar
+parquet-hadoop-bundle/1.6.0//parquet-hadoop-bundle-1.6.0.jar
+parquet-hadoop/1.10.1//parquet-hadoop-1.10.1.jar
+parquet-jackson/1.10.1//parquet-jackson-1.10.1.jar
+protobuf-java/2.5.0//protobuf-java-2.5.0.jar
+py4j/0.10.8.1//py4j-0.10.8.1.jar
+pyrolite/4.30//pyrolite-4.30.jar
+scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
+scala-compiler/2.12.10//scala-compiler-2.12.10.jar
+scala-library/2.12.10//scala-library-2.12.10.jar
+scala-parser-combinators_2.12/1.1.2//scala-parser-combinators_2.12-1.1.2.jar
+scala-reflect/2.12.10//scala-reflect-2.12.10.jar
+scala-xml_2.12/1.2.0//scala-xml_2.12-1.2.0.jar
+shapeless_2.12/2.3.3//shapeless_2.12-2.3.3.jar
+shims/0.7.45//shims-0.7.45.jar
+slf4j-api/1.7.16//slf4j-api-1.7.16.jar
+slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
+snakeyaml/1.24//snakeyaml-1.24.jar
+snappy-java/1.1.7.3//snappy-java-1.1.7.3.jar
+snappy/0.2//snappy-0.2.jar
+spire-macros_2.12/0.17.0-M1//spire-macros_2.12-0.17.0-M1.jar
+spire-platform_2.12/0.17.0-M1//spire-platform_2.12-0.17.0-M1.jar
+spire-util_2.12/0.17.0-M1//spire-util_2.12-0.17.0-M1.jar
+spire_2.12/0.17.0-M1//spire_2.12-0.17.0-M1.jar
+stax-api/1.0-2//stax-api-1.0-2.jar
+stax-api/1.0.1//stax-api-1.0.1.jar
+stream/2.9.6//stream-2.9.6.jar
+stringtemplate/3.2.1//stringtemplate-3.2.1.jar
+super-csv/2.2.0//super-csv-2.2.0.jar
+univocity-parsers/2.8.3//univocity-parsers-2.8.3.jar
+xbean-asm7-shaded/4.15//xbean-asm7-shaded-4.15.jar
+xercesImpl/2.9.1//xercesImpl-2.9.1.jar
+xmlenc/0.52//xmlenc-0.52.jar
+xz/1.5//xz-1.5.jar
+zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
+zookeeper/3.4.14//zookeeper-3.4.14.jar
+zstd-jni/1.4.4-3//zstd-jni-1.4.4-3.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
index 300c3ee..9bfd801 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
@@ -1,223 +1,223 @@
-HikariCP-2.5.1.jar
-JLargeArrays-1.5.jar
-JTransforms-3.1.jar
-RoaringBitmap-0.7.45.jar
-ST4-4.0.4.jar
-activation-1.1.1.jar
-aircompressor-0.10.jar
-algebra_2.12-2.0.0-M2.jar
-antlr-runtime-3.5.2.jar
-antlr4-runtime-4.7.1.jar
-aopalliance-1.0.jar
-aopalliance-repackaged-2.6.1.jar
-apacheds-i18n-2.0.0-M15.jar
-apacheds-kerberos-codec-2.0.0-M15.jar
-api-asn1-api-1.0.0-M20.jar
-api-util-1.0.0-M20.jar
-arpack_combined_all-0.1.jar
-arrow-format-0.15.1.jar
-arrow-memory-0.15.1.jar
-arrow-vector-0.15.1.jar
-audience-annotations-0.5.0.jar
-automaton-1.11-8.jar
-avro-1.8.2.jar
-avro-ipc-1.8.2.jar
-avro-mapred-1.8.2-hadoop2.jar
-bonecp-0.8.0.RELEASE.jar
-breeze-macros_2.12-1.0.jar
-breeze_2.12-1.0.jar
-cats-kernel_2.12-2.0.0-M4.jar
-chill-java-0.9.3.jar
-chill_2.12-0.9.3.jar
-commons-beanutils-1.9.4.jar
-commons-cli-1.2.jar
-commons-codec-1.10.jar
-commons-collections-3.2.2.jar
-commons-compiler-3.0.15.jar
-commons-compress-1.8.1.jar
-commons-configuration-1.6.jar
-commons-crypto-1.0.0.jar
-commons-dbcp-1.4.jar
-commons-digester-1.8.jar
-commons-httpclient-3.1.jar
-commons-io-2.4.jar
-commons-lang-2.6.jar
-commons-lang3-3.9.jar
-commons-logging-1.1.3.jar
-commons-math3-3.4.1.jar
-commons-net-3.1.jar
-commons-pool-1.5.4.jar
-commons-text-1.6.jar
-compress-lzf-1.0.3.jar
-core-1.1.2.jar
-curator-client-2.7.1.jar
-curator-framework-2.7.1.jar
-curator-recipes-2.7.1.jar
-datanucleus-api-jdo-4.2.4.jar
-datanucleus-core-4.1.17.jar
-datanucleus-rdbms-4.1.19.jar
-derby-10.12.1.1.jar
-dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
-flatbuffers-java-1.9.0.jar
-generex-1.0.2.jar
-gson-2.2.4.jar
-guava-14.0.1.jar
-guice-3.0.jar
-guice-servlet-3.0.jar
-hadoop-annotations-2.7.4.jar
-hadoop-auth-2.7.4.jar
-hadoop-client-2.7.4.jar
-hadoop-common-2.7.4.jar
-hadoop-hdfs-2.7.4.jar
-hadoop-mapreduce-client-app-2.7.4.jar
-hadoop-mapreduce-client-common-2.7.4.jar
-hadoop-mapreduce-client-core-2.7.4.jar
-hadoop-mapreduce-client-jobclient-2.7.4.jar
-hadoop-mapreduce-client-shuffle-2.7.4.jar
-hadoop-yarn-api-2.7.4.jar
-hadoop-yarn-client-2.7.4.jar
-hadoop-yarn-common-2.7.4.jar
-hadoop-yarn-server-common-2.7.4.jar
-hadoop-yarn-server-web-proxy-2.7.4.jar
-hive-beeline-2.3.6.jar
-hive-cli-2.3.6.jar
-hive-common-2.3.6.jar
-hive-exec-2.3.6-core.jar
-hive-jdbc-2.3.6.jar
-hive-llap-common-2.3.6.jar
-hive-metastore-2.3.6.jar
-hive-serde-2.3.6.jar
-hive-service-rpc-2.3.6.jar
-hive-shims-0.23-2.3.6.jar
-hive-shims-2.3.6.jar
-hive-shims-common-2.3.6.jar
-hive-shims-scheduler-2.3.6.jar
-hive-storage-api-2.6.0.jar
-hive-vector-code-gen-2.3.6.jar
-hk2-api-2.6.1.jar
-hk2-locator-2.6.1.jar
-hk2-utils-2.6.1.jar
-htrace-core-3.1.0-incubating.jar
-httpclient-4.5.6.jar
-httpcore-4.4.12.jar
-istack-commons-runtime-3.0.8.jar
-ivy-2.4.0.jar
-jackson-annotations-2.10.0.jar
-jackson-core-2.10.0.jar
-jackson-core-asl-1.9.13.jar
-jackson-databind-2.10.0.jar
-jackson-dataformat-yaml-2.10.0.jar
-jackson-jaxrs-1.9.13.jar
-jackson-mapper-asl-1.9.13.jar
-jackson-module-jaxb-annotations-2.10.0.jar
-jackson-module-paranamer-2.10.0.jar
-jackson-module-scala_2.12-2.10.0.jar
-jackson-xc-1.9.13.jar
-jakarta.activation-api-1.2.1.jar
-jakarta.annotation-api-1.3.5.jar
-jakarta.inject-2.6.1.jar
-jakarta.validation-api-2.0.2.jar
-jakarta.ws.rs-api-2.1.6.jar
-jakarta.xml.bind-api-2.3.2.jar
-janino-3.0.15.jar
-javassist-3.22.0-CR2.jar
-javax.inject-1.jar
-javax.jdo-3.2.0-m3.jar
-javax.servlet-api-3.1.0.jar
-javolution-5.5.1.jar
-jaxb-api-2.2.2.jar
-jaxb-runtime-2.3.2.jar
-jcl-over-slf4j-1.7.16.jar
-jdo-api-3.0.1.jar
-jersey-client-2.29.1.jar
-jersey-common-2.29.1.jar
-jersey-container-servlet-2.29.1.jar
-jersey-container-servlet-core-2.29.1.jar
-jersey-hk2-2.29.1.jar
-jersey-media-jaxb-2.29.1.jar
-jersey-server-2.29.1.jar
-jetty-6.1.26.jar
-jetty-sslengine-6.1.26.jar
-jetty-util-6.1.26.jar
-jline-2.14.6.jar
-joda-time-2.10.5.jar
-jodd-core-3.5.2.jar
-jpam-1.1.jar
-json-1.8.jar
-json4s-ast_2.12-3.6.6.jar
-json4s-core_2.12-3.6.6.jar
-json4s-jackson_2.12-3.6.6.jar
-json4s-scalap_2.12-3.6.6.jar
-jsp-api-2.1.jar
-jsr305-3.0.0.jar
-jta-1.1.jar
-jul-to-slf4j-1.7.16.jar
-kryo-shaded-4.0.2.jar
-kubernetes-client-4.6.4.jar
-kubernetes-model-4.6.4.jar
-kubernetes-model-common-4.6.4.jar
-leveldbjni-all-1.8.jar
-libfb303-0.9.3.jar
-libthrift-0.12.0.jar
-log4j-1.2.17.jar
-logging-interceptor-3.12.6.jar
-lz4-java-1.7.0.jar
-machinist_2.12-0.6.8.jar
-macro-compat_2.12-1.1.1.jar
-mesos-1.4.0-shaded-protobuf.jar
-metrics-core-4.1.1.jar
-metrics-graphite-4.1.1.jar
-metrics-jmx-4.1.1.jar
-metrics-json-4.1.1.jar
-metrics-jvm-4.1.1.jar
-minlog-1.3.0.jar
-netty-all-4.1.42.Final.jar
-objenesis-2.5.1.jar
-okhttp-3.12.6.jar
-okio-1.15.0.jar
-opencsv-2.3.jar
-orc-core-1.5.8.jar
-orc-mapreduce-1.5.8.jar
-orc-shims-1.5.8.jar
-oro-2.0.8.jar
-osgi-resource-locator-1.0.3.jar
-paranamer-2.8.jar
-parquet-column-1.10.1.jar
-parquet-common-1.10.1.jar
-parquet-encoding-1.10.1.jar
-parquet-format-2.4.0.jar
-parquet-hadoop-1.10.1.jar
-parquet-jackson-1.10.1.jar
-protobuf-java-2.5.0.jar
-py4j-0.10.8.1.jar
-pyrolite-4.30.jar
-scala-collection-compat_2.12-2.1.1.jar
-scala-compiler-2.12.10.jar
-scala-library-2.12.10.jar
-scala-parser-combinators_2.12-1.1.2.jar
-scala-reflect-2.12.10.jar
-scala-xml_2.12-1.2.0.jar
-shapeless_2.12-2.3.3.jar
-shims-0.7.45.jar
-slf4j-api-1.7.16.jar
-slf4j-log4j12-1.7.16.jar
-snakeyaml-1.24.jar
-snappy-java-1.1.7.3.jar
-spire-macros_2.12-0.17.0-M1.jar
-spire-platform_2.12-0.17.0-M1.jar
-spire-util_2.12-0.17.0-M1.jar
-spire_2.12-0.17.0-M1.jar
-stax-api-1.0-2.jar
-stax-api-1.0.1.jar
-stream-2.9.6.jar
-super-csv-2.2.0.jar
-transaction-api-1.1.jar
-univocity-parsers-2.8.3.jar
-velocity-1.5.jar
-xbean-asm7-shaded-4.15.jar
-xercesImpl-2.9.1.jar
-xmlenc-0.52.jar
-xz-1.5.jar
-zjsonpatch-0.3.0.jar
-zookeeper-3.4.14.jar
-zstd-jni-1.4.4-3.jar
+HikariCP/2.5.1//HikariCP-2.5.1.jar
+JLargeArrays/1.5//JLargeArrays-1.5.jar
+JTransforms/3.1//JTransforms-3.1.jar
+RoaringBitmap/0.7.45//RoaringBitmap-0.7.45.jar
+ST4/4.0.4//ST4-4.0.4.jar
+activation/1.1.1//activation-1.1.1.jar
+aircompressor/0.10//aircompressor-0.10.jar
+algebra_2.12/2.0.0-M2//algebra_2.12-2.0.0-M2.jar
+antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar
+antlr4-runtime/4.7.1//antlr4-runtime-4.7.1.jar
+aopalliance-repackaged/2.6.1//aopalliance-repackaged-2.6.1.jar
+aopalliance/1.0//aopalliance-1.0.jar
+apacheds-i18n/2.0.0-M15//apacheds-i18n-2.0.0-M15.jar
+apacheds-kerberos-codec/2.0.0-M15//apacheds-kerberos-codec-2.0.0-M15.jar
+api-asn1-api/1.0.0-M20//api-asn1-api-1.0.0-M20.jar
+api-util/1.0.0-M20//api-util-1.0.0-M20.jar
+arpack_combined_all/0.1//arpack_combined_all-0.1.jar
+arrow-format/0.15.1//arrow-format-0.15.1.jar
+arrow-memory/0.15.1//arrow-memory-0.15.1.jar
+arrow-vector/0.15.1//arrow-vector-0.15.1.jar
+audience-annotations/0.5.0//audience-annotations-0.5.0.jar
+automaton/1.11-8//automaton-1.11-8.jar
+avro-ipc/1.8.2//avro-ipc-1.8.2.jar
+avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar
+avro/1.8.2//avro-1.8.2.jar
+bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar
+breeze-macros_2.12/1.0//breeze-macros_2.12-1.0.jar
+breeze_2.12/1.0//breeze_2.12-1.0.jar
+cats-kernel_2.12/2.0.0-M4//cats-kernel_2.12-2.0.0-M4.jar
+chill-java/0.9.3//chill-java-0.9.3.jar
+chill_2.12/0.9.3//chill_2.12-0.9.3.jar
+commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar
+commons-cli/1.2//commons-cli-1.2.jar
+commons-codec/1.10//commons-codec-1.10.jar
+commons-collections/3.2.2//commons-collections-3.2.2.jar
+commons-compiler/3.0.15//commons-compiler-3.0.15.jar
+commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-configuration/1.6//commons-configuration-1.6.jar
+commons-crypto/1.0.0//commons-crypto-1.0.0.jar
+commons-dbcp/1.4//commons-dbcp-1.4.jar
+commons-digester/1.8//commons-digester-1.8.jar
+commons-httpclient/3.1//commons-httpclient-3.1.jar
+commons-io/2.4//commons-io-2.4.jar
+commons-lang/2.6//commons-lang-2.6.jar
+commons-lang3/3.9//commons-lang3-3.9.jar
+commons-logging/1.1.3//commons-logging-1.1.3.jar
+commons-math3/3.4.1//commons-math3-3.4.1.jar
+commons-net/3.1//commons-net-3.1.jar
+commons-pool/1.5.4//commons-pool-1.5.4.jar
+commons-text/1.6//commons-text-1.6.jar
+compress-lzf/1.0.3//compress-lzf-1.0.3.jar
+core/1.1.2//core-1.1.2.jar
+curator-client/2.7.1//curator-client-2.7.1.jar
+curator-framework/2.7.1//curator-framework-2.7.1.jar
+curator-recipes/2.7.1//curator-recipes-2.7.1.jar
+datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar
+datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
+datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
+derby/10.12.1.1//derby-10.12.1.1.jar
+dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+flatbuffers-java/1.9.0//flatbuffers-java-1.9.0.jar
+generex/1.0.2//generex-1.0.2.jar
+gson/2.2.4//gson-2.2.4.jar
+guava/14.0.1//guava-14.0.1.jar
+guice-servlet/3.0//guice-servlet-3.0.jar
+guice/3.0//guice-3.0.jar
+hadoop-annotations/2.7.4//hadoop-annotations-2.7.4.jar
+hadoop-auth/2.7.4//hadoop-auth-2.7.4.jar
+hadoop-client/2.7.4//hadoop-client-2.7.4.jar
+hadoop-common/2.7.4//hadoop-common-2.7.4.jar
+hadoop-hdfs/2.7.4//hadoop-hdfs-2.7.4.jar
+hadoop-mapreduce-client-app/2.7.4//hadoop-mapreduce-client-app-2.7.4.jar
+hadoop-mapreduce-client-common/2.7.4//hadoop-mapreduce-client-common-2.7.4.jar
+hadoop-mapreduce-client-core/2.7.4//hadoop-mapreduce-client-core-2.7.4.jar
+hadoop-mapreduce-client-jobclient/2.7.4//hadoop-mapreduce-client-jobclient-2.7.4.jar
+hadoop-mapreduce-client-shuffle/2.7.4//hadoop-mapreduce-client-shuffle-2.7.4.jar
+hadoop-yarn-api/2.7.4//hadoop-yarn-api-2.7.4.jar
+hadoop-yarn-client/2.7.4//hadoop-yarn-client-2.7.4.jar
+hadoop-yarn-common/2.7.4//hadoop-yarn-common-2.7.4.jar
+hadoop-yarn-server-common/2.7.4//hadoop-yarn-server-common-2.7.4.jar
+hadoop-yarn-server-web-proxy/2.7.4//hadoop-yarn-server-web-proxy-2.7.4.jar
+hive-beeline/2.3.6//hive-beeline-2.3.6.jar
+hive-cli/2.3.6//hive-cli-2.3.6.jar
+hive-common/2.3.6//hive-common-2.3.6.jar
+hive-exec/2.3.6/core/hive-exec-2.3.6-core.jar
+hive-jdbc/2.3.6//hive-jdbc-2.3.6.jar
+hive-llap-common/2.3.6//hive-llap-common-2.3.6.jar
+hive-metastore/2.3.6//hive-metastore-2.3.6.jar
+hive-serde/2.3.6//hive-serde-2.3.6.jar
+hive-service-rpc/2.3.6//hive-service-rpc-2.3.6.jar
+hive-shims-0.23/2.3.6//hive-shims-0.23-2.3.6.jar
+hive-shims-common/2.3.6//hive-shims-common-2.3.6.jar
+hive-shims-scheduler/2.3.6//hive-shims-scheduler-2.3.6.jar
+hive-shims/2.3.6//hive-shims-2.3.6.jar
+hive-storage-api/2.6.0//hive-storage-api-2.6.0.jar
+hive-vector-code-gen/2.3.6//hive-vector-code-gen-2.3.6.jar
+hk2-api/2.6.1//hk2-api-2.6.1.jar
+hk2-locator/2.6.1//hk2-locator-2.6.1.jar
+hk2-utils/2.6.1//hk2-utils-2.6.1.jar
+htrace-core/3.1.0-incubating//htrace-core-3.1.0-incubating.jar
+httpclient/4.5.6//httpclient-4.5.6.jar
+httpcore/4.4.12//httpcore-4.4.12.jar
+istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
+ivy/2.4.0//ivy-2.4.0.jar
+jackson-annotations/2.10.0//jackson-annotations-2.10.0.jar
+jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
+jackson-core/2.10.0//jackson-core-2.10.0.jar
+jackson-databind/2.10.0//jackson-databind-2.10.0.jar
+jackson-dataformat-yaml/2.10.0//jackson-dataformat-yaml-2.10.0.jar
+jackson-jaxrs/1.9.13//jackson-jaxrs-1.9.13.jar
+jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
+jackson-module-jaxb-annotations/2.10.0//jackson-module-jaxb-annotations-2.10.0.jar
+jackson-module-paranamer/2.10.0//jackson-module-paranamer-2.10.0.jar
+jackson-module-scala_2.12/2.10.0//jackson-module-scala_2.12-2.10.0.jar
+jackson-xc/1.9.13//jackson-xc-1.9.13.jar
+jakarta.activation-api/1.2.1//jakarta.activation-api-1.2.1.jar
+jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
+jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
+jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar
+jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar
+jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
+janino/3.0.15//janino-3.0.15.jar
+javassist/3.22.0-CR2//javassist-3.22.0-CR2.jar
+javax.inject/1//javax.inject-1.jar
+javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
+javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar
+javolution/5.5.1//javolution-5.5.1.jar
+jaxb-api/2.2.2//jaxb-api-2.2.2.jar
+jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
+jcl-over-slf4j/1.7.16//jcl-over-slf4j-1.7.16.jar
+jdo-api/3.0.1//jdo-api-3.0.1.jar
+jersey-client/2.29.1//jersey-client-2.29.1.jar
+jersey-common/2.29.1//jersey-common-2.29.1.jar
+jersey-container-servlet-core/2.29.1//jersey-container-servlet-core-2.29.1.jar
+jersey-container-servlet/2.29.1//jersey-container-servlet-2.29.1.jar
+jersey-hk2/2.29.1//jersey-hk2-2.29.1.jar
+jersey-media-jaxb/2.29.1//jersey-media-jaxb-2.29.1.jar
+jersey-server/2.29.1//jersey-server-2.29.1.jar
+jetty-sslengine/6.1.26//jetty-sslengine-6.1.26.jar
+jetty-util/6.1.26//jetty-util-6.1.26.jar
+jetty/6.1.26//jetty-6.1.26.jar
+jline/2.14.6//jline-2.14.6.jar
+joda-time/2.10.5//joda-time-2.10.5.jar
+jodd-core/3.5.2//jodd-core-3.5.2.jar
+jpam/1.1//jpam-1.1.jar
+json/1.8//json-1.8.jar
+json4s-ast_2.12/3.6.6//json4s-ast_2.12-3.6.6.jar
+json4s-core_2.12/3.6.6//json4s-core_2.12-3.6.6.jar
+json4s-jackson_2.12/3.6.6//json4s-jackson_2.12-3.6.6.jar
+json4s-scalap_2.12/3.6.6//json4s-scalap_2.12-3.6.6.jar
+jsp-api/2.1//jsp-api-2.1.jar
+jsr305/3.0.0//jsr305-3.0.0.jar
+jta/1.1//jta-1.1.jar
+jul-to-slf4j/1.7.16//jul-to-slf4j-1.7.16.jar
+kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
+kubernetes-client/4.6.4//kubernetes-client-4.6.4.jar
+kubernetes-model-common/4.6.4//kubernetes-model-common-4.6.4.jar
+kubernetes-model/4.6.4//kubernetes-model-4.6.4.jar
+leveldbjni-all/1.8//leveldbjni-all-1.8.jar
+libfb303/0.9.3//libfb303-0.9.3.jar
+libthrift/0.12.0//libthrift-0.12.0.jar
+log4j/1.2.17//log4j-1.2.17.jar
+logging-interceptor/3.12.6//logging-interceptor-3.12.6.jar
+lz4-java/1.7.0//lz4-java-1.7.0.jar
+machinist_2.12/0.6.8//machinist_2.12-0.6.8.jar
+macro-compat_2.12/1.1.1//macro-compat_2.12-1.1.1.jar
+mesos/1.4.0/shaded-protobuf/mesos-1.4.0-shaded-protobuf.jar
+metrics-core/4.1.1//metrics-core-4.1.1.jar
+metrics-graphite/4.1.1//metrics-graphite-4.1.1.jar
+metrics-jmx/4.1.1//metrics-jmx-4.1.1.jar
+metrics-json/4.1.1//metrics-json-4.1.1.jar
+metrics-jvm/4.1.1//metrics-jvm-4.1.1.jar
+minlog/1.3.0//minlog-1.3.0.jar
+netty-all/4.1.42.Final//netty-all-4.1.42.Final.jar
+objenesis/2.5.1//objenesis-2.5.1.jar
+okhttp/3.12.6//okhttp-3.12.6.jar
+okio/1.15.0//okio-1.15.0.jar
+opencsv/2.3//opencsv-2.3.jar
+orc-core/1.5.8//orc-core-1.5.8.jar
+orc-mapreduce/1.5.8//orc-mapreduce-1.5.8.jar
+orc-shims/1.5.8//orc-shims-1.5.8.jar
+oro/2.0.8//oro-2.0.8.jar
+osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
+paranamer/2.8//paranamer-2.8.jar
+parquet-column/1.10.1//parquet-column-1.10.1.jar
+parquet-common/1.10.1//parquet-common-1.10.1.jar
+parquet-encoding/1.10.1//parquet-encoding-1.10.1.jar
+parquet-format/2.4.0//parquet-format-2.4.0.jar
+parquet-hadoop/1.10.1//parquet-hadoop-1.10.1.jar
+parquet-jackson/1.10.1//parquet-jackson-1.10.1.jar
+protobuf-java/2.5.0//protobuf-java-2.5.0.jar
+py4j/0.10.8.1//py4j-0.10.8.1.jar
+pyrolite/4.30//pyrolite-4.30.jar
+scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
+scala-compiler/2.12.10//scala-compiler-2.12.10.jar
+scala-library/2.12.10//scala-library-2.12.10.jar
+scala-parser-combinators_2.12/1.1.2//scala-parser-combinators_2.12-1.1.2.jar
+scala-reflect/2.12.10//scala-reflect-2.12.10.jar
+scala-xml_2.12/1.2.0//scala-xml_2.12-1.2.0.jar
+shapeless_2.12/2.3.3//shapeless_2.12-2.3.3.jar
+shims/0.7.45//shims-0.7.45.jar
+slf4j-api/1.7.16//slf4j-api-1.7.16.jar
+slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
+snakeyaml/1.24//snakeyaml-1.24.jar
+snappy-java/1.1.7.3//snappy-java-1.1.7.3.jar
+spire-macros_2.12/0.17.0-M1//spire-macros_2.12-0.17.0-M1.jar
+spire-platform_2.12/0.17.0-M1//spire-platform_2.12-0.17.0-M1.jar
+spire-util_2.12/0.17.0-M1//spire-util_2.12-0.17.0-M1.jar
+spire_2.12/0.17.0-M1//spire_2.12-0.17.0-M1.jar
+stax-api/1.0-2//stax-api-1.0-2.jar
+stax-api/1.0.1//stax-api-1.0.1.jar
+stream/2.9.6//stream-2.9.6.jar
+super-csv/2.2.0//super-csv-2.2.0.jar
+transaction-api/1.1//transaction-api-1.1.jar
+univocity-parsers/2.8.3//univocity-parsers-2.8.3.jar
+velocity/1.5//velocity-1.5.jar
+xbean-asm7-shaded/4.15//xbean-asm7-shaded-4.15.jar
+xercesImpl/2.9.1//xercesImpl-2.9.1.jar
+xmlenc/0.52//xmlenc-0.52.jar
+xz/1.5//xz-1.5.jar
+zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
+zookeeper/3.4.14//zookeeper-3.4.14.jar
+zstd-jni/1.4.4-3//zstd-jni-1.4.4-3.jar
diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
index aa1a8cc..da2020b 100644
--- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
@@ -1,239 +1,239 @@
-HikariCP-2.5.1.jar
-JLargeArrays-1.5.jar
-JTransforms-3.1.jar
-RoaringBitmap-0.7.45.jar
-ST4-4.0.4.jar
-accessors-smart-1.2.jar
-activation-1.1.1.jar
-aircompressor-0.10.jar
-algebra_2.12-2.0.0-M2.jar
-antlr-runtime-3.5.2.jar
-antlr4-runtime-4.7.1.jar
-aopalliance-1.0.jar
-aopalliance-repackaged-2.6.1.jar
-arpack_combined_all-0.1.jar
-arrow-format-0.15.1.jar
-arrow-memory-0.15.1.jar
-arrow-vector-0.15.1.jar
-audience-annotations-0.5.0.jar
-automaton-1.11-8.jar
-avro-1.8.2.jar
-avro-ipc-1.8.2.jar
-avro-mapred-1.8.2-hadoop2.jar
-bonecp-0.8.0.RELEASE.jar
-breeze-macros_2.12-1.0.jar
-breeze_2.12-1.0.jar
-cats-kernel_2.12-2.0.0-M4.jar
-chill-java-0.9.3.jar
-chill_2.12-0.9.3.jar
-commons-beanutils-1.9.4.jar
-commons-cli-1.2.jar
-commons-codec-1.10.jar
-commons-collections-3.2.2.jar
-commons-compiler-3.0.15.jar
-commons-compress-1.8.1.jar
-commons-configuration2-2.1.1.jar
-commons-crypto-1.0.0.jar
-commons-daemon-1.0.13.jar
-commons-dbcp-1.4.jar
-commons-httpclient-3.1.jar
-commons-io-2.4.jar
-commons-lang-2.6.jar
-commons-lang3-3.9.jar
-commons-logging-1.1.3.jar
-commons-math3-3.4.1.jar
-commons-net-3.1.jar
-commons-pool-1.5.4.jar
-commons-text-1.6.jar
-compress-lzf-1.0.3.jar
-core-1.1.2.jar
-curator-client-2.13.0.jar
-curator-framework-2.13.0.jar
-curator-recipes-2.13.0.jar
-datanucleus-api-jdo-4.2.4.jar
-datanucleus-core-4.1.17.jar
-datanucleus-rdbms-4.1.19.jar
-derby-10.12.1.1.jar
-dnsjava-2.1.7.jar
-dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
-ehcache-3.3.1.jar
-flatbuffers-java-1.9.0.jar
-generex-1.0.2.jar
-geronimo-jcache_1.0_spec-1.0-alpha-1.jar
-gson-2.2.4.jar
-guava-14.0.1.jar
-guice-4.0.jar
-guice-servlet-4.0.jar
-hadoop-annotations-3.2.0.jar
-hadoop-auth-3.2.0.jar
-hadoop-client-3.2.0.jar
-hadoop-common-3.2.0.jar
-hadoop-hdfs-client-3.2.0.jar
-hadoop-mapreduce-client-common-3.2.0.jar
-hadoop-mapreduce-client-core-3.2.0.jar
-hadoop-mapreduce-client-jobclient-3.2.0.jar
-hadoop-yarn-api-3.2.0.jar
-hadoop-yarn-client-3.2.0.jar
-hadoop-yarn-common-3.2.0.jar
-hadoop-yarn-registry-3.2.0.jar
-hadoop-yarn-server-common-3.2.0.jar
-hadoop-yarn-server-web-proxy-3.2.0.jar
-hive-beeline-2.3.6.jar
-hive-cli-2.3.6.jar
-hive-common-2.3.6.jar
-hive-exec-2.3.6-core.jar
-hive-jdbc-2.3.6.jar
-hive-llap-common-2.3.6.jar
-hive-metastore-2.3.6.jar
-hive-serde-2.3.6.jar
-hive-service-rpc-2.3.6.jar
-hive-shims-0.23-2.3.6.jar
-hive-shims-2.3.6.jar
-hive-shims-common-2.3.6.jar
-hive-shims-scheduler-2.3.6.jar
-hive-storage-api-2.6.0.jar
-hive-vector-code-gen-2.3.6.jar
-hk2-api-2.6.1.jar
-hk2-locator-2.6.1.jar
-hk2-utils-2.6.1.jar
-htrace-core4-4.1.0-incubating.jar
-httpclient-4.5.6.jar
-httpcore-4.4.12.jar
-istack-commons-runtime-3.0.8.jar
-ivy-2.4.0.jar
-jackson-annotations-2.10.0.jar
-jackson-core-2.10.0.jar
-jackson-core-asl-1.9.13.jar
-jackson-databind-2.10.0.jar
-jackson-dataformat-yaml-2.10.0.jar
-jackson-jaxrs-base-2.9.5.jar
-jackson-jaxrs-json-provider-2.9.5.jar
-jackson-mapper-asl-1.9.13.jar
-jackson-module-jaxb-annotations-2.10.0.jar
-jackson-module-paranamer-2.10.0.jar
-jackson-module-scala_2.12-2.10.0.jar
-jakarta.activation-api-1.2.1.jar
-jakarta.annotation-api-1.3.5.jar
-jakarta.inject-2.6.1.jar
-jakarta.validation-api-2.0.2.jar
-jakarta.ws.rs-api-2.1.6.jar
-jakarta.xml.bind-api-2.3.2.jar
-janino-3.0.15.jar
-javassist-3.22.0-CR2.jar
-javax.inject-1.jar
-javax.jdo-3.2.0-m3.jar
-javax.servlet-api-3.1.0.jar
-javolution-5.5.1.jar
-jaxb-api-2.2.11.jar
-jaxb-runtime-2.3.2.jar
-jcip-annotations-1.0-1.jar
-jcl-over-slf4j-1.7.16.jar
-jdo-api-3.0.1.jar
-jersey-client-2.29.1.jar
-jersey-common-2.29.1.jar
-jersey-container-servlet-2.29.1.jar
-jersey-container-servlet-core-2.29.1.jar
-jersey-hk2-2.29.1.jar
-jersey-media-jaxb-2.29.1.jar
-jersey-server-2.29.1.jar
-jline-2.14.6.jar
-joda-time-2.10.5.jar
-jodd-core-3.5.2.jar
-jpam-1.1.jar
-json-1.8.jar
-json-smart-2.3.jar
-json4s-ast_2.12-3.6.6.jar
-json4s-core_2.12-3.6.6.jar
-json4s-jackson_2.12-3.6.6.jar
-json4s-scalap_2.12-3.6.6.jar
-jsp-api-2.1.jar
-jsr305-3.0.0.jar
-jta-1.1.jar
-jul-to-slf4j-1.7.16.jar
-kerb-admin-1.0.1.jar
-kerb-client-1.0.1.jar
-kerb-common-1.0.1.jar
-kerb-core-1.0.1.jar
-kerb-crypto-1.0.1.jar
-kerb-identity-1.0.1.jar
-kerb-server-1.0.1.jar
-kerb-simplekdc-1.0.1.jar
-kerb-util-1.0.1.jar
-kerby-asn1-1.0.1.jar
-kerby-config-1.0.1.jar
-kerby-pkix-1.0.1.jar
-kerby-util-1.0.1.jar
-kerby-xdr-1.0.1.jar
-kryo-shaded-4.0.2.jar
-kubernetes-client-4.6.4.jar
-kubernetes-model-4.6.4.jar
-kubernetes-model-common-4.6.4.jar
-leveldbjni-all-1.8.jar
-libfb303-0.9.3.jar
-libthrift-0.12.0.jar
-log4j-1.2.17.jar
-logging-interceptor-3.12.6.jar
-lz4-java-1.7.0.jar
-machinist_2.12-0.6.8.jar
-macro-compat_2.12-1.1.1.jar
-mesos-1.4.0-shaded-protobuf.jar
-metrics-core-4.1.1.jar
-metrics-graphite-4.1.1.jar
-metrics-jmx-4.1.1.jar
-metrics-json-4.1.1.jar
-metrics-jvm-4.1.1.jar
-minlog-1.3.0.jar
-mssql-jdbc-6.2.1.jre7.jar
-netty-all-4.1.42.Final.jar
-nimbus-jose-jwt-4.41.1.jar
-objenesis-2.5.1.jar
-okhttp-2.7.5.jar
-okhttp-3.12.6.jar
-okio-1.15.0.jar
-opencsv-2.3.jar
-orc-core-1.5.8.jar
-orc-mapreduce-1.5.8.jar
-orc-shims-1.5.8.jar
-oro-2.0.8.jar
-osgi-resource-locator-1.0.3.jar
-paranamer-2.8.jar
-parquet-column-1.10.1.jar
-parquet-common-1.10.1.jar
-parquet-encoding-1.10.1.jar
-parquet-format-2.4.0.jar
-parquet-hadoop-1.10.1.jar
-parquet-jackson-1.10.1.jar
-protobuf-java-2.5.0.jar
-py4j-0.10.8.1.jar
-pyrolite-4.30.jar
-re2j-1.1.jar
-scala-collection-compat_2.12-2.1.1.jar
-scala-compiler-2.12.10.jar
-scala-library-2.12.10.jar
-scala-parser-combinators_2.12-1.1.2.jar
-scala-reflect-2.12.10.jar
-scala-xml_2.12-1.2.0.jar
-shapeless_2.12-2.3.3.jar
-shims-0.7.45.jar
-slf4j-api-1.7.16.jar
-slf4j-log4j12-1.7.16.jar
-snakeyaml-1.24.jar
-snappy-java-1.1.7.3.jar
-spire-macros_2.12-0.17.0-M1.jar
-spire-platform_2.12-0.17.0-M1.jar
-spire-util_2.12-0.17.0-M1.jar
-spire_2.12-0.17.0-M1.jar
-stax-api-1.0.1.jar
-stax2-api-3.1.4.jar
-stream-2.9.6.jar
-super-csv-2.2.0.jar
-token-provider-1.0.1.jar
-transaction-api-1.1.jar
-univocity-parsers-2.8.3.jar
-velocity-1.5.jar
-woodstox-core-5.0.3.jar
-xbean-asm7-shaded-4.15.jar
-xz-1.5.jar
-zjsonpatch-0.3.0.jar
-zookeeper-3.4.14.jar
-zstd-jni-1.4.4-3.jar
+HikariCP/2.5.1//HikariCP-2.5.1.jar
+JLargeArrays/1.5//JLargeArrays-1.5.jar
+JTransforms/3.1//JTransforms-3.1.jar
+RoaringBitmap/0.7.45//RoaringBitmap-0.7.45.jar
+ST4/4.0.4//ST4-4.0.4.jar
+accessors-smart/1.2//accessors-smart-1.2.jar
+activation/1.1.1//activation-1.1.1.jar
+aircompressor/0.10//aircompressor-0.10.jar
+algebra_2.12/2.0.0-M2//algebra_2.12-2.0.0-M2.jar
+antlr-runtime/3.5.2//antlr-runtime-3.5.2.jar
+antlr4-runtime/4.7.1//antlr4-runtime-4.7.1.jar
+aopalliance-repackaged/2.6.1//aopalliance-repackaged-2.6.1.jar
+aopalliance/1.0//aopalliance-1.0.jar
+arpack_combined_all/0.1//arpack_combined_all-0.1.jar
+arrow-format/0.15.1//arrow-format-0.15.1.jar
+arrow-memory/0.15.1//arrow-memory-0.15.1.jar
+arrow-vector/0.15.1//arrow-vector-0.15.1.jar
+audience-annotations/0.5.0//audience-annotations-0.5.0.jar
+automaton/1.11-8//automaton-1.11-8.jar
+avro-ipc/1.8.2//avro-ipc-1.8.2.jar
+avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar
+avro/1.8.2//avro-1.8.2.jar
+bonecp/0.8.0.RELEASE//bonecp-0.8.0.RELEASE.jar
+breeze-macros_2.12/1.0//breeze-macros_2.12-1.0.jar
+breeze_2.12/1.0//breeze_2.12-1.0.jar
+cats-kernel_2.12/2.0.0-M4//cats-kernel_2.12-2.0.0-M4.jar
+chill-java/0.9.3//chill-java-0.9.3.jar
+chill_2.12/0.9.3//chill_2.12-0.9.3.jar
+commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar
+commons-cli/1.2//commons-cli-1.2.jar
+commons-codec/1.10//commons-codec-1.10.jar
+commons-collections/3.2.2//commons-collections-3.2.2.jar
+commons-compiler/3.0.15//commons-compiler-3.0.15.jar
+commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
+commons-crypto/1.0.0//commons-crypto-1.0.0.jar
+commons-daemon/1.0.13//commons-daemon-1.0.13.jar
+commons-dbcp/1.4//commons-dbcp-1.4.jar
+commons-httpclient/3.1//commons-httpclient-3.1.jar
+commons-io/2.4//commons-io-2.4.jar
+commons-lang/2.6//commons-lang-2.6.jar
+commons-lang3/3.9//commons-lang3-3.9.jar
+commons-logging/1.1.3//commons-logging-1.1.3.jar
+commons-math3/3.4.1//commons-math3-3.4.1.jar
+commons-net/3.1//commons-net-3.1.jar
+commons-pool/1.5.4//commons-pool-1.5.4.jar
+commons-text/1.6//commons-text-1.6.jar
+compress-lzf/1.0.3//compress-lzf-1.0.3.jar
+core/1.1.2//core-1.1.2.jar
+curator-client/2.13.0//curator-client-2.13.0.jar
+curator-framework/2.13.0//curator-framework-2.13.0.jar
+curator-recipes/2.13.0//curator-recipes-2.13.0.jar
+datanucleus-api-jdo/4.2.4//datanucleus-api-jdo-4.2.4.jar
+datanucleus-core/4.1.17//datanucleus-core-4.1.17.jar
+datanucleus-rdbms/4.1.19//datanucleus-rdbms-4.1.19.jar
+derby/10.12.1.1//derby-10.12.1.1.jar
+dnsjava/2.1.7//dnsjava-2.1.7.jar
+dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
+ehcache/3.3.1//ehcache-3.3.1.jar
+flatbuffers-java/1.9.0//flatbuffers-java-1.9.0.jar
+generex/1.0.2//generex-1.0.2.jar
+geronimo-jcache_1.0_spec/1.0-alpha-1//geronimo-jcache_1.0_spec-1.0-alpha-1.jar
+gson/2.2.4//gson-2.2.4.jar
+guava/14.0.1//guava-14.0.1.jar
+guice-servlet/4.0//guice-servlet-4.0.jar
+guice/4.0//guice-4.0.jar
+hadoop-annotations/3.2.0//hadoop-annotations-3.2.0.jar
+hadoop-auth/3.2.0//hadoop-auth-3.2.0.jar
+hadoop-client/3.2.0//hadoop-client-3.2.0.jar
+hadoop-common/3.2.0//hadoop-common-3.2.0.jar
+hadoop-hdfs-client/3.2.0//hadoop-hdfs-client-3.2.0.jar
+hadoop-mapreduce-client-common/3.2.0//hadoop-mapreduce-client-common-3.2.0.jar
+hadoop-mapreduce-client-core/3.2.0//hadoop-mapreduce-client-core-3.2.0.jar
+hadoop-mapreduce-client-jobclient/3.2.0//hadoop-mapreduce-client-jobclient-3.2.0.jar
+hadoop-yarn-api/3.2.0//hadoop-yarn-api-3.2.0.jar
+hadoop-yarn-client/3.2.0//hadoop-yarn-client-3.2.0.jar
+hadoop-yarn-common/3.2.0//hadoop-yarn-common-3.2.0.jar
+hadoop-yarn-registry/3.2.0//hadoop-yarn-registry-3.2.0.jar
+hadoop-yarn-server-common/3.2.0//hadoop-yarn-server-common-3.2.0.jar
+hadoop-yarn-server-web-proxy/3.2.0//hadoop-yarn-server-web-proxy-3.2.0.jar
+hive-beeline/2.3.6//hive-beeline-2.3.6.jar
+hive-cli/2.3.6//hive-cli-2.3.6.jar
+hive-common/2.3.6//hive-common-2.3.6.jar
+hive-exec/2.3.6/core/hive-exec-2.3.6-core.jar
+hive-jdbc/2.3.6//hive-jdbc-2.3.6.jar
+hive-llap-common/2.3.6//hive-llap-common-2.3.6.jar
+hive-metastore/2.3.6//hive-metastore-2.3.6.jar
+hive-serde/2.3.6//hive-serde-2.3.6.jar
+hive-service-rpc/2.3.6//hive-service-rpc-2.3.6.jar
+hive-shims-0.23/2.3.6//hive-shims-0.23-2.3.6.jar
+hive-shims-common/2.3.6//hive-shims-common-2.3.6.jar
+hive-shims-scheduler/2.3.6//hive-shims-scheduler-2.3.6.jar
+hive-shims/2.3.6//hive-shims-2.3.6.jar
+hive-storage-api/2.6.0//hive-storage-api-2.6.0.jar
+hive-vector-code-gen/2.3.6//hive-vector-code-gen-2.3.6.jar
+hk2-api/2.6.1//hk2-api-2.6.1.jar
+hk2-locator/2.6.1//hk2-locator-2.6.1.jar
+hk2-utils/2.6.1//hk2-utils-2.6.1.jar
+htrace-core4/4.1.0-incubating//htrace-core4-4.1.0-incubating.jar
+httpclient/4.5.6//httpclient-4.5.6.jar
+httpcore/4.4.12//httpcore-4.4.12.jar
+istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
+ivy/2.4.0//ivy-2.4.0.jar
+jackson-annotations/2.10.0//jackson-annotations-2.10.0.jar
+jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
+jackson-core/2.10.0//jackson-core-2.10.0.jar
+jackson-databind/2.10.0//jackson-databind-2.10.0.jar
+jackson-dataformat-yaml/2.10.0//jackson-dataformat-yaml-2.10.0.jar
+jackson-jaxrs-base/2.9.5//jackson-jaxrs-base-2.9.5.jar
+jackson-jaxrs-json-provider/2.9.5//jackson-jaxrs-json-provider-2.9.5.jar
+jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
+jackson-module-jaxb-annotations/2.10.0//jackson-module-jaxb-annotations-2.10.0.jar
+jackson-module-paranamer/2.10.0//jackson-module-paranamer-2.10.0.jar
+jackson-module-scala_2.12/2.10.0//jackson-module-scala_2.12-2.10.0.jar
+jakarta.activation-api/1.2.1//jakarta.activation-api-1.2.1.jar
+jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
+jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
+jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar
+jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar
+jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
+janino/3.0.15//janino-3.0.15.jar
+javassist/3.22.0-CR2//javassist-3.22.0-CR2.jar
+javax.inject/1//javax.inject-1.jar
+javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
+javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar
+javolution/5.5.1//javolution-5.5.1.jar
+jaxb-api/2.2.11//jaxb-api-2.2.11.jar
+jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
+jcip-annotations/1.0-1//jcip-annotations-1.0-1.jar
+jcl-over-slf4j/1.7.16//jcl-over-slf4j-1.7.16.jar
+jdo-api/3.0.1//jdo-api-3.0.1.jar
+jersey-client/2.29.1//jersey-client-2.29.1.jar
+jersey-common/2.29.1//jersey-common-2.29.1.jar
+jersey-container-servlet-core/2.29.1//jersey-container-servlet-core-2.29.1.jar
+jersey-container-servlet/2.29.1//jersey-container-servlet-2.29.1.jar
+jersey-hk2/2.29.1//jersey-hk2-2.29.1.jar
+jersey-media-jaxb/2.29.1//jersey-media-jaxb-2.29.1.jar
+jersey-server/2.29.1//jersey-server-2.29.1.jar
+jline/2.14.6//jline-2.14.6.jar
+joda-time/2.10.5//joda-time-2.10.5.jar
+jodd-core/3.5.2//jodd-core-3.5.2.jar
+jpam/1.1//jpam-1.1.jar
+json-smart/2.3//json-smart-2.3.jar
+json/1.8//json-1.8.jar
+json4s-ast_2.12/3.6.6//json4s-ast_2.12-3.6.6.jar
+json4s-core_2.12/3.6.6//json4s-core_2.12-3.6.6.jar
+json4s-jackson_2.12/3.6.6//json4s-jackson_2.12-3.6.6.jar
+json4s-scalap_2.12/3.6.6//json4s-scalap_2.12-3.6.6.jar
+jsp-api/2.1//jsp-api-2.1.jar
+jsr305/3.0.0//jsr305-3.0.0.jar
+jta/1.1//jta-1.1.jar
+jul-to-slf4j/1.7.16//jul-to-slf4j-1.7.16.jar
+kerb-admin/1.0.1//kerb-admin-1.0.1.jar
+kerb-client/1.0.1//kerb-client-1.0.1.jar
+kerb-common/1.0.1//kerb-common-1.0.1.jar
+kerb-core/1.0.1//kerb-core-1.0.1.jar
+kerb-crypto/1.0.1//kerb-crypto-1.0.1.jar
+kerb-identity/1.0.1//kerb-identity-1.0.1.jar
+kerb-server/1.0.1//kerb-server-1.0.1.jar
+kerb-simplekdc/1.0.1//kerb-simplekdc-1.0.1.jar
+kerb-util/1.0.1//kerb-util-1.0.1.jar
+kerby-asn1/1.0.1//kerby-asn1-1.0.1.jar
+kerby-config/1.0.1//kerby-config-1.0.1.jar
+kerby-pkix/1.0.1//kerby-pkix-1.0.1.jar
+kerby-util/1.0.1//kerby-util-1.0.1.jar
+kerby-xdr/1.0.1//kerby-xdr-1.0.1.jar
+kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
+kubernetes-client/4.6.4//kubernetes-client-4.6.4.jar
+kubernetes-model-common/4.6.4//kubernetes-model-common-4.6.4.jar
+kubernetes-model/4.6.4//kubernetes-model-4.6.4.jar
+leveldbjni-all/1.8//leveldbjni-all-1.8.jar
+libfb303/0.9.3//libfb303-0.9.3.jar
+libthrift/0.12.0//libthrift-0.12.0.jar
+log4j/1.2.17//log4j-1.2.17.jar
+logging-interceptor/3.12.6//logging-interceptor-3.12.6.jar
+lz4-java/1.7.0//lz4-java-1.7.0.jar
+machinist_2.12/0.6.8//machinist_2.12-0.6.8.jar
+macro-compat_2.12/1.1.1//macro-compat_2.12-1.1.1.jar
+mesos/1.4.0/shaded-protobuf/mesos-1.4.0-shaded-protobuf.jar
+metrics-core/4.1.1//metrics-core-4.1.1.jar
+metrics-graphite/4.1.1//metrics-graphite-4.1.1.jar
+metrics-jmx/4.1.1//metrics-jmx-4.1.1.jar
+metrics-json/4.1.1//metrics-json-4.1.1.jar
+metrics-jvm/4.1.1//metrics-jvm-4.1.1.jar
+minlog/1.3.0//minlog-1.3.0.jar
+mssql-jdbc/6.2.1.jre7//mssql-jdbc-6.2.1.jre7.jar
+netty-all/4.1.42.Final//netty-all-4.1.42.Final.jar
+nimbus-jose-jwt/4.41.1//nimbus-jose-jwt-4.41.1.jar
+objenesis/2.5.1//objenesis-2.5.1.jar
+okhttp/2.7.5//okhttp-2.7.5.jar
+okhttp/3.12.6//okhttp-3.12.6.jar
+okio/1.15.0//okio-1.15.0.jar
+opencsv/2.3//opencsv-2.3.jar
+orc-core/1.5.8//orc-core-1.5.8.jar
+orc-mapreduce/1.5.8//orc-mapreduce-1.5.8.jar
+orc-shims/1.5.8//orc-shims-1.5.8.jar
+oro/2.0.8//oro-2.0.8.jar
+osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
+paranamer/2.8//paranamer-2.8.jar
+parquet-column/1.10.1//parquet-column-1.10.1.jar
+parquet-common/1.10.1//parquet-common-1.10.1.jar
+parquet-encoding/1.10.1//parquet-encoding-1.10.1.jar
+parquet-format/2.4.0//parquet-format-2.4.0.jar
+parquet-hadoop/1.10.1//parquet-hadoop-1.10.1.jar
+parquet-jackson/1.10.1//parquet-jackson-1.10.1.jar
+protobuf-java/2.5.0//protobuf-java-2.5.0.jar
+py4j/0.10.8.1//py4j-0.10.8.1.jar
+pyrolite/4.30//pyrolite-4.30.jar
+re2j/1.1//re2j-1.1.jar
+scala-collection-compat_2.12/2.1.1//scala-collection-compat_2.12-2.1.1.jar
+scala-compiler/2.12.10//scala-compiler-2.12.10.jar
+scala-library/2.12.10//scala-library-2.12.10.jar
+scala-parser-combinators_2.12/1.1.2//scala-parser-combinators_2.12-1.1.2.jar
+scala-reflect/2.12.10//scala-reflect-2.12.10.jar
+scala-xml_2.12/1.2.0//scala-xml_2.12-1.2.0.jar
+shapeless_2.12/2.3.3//shapeless_2.12-2.3.3.jar
+shims/0.7.45//shims-0.7.45.jar
+slf4j-api/1.7.16//slf4j-api-1.7.16.jar
+slf4j-log4j12/1.7.16//slf4j-log4j12-1.7.16.jar
+snakeyaml/1.24//snakeyaml-1.24.jar
+snappy-java/1.1.7.3//snappy-java-1.1.7.3.jar
+spire-macros_2.12/0.17.0-M1//spire-macros_2.12-0.17.0-M1.jar
+spire-platform_2.12/0.17.0-M1//spire-platform_2.12-0.17.0-M1.jar
+spire-util_2.12/0.17.0-M1//spire-util_2.12-0.17.0-M1.jar
+spire_2.12/0.17.0-M1//spire_2.12-0.17.0-M1.jar
+stax-api/1.0.1//stax-api-1.0.1.jar
+stax2-api/3.1.4//stax2-api-3.1.4.jar
+stream/2.9.6//stream-2.9.6.jar
+super-csv/2.2.0//super-csv-2.2.0.jar
+token-provider/1.0.1//token-provider-1.0.1.jar
+transaction-api/1.1//transaction-api-1.1.jar
+univocity-parsers/2.8.3//univocity-parsers-2.8.3.jar
+velocity/1.5//velocity-1.5.jar
+woodstox-core/5.0.3//woodstox-core-5.0.3.jar
+xbean-asm7-shaded/4.15//xbean-asm7-shaded-4.15.jar
+xz/1.5//xz-1.5.jar
+zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
+zookeeper/3.4.14//zookeeper-3.4.14.jar
+zstd-jni/1.4.4-3//zstd-jni-1.4.4-3.jar
diff --git a/dev/test-dependencies.sh b/dev/test-dependencies.sh
index ad5cc75..936ac00 100755
--- a/dev/test-dependencies.sh
+++ b/dev/test-dependencies.sh
@@ -88,8 +88,24 @@ for HADOOP_HIVE_PROFILE in "${HADOOP_HIVE_PROFILES[@]}"; do
mkdir -p dev/pr-deps
$MVN $HADOOP2_MODULE_PROFILES -P$HADOOP_PROFILE -P$HIVE_PROFILE
dependency:build-classpath -pl assembly -am \
| grep "Dependencies classpath:" -A 1 \
- | tail -n 1 | tr ":" "\n" | rev | cut -d "/" -f 1 | rev | sort \
- | grep -v spark > dev/pr-deps/spark-deps-$HADOOP_HIVE_PROFILE
+ | tail -n 1 | tr ":" "\n" | awk -F '/' '{
+ # For each dependency classpath, we fetch the last three parts split by
"/": artifact id, version, and jar name.
+ # Since classifier, if exists, always sits between
"artifact_id-version-" and ".jar" suffix in the jar name,
+ # we extract classifier and put it right before the jar name explicitly.
+ # For example, `orc-core/1.5.5/nohive/orc-core-1.5.5-nohive.jar`
+ # ^^^^^^
+ # extracted classifier
+ # `okio/1.15.0//okio-1.15.0.jar`
+ # ^
+ # empty for dependencies without classifier
+ artifact_id=$(NF-2);
+ version=$(NF-1);
+ jar_name=$NF;
+ classifier_start_index=length(artifact_id"-"version"-") + 1;
+ classifier_end_index=index(jar_name, ".jar") - 1;
+ classifier=substr(jar_name, classifier_start_index, classifier_end_index
- classifier_start_index + 1);
+ print artifact_id"/"version"/"classifier"/"jar_name
+ }' | sort | grep -v spark > dev/pr-deps/spark-deps-$HADOOP_HIVE_PROFILE
done
if [[ $@ == **replace-manifest** ]]; then
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
