This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8a7609f1cb2d [SPARK-46718][BUILD] Upgrade Arrow to 15.0.0
8a7609f1cb2d is described below
commit 8a7609f1cb2dd92ee30ec8172a1c1501d5810dae
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Mon Jan 22 21:25:21 2024 -0800
[SPARK-46718][BUILD] Upgrade Arrow to 15.0.0
### What changes were proposed in this pull request?
This pr aims to upgrade Arrow from 14.0.2 to 15.0.0, this version fixes the
compatibility issue with Netty 4.1.104.Final(GH-39265).
Additionally, since the `arrow-vector` module uses `eclipse-collections` to
replace `netty-common` as a compile-level dependency, Apache Spark has added a
dependency on `eclipse-collections` after upgrading to use Arrow 15.0.0.
### Why are the changes needed?
The new version brings the following major changes:
Bug Fixes
GH-34610 - [Java] Fix valueCount and field name when loading/transferring
NullVector
GH-38242 - [Java] Fix incorrect internal struct accounting for
DenseUnionVector#getBufferSizeFor
GH-38254 - [Java] Add reusable buffer getters to char/binary vectors
GH-38366 - [Java] Fix Murmur hash on buffers less than 4 bytes
GH-38387 - [Java] Fix JDK8 compilation issue with TestAllTypes
GH-38614 - [Java] Add VarBinary and VarCharWriter helper methods to more
writers
GH-38725 - [Java] decompression in Lz4CompressionCodec.java does not set
writer index
New Features and Improvements
GH-38511 - [Java] Add getTransferPair(Field, BufferAllocator, CallBack) for
StructVector and MapVector
GH-14936 - [Java] Remove netty dependency from arrow-vector
GH-38990 - [Java] Upgrade to flatc version 23.5.26
GH-39265 - [Java] Make it run well with the netty newest version 4.1.104
The full release notes as follows:
- https://arrow.apache.org/release/15.0.0.html
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GitHub Actions
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #44797 from LuciferYang/SPARK-46718.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 12 +++++++-----
pom.xml | 2 +-
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 6220626069af..4ee0f5a41191 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -16,10 +16,10 @@ antlr4-runtime/4.13.1//antlr4-runtime-4.13.1.jar
aopalliance-repackaged/2.6.1//aopalliance-repackaged-2.6.1.jar
arpack/3.0.3//arpack-3.0.3.jar
arpack_combined_all/0.1//arpack_combined_all-0.1.jar
-arrow-format/14.0.2//arrow-format-14.0.2.jar
-arrow-memory-core/14.0.2//arrow-memory-core-14.0.2.jar
-arrow-memory-netty/14.0.2//arrow-memory-netty-14.0.2.jar
-arrow-vector/14.0.2//arrow-vector-14.0.2.jar
+arrow-format/15.0.0//arrow-format-15.0.0.jar
+arrow-memory-core/15.0.0//arrow-memory-core-15.0.0.jar
+arrow-memory-netty/15.0.0//arrow-memory-netty-15.0.0.jar
+arrow-vector/15.0.0//arrow-vector-15.0.0.jar
audience-annotations/0.12.0//audience-annotations-0.12.0.jar
avro-ipc/1.11.3//avro-ipc-1.11.3.jar
avro-mapred/1.11.3//avro-mapred-1.11.3.jar
@@ -63,7 +63,9 @@ derby/10.16.1.1//derby-10.16.1.1.jar
derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar
derbytools/10.16.1.1//derbytools-10.16.1.1.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
-flatbuffers-java/1.12.0//flatbuffers-java-1.12.0.jar
+eclipse-collections-api/11.1.0//eclipse-collections-api-11.1.0.jar
+eclipse-collections/11.1.0//eclipse-collections-11.1.0.jar
+flatbuffers-java/23.5.26//flatbuffers-java-23.5.26.jar
gcs-connector/hadoop3-2.2.18/shaded/gcs-connector-hadoop3-2.2.18-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
diff --git a/pom.xml b/pom.xml
index e290273543c6..5f33dd7d8ebc 100644
--- a/pom.xml
+++ b/pom.xml
@@ -230,7 +230,7 @@
If you are changing Arrow version specification, please check
./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too.
-->
- <arrow.version>14.0.2</arrow.version>
+ <arrow.version>15.0.0</arrow.version>
<ammonite.version>2.5.11</ammonite.version>
<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
