This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 65c5775a3365 [SPARK-53420][BUILD] Upgrade Parquet to 1.16.0
65c5775a3365 is described below
commit 65c5775a33656dc81d6cfba857a66821c3b42ec8
Author: Cheng Pan <[email protected]>
AuthorDate: Wed Sep 3 19:47:28 2025 -0700
[SPARK-53420][BUILD] Upgrade Parquet to 1.16.0
### What changes were proposed in this pull request?
Parquet Java 1.16.0 Release Notes:
https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.16.0
### Why are the changes needed?
Keep Parquet update to date, benefit from upstream bugfixes and
improvements.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GHA.
Run TPC-DS 300G (query time in seconds), no surprise compared to Parquet
1.15.2
| query_name | parquet 1.15.2 | parquet 1.16.0 RC2 |
| :--------- | :----------- | :----------- |
| q01.sql | 31 | 29 |
| q02.sql | 26 | 20 |
| q03.sql | 9 | 9 |
| q04.sql | 67 | 67 |
| q05.sql | 14 | 14 |
| q06.sql | 8 | 8 |
| q07.sql | 5 | 5 |
| q08.sql | 6 | 6 |
| q09.sql | 13 | 12 |
| q10.sql | 6 | 6 |
| q11.sql | 31 | 31 |
| q12.sql | 4 | 3 |
| q13.sql | 5 | 5 |
| q14a.sql | 52 | 53 |
| q14b.sql | 41 | 41 |
| q15.sql | 4 | 4 |
| q16.sql | 16 | 16 |
| q17.sql | 7 | 7 |
| q18.sql | 6 | 6 |
| q19.sql | 6 | 6 |
| q20.sql | 3 | 3 |
| q21.sql | 5 | 4 |
| q22.sql | 10 | 10 |
| q23a.sql | 76 | 74 |
| q23b.sql | 80 | 80 |
| q24a.sql | 39 | 39 |
| q24b.sql | 32 | 31 |
| q25.sql | 7 | 7 |
| q26.sql | 4 | 3 |
| q27.sql | 3 | 3 |
| q28.sql | 17 | 16 |
| q29.sql | 9 | 9 |
| q30.sql | 9 | 9 |
| q31.sql | 13 | 13 |
| q32.sql | 2 | 2 |
| q33.sql | 5 | 6 |
| q34.sql | 5 | 5 |
| q35.sql | 13 | 13 |
| q36.sql | 4 | 4 |
| q37.sql | 7 | 6 |
| q38.sql | 13 | 12 |
| q39a.sql | 7 | 7 |
| q39b.sql | 7 | 6 |
| q40.sql | 5 | 6 |
| q41.sql | 0 | 0 |
| q42.sql | 1 | 2 |
| q43.sql | 4 | 4 |
| q44.sql | 7 | 7 |
| q45.sql | 3 | 3 |
| q46.sql | 7 | 7 |
| q47.sql | 11 | 12 |
| q48.sql | 5 | 5 |
| q49.sql | 7 | 7 |
| q50.sql | 16 | 16 |
| q51.sql | 15 | 14 |
| q52.sql | 2 | 2 |
| q53.sql | 2 | 2 |
| q54.sql | 4 | 5 |
| q55.sql | 1 | 2 |
| q56.sql | 4 | 4 |
| q57.sql | 12 | 9 |
| q58.sql | 4 | 6 |
| q59.sql | 14 | 10 |
| q60.sql | 7 | 7 |
| q61.sql | 4 | 3 |
| q62.sql | 4 | 5 |
| q63.sql | 2 | 2 |
| q64.sql | 21 | 21 |
| q65.sql | 12 | 12 |
| q66.sql | 6 | 7 |
| q67.sql | 46 | 45 |
| q68.sql | 7 | 6 |
| q69.sql | 10 | 10 |
| q70.sql | 5 | 5 |
| q71.sql | 6 | 6 |
| q72.sql | 17 | 16 |
| q73.sql | 4 | 4 |
| q74.sql | 24 | 24 |
| q75.sql | 24 | 21 |
| q76.sql | 12 | 10 |
| q77.sql | 6 | 5 |
| q78.sql | 31 | 31 |
| q79.sql | 5 | 5 |
| q80.sql | 9 | 8 |
| q81.sql | 8 | 7 |
| q82.sql | 11 | 11 |
| q83.sql | 4 | 4 |
| q84.sql | 5 | 4 |
| q85.sql | 8 | 7 |
| q86.sql | 3 | 3 |
| q87.sql | 14 | 13 |
| q88.sql | 13 | 13 |
| q89.sql | 4 | 3 |
| q90.sql | 4 | 3 |
| q91.sql | 3 | 4 |
| q92.sql | 2 | 2 |
| q93.sql | 21 | 20 |
| q94.sql | 9 | 9 |
| q95.sql | 62 | 61 |
| q96.sql | 5 | 3 |
| q97.sql | 12 | 11 |
| q98.sql | 4 | 3 |
| q99.sql | 6 | 6 |
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #52165 from pan3793/parquet-1.16.0.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 12 ++++++------
pom.xml | 2 +-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 9e117bbbb1a5..e5e82bc067dc 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -240,12 +240,12 @@ orc-shims/2.2.0//orc-shims-2.2.0.jar
oro/2.0.8//oro-2.0.8.jar
osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
paranamer/2.8.3//paranamer-2.8.3.jar
-parquet-column/1.15.2//parquet-column-1.15.2.jar
-parquet-common/1.15.2//parquet-common-1.15.2.jar
-parquet-encoding/1.15.2//parquet-encoding-1.15.2.jar
-parquet-format-structures/1.15.2//parquet-format-structures-1.15.2.jar
-parquet-hadoop/1.15.2//parquet-hadoop-1.15.2.jar
-parquet-jackson/1.15.2//parquet-jackson-1.15.2.jar
+parquet-column/1.16.0//parquet-column-1.16.0.jar
+parquet-common/1.16.0//parquet-common-1.16.0.jar
+parquet-encoding/1.16.0//parquet-encoding-1.16.0.jar
+parquet-format-structures/1.16.0//parquet-format-structures-1.16.0.jar
+parquet-hadoop/1.16.0//parquet-hadoop-1.16.0.jar
+parquet-jackson/1.16.0//parquet-jackson-1.16.0.jar
pickle/1.5//pickle-1.5.jar
py4j/0.10.9.9//py4j-0.10.9.9.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
diff --git a/pom.xml b/pom.xml
index 01c685bb0cff..eb464acce8ba 100644
--- a/pom.xml
+++ b/pom.xml
@@ -140,7 +140,7 @@
<kafka.version>3.9.1</kafka.version>
<!-- After 10.17.1.0, the minimum required version is JDK19 -->
<derby.version>10.16.1.1</derby.version>
- <parquet.version>1.15.2</parquet.version>
+ <parquet.version>1.16.0</parquet.version>
<orc.version>2.2.0</orc.version>
<orc.classifier>shaded-protobuf</orc.classifier>
<jetty.version>11.0.25</jetty.version>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
