This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new de0e4cce034 [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3 de0e4cce034 is described below commit de0e4cce0343cd82870b2904d75352ca50647706 Author: yangjie01 <yangji...@baidu.com> AuthorDate: Sun Sep 4 08:27:20 2022 -0500 [SPARK-40321][BUILD] Upgrade rocksdbjni to 7.5.3 ### What changes were proposed in this pull request? This PR aims to upgrade RocksDB JNI library from 7.4.5 to 7.5.3. ### Why are the changes needed? This version bring performance improvements(related to write throughput and compaction) and some bug fix,the release note as follows: - https://github.com/facebook/rocksdb/releases/tag/v7.5.3 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GA - The benchmark result : **Before 7.4.5** ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.340 0.265 0.510 0.510 dbCreation 4 78.628 3.499 303.732 303.732 naturalIndexCreateIterator 1024 0.005 0.002 1.473 0.006 naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.062 0.007 naturalIndexDescendingIteration 1024 0.006 0.004 0.268 0.008 naturalIndexIteration 1024 0.006 0.004 0.053 0.009 randomDeleteIndexed 1024 0.025 0.019 0.183 0.033 randomDeletesNoIndex 1024 0.015 0.012 0.037 0.017 randomUpdatesIndexed 1024 0.081 0.033 30.773 0.084 randomUpdatesNoIndex 1024 0.034 0.031 0.580 0.037 randomWritesIndexed 1024 0.117 0.035 52.383 0.120 randomWritesNoIndex 1024 0.039 0.034 1.556 0.044 refIndexCreateIterator 1024 0.005 0.004 0.019 0.006 refIndexDescendingCreateIterator 1024 0.003 0.002 0.028 0.004 refIndexDescendingIteration 1024 0.006 0.005 0.043 0.008 refIndexIteration 1024 0.007 0.005 0.068 0.009 sequentialDeleteIndexed 1024 0.021 0.017 0.096 0.026 sequentialDeleteNoIndex 1024 0.015 0.012 0.041 0.019 sequentialUpdatesIndexed 1024 0.044 0.037 0.747 0.051 sequentialUpdatesNoIndex 1024 0.039 0.032 0.771 0.046 sequentialWritesIndexed 1024 0.049 0.042 1.766 0.056 sequentialWritesNoIndex 1024 0.039 0.032 2.282 0.041 ``` **After 7.5.3** ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.342 0.212 0.683 0.683 dbCreation 4 77.603 3.403 301.357 301.357 naturalIndexCreateIterator 1024 0.005 0.002 1.507 0.007 naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.076 0.007 naturalIndexDescendingIteration 1024 0.006 0.004 0.244 0.010 naturalIndexIteration 1024 0.006 0.004 0.065 0.009 randomDeleteIndexed 1024 0.028 0.019 1.385 0.038 randomDeletesNoIndex 1024 0.016 0.013 0.042 0.019 randomUpdatesIndexed 1024 0.083 0.033 32.094 0.086 randomUpdatesNoIndex 1024 0.038 0.034 0.469 0.042 randomWritesIndexed 1024 0.122 0.034 54.517 0.126 randomWritesNoIndex 1024 0.042 0.037 1.369 0.048 refIndexCreateIterator 1024 0.005 0.005 0.019 0.007 refIndexDescendingCreateIterator 1024 0.003 0.003 0.034 0.005 refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008 refIndexIteration 1024 0.007 0.005 0.072 0.011 sequentialDeleteIndexed 1024 0.022 0.017 0.104 0.027 sequentialDeleteNoIndex 1024 0.015 0.012 0.038 0.019 sequentialUpdatesIndexed 1024 0.046 0.039 0.862 0.054 sequentialUpdatesNoIndex 1024 0.043 0.030 0.752 0.055 sequentialWritesIndexed 1024 0.051 0.039 1.949 0.060 sequentialWritesNoIndex 1024 0.038 0.032 2.858 0.042 ``` Closes #37783 from LuciferYang/SPARK-40321. Authored-by: yangjie01 <yangji...@baidu.com> Signed-off-by: Sean Owen <sro...@gmail.com> --- dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 b/dev/deps/spark-deps-hadoop-2-hive-2.3 index 9f91c5e075b..66882b2a4b4 100644 --- a/dev/deps/spark-deps-hadoop-2-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-2-hive-2.3 @@ -234,7 +234,7 @@ pickle/1.2//pickle-1.2.jar protobuf-java/2.5.0//protobuf-java-2.5.0.jar py4j/0.10.9.7//py4j-0.10.9.7.jar remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar -rocksdbjni/7.4.5//rocksdbjni-7.4.5.jar +rocksdbjni/7.5.3//rocksdbjni-7.5.3.jar scala-collection-compat_2.12/2.7.0//scala-collection-compat_2.12-2.7.0.jar scala-compiler/2.12.16//scala-compiler-2.12.16.jar scala-library/2.12.16//scala-library-2.12.16.jar diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 514afc41800..60dd8b190de 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -221,7 +221,7 @@ pickle/1.2//pickle-1.2.jar protobuf-java/2.5.0//protobuf-java-2.5.0.jar py4j/0.10.9.7//py4j-0.10.9.7.jar remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar -rocksdbjni/7.4.5//rocksdbjni-7.4.5.jar +rocksdbjni/7.5.3//rocksdbjni-7.5.3.jar scala-collection-compat_2.12/2.7.0//scala-collection-compat_2.12-2.7.0.jar scala-compiler/2.12.16//scala-compiler-2.12.16.jar scala-library/2.12.16//scala-library-2.12.16.jar diff --git a/pom.xml b/pom.xml index 1fc694ba43f..1ac8dbd1d00 100644 --- a/pom.xml +++ b/pom.xml @@ -674,7 +674,7 @@ <dependency> <groupId>org.rocksdb</groupId> <artifactId>rocksdbjni</artifactId> - <version>7.4.5</version> + <version>7.5.3</version> </dependency> <dependency> <groupId>${leveldbjni.group}</groupId> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org