This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 9381fe6891d [SPARK-39975][BUILD] Upgrade rocksdbjni to 7.4.5 9381fe6891d is described below commit 9381fe6891de5a9898a6cfd10e7ed846c4b47089 Author: yangjie01 <yangji...@baidu.com> AuthorDate: Thu Aug 18 08:47:40 2022 -0500 [SPARK-39975][BUILD] Upgrade rocksdbjni to 7.4.5 ### What changes were proposed in this pull request? This PR aims to upgrade RocksDB JNI library from 7.4.4 to 7.4.5. ### Why are the changes needed? Rocksdbjni 7.4.5 only contains one patch compared with version 7.4.4 and this patch fixes a bug that causes data loss: - [Fix serious FSDirectory use-after-Close bug (missing fsync) ](https://github.com/facebook/rocksdb/pull/10460) ``` Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to FSDirectory::Fsync or FSDirectory::Close after the first FSDirectory::Close; Also, valgrind could report call to close() with fd=-1.) ``` The release note as follows: - https://github.com/facebook/rocksdb/releases/tag/v7.4.5 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GA - The benchmark result : **Before 7.4.4** ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.362 0.292 0.525 0.525 dbCreation 4 77.126 3.527 299.975 299.975 naturalIndexCreateIterator 1024 0.005 0.002 1.465 0.006 naturalIndexDescendingCreateIterator 1024 0.006 0.005 0.064 0.008 naturalIndexDescendingIteration 1024 0.006 0.004 0.266 0.008 naturalIndexIteration 1024 0.006 0.004 0.049 0.009 randomDeleteIndexed 1024 0.026 0.018 0.288 0.034 randomDeletesNoIndex 1024 0.015 0.013 0.041 0.018 randomUpdatesIndexed 1024 0.082 0.033 31.463 0.083 randomUpdatesNoIndex 1024 0.035 0.031 0.618 0.039 randomWritesIndexed 1024 0.118 0.035 52.198 0.122 randomWritesNoIndex 1024 0.042 0.035 1.650 0.048 refIndexCreateIterator 1024 0.005 0.004 0.020 0.007 refIndexDescendingCreateIterator 1024 0.003 0.002 0.027 0.004 refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008 refIndexIteration 1024 0.007 0.005 0.071 0.011 sequentialDeleteIndexed 1024 0.022 0.018 0.222 0.027 sequentialDeleteNoIndex 1024 0.015 0.012 0.044 0.018 sequentialUpdatesIndexed 1024 0.048 0.040 0.860 0.062 sequentialUpdatesNoIndex 1024 0.038 0.030 0.871 0.045 sequentialWritesIndexed 1024 0.051 0.039 2.405 0.058 sequentialWritesNoIndex 1024 0.038 0.032 2.937 0.042 ``` **After 7.4.5** ``` [INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark count mean min max 95th dbClose 4 0.392 0.239 0.738 0.738 dbCreation 4 78.741 3.764 303.031 303.031 naturalIndexCreateIterator 1024 0.005 0.002 1.549 0.006 naturalIndexDescendingCreateIterator 1024 0.005 0.005 0.068 0.007 naturalIndexDescendingIteration 1024 0.006 0.004 0.295 0.010 naturalIndexIteration 1024 0.006 0.004 0.084 0.009 randomDeleteIndexed 1024 0.027 0.020 0.335 0.037 randomDeletesNoIndex 1024 0.015 0.013 0.039 0.017 randomUpdatesIndexed 1024 0.084 0.034 31.642 0.088 randomUpdatesNoIndex 1024 0.037 0.035 0.573 0.041 randomWritesIndexed 1024 0.122 0.035 53.724 0.126 randomWritesNoIndex 1024 0.043 0.036 1.694 0.047 refIndexCreateIterator 1024 0.005 0.005 0.023 0.007 refIndexDescendingCreateIterator 1024 0.003 0.003 0.029 0.005 refIndexDescendingIteration 1024 0.006 0.005 0.044 0.008 refIndexIteration 1024 0.007 0.005 0.084 0.011 sequentialDeleteIndexed 1024 0.024 0.018 2.101 0.027 sequentialDeleteNoIndex 1024 0.016 0.013 0.044 0.020 sequentialUpdatesIndexed 1024 0.046 0.040 0.870 0.053 sequentialUpdatesNoIndex 1024 0.040 0.031 0.750 0.048 sequentialWritesIndexed 1024 0.051 0.039 2.093 0.060 sequentialWritesNoIndex 1024 0.038 0.031 2.827 0.041 ``` Closes #37543 from LuciferYang/SPARK-39975. Authored-by: yangjie01 <yangji...@baidu.com> Signed-off-by: Sean Owen <sro...@gmail.com> --- dev/deps/spark-deps-hadoop-2-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 b/dev/deps/spark-deps-hadoop-2-hive-2.3 index b68295f4eb5..e2b193fb3f4 100644 --- a/dev/deps/spark-deps-hadoop-2-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-2-hive-2.3 @@ -234,7 +234,7 @@ pickle/1.2//pickle-1.2.jar protobuf-java/2.5.0//protobuf-java-2.5.0.jar py4j/0.10.9.7//py4j-0.10.9.7.jar remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar -rocksdbjni/7.4.4//rocksdbjni-7.4.4.jar +rocksdbjni/7.4.5//rocksdbjni-7.4.5.jar scala-collection-compat_2.12/2.2.0//scala-collection-compat_2.12-2.2.0.jar scala-compiler/2.12.16//scala-compiler-2.12.16.jar scala-library/2.12.16//scala-library-2.12.16.jar diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 02b065f10df..550336aa59e 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -221,7 +221,7 @@ pickle/1.2//pickle-1.2.jar protobuf-java/2.5.0//protobuf-java-2.5.0.jar py4j/0.10.9.7//py4j-0.10.9.7.jar remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar -rocksdbjni/7.4.4//rocksdbjni-7.4.4.jar +rocksdbjni/7.4.5//rocksdbjni-7.4.5.jar scala-collection-compat_2.12/2.2.0//scala-collection-compat_2.12-2.2.0.jar scala-compiler/2.12.16//scala-compiler-2.12.16.jar scala-library/2.12.16//scala-library-2.12.16.jar diff --git a/pom.xml b/pom.xml index 77d676051d2..261d5278c7a 100644 --- a/pom.xml +++ b/pom.xml @@ -674,7 +674,7 @@ <dependency> <groupId>org.rocksdb</groupId> <artifactId>rocksdbjni</artifactId> - <version>7.4.4</version> + <version>7.4.5</version> </dependency> <dependency> <groupId>${leveldbjni.group}</groupId> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org