Hi all, Following the latest comments in SPARK-34198 <https://issues.apache.org/jira/browse/SPARK-34198>, Databricks decided to donate the commercial implementation of the RocksDBStateStore. Compared with the original decision, there’s only one topic we want to raise again for discussion: can we directly add the RockDBStateStoreProvider in the sql/core module? This suggestion based on the following reasons:
1. The RocksDBStateStore aims to solve the problem of the original HDFSBasedStateStore, which is built-in. 2. End users can conveniently set the config to use the new implementation. 3. We can set the RocksDB one as the default one in the future. For the consideration of the dependency, I also checked the rocksdbjni we might introduce. As a JNI package <https://repo1.maven.org/maven2/org/rocksdb/rocksdbjni/6.2.2/rocksdbjni-6.2.2.pom>, it should not have any dependency conflicts with Apache Spark. Any suggestions are welcome! Best, Yuanjian Reynold Xin <r...@databricks.com> 于2021年2月14日周日 上午6:54写道： > Late +1 > > > On Sat, Feb 13 2021 at 2:49 PM, Liang-Chi Hsieh <vii...@gmail.com> wrote: > >> Hi devs, >> >> Thanks for all the inputs. I think overall there are positive inputs in >> Spark community about having RocksDB state store as external module. Then >> let's go forward with this direction and to improve structured streaming. I >> will keep update to the JIRA SPARK-34198. >> >> Thanks all again for the inputs and discussion. >> >> Liang-Chi Hsieh >> >> -- >> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >> >> --------------------------------------------------------------------- To >> unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >
