Re: [DISCUSS] Add RocksDB StateStore

I am fine with RocksDB state store as built-in state store. Actually the proposal to have it as external module is to avoid the raised concern in the previous effort. The need to have it as experimental doesn't necessarily mean to have it as external module, I think. They are two things. So I

Re: [DISCUSS] Add RocksDB StateStore

I think adding RocksDB state store to sql/core directly would be OK. Personally I also voted "either way is fine with me" against RocksDB state store implementation in Spark ecosystem. The overall stance hasn't changed, but I'd like to point out that the risk becomes quite lower than before, given

Re: [DISCUSS] Add RocksDB StateStore

Hi all, Following the latest comments in SPARK-34198 , Databricks decided to donate the commercial implementation of the RocksDBStateStore. Compared with the original decision, there’s only one topic we want to raise again for discussion: can we

Re: [DISCUSS] Add RocksDB StateStore

Late +1 On Sat, Feb 13 2021 at 2:49 PM, Liang-Chi Hsieh < vii...@gmail.com > wrote: > > > > Hi devs, > > > > Thanks for all the inputs. I think overall there are positive inputs in > Spark community about having RocksDB state store as external module. Then > let's go forward with this

Re: [DISCUSS] Add RocksDB StateStore

Hi devs, Thanks for all the inputs. I think overall there are positive inputs in Spark community about having RocksDB state store as external module. Then let's go forward with this direction and to improve structured streaming. I will keep update to the JIRA SPARK-34198. Thanks all again for

Re: [DISCUSS] Add RocksDB StateStore

conflict with previous concern for dependency. >> >>> >> >>> >> >>> >> >>> Thanks, >> >>> >> >>> Cheng Su >> >>> >> >>> >> >>> >> >>> From: Dongjoo

Re: [DISCUSS] Add RocksDB StateStore

t; I think this feature is useful and popular in practice, and option 2 > is not conflict with previous concern for dependency. > >>> > >>> > >>> > >>> Thanks, > >>> > >>> Cheng Su > >>> > >>> > >>

Re: [DISCUSS] Add RocksDB StateStore

>>> >>> Thanks, >>> >>> Cheng Su >>> >>> >>> >>> From: Dongjoon Hyun >>> Date: Monday, February 8, 2021 at 10:39 AM >>> To: Jacek Laskowski >>> Cc: Liang-Chi Hsieh , dev >>> Subject: Re: [D

Re: [DISCUSS] Add RocksDB StateStore

;>> >>> >>> Thanks, >>> >>> Cheng Su >>> >>> >>> >>> *From: *Dongjoon Hyun >>> *Date: *Monday, February 8, 2021 at 10:39 AM >>> *To: *Jacek Laskowski >>> *Cc: *Liang-Chi Hsieh , dev >&

Re: [DISCUSS] Add RocksDB StateStore

nd option 2 is >> not conflict with previous concern for dependency. >> >> >> >> Thanks, >> >> Cheng Su >> >> >> >> *From: *Dongjoon Hyun >> *Date: *Monday, February 8, 2021 at 10:39 AM >> *To: *Jacek Laskowski >> *Cc: *

Re: [DISCUSS] Add RocksDB StateStore

t; > Cheng Su > > > > *From: *Dongjoon Hyun > *Date: *Monday, February 8, 2021 at 10:39 AM > *To: *Jacek Laskowski > *Cc: *Liang-Chi Hsieh , dev > *Subject: *Re: [DISCUSS] Add RocksDB StateStore > > > > Thank you, Liang-chi and all. > > > > +1 for (2)

Re: [DISCUSS] Add RocksDB StateStore

Subject: Re: [DISCUSS] Add RocksDB StateStore Thank you, Liang-chi and all. +1 for (2) external module design because it can deliver the new feature in a safe way. Bests, Dongjoon On Mon, Feb 8, 2021 at 9:00 AM Jacek Laskowski mailto:ja...@japila.pl>> wrote: Hi, I'm "okay to add RocksDB

Re: [DISCUSS] Add RocksDB StateStore

Thank you, Liang-chi and all. +1 for (2) external module design because it can deliver the new feature in a safe way. Bests, Dongjoon On Mon, Feb 8, 2021 at 9:00 AM Jacek Laskowski wrote: > Hi, > > I'm "okay to add RocksDB StateStore as external module". See no reason not > to. > >

Re: [DISCUSS] Add RocksDB StateStore

Hi, I'm "okay to add RocksDB StateStore as external module". See no reason not to. Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on https://twitter.com/jaceklaskowski

Re: [DISCUSS] Add RocksDB StateStore

Thank you for the inputs! Yikun. Let's take these inputs when we are ready to have rocksdb state store in Spark SS. Yikun Jiang wrote > I worked on some work about rocksdb multi-arch support and version upgrade > on > Kafka/Storm/Flink[1][2][3].To avoid these issues happened in spark again, > I

Re: [DISCUSS] Add RocksDB StateStore

I worked on some work about rocksdb multi-arch support and version upgrade on Kafka/Storm/Flink[1][2][3].To avoid these issues happened in spark again, I want to give some inputs in here about rocksdb version selection from multi-arch support view. Hope it helps. The Rocksdb adds Arm64 support

Re: [DISCUSS] Add RocksDB StateStore

Hi, FYI, I have been using the project at https://github.com/chermenin/spark-states for a few months and it has been working well for me. -Nico -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To

[DISCUSS] Add RocksDB StateStore

Hi devs, In Spark structured streaming, we need state store for state management for stateful operators such streaming aggregates, joins, etc. We have one and only one state store implementation now. It is in-memory hashmap which was backed up in HDFS complaint file system at the end of every