Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-23 Thread laglangyue
+1 发自我的iPhone -- Original -- From: Jungtaek Lim https://lists.apache.org/thread/7ohctj1gmqbhds56bntf4s2zst5qpll1;(committer+can login to reply) or search with "[VOTE] SPIP: State Data Source - Reader" in your inbox. Every vote would be really appreciated! On

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-23 Thread Jungtaek Lim
FYI: VOTE thread is open, please check the link https://lists.apache.org/thread/7ohctj1gmqbhds56bntf4s2zst5qpll1 (committer+ can login to reply) or search with "[VOTE] SPIP: State Data Source - Reader" in your inbox. Every vote would be really appreciated! On Mon, Oct 23, 2023 at 1:06 PM Jungtaek

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-22 Thread Jungtaek Lim
I don't see major comments as of now. Given that the thread was initiated more than 10 days ago and I see multiple supporters, I'm going to initiate a VOTE thread. Please participate in the VOTE thread as well. Thanks! On Thu, Oct 19, 2023 at 11:39 AM Jungtaek Lim wrote: > Also, I want to

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-18 Thread Jungtaek Lim
Also, I want to replicate the comment Liang-Chi put into SPIP doc, as it is a rather general and usual question for every new addition of data source. Hence I want to sort it out for everyone. As I know, the author implemented a third-party tool for query state store > as a data source long time

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-18 Thread Jungtaek Lim
Thanks Raghu for your support! Btw, I'd like to replicate the support from JIRA ticket itself, I see support from Chaoqin and Praveen. Thanks both! On Thu, Oct 19, 2023 at 5:56 AM Raghu Angadi wrote: > +1 overall and a big +1 to keeping offline state-rebalancing as a primary > use case. > >

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-18 Thread Raghu Angadi
+1 overall and a big +1 to keeping offline state-rebalancing as a primary use case. Raghu. On Mon, Oct 16, 2023 at 11:25 AM Bartosz Konieczny wrote: > Thank you, Jungtaek, for your answers! It's clear now. > > +1 for me. It seems like a prerequisite for further ops-related > improvements for

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-18 Thread Jungtaek Lim
Thanks Yuanjian for your support! I've left a comment but to replicate here - I agree with your point. It's really uneasy for a new feature to be stable from the initial version and we might want to decide on breaking backward compatibility for (semantic) bug fixes/improvements. Maybe we could

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-18 Thread Yuanjian Li
+1, I have no issues with the practicality and value of this feature itself. I've left some comments concerning ongoing maintenance and compatibility-related matters, which we can continue to discuss. Jungtaek Lim 于2023年10月17日周二 05:23写道: > Thanks Bartosz and Anish for your support! > > I'll

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-16 Thread Jungtaek Lim
Thanks Bartosz and Anish for your support! I'll wait for a couple more days to see whether we can hear more voices on this. We could probably look for initiating a VOTE thread if there is no objection. On Tue, Oct 17, 2023 at 5:48 AM Anish Shrigondekar < anish.shrigonde...@databricks.com> wrote:

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-16 Thread Anish Shrigondekar
Hi Jungtaek, Thanks for putting this together. +1 from me and looks good overall. Posted some minor comments/questions to the doc. Thanks, Anish On Mon, Oct 16, 2023 at 11:25 AM Bartosz Konieczny wrote: > Thank you, Jungtaek, for your answers! It's clear now. > > +1 for me. It seems like a

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-16 Thread Bartosz Konieczny
Thank you, Jungtaek, for your answers! It's clear now. +1 for me. It seems like a prerequisite for further ops-related improvements for the state store management. I mean especially here the state rebalancing that could rely on this read+write state store API. I don't mean here the dynamic state

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-16 Thread Jungtaek Lim
bump for better reach On Thu, Oct 12, 2023 at 4:26 PM Jungtaek Lim wrote: > Sorry, please use this link instead for SPIP doc: > https://docs.google.com/document/d/1_iVf_CIu2RZd3yWWF6KoRNlBiz5NbSIK0yThqG0EvPY/edit?usp=sharing > > > On Thu, Oct 12, 2023 at 3:58 PM Jungtaek Lim > wrote: > >> Hi

Re: [DISCUSS] SPIP: State Data Source - Reader

2023-10-12 Thread Jungtaek Lim
Sorry, please use this link instead for SPIP doc: https://docs.google.com/document/d/1_iVf_CIu2RZd3yWWF6KoRNlBiz5NbSIK0yThqG0EvPY/edit?usp=sharing On Thu, Oct 12, 2023 at 3:58 PM Jungtaek Lim wrote: > Hi dev, > > I'd like to start a discussion on "State Data Source - Reader". > > This proposal

[DISCUSS] SPIP: State Data Source - Reader

2023-10-12 Thread Jungtaek Lim
Hi dev, I'd like to start a discussion on "State Data Source - Reader". This proposal aims to introduce a new data source "statestore" which enables reading the state rows from existing checkpoint via offline (batch) query. This will enable users to 1) create unit tests against stateful query