+1. Looking forward to seeing how the new API brings in new streaming use cases!
Best Regards, Shixiong Zhu On Wed, Nov 29, 2023 at 6:42 PM Anish Shrigondekar <anish.shrigonde...@databricks.com.invalid> wrote: > Hi dev, > > Addressed the comments that Jungtaek had on the doc. Bumping the thread > once again to see if other folks have any feedback on the proposal. > > Thanks, > Anish > > On Mon, Nov 27, 2023 at 8:15 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> Kindly bump for better reach after the long holiday. Please kindly review >> the proposal which opens the chance to address complex use cases of >> streaming. Thanks! >> >> On Thu, Nov 23, 2023 at 8:19 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Thanks Anish for proposing SPIP and initiating this thread! I believe >>> this SPIP will help a bunch of complex use cases on streaming. >>> >>> dev@: We are coincidentally initiating this discussion in thanksgiving >>> holidays. We understand people in the US may not have time to review the >>> SPIP, and we plan to bump this thread in early next week. We are open for >>> any feedback from non-US during the holiday. We can either address feedback >>> altogether after the holiday (Anish is in the US) or I can answer if the >>> feedback is more about the question. Thanks! >>> >>> On Thu, Nov 23, 2023 at 5:27 AM Anish Shrigondekar < >>> anish.shrigonde...@databricks.com> wrote: >>> >>>> Hi dev, >>>> >>>> I would like to start a discussion on "Structured Streaming - Arbitrary >>>> State API v2". This proposal aims to address a bunch of limitations we see >>>> today using mapGroupsWithState/flatMapGroupsWithState operator. The >>>> detailed set of limitations is described in the SPIP doc. >>>> >>>> We propose to support various features such as multiple state variables >>>> (flexible data modeling), composite types, enhanced timer functionality, >>>> support for chaining operators after new operator, handling initial state >>>> along with state data source, schema evolution etc This will allow users to >>>> write more powerful streaming state management logic primarily used in >>>> operational use-cases. Other built-in stateful operators could also benefit >>>> from such changes in the future. >>>> >>>> JIRA: https://issues.apache.org/jira/browse/SPARK-45939 >>>> SPIP: >>>> https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig/edit?usp=sharing >>>> Design Doc: >>>> https://docs.google.com/document/d/1QjZmNZ-fHBeeCYKninySDIoOEWfX6EmqXs2lK097u9o/edit?usp=sharing >>>> >>>> cc - @Jungtaek Lim <kabhwan.opensou...@gmail.com> who has graciously >>>> agreed to be the shepherd for this project >>>> >>>> Looking forward to your feedback ! >>>> >>>> Thanks, >>>> Anish >>>> >>>
