+1 I left some comments in the SPIP doc and got replies quickly. The new API looks good and more comprehensive. I think it will help Spark Structured Streaming to be more useful in more complicated streaming use cases.
On Fri, Jan 5, 2024 at 8:15 PM Burak Yavuz <brk...@gmail.com> wrote: > > I'm also a +1 on the newer APIs. We had a lot of learnings from using > flatMapGroupsWithState and I believe that we can make the APIs a lot easier > to use. > > On Wed, Nov 29, 2023 at 6:43 PM Anish Shrigondekar > <anish.shrigonde...@databricks.com.invalid> wrote: >> >> Hi dev, >> >> Addressed the comments that Jungtaek had on the doc. Bumping the thread once >> again to see if other folks have any feedback on the proposal. >> >> Thanks, >> Anish >> >> On Mon, Nov 27, 2023 at 8:15 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> >> wrote: >>> >>> Kindly bump for better reach after the long holiday. Please kindly review >>> the proposal which opens the chance to address complex use cases of >>> streaming. Thanks! >>> >>> On Thu, Nov 23, 2023 at 8:19 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> >>> wrote: >>>> >>>> Thanks Anish for proposing SPIP and initiating this thread! I believe this >>>> SPIP will help a bunch of complex use cases on streaming. >>>> >>>> dev@: We are coincidentally initiating this discussion in thanksgiving >>>> holidays. We understand people in the US may not have time to review the >>>> SPIP, and we plan to bump this thread in early next week. We are open for >>>> any feedback from non-US during the holiday. We can either address >>>> feedback altogether after the holiday (Anish is in the US) or I can answer >>>> if the feedback is more about the question. Thanks! >>>> >>>> On Thu, Nov 23, 2023 at 5:27 AM Anish Shrigondekar >>>> <anish.shrigonde...@databricks.com> wrote: >>>>> >>>>> Hi dev, >>>>> >>>>> I would like to start a discussion on "Structured Streaming - Arbitrary >>>>> State API v2". This proposal aims to address a bunch of limitations we >>>>> see today using mapGroupsWithState/flatMapGroupsWithState operator. The >>>>> detailed set of limitations is described in the SPIP doc. >>>>> >>>>> We propose to support various features such as multiple state variables >>>>> (flexible data modeling), composite types, enhanced timer functionality, >>>>> support for chaining operators after new operator, handling initial state >>>>> along with state data source, schema evolution etc This will allow users >>>>> to write more powerful streaming state management logic primarily used in >>>>> operational use-cases. Other built-in stateful operators could also >>>>> benefit from such changes in the future. >>>>> >>>>> JIRA: https://issues.apache.org/jira/browse/SPARK-45939 >>>>> SPIP: >>>>> https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig/edit?usp=sharing >>>>> Design Doc: >>>>> https://docs.google.com/document/d/1QjZmNZ-fHBeeCYKninySDIoOEWfX6EmqXs2lK097u9o/edit?usp=sharing >>>>> >>>>> cc - @Jungtaek Lim who has graciously agreed to be the shepherd for this >>>>> project >>>>> >>>>> Looking forward to your feedback ! >>>>> >>>>> Thanks, >>>>> Anish --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
