Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]
Hi Pavan, I looked at the PR, and the changes look simple and contained. It would be useful to add dynamic resource allocation to Spark Structured Streaming. Jungtaek. Would you be able to shepherd this change? On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni wrote: > Thanks a lot for creating the risk table Pavan. My apologies. I was tied > up with high priority items for the last couple weeks and could not > respond. I will review the PR by tomorrow's end, and get back to you. > > Appreciate your patience. > > Thanks > Bhuwan Sahni > > On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi < > pkotikalap...@twilio.com> wrote: > >> Hi Bhuwan, >> >> I hope the team got a chance to review the draft PR, looking for some >> comments to see if the plan looks alright?. I have updated the document >> about the risks >> <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit#heading=h.577aawlyiedf>.(also >> mentioned below). Please confirm if it looks alright? >> >> *Spark application type* >> >> *auto-scaling capability* >> >> *with New auto-scaling capability* >> >> Spark Batch job >> >> Works with current DRA >> >> No - change >> >> Streaming query without trigger interval >> >> No implementation >> >> Can work with this implementation - (have to set certain scale back >> configs based on previous usage pattern) - maybe automate with future work? >> >> Spark Streaming query with Trigger interval >> >> No implementation >> >> With this implementation >> >> Spark Streaming query with one-time micro batch >> >> Works with current DRA >> >> No - change >> >> Spark Streaming query with >> >> Availablenow micro batch >> >> Works with current DRA >> >> No - change >> >> Batch + Streaming query ( >> >> default/ >> >> triggger-interval/ >> >> once/ >> >> availablenow modes), other notebook use cases. >> >> No implementation >> >> No implementation >> >> >> >> We are more than happy to collaborate on a call to make better progress >> on this enhancement. Please let us know. >> >> Thank you, >> >> Pavan >> >> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> >>> Hi Bhuwan et al, >>> >>> Thank you for passing on the DataBricks Structured Streaming team's >>> review of the SPIP document. FYI, I work closely with Pawan and other >>> members to help deliver this piece of work. We appreciate your insights, >>> especially regarding the cost savings potential from the PoC. >>> >>> Pavan already furnished you with some additional info. Your team's point >>> about the SPIP currently addressing a specific use case (single streaming >>> query with Processing Time trigger) is well-taken. We agree that >>> maintaining simplicity is key, particularly as we explore more general >>> resource allocation mechanisms in the future. To address the concerns and >>> foster open discussion, The DataBricks team are invited to directly add >>> their comments and suggestions to the Jira itself >>> >>> [SPARK-24815] Structured Streaming should support dynamic allocation - >>> ASF JIRA (apache.org) >>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>> This will ensure everyone involved can benefit from your team's >>> expertise and facilitate further collaboration. >>> >>> Thanks >>> >>> Mich Talebzadeh, >>> Dad | Technologist | Solutions Architect | Engineer >>> London >>> United Kingdom >>> >>> >>>view my Linkedin profile >>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of c
Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]
Thanks a lot for creating the risk table Pavan. My apologies. I was tied up with high priority items for the last couple weeks and could not respond. I will review the PR by tomorrow's end, and get back to you. Appreciate your patience. Thanks Bhuwan Sahni On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi wrote: > Hi Bhuwan, > > I hope the team got a chance to review the draft PR, looking for some > comments to see if the plan looks alright?. I have updated the document > about the risks > <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit#heading=h.577aawlyiedf>.(also > mentioned below). Please confirm if it looks alright? > > *Spark application type* > > *auto-scaling capability* > > *with New auto-scaling capability* > > Spark Batch job > > Works with current DRA > > No - change > > Streaming query without trigger interval > > No implementation > > Can work with this implementation - (have to set certain scale back > configs based on previous usage pattern) - maybe automate with future work? > > Spark Streaming query with Trigger interval > > No implementation > > With this implementation > > Spark Streaming query with one-time micro batch > > Works with current DRA > > No - change > > Spark Streaming query with > > Availablenow micro batch > > Works with current DRA > > No - change > > Batch + Streaming query ( > > default/ > > triggger-interval/ > > once/ > > availablenow modes), other notebook use cases. > > No implementation > > No implementation > > > > We are more than happy to collaborate on a call to make better progress > on this enhancement. Please let us know. > > Thank you, > > Pavan > > On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh > wrote: > >> >> Hi Bhuwan et al, >> >> Thank you for passing on the DataBricks Structured Streaming team's >> review of the SPIP document. FYI, I work closely with Pawan and other >> members to help deliver this piece of work. We appreciate your insights, >> especially regarding the cost savings potential from the PoC. >> >> Pavan already furnished you with some additional info. Your team's point >> about the SPIP currently addressing a specific use case (single streaming >> query with Processing Time trigger) is well-taken. We agree that >> maintaining simplicity is key, particularly as we explore more general >> resource allocation mechanisms in the future. To address the concerns and >> foster open discussion, The DataBricks team are invited to directly add >> their comments and suggestions to the Jira itself >> >> [SPARK-24815] Structured Streaming should support dynamic allocation - >> ASF JIRA (apache.org) >> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >> This will ensure everyone involved can benefit from your team's expertise >> and facilitate further collaboration. >> >> Thanks >> >> Mich Talebzadeh, >> Dad | Technologist | Solutions Architect | Engineer >> London >> United Kingdom >> >> >>view my Linkedin profile >> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> >> >> >> >> *Disclaimer:* The information provided is correct to the best of my >> knowledge but of course cannot be guaranteed . It is essential to note >> that, as with any advice, quote "one test result is worth one-thousand >> expert opinions (Werner >> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von >> Braun >> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$> >> )". >> >> >> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi >> wrote: >> >>> Thanks Bhuwan and rest of the databricks team for the reviews, >>> >>> I appreciate your reviews, w
Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]
PR was closed due to inactivity by github actions.. >>>>>> >>>>>> The msg >>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>> also >>>>>> says >>>>>> >>>>>> > If you'd like to revive this PR, please reopen it and ask a >>>>>> committer to remove the Stale tag! >>>>>> >>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>> >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>> London >>>>>>> United Kingdom >>>>>>> >>>>>>> >>>>>>>view my Linkedin profile >>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>> Braun >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>> )". >>>>>>> >>>>>>> >>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Spark PMC members, >>>>>>>> >>>>>>>> I think we have few upvotes for this effort here and more people >>>>>>>> are showing interest (see PR comments >>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>> .) >>>>>>>> >>>>>>>> Is anyone interested in mentoring and reviewing this effort? >>>>>>>> >>>>>>>> Also can the repository admin/owner re-open the PR? ( I guess >>>>>>>> people only with admin access to the repository can do that). >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Pavan >>>>>>>> >>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>> > >>>>>>>>> > adding some references to the thread. >>>>>>>>> > >>>>>>>>> >- Jira ticket - SPARK-24815 >>>>>>>>> ><https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>> >- Design Doc >>>>>>>>> >< >>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>> > >>>>>>>>> >- discussion thread >>>>>>>>> >< >>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>> >- PR with initial implementation - >>>>>>>>> >https://github.com/apache/spark/pull/42352 >>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>> > >>>>>>>>> > Please vote with: >>>>>>>>> > >>>>>>>>> > [ ] +1: Accept the proposal and start with the development. >>>>>>>>> > [ ] +0 >>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>> > >>>>>>>>> > Thank you, >>>>>>>>> > >>>>>>>>> > Pavan >>>>>>>>> > >>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh >>>>>>>> > >>>>>>>>> > wrote: >>>>>>>>> > >>>>>>>>> > > >>>>>>>>> > > +1 for me (non binding) >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>> responsibility for any >>>>>>>>> > > loss, damage or destruction of data or any other property >>>>>>>>> which may arise >>>>>>>>> > > from relying on this email's technical content is explicitly >>>>>>>>> disclaimed. >>>>>>>>> > > The author will in no case be liable for any monetary damages >>>>>>>>> arising from >>>>>>>>> > > such loss, damage or destruction. >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> >>>>>>>> -- <http://www.databricks.com> *Bhuwan Sahni* Staff Software Engineer bhuwan.sa...@databricks.com 500 108th Ave. NE Bellevue, WA 98004 USA
Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2
+1. This is a good addition. <http://www.databricks.com> *Bhuwan Sahni* Staff Software Engineer bhuwan.sa...@databricks.com 500 108th Ave. NE Bellevue, WA 98004 USA On Wed, Jan 10, 2024 at 9:00 AM Burak Yavuz wrote: > +1. Excited to see more stateful workloads with Structured Streaming! > > > Best, > Burak > > On Wed, Jan 10, 2024 at 8:21 AM Praveen Gattu > wrote: > >> +1. This brings Structured Streaming a good solution for >> customers wanting to build stateful stream processing applications. >> >> On Wed, Jan 10, 2024 at 7:30 AM Bartosz Konieczny < >> bartkoniec...@gmail.com> wrote: >> >>> +1 :) >>> >>> On Wed, Jan 10, 2024 at 9:57 AM Shixiong Zhu wrote: >>> >>>> +1 (binding) >>>> >>>> Best Regards, >>>> Shixiong Zhu >>>> >>>> >>>> On Tue, Jan 9, 2024 at 6:47 PM 刘唯 wrote: >>>> >>>>> This is a good addition! +1 >>>>> >>>>> Raghu Angadi 于2024年1月9日周二 >>>>> 13:17写道： >>>>> >>>>>> +1. This is a major improvement to the state API. >>>>>> >>>>>> Raghu. >>>>>> >>>>>> On Tue, Jan 9, 2024 at 1:42 AM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> +1 for me as well >>>>>>> >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>> London >>>>>>> United Kingdom >>>>>>> >>>>>>> >>>>>>>view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>> for any loss, damage or destruction of data or any other property which >>>>>>> may >>>>>>> arise from relying on this email's technical content is explicitly >>>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>>> damages >>>>>>> arising from such loss, damage or destruction. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, 9 Jan 2024 at 03:24, Anish Shrigondekar >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks Jungtaek for creating the Vote thread. >>>>>>>> >>>>>>>> +1 (non-binding) from my side too. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Anish >>>>>>>> >>>>>>>> On Tue, Jan 9, 2024 at 6:09 AM Jungtaek Lim < >>>>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Starting with my +1 (non-binding). Thanks! >>>>>>>>> >>>>>>>>> On Tue, Jan 9, 2024 at 9:37 AM Jungtaek Lim < >>>>>>>>> kabhwan.opensou...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I'd like to start the vote for SPIP: Structured Streaming - >>>>>>>>>> Arbitrary State API v2. >>>>>>>>>> >>>>>>>>>> References: >>>>>>>>>> >>>>>>>>>>- JIRA ticket >>>>>>>>>><https://issues.apache.org/jira/browse/SPARK-45939> >>>>>>>>>>- SPIP doc >>>>>>>>>> >>>>>>>>>> <https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig/edit?usp=sharing> >>>>>>>>>>- Discussion thread >>>>>>>>>><https://lists.apache.org/thread/3jyjdgk1m5zyqfmrocnt6t415703nc8l> >>>>>>>>>> >>>>>>>>>> Please vote on the SPIP for the next 72 hours: >>>>>>>>>> >>>>>>>>>> [ ] +1: Accept the proposal as an official SPIP >>>>>>>>>> [ ] +0 >>>>>>>>>> [ ] -1: I don’t think this is a good idea because … >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> Jungtaek Lim (HeartSaVioR) >>>>>>>>>> >>>>>>>>> >>> >>> -- >>> Bartosz Konieczny >>> freelance data engineer >>> https://www.waitingforcode.com >>> https://github.com/bartosz25/ >>> https://twitter.com/waitingforcode >>> >>>
Re: [DISCUSS] SPIP: Structured Streaming - Arbitrary State API v2
+1 on the newer APIs. I believe these APIs provide a much powerful mechanism for the user to perform arbitrary state management in Structured Streaming queries. Thanks Bhuwan Sahni On Mon, Jan 8, 2024 at 10:07 AM L. C. Hsieh wrote: > +1 > > I left some comments in the SPIP doc and got replies quickly. The new > API looks good and more comprehensive. I think it will help Spark > Structured Streaming to be more useful in more complicated streaming > use cases. > > On Fri, Jan 5, 2024 at 8:15 PM Burak Yavuz wrote: > > > > I'm also a +1 on the newer APIs. We had a lot of learnings from using > flatMapGroupsWithState and I believe that we can make the APIs a lot easier > to use. > > > > On Wed, Nov 29, 2023 at 6:43 PM Anish Shrigondekar > wrote: > >> > >> Hi dev, > >> > >> Addressed the comments that Jungtaek had on the doc. Bumping the thread > once again to see if other folks have any feedback on the proposal. > >> > >> Thanks, > >> Anish > >> > >> On Mon, Nov 27, 2023 at 8:15 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >>> > >>> Kindly bump for better reach after the long holiday. Please kindly > review the proposal which opens the chance to address complex use cases of > streaming. Thanks! > >>> > >>> On Thu, Nov 23, 2023 at 8:19 AM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >>>> > >>>> Thanks Anish for proposing SPIP and initiating this thread! I believe > this SPIP will help a bunch of complex use cases on streaming. > >>>> > >>>> dev@: We are coincidentally initiating this discussion in > thanksgiving holidays. We understand people in the US may not have time to > review the SPIP, and we plan to bump this thread in early next week. We are > open for any feedback from non-US during the holiday. We can either address > feedback altogether after the holiday (Anish is in the US) or I can answer > if the feedback is more about the question. Thanks! > >>>> > >>>> On Thu, Nov 23, 2023 at 5:27 AM Anish Shrigondekar < > anish.shrigonde...@databricks.com> wrote: > >>>>> > >>>>> Hi dev, > >>>>> > >>>>> I would like to start a discussion on "Structured Streaming - > Arbitrary State API v2". This proposal aims to address a bunch of > limitations we see today using mapGroupsWithState/flatMapGroupsWithState > operator. The detailed set of limitations is described in the SPIP doc. > >>>>> > >>>>> We propose to support various features such as multiple state > variables (flexible data modeling), composite types, enhanced timer > functionality, support for chaining operators after new operator, handling > initial state along with state data source, schema evolution etc This will > allow users to write more powerful streaming state management logic > primarily used in operational use-cases. Other built-in stateful operators > could also benefit from such changes in the future. > >>>>> > >>>>> JIRA: https://issues.apache.org/jira/browse/SPARK-45939 > >>>>> SPIP: > https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig/edit?usp=sharing > >>>>> Design Doc: > https://docs.google.com/document/d/1QjZmNZ-fHBeeCYKninySDIoOEWfX6EmqXs2lK097u9o/edit?usp=sharing > >>>>> > >>>>> cc - @Jungtaek Lim who has graciously agreed to be the shepherd for > this project > >>>>> > >>>>> Looking forward to your feedback ! > >>>>> > >>>>> Thanks, > >>>>> Anish > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- <http://www.databricks.com> *Bhuwan Sahni* Staff Software Engineer bhuwan.sa...@databricks.com 500 108th Ave. NE Bellevue, WA 98004 USA
