Hi Pavan, I am from the DataBricks Structured Streaming team, and we did a review of the SPIP internally. Wanted to pass on the points discussed in the meeting.
Thanks for putting together the SPIP document. It's useful to have dynamic resource allocation for Streaming queries, and it's exciting to see the cost saving numbers from your PoC. However, in general we discovered that the SPIP addresses a very particular use-case (single streaming query in Spark cluster with Processing time Trigger). Keeping that in mind, it's useful to make sure that the implementation is simple. Can you please share your PoC implementation to understand the code complexity. This would help us to ensure that dynamic resource allocation mechanism for Streaming queries does not become complicated in the future (if more resource allocation mechanisms are added to address other use-cases). Thanks, Bhuwan On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Pavan and those who kindly voted for this SPIP > > Great to have 6+ votes and no -1 and 0. The so-called mass volume is > there. The rest is admin matter and how to drive the project forward and > yes there is more than one way of skinning the cat. I think we need some > flexibility in the rules given the dwindling (IMO) number of comitters who > are willing or actively participating. For example, on a similar matter I > approached Codi Koeninger who was one of the founders of Spark Streaming, > to shepherd a project almost a year back. Sadly he is no longer active and > quotes "I haven't been involved lately and would be missing a lot of > context." So we need to improvise and see how best we can drive this and > similar ones. We wait a short while for a response otherwise I am happy to > give a hand if needed and work with you guys to drive this. It is something > worthwhile. > > HTH > > T > Mich Talebzadeh, > Dad | Technologist | Solutions Architect | Engineer > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi > <pkotikalap...@twilio.com.invalid> wrote: > >> Thanks for the pointers Mich, will wait for Jungtaek Lee or any other PMC >> members to respond. >> >> aggregating upvotes to this email thread >> >> +6 >> Mich Talebzadeh >> Adam Hobbs >> Pavan Kotikalapudi >> Krystal Mitchell >> Sona Torosyan >> Aaron Kern >> >> Thank you, >> >> Pavan >> >> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> Hi, >>> >>> please check this doc >>> >>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>> <https://urldefense.com/v3/__https://spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>> >>> and specifically the below extract >>> >>> Discussing an SPIP >>> >>> All discussion of an SPIP should take place in a public forum, >>> preferably the discussion attached to the Jira. Any discussions that happen >>> offline should be made available online for the public via meeting notes >>> summarizing the discussions.(done) >>> >>> During this discussion, one or more shepherds should be identified among >>> PMC members. (outstanding) >>> >>> Once the discussion settles, the shepherd(s) should call for a vote on >>> the SPIP moving forward on the dev@ list. The vote should be open for >>> at least 72 hours and follows the typical Apache vote process and passes >>> upon consensus (at least 3 +1 votes from PMC members and no -1 votes from >>> PMC members). dev@ should be notified of the vote result. >>> >>> If there does not exist at least one PMC member that is committed to >>> shepherding the change within a month, the SPIP is rejected. >>> >>> If a committer does not think a SPIP aligns with long-term project >>> goals, or is not practical at the point of proposal, the committer should >>> -1 the SPIP explicitly and give technical justifications. >>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee can >>> kindly help the process >>> >>> cheers >>> >>> Mich Talebzadeh, >>> Dad | Technologist | Solutions Architect | Engineer >>> London >>> United Kingdom >>> >>> >>> view my Linkedin profile >>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>> Braun >>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>> )". >>> >>> >>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>> <pkotikalap...@twilio.com.invalid> wrote: >>> >>>> Hi Mich, >>>> >>>> We have >>>> >>>> five +1s till now. >>>> >>>> Mich Talebzadeh >>>> Adam Hobbs >>>> Pavan Kotikalapudi >>>> Krystal Mitchell >>>> Sona Torosyan >>>> (few more in github pr) >>>> +0: None >>>> >>>> -1: None >>>> >>>> Does it pass the required condition as approved? >>>> >>>> >>>> Not sure of that though, nothing about minimum required is mentioned in >>>> the past emails. >>>> >>>> I would request spark PMC members or any others who have done this in >>>> the past to understand the process better. >>>> >>>> Thank you, >>>> >>>> Pavan >>>> >>>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Hi Pavan, >>>>> >>>>> Do you have a list of votes for this feature by any chance? Does it >>>>> pass the required condition as approved? >>>>> >>>>> HTH >>>>> >>>>> Mich Talebzadeh, >>>>> Dad | Technologist | Solutions Architect | Engineer >>>>> London >>>>> United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>>>> >>>>> >>>>> >>>>> *Disclaimer:* The information provided is correct to the best of my >>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>> expert opinions (Werner >>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$>Von >>>>> Braun >>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$> >>>>> )". >>>>> >>>>> >>>>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi >>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>> >>>>>> Yes. The PR was closed due to inactivity by github actions.. >>>>>> >>>>>> The msg >>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>> also >>>>>> says >>>>>> >>>>>> > If you'd like to revive this PR, please reopen it and ask a >>>>>> committer to remove the Stale tag! >>>>>> >>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>> >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>> London >>>>>>> United Kingdom >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>> Braun >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>> )". >>>>>>> >>>>>>> >>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>> >>>>>>>> Hi Spark PMC members, >>>>>>>> >>>>>>>> I think we have few upvotes for this effort here and more people >>>>>>>> are showing interest (see PR comments >>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>> .) >>>>>>>> >>>>>>>> Is anyone interested in mentoring and reviewing this effort? >>>>>>>> >>>>>>>> Also can the repository admin/owner re-open the PR? ( I guess >>>>>>>> people only with admin access to the repository can do that). >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Pavan >>>>>>>> >>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>> <kryst...@adobe.com.invalid> wrote: >>>>>>>> >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>> > >>>>>>>>> > adding some references to the thread. >>>>>>>>> > >>>>>>>>> > - Jira ticket - SPARK-24815 >>>>>>>>> > <https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>> > - Design Doc >>>>>>>>> > < >>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>> > >>>>>>>>> > - discussion thread >>>>>>>>> > < >>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>> > - PR with initial implementation - >>>>>>>>> > https://github.com/apache/spark/pull/42352 >>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>> > >>>>>>>>> > Please vote with: >>>>>>>>> > >>>>>>>>> > [ ] +1: Accept the proposal and start with the development. >>>>>>>>> > [ ] +0 >>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>> > >>>>>>>>> > Thank you, >>>>>>>>> > >>>>>>>>> > Pavan >>>>>>>>> > >>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh <mi...@gmail.com >>>>>>>>> > >>>>>>>>> > wrote: >>>>>>>>> > >>>>>>>>> > > >>>>>>>>> > > +1 for me (non binding) >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>> responsibility for any >>>>>>>>> > > loss, damage or destruction of data or any other property >>>>>>>>> which may arise >>>>>>>>> > > from relying on this email's technical content is explicitly >>>>>>>>> disclaimed. >>>>>>>>> > > The author will in no case be liable for any monetary damages >>>>>>>>> arising from >>>>>>>>> > > such loss, damage or destruction. >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> >>>>>>>> -- <http://www.databricks.com> *Bhuwan Sahni* Staff Software Engineer bhuwan.sa...@databricks.com 500 108th Ave. NE Bellevue, WA 98004 USA
