Sounds good. Thanks again for your help on guiding the effort from discussion/review through voting phases in the spark dev community.
Thank you, Pavan On Tue, Mar 26, 2024 at 4:20 AM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Pavan, > > Thanks for instigating this proposal. Looks like the proposal is ready and > has enough votes to be implemented. Having a sheppard will make it more > fruitful. > > I will leave it to @Jungtaek Lim <kabhwan.opensou...@gmail.com> 's > capable hands to drive it forward. > > Will be there to help if needed. > > Cheers > > Mich Talebzadeh, > Technologist | Solutions Architect | Data Engineer | Generative AI > London > United Kingdom > > > view my Linkedin profile > <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!fi2RXZxkkJ-KxDWvHO9lRdlxh1Nu7smOSqX0Wbi6Gq0s7elQplrqshAH89qRNXw44q1o3Uk1q7FXrFHobRp6rkB9dUUgig$> > > > https://en.everybodywiki.com/Mich_Talebzadeh > <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!fi2RXZxkkJ-KxDWvHO9lRdlxh1Nu7smOSqX0Wbi6Gq0s7elQplrqshAH89qRNXw44q1o3Uk1q7FXrFHobRp6rkD7rLCHOA$> > > > > *Disclaimer:* The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner > <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!fi2RXZxkkJ-KxDWvHO9lRdlxh1Nu7smOSqX0Wbi6Gq0s7elQplrqshAH89qRNXw44q1o3Uk1q7FXrFHobRp6rkC5hzctXw$>Von > Braun > <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!fi2RXZxkkJ-KxDWvHO9lRdlxh1Nu7smOSqX0Wbi6Gq0s7elQplrqshAH89qRNXw44q1o3Uk1q7FXrFHobRp6rkC5hzctXw$> > )". > > > On Tue, 26 Mar 2024 at 10:02, Pavan Kotikalapudi <pkotikalap...@twilio.com> > wrote: > >> Hi Bhuwan, >> >> Glad to hear back from you! Very much appreciate your help on reviewing >> the design doc/PR and endorsing this proposal. >> >> Thank you so much @Jungtaek Lim <kabhwan.opensou...@gmail.com> , @Mich >> Talebzadeh <mich.talebza...@gmail.com> for graciously agreeing to >> mentor/shepherd this effort. >> >> Regarding Twilio copyright in Notice binary file: >> Twilio Opensource counsel was involved all through the process, I have >> placed it in the project file prior to Twilio signing a CCLA for the spark >> project contribution( Aug '23). >> >> Since the CCLA is signed now, I have removed the twilio copyright from >> that file. I didn't get a chance to update the PR after github-actions >> closed it. >> >> Please let me know of next steps needed to bring this draft PR/effort to >> completion. >> >> Thank you, >> >> Pavan >> >> >> On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> I'm happy to, but it looks like I need to check one more thing about the >>> license, according to the WIP PR >>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrA6JBQ8S0$> >>> . >>> >>> @Pavan Kotikalapudi <pkotikalap...@twilio.com> >>> I see you've added the copyright of Twilio in the NOTICE-binary file, >>> which makes me wonder if Twilio had filed CCLA to the Apache Software >>> Foundation. >>> >>> PMC members can correct me if I'm mistaken, but from my understanding >>> (and experiences of PMC member in other ASF project), code contribution is >>> considered as code donation and copyright belongs to ASF. That's why you >>> can't find the copyright of employers for contributors in the codebase. >>> What you see copyrights in NOTICE-binary is due to the fact we have binary >>> dependency and their licenses may require to explicitly mention about >>> copyright. It's not about direct code contribution. >>> >>> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior, >>> could you please engage with a relevant group in the company (could be a >>> legal team, or similar with OSS advocate team if there is any) and ensure >>> that CCLA is filed? The copyright issue is a legal issue, so we have to be >>> conservative and 100% sure that the employer is aware of what is the >>> meaning of donating the code to ASF via reviewing CCLA and relevant doc, >>> and explicitly express that they are OK with it via filing CCLA. >>> >>> You can read the description of agreements on contribution and ICLA/CCLA >>> form from this page. >>> https://www.apache.org/licenses/contributor-agreements.html >>> <https://urldefense.com/v3/__https://www.apache.org/licenses/contributor-agreements.html__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAktmm6BY$> >>> >>> Please let me know if this is resolved. This seems to me as a blocker to >>> move on. Please also let me know if the contribution is withdrawn from the >>> employer. >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>> >>> On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni >>> <bhuwan.sa...@databricks.com.invalid> wrote: >>> >>>> Hi Pavan, >>>> >>>> I looked at the PR, and the changes look simple and contained. It would >>>> be useful to add dynamic resource allocation to Spark Structured Streaming. >>>> >>>> Jungtaek. Would you be able to shepherd this change? >>>> >>>> >>>> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni < >>>> bhuwan.sa...@databricks.com> wrote: >>>> >>>>> Thanks a lot for creating the risk table Pavan. My apologies. I was >>>>> tied up with high priority items for the last couple weeks and could not >>>>> respond. I will review the PR by tomorrow's end, and get back to you. >>>>> >>>>> Appreciate your patience. >>>>> >>>>> Thanks >>>>> Bhuwan Sahni >>>>> >>>>> On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi < >>>>> pkotikalap...@twilio.com> wrote: >>>>> >>>>>> Hi Bhuwan, >>>>>> >>>>>> I hope the team got a chance to review the draft PR, looking for some >>>>>> comments to see if the plan looks alright?. I have updated the document >>>>>> about the risks >>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit*heading=h.577aawlyiedf__;Iw!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAzuRa_bM$>.(also >>>>>> mentioned below). Please confirm if it looks alright? >>>>>> >>>>>> *Spark application type* >>>>>> >>>>>> *auto-scaling capability* >>>>>> >>>>>> *with New auto-scaling capability* >>>>>> >>>>>> Spark Batch job >>>>>> >>>>>> Works with current DRA >>>>>> >>>>>> No - change >>>>>> >>>>>> Streaming query without trigger interval >>>>>> >>>>>> No implementation >>>>>> >>>>>> Can work with this implementation - (have to set certain scale back >>>>>> configs based on previous usage pattern) - maybe automate with future >>>>>> work? >>>>>> >>>>>> Spark Streaming query with Trigger interval >>>>>> >>>>>> No implementation >>>>>> >>>>>> With this implementation >>>>>> >>>>>> Spark Streaming query with one-time micro batch >>>>>> >>>>>> Works with current DRA >>>>>> >>>>>> No - change >>>>>> >>>>>> Spark Streaming query with >>>>>> >>>>>> Availablenow micro batch >>>>>> >>>>>> Works with current DRA >>>>>> >>>>>> No - change >>>>>> >>>>>> Batch + Streaming query ( >>>>>> >>>>>> default/ >>>>>> >>>>>> triggger-interval/ >>>>>> >>>>>> once/ >>>>>> >>>>>> availablenow modes), other notebook use cases. >>>>>> >>>>>> No implementation >>>>>> >>>>>> No implementation >>>>>> >>>>>> >>>>>> >>>>>> We are more than happy to collaborate on a call to make better >>>>>> progress on this enhancement. Please let us know. >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Pavan >>>>>> >>>>>> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> Hi Bhuwan et al, >>>>>>> >>>>>>> Thank you for passing on the DataBricks Structured Streaming team's >>>>>>> review of the SPIP document. FYI, I work closely with Pawan and other >>>>>>> members to help deliver this piece of work. We appreciate your insights, >>>>>>> especially regarding the cost savings potential from the PoC. >>>>>>> >>>>>>> Pavan already furnished you with some additional info. Your team's >>>>>>> point about the SPIP currently addressing a specific use case (single >>>>>>> streaming query with Processing Time trigger) is well-taken. We agree >>>>>>> that >>>>>>> maintaining simplicity is key, particularly as we explore more general >>>>>>> resource allocation mechanisms in the future. To address the concerns >>>>>>> and >>>>>>> foster open discussion, The DataBricks team are invited to directly add >>>>>>> their comments and suggestions to the Jira itself >>>>>>> >>>>>>> [SPARK-24815] Structured Streaming should support dynamic allocation >>>>>>> - ASF JIRA (apache.org) >>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>>>>>> This will ensure everyone involved can benefit from your team's >>>>>>> expertise and facilitate further collaboration. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>> London >>>>>>> United Kingdom >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von >>>>>>> Braun >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$> >>>>>>> )". >>>>>>> >>>>>>> >>>>>>> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi >>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>> >>>>>>>> Thanks Bhuwan and rest of the databricks team for the reviews, >>>>>>>> >>>>>>>> I appreciate your reviews, was very helpful in evaluating a few >>>>>>>> options that were overlooked earlier (especially about mixed spark apps >>>>>>>> running on notebooks). Regarding the use-cases, It could handle >>>>>>>> multiple >>>>>>>> streaming queries provided that they are run on the same trigger >>>>>>>> interval >>>>>>>> processing time (very similar to how current batch dra is set up)..but >>>>>>>> I >>>>>>>> felt like it would be beneficial if we separate out streaming queries >>>>>>>> when >>>>>>>> setting up production pipelines. >>>>>>>> >>>>>>>> Regarding the implementation, here is the draft PR >>>>>>>> https://github.com/apache/spark/pull/42352 >>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UC8iQomlg$>. >>>>>>>> (already mentioned in ticket SPARK-24815 >>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>>>>>>> ) >>>>>>>> >>>>>>>> I have built it on top of the current Dynamic resource allocation >>>>>>>> (DRA) algorithm >>>>>>>> <https://urldefense.com/v3/__https://spark.apache.org/docs/latest/job-scheduling.html*dynamic-resource-allocation__;Iw!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBK8f68fQ$> >>>>>>>> . >>>>>>>> While current DRA is catered towards batch jobs. This >>>>>>>> implementation just makes few changes to that algorithm to >>>>>>>> - do gradual scale-back. The remove-policy still applies (uses 2 >>>>>>>> old configs we currently have), but we now remove few executors per >>>>>>>> round >>>>>>>> of evaluation ( I have added 2 configs to tune that) >>>>>>>> - The scale-out process also still uses the same request policy >>>>>>>> (same uses 2 old configs we currently have). >>>>>>>> - while we are using the old configs in the both scale-out/back, >>>>>>>> the difference is that we are now giving configs to them based on the >>>>>>>> trigger interval as our north star. >>>>>>>> >>>>>>>> This implementation is just changes in 2 files to make it work. I >>>>>>>> have made the changes minimal/limited to just the core module of the >>>>>>>> spark >>>>>>>> repo. >>>>>>>> 1) to make sure it is applied on primitives of task, stage, job >>>>>>>> which the current dra is already doing. (This will enable us to think >>>>>>>> about >>>>>>>> other cases like default and continuous mode can still work provided >>>>>>>> we >>>>>>>> have a target processing time range we want to achieve) >>>>>>>> 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and >>>>>>>> listeners which are already well tested and working well for batch job >>>>>>>> use >>>>>>>> case. >>>>>>>> >>>>>>>> We internally (in the company) have also added helpers so that we >>>>>>>> have less configs to tune. I can contribute that as well, if it makes >>>>>>>> the >>>>>>>> dev experience better. >>>>>>>> >>>>>>>> Feel free to review the PR, when we decide the direction is alright >>>>>>>> I will start adding the tests as well. >>>>>>>> >>>>>>>> On a side note. Maybe we should consider some future work to have >>>>>>>> DRA algo per query (batch, streaming queries, mixed etc) rather than >>>>>>>> per >>>>>>>> spark context. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Pavan >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni >>>>>>>> <bhuwan.sa...@databricks.com.invalid> wrote: >>>>>>>> >>>>>>>>> Hi Pavan, >>>>>>>>> >>>>>>>>> I am from the DataBricks Structured Streaming team, and we did a >>>>>>>>> review of the SPIP internally. Wanted to pass on the points discussed >>>>>>>>> in >>>>>>>>> the meeting. >>>>>>>>> >>>>>>>>> Thanks for putting together the SPIP document. It's useful to have >>>>>>>>> dynamic resource allocation for Streaming queries, and it's exciting >>>>>>>>> to see >>>>>>>>> the cost saving numbers from your PoC. However, in general we >>>>>>>>> discovered >>>>>>>>> that the SPIP addresses a very particular use-case (single streaming >>>>>>>>> query >>>>>>>>> in Spark cluster with Processing time Trigger). Keeping that in mind, >>>>>>>>> it's >>>>>>>>> useful to make sure that the implementation is simple. Can you please >>>>>>>>> share >>>>>>>>> your PoC implementation to understand the code complexity. This would >>>>>>>>> help >>>>>>>>> us to ensure that dynamic resource allocation mechanism for Streaming >>>>>>>>> queries does not become complicated in the future (if more resource >>>>>>>>> allocation mechanisms are added to address other use-cases). >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Bhuwan >>>>>>>>> >>>>>>>>> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh < >>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Pavan and those who kindly voted for this SPIP >>>>>>>>>> >>>>>>>>>> Great to have 6+ votes and no -1 and 0. The so-called mass volume >>>>>>>>>> is there. The rest is admin matter and how to drive the project >>>>>>>>>> forward and >>>>>>>>>> yes there is more than one way of skinning the cat. I think we need >>>>>>>>>> some >>>>>>>>>> flexibility in the rules given the dwindling (IMO) number of >>>>>>>>>> comitters who >>>>>>>>>> are willing or actively participating. For example, on a similar >>>>>>>>>> matter I >>>>>>>>>> approached Codi Koeninger who was one of the founders of Spark >>>>>>>>>> Streaming, >>>>>>>>>> to shepherd a project almost a year back. Sadly he is no longer >>>>>>>>>> active and >>>>>>>>>> quotes "I haven't been involved lately and would be missing a >>>>>>>>>> lot of context." So we need to improvise and see how best we can >>>>>>>>>> drive this and similar ones. We wait a short while for a response >>>>>>>>>> otherwise >>>>>>>>>> I am happy to give a hand if needed and work with you guys to drive >>>>>>>>>> this. >>>>>>>>>> It is something worthwhile. >>>>>>>>>> >>>>>>>>>> HTH >>>>>>>>>> >>>>>>>>>> T >>>>>>>>>> Mich Talebzadeh, >>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>> London >>>>>>>>>> United Kingdom >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> view my Linkedin profile >>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfbGToAE$> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWTA0_mlE$> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>>> note >>>>>>>>>> that, as with any advice, quote "one test result is worth >>>>>>>>>> one-thousand >>>>>>>>>> expert opinions (Werner >>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >>>>>>>>>> Braun >>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$> >>>>>>>>>> )". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi >>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks for the pointers Mich, will wait for Jungtaek Lee or any >>>>>>>>>>> other PMC members to respond. >>>>>>>>>>> >>>>>>>>>>> aggregating upvotes to this email thread >>>>>>>>>>> >>>>>>>>>>> +6 >>>>>>>>>>> Mich Talebzadeh >>>>>>>>>>> Adam Hobbs >>>>>>>>>>> Pavan Kotikalapudi >>>>>>>>>>> Krystal Mitchell >>>>>>>>>>> Sona Torosyan >>>>>>>>>>> Aaron Kern >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> >>>>>>>>>>> Pavan >>>>>>>>>>> >>>>>>>>>>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> please check this doc >>>>>>>>>>>> >>>>>>>>>>>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>>>>>>>>>>> <https://urldefense.com/v3/__https://spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>>>>>>>>>>> >>>>>>>>>>>> and specifically the below extract >>>>>>>>>>>> >>>>>>>>>>>> Discussing an SPIP >>>>>>>>>>>> >>>>>>>>>>>> All discussion of an SPIP should take place in a public forum, >>>>>>>>>>>> preferably the discussion attached to the Jira. Any discussions >>>>>>>>>>>> that happen >>>>>>>>>>>> offline should be made available online for the public via meeting >>>>>>>>>>>> notes >>>>>>>>>>>> summarizing the discussions.(done) >>>>>>>>>>>> >>>>>>>>>>>> During this discussion, one or more shepherds should be >>>>>>>>>>>> identified among PMC members. (outstanding) >>>>>>>>>>>> >>>>>>>>>>>> Once the discussion settles, the shepherd(s) should call for a >>>>>>>>>>>> vote on the SPIP moving forward on the dev@ list. The vote >>>>>>>>>>>> should be open for at least 72 hours and follows the typical >>>>>>>>>>>> Apache vote >>>>>>>>>>>> process and passes upon consensus (at least 3 +1 votes from PMC >>>>>>>>>>>> members and >>>>>>>>>>>> no -1 votes from PMC members). dev@ should be notified of the >>>>>>>>>>>> vote result. >>>>>>>>>>>> >>>>>>>>>>>> If there does not exist at least one PMC member that is >>>>>>>>>>>> committed to shepherding the change within a month, the SPIP is >>>>>>>>>>>> rejected. >>>>>>>>>>>> >>>>>>>>>>>> If a committer does not think a SPIP aligns with long-term >>>>>>>>>>>> project goals, or is not practical at the point of proposal, the >>>>>>>>>>>> committer >>>>>>>>>>>> should -1 the SPIP explicitly and give technical justifications. >>>>>>>>>>>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee >>>>>>>>>>>> can kindly help the process >>>>>>>>>>>> >>>>>>>>>>>> cheers >>>>>>>>>>>> >>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>> London >>>>>>>>>>>> United Kingdom >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>> essential to >>>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>>>>>>>>>>> Braun >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>>>>>>>>>>> )". >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Mich, >>>>>>>>>>>>> >>>>>>>>>>>>> We have >>>>>>>>>>>>> >>>>>>>>>>>>> five +1s till now. >>>>>>>>>>>>> >>>>>>>>>>>>> Mich Talebzadeh >>>>>>>>>>>>> Adam Hobbs >>>>>>>>>>>>> Pavan Kotikalapudi >>>>>>>>>>>>> Krystal Mitchell >>>>>>>>>>>>> Sona Torosyan >>>>>>>>>>>>> (few more in github pr) >>>>>>>>>>>>> +0: None >>>>>>>>>>>>> >>>>>>>>>>>>> -1: None >>>>>>>>>>>>> >>>>>>>>>>>>> Does it pass the required condition as approved? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Not sure of that though, nothing about minimum required is >>>>>>>>>>>>> mentioned in the past emails. >>>>>>>>>>>>> >>>>>>>>>>>>> I would request spark PMC members or any others who have done >>>>>>>>>>>>> this in the past to understand the process better. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> >>>>>>>>>>>>> Pavan >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Pavan, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Do you have a list of votes for this feature by any >>>>>>>>>>>>>> chance? Does it pass the required condition as approved? >>>>>>>>>>>>>> >>>>>>>>>>>>>> HTH >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>>>> London >>>>>>>>>>>>>> United Kingdom >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>>>> essential >>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$>Von >>>>>>>>>>>>>> Braun >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$> >>>>>>>>>>>>>> )". >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi >>>>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes. The PR was closed due to inactivity by github actions.. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The msg >>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>>>>>>>>>>> also >>>>>>>>>>>>>>> says >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > If you'd like to revive this PR, please reopen it and ask >>>>>>>>>>>>>>> a committer to remove the Stale tag! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>>>>>> London >>>>>>>>>>>>>>>> United Kingdom >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It >>>>>>>>>>>>>>>> is essential >>>>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>>>>>>>>>>> Braun >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>>>>>>>>>>> )". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Spark PMC members, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I think we have few upvotes for this effort here and more >>>>>>>>>>>>>>>>> people are showing interest (see PR comments >>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>>>>>>>>>>> .) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is anyone interested in mentoring and reviewing this >>>>>>>>>>>>>>>>> effort? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Also can the repository admin/owner re-open the PR? ( I >>>>>>>>>>>>>>>>> guess people only with admin access to the repository can do >>>>>>>>>>>>>>>>> that). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Pavan >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>>>>>>>>>>> <kryst...@adobe.com.invalid> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>>>>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > adding some references to the thread. >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > - Jira ticket - SPARK-24815 >>>>>>>>>>>>>>>>>> > <https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>>>>>>>>>>> > - Design Doc >>>>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > - discussion thread >>>>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>>>>>>>>>>> > - PR with initial implementation - >>>>>>>>>>>>>>>>>> > https://github.com/apache/spark/pull/42352 >>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > Please vote with: >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > [ ] +1: Accept the proposal and start with the >>>>>>>>>>>>>>>>>> development. >>>>>>>>>>>>>>>>>> > [ ] +0 >>>>>>>>>>>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > Thank you, >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > Pavan >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh < >>>>>>>>>>>>>>>>>> mi...@gmail.com> >>>>>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > +1 for me (non binding) >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>>>>>>>>> responsibility for any >>>>>>>>>>>>>>>>>> > > loss, damage or destruction of data or any other >>>>>>>>>>>>>>>>>> property which may arise >>>>>>>>>>>>>>>>>> > > from relying on this email's technical content is >>>>>>>>>>>>>>>>>> explicitly disclaimed. >>>>>>>>>>>>>>>>>> > > The author will in no case be liable for any monetary >>>>>>>>>>>>>>>>>> damages arising from >>>>>>>>>>>>>>>>>> > > such loss, damage or destruction. >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfJvj8wi$> >>>>>>>>> *Bhuwan Sahni* >>>>>>>>> Staff Software Engineer >>>>>>>>> >>>>>>>>> bhuwan.sa...@databricks.com >>>>>>>>> 500 108th Ave. NE >>>>>>>>> Bellevue, WA 98004 >>>>>>>>> USA >>>>>>>>> >>>>>>>> >>>>> >>>>> -- >>>>> >>>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrARPj-LXk$> >>>>> *Bhuwan Sahni* >>>>> Staff Software Engineer >>>>> >>>>> bhuwan.sa...@databricks.com >>>>> 500 108th Ave. NE >>>>> Bellevue, WA 98004 >>>>> USA >>>>> >>>> >>>> >>>> -- >>>> >>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrARPj-LXk$> >>>> *Bhuwan Sahni* >>>> Staff Software Engineer >>>> >>>> bhuwan.sa...@databricks.com >>>> 500 108th Ave. NE >>>> Bellevue, WA 98004 >>>> USA >>>> >>>
