Hi Bhuwan et al, Thank you for passing on the DataBricks Structured Streaming team's review of the SPIP document. FYI, I work closely with Pawan and other members to help deliver this piece of work. We appreciate your insights, especially regarding the cost savings potential from the PoC.
Pavan already furnished you with some additional info. Your team's point about the SPIP currently addressing a specific use case (single streaming query with Processing Time trigger) is well-taken. We agree that maintaining simplicity is key, particularly as we explore more general resource allocation mechanisms in the future. To address the concerns and foster open discussion, The DataBricks team are invited to directly add their comments and suggestions to the Jira itself [SPARK-24815] Structured Streaming should support dynamic allocation - ASF JIRA (apache.org) <https://issues.apache.org/jira/browse/SPARK-24815> This will ensure everyone involved can benefit from your team's expertise and facilitate further collaboration. Thanks Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi <pkotikalap...@twilio.com.invalid> wrote: > Thanks Bhuwan and rest of the databricks team for the reviews, > > I appreciate your reviews, was very helpful in evaluating a few options > that were overlooked earlier (especially about mixed spark apps running on > notebooks). Regarding the use-cases, It could handle multiple streaming > queries provided that they are run on the same trigger interval processing > time (very similar to how current batch dra is set up)..but I felt like it > would be beneficial if we separate out streaming queries when setting up > production pipelines. > > Regarding the implementation, here is the draft PR > https://github.com/apache/spark/pull/42352. (already mentioned in ticket > SPARK-24815 <https://issues.apache.org/jira/browse/SPARK-24815>) > > I have built it on top of the current Dynamic resource allocation (DRA) > algorithm > <https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation> > . > While current DRA is catered towards batch jobs. This implementation just > makes few changes to that algorithm to > - do gradual scale-back. The remove-policy still applies (uses 2 old > configs we currently have), but we now remove few executors per round of > evaluation ( I have added 2 configs to tune that) > - The scale-out process also still uses the same request policy (same uses > 2 old configs we currently have). > - while we are using the old configs in the both scale-out/back, the > difference is that we are now giving configs to them based on the trigger > interval as our north star. > > This implementation is just changes in 2 files to make it work. I have > made the changes minimal/limited to just the core module of the spark repo. > 1) to make sure it is applied on primitives of task, stage, job which the > current dra is already doing. (This will enable us to think about other > cases like default and continuous mode can still work provided we have a > target processing time range we want to achieve) > 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and listeners > which are already well tested and working well for batch job use case. > > We internally (in the company) have also added helpers so that we have > less configs to tune. I can contribute that as well, if it makes the dev > experience better. > > Feel free to review the PR, when we decide the direction is alright I will > start adding the tests as well. > > On a side note. Maybe we should consider some future work to have DRA > algo per query (batch, streaming queries, mixed etc) rather than per spark > context. > > Thank you, > > Pavan > > > On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni > <bhuwan.sa...@databricks.com.invalid> wrote: > >> Hi Pavan, >> >> I am from the DataBricks Structured Streaming team, and we did a review >> of the SPIP internally. Wanted to pass on the points discussed in the >> meeting. >> >> Thanks for putting together the SPIP document. It's useful to have >> dynamic resource allocation for Streaming queries, and it's exciting to see >> the cost saving numbers from your PoC. However, in general we discovered >> that the SPIP addresses a very particular use-case (single streaming query >> in Spark cluster with Processing time Trigger). Keeping that in mind, it's >> useful to make sure that the implementation is simple. Can you please share >> your PoC implementation to understand the code complexity. This would help >> us to ensure that dynamic resource allocation mechanism for Streaming >> queries does not become complicated in the future (if more resource >> allocation mechanisms are added to address other use-cases). >> >> Thanks, >> Bhuwan >> >> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> Hi Pavan and those who kindly voted for this SPIP >>> >>> Great to have 6+ votes and no -1 and 0. The so-called mass volume is >>> there. The rest is admin matter and how to drive the project forward and >>> yes there is more than one way of skinning the cat. I think we need some >>> flexibility in the rules given the dwindling (IMO) number of comitters who >>> are willing or actively participating. For example, on a similar matter I >>> approached Codi Koeninger who was one of the founders of Spark Streaming, >>> to shepherd a project almost a year back. Sadly he is no longer active and >>> quotes "I haven't been involved lately and would be missing a lot of >>> context." So we need to improvise and see how best we can drive this >>> and similar ones. We wait a short while for a response otherwise I am happy >>> to give a hand if needed and work with you guys to drive this. It is >>> something worthwhile. >>> >>> HTH >>> >>> T >>> Mich Talebzadeh, >>> Dad | Technologist | Solutions Architect | Engineer >>> London >>> United Kingdom >>> >>> >>> view my Linkedin profile >>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfbGToAE$> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWTA0_mlE$> >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >>> Braun >>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$> >>> )". >>> >>> >>> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi >>> <pkotikalap...@twilio.com.invalid> wrote: >>> >>>> Thanks for the pointers Mich, will wait for Jungtaek Lee or any other >>>> PMC members to respond. >>>> >>>> aggregating upvotes to this email thread >>>> >>>> +6 >>>> Mich Talebzadeh >>>> Adam Hobbs >>>> Pavan Kotikalapudi >>>> Krystal Mitchell >>>> Sona Torosyan >>>> Aaron Kern >>>> >>>> Thank you, >>>> >>>> Pavan >>>> >>>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> please check this doc >>>>> >>>>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>>>> <https://urldefense.com/v3/__https://spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>>>> >>>>> and specifically the below extract >>>>> >>>>> Discussing an SPIP >>>>> >>>>> All discussion of an SPIP should take place in a public forum, >>>>> preferably the discussion attached to the Jira. Any discussions that >>>>> happen >>>>> offline should be made available online for the public via meeting notes >>>>> summarizing the discussions.(done) >>>>> >>>>> During this discussion, one or more shepherds should be identified >>>>> among PMC members. (outstanding) >>>>> >>>>> Once the discussion settles, the shepherd(s) should call for a vote on >>>>> the SPIP moving forward on the dev@ list. The vote should be open for >>>>> at least 72 hours and follows the typical Apache vote process and passes >>>>> upon consensus (at least 3 +1 votes from PMC members and no -1 votes from >>>>> PMC members). dev@ should be notified of the vote result. >>>>> >>>>> If there does not exist at least one PMC member that is committed to >>>>> shepherding the change within a month, the SPIP is rejected. >>>>> >>>>> If a committer does not think a SPIP aligns with long-term project >>>>> goals, or is not practical at the point of proposal, the committer should >>>>> -1 the SPIP explicitly and give technical justifications. >>>>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee can >>>>> kindly help the process >>>>> >>>>> cheers >>>>> >>>>> Mich Talebzadeh, >>>>> Dad | Technologist | Solutions Architect | Engineer >>>>> London >>>>> United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>>>> >>>>> >>>>> >>>>> *Disclaimer:* The information provided is correct to the best of my >>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>> expert opinions (Werner >>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>>>> Braun >>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>>>> )". >>>>> >>>>> >>>>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>> >>>>>> Hi Mich, >>>>>> >>>>>> We have >>>>>> >>>>>> five +1s till now. >>>>>> >>>>>> Mich Talebzadeh >>>>>> Adam Hobbs >>>>>> Pavan Kotikalapudi >>>>>> Krystal Mitchell >>>>>> Sona Torosyan >>>>>> (few more in github pr) >>>>>> +0: None >>>>>> >>>>>> -1: None >>>>>> >>>>>> Does it pass the required condition as approved? >>>>>> >>>>>> >>>>>> Not sure of that though, nothing about minimum required is mentioned >>>>>> in the past emails. >>>>>> >>>>>> I would request spark PMC members or any others who have done this in >>>>>> the past to understand the process better. >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Pavan >>>>>> >>>>>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> Hi Pavan, >>>>>>> >>>>>>> Do you have a list of votes for this feature by any chance? Does it >>>>>>> pass the required condition as approved? >>>>>>> >>>>>>> HTH >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>> London >>>>>>> United Kingdom >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$>Von >>>>>>> Braun >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$> >>>>>>> )". >>>>>>> >>>>>>> >>>>>>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi >>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>> >>>>>>>> Yes. The PR was closed due to inactivity by github actions.. >>>>>>>> >>>>>>>> The msg >>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>>>> also >>>>>>>> says >>>>>>>> >>>>>>>> > If you'd like to revive this PR, please reopen it and ask a >>>>>>>> committer to remove the Stale tag! >>>>>>>> >>>>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>>>> >>>>>>>>> >>>>>>>>> Mich Talebzadeh, >>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>> London >>>>>>>>> United Kingdom >>>>>>>>> >>>>>>>>> >>>>>>>>> view my Linkedin profile >>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>> note >>>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>>> expert opinions (Werner >>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>>>> Braun >>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>>>> )". >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>> >>>>>>>>>> Hi Spark PMC members, >>>>>>>>>> >>>>>>>>>> I think we have few upvotes for this effort here and more people >>>>>>>>>> are showing interest (see PR comments >>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>>>> .) >>>>>>>>>> >>>>>>>>>> Is anyone interested in mentoring and reviewing this effort? >>>>>>>>>> >>>>>>>>>> Also can the repository admin/owner re-open the PR? ( I guess >>>>>>>>>> people only with admin access to the repository can do that). >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> >>>>>>>>>> Pavan >>>>>>>>>> >>>>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>>>> <kryst...@adobe.com.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> +1 >>>>>>>>>>> >>>>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>>>> > >>>>>>>>>>> > adding some references to the thread. >>>>>>>>>>> > >>>>>>>>>>> > - Jira ticket - SPARK-24815 >>>>>>>>>>> > <https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>>>> > - Design Doc >>>>>>>>>>> > < >>>>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>>>> > >>>>>>>>>>> > - discussion thread >>>>>>>>>>> > < >>>>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>>>> > - PR with initial implementation - >>>>>>>>>>> > https://github.com/apache/spark/pull/42352 >>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>>>> > >>>>>>>>>>> > Please vote with: >>>>>>>>>>> > >>>>>>>>>>> > [ ] +1: Accept the proposal and start with the development. >>>>>>>>>>> > [ ] +0 >>>>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>>>> > >>>>>>>>>>> > Thank you, >>>>>>>>>>> > >>>>>>>>>>> > Pavan >>>>>>>>>>> > >>>>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh < >>>>>>>>>>> mi...@gmail.com> >>>>>>>>>>> > wrote: >>>>>>>>>>> > >>>>>>>>>>> > > >>>>>>>>>>> > > +1 for me (non binding) >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>> responsibility for any >>>>>>>>>>> > > loss, damage or destruction of data or any other property >>>>>>>>>>> which may arise >>>>>>>>>>> > > from relying on this email's technical content is explicitly >>>>>>>>>>> disclaimed. >>>>>>>>>>> > > The author will in no case be liable for any monetary >>>>>>>>>>> damages arising from >>>>>>>>>>> > > such loss, damage or destruction. >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>> >> >> -- >> >> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfJvj8wi$> >> *Bhuwan Sahni* >> Staff Software Engineer >> >> bhuwan.sa...@databricks.com >> 500 108th Ave. NE >> Bellevue, WA 98004 >> USA >> >