Re: Something about Spark which has bothered me for a very long time, which I've never understood

2022-05-05 Thread Lalwani, Jayesh
objects. I suspect you’ll just see the threads waiting for S3 to send back a response. From: Denarian Kislata Date: Thursday, May 5, 2022 at 8:03 AM To: "user@spark.apache.org" Subject: [EXTERNAL] Something about Spark which has bothered me for a very long time, which I've n

Something about Spark which has bothered me for a very long time, which I've never understood

2022-05-05 Thread Denarian Kislata
Greetings, and thanks in advance. For ~8 years i've been a spark user, and i've seen this same problem at more SaaS startups than I can count, and although it's straightforward to fix, I've never understood _why_ it happens. I'm hoping someone can explain the why behind it. Unfortunately I don't