Hi, in the Spark UI, one of the metrics is "shuffle spill (memory)". What is it exactly? Spilling to disk when the shuffle data doesn't fit in memory I get it, but what does it mean to spill to memory?
Thanks, - Sebastien
Hi, in the Spark UI, one of the metrics is "shuffle spill (memory)". What is it exactly? Spilling to disk when the shuffle data doesn't fit in memory I get it, but what does it mean to spill to memory?
Thanks, - Sebastien