I have a 150TB CSV file. I have a total of 100 TB RAM and 100TB disk. So If I do something like this
spark.read.option("header","true").csv(filepath).show(false) Will it lead to an OOM error since it doesn't have enough memory? or it will spill data onto the disk and process it? Thanks, Sid