I have a 150TB CSV file.

I have a total of 100 TB RAM and 100TB disk. So If I do something like this

spark.read.option("header","true").csv(filepath).show(false)

Will it lead to an OOM error since it doesn't have enough memory? or it
will spill data onto the disk and process it?

Thanks,
Sid

Reply via email to