Re: Will it lead to OOM error?

f 100 TB RAM and 100TB disk. So If I do something like >> this >> >> spark.read.option("header","true").csv(filepath).show(false) >> >> Will it lead to an OOM error since it doesn't have enough memory? or it >> will spill data onto the disk and process it? >> >> Thanks, >> Sid >> >

Re: Will it lead to OOM error?

epath).show(false) > > Will it lead to an OOM error since it doesn't have enough memory? or it > will spill data onto the disk and process it? > > Thanks, > Sid >

Re: Will it lead to OOM error?

lead to OOM error? Thanks, Sid On Wed, Jun 22, 2022 at 6:40 PM Enrico Minack wrote: The RAM and disk memory consumtion depends on what you do with the data after reading them. Your particular action will read 20 lines from the first partition and show them. So it will not use

Re: Will it lead to OOM error?

Hi Enrico, Thanks for the insights. Could you please help me to understand with one example of compressed files where the file wouldn't be split in partitions and will put load on a single partition and might lead to OOM error? Thanks, Sid On Wed, Jun 22, 2022 at 6:40 PM Enrico Minack wrote

Re: Will it lead to OOM error?

quot;true").csv(filepath).show(false) Will it lead to an OOM error since it doesn't have enough memory? or it will spill data onto the disk and process it? Thanks, Sid -- Thanks Deepak www.bigdatabig.com <http://www.bigdatabig.com> www.keosha.net <http://www.keosha.net>

Re: Will it lead to OOM error?

rue").csv(filepath).show(false) > > Will it lead to an OOM error since it doesn't have enough memory? or it > will spill data onto the disk and process it? > > Thanks, > Sid > -- Thanks Deepak www.bigdatabig.com www.keosha.net

Will it lead to OOM error?

I have a 150TB CSV file. I have a total of 100 TB RAM and 100TB disk. So If I do something like this spark.read.option("header","true").csv(filepath).show(false) Will it lead to an OOM error since it doesn't have enough memory? or it will spill data onto the disk and process it? Thanks, Sid