Re: about memory size for loading file

for this case i have 3 partitions, each process 3.333 GB data, am i right? On 2022/1/14 2:20, Sonal Goyal wrote: No it should not. The file would be partitioned and read across each node. On Fri, 14 Jan 2022 at 11:48 AM, frakass > wrote: Hello list

Re: about memory size for loading file

No it should not. The file would be partitioned and read across each node. On Fri, 14 Jan 2022 at 11:48 AM, frakass wrote: > Hello list > > Given the case I have a file whose size is 10GB. The ram of total > cluster is 24GB, three nodes. So the local node has only 8GB. > If I load this file

about memory size for loading file

Hello list Given the case I have a file whose size is 10GB. The ram of total cluster is 24GB, three nodes. So the local node has only 8GB. If I load this file into Spark as a RDD via sc.textFile interface, will this operation run into "out of memory" issue? Thank you.