for this case i have 3 partitions, each process 3.333 GB data, am i right?
On 2022/1/14 2:20, Sonal Goyal wrote:
No it should not. The file would be partitioned and read across each node.
On Fri, 14 Jan 2022 at 11:48 AM, frakass <capitnfrak...@free.fr
<mailto:capitnfrak...@free.fr>> wrote:
Hello list
Given the case I have a file whose size is 10GB. The ram of total
cluster is 24GB, three nodes. So the local node has only 8GB.
If I load this file into Spark as a RDD via sc.textFile interface, will
this operation run into "out of memory" issue?
Thank you.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
<mailto:user-unsubscr...@spark.apache.org>
--
Cheers,
Sonal
https://github.com/zinggAI/zingg <https://github.com/zinggAI/zingg>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
