I just did a test, even for a single node (local deployment), spark can
handle the data whose size is much larger than the total memory.
My test VM (2g ram, 2 cores):
$ free -m
totalusedfree shared buff/cache
available
Mem: 19921845
With autoscaling can have any numbers of executors.
Thanks
On Fri, Apr 8, 2022, 08:27 Wes Peng wrote:
> I once had a file which is 100+GB getting computed in 3 nodes, each node
> has 24GB memory only. And the job could be done well. So from my
> experience spark cluster seems to work correctly
I once had a file which is 100+GB getting computed in 3 nodes, each node
has 24GB memory only. And the job could be done well. So from my
experience spark cluster seems to work correctly for big files larger
than memory by swapping them to disk.
Thanks
rajat kumar wrote:
Tested this with
how many executors do you have?
rajat kumar wrote:
Tested this with executors of size 5 cores, 17GB memory. Data vol is
really high around 1TB
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Tested this with executors of size 5 cores, 17GB memory. Data vol is really
high around 1TB
Thanks
Rajat
On Thu, Apr 7, 2022, 23:43 rajat kumar wrote:
> Hello Users,
>
> I got following error, tried increasing executor memory and memory
> overhead that also did not help .
>
Hello Users,
I got following error, tried increasing executor memory and memory overhead
that also did not help .
ExecutorLost Failure(executor1 exited caused by one of the following tasks)
Reason: container from a bad node:
java.lang.OutOfMemoryError: enough memory for aggregation
Can
