Hi,
I am reading a Hive Orc table into memory, StorageLevel is set to
(StorageLevel.MEMORY_AND_DISK_SER)
Total size of the Hive table is 5GB
Started the spark-shell as below
spark-shell --master yarn --deploy-mode client --num-executors 8
--driver-memory 5G --executor-memory 7G --executor-cores
, May 11, 2017 1:34 PM
To: Anantharaman, Srinatha (Contractor) ;
user
Subject: Re: Spark consumes more memory
I would try to track down the "no space left on device" - find out where that
originates from, since you should be able to allocate 10 executors with 4 cores
and 15GB RAM
