Hi,
Our Spark is deployed on YARN and I found there were lots of spark-assembly
jars in the Spark heavy user filecache directory (aka
/usercache/username/filecache), and you know the assembly jar is bigger
than 100 MB before Spark v2. So all of them take 26GB (1/4 reserved
space) in most of Datanodes. And that's only one Spark heavy user. We have
lot's of spark heavy user.


BTW, I didn't put the assembly jar to HDFS. It is in $SPARK_HOME/lib and
submitted every time.

Best regards
Alan

Reply via email to