Hi, Our Spark is deployed on YARN and I found there were lots of spark-assembly jars in the Spark heavy user filecache directory (aka /usercache/username/filecache), and you know the assembly jar is bigger than 100 MB before Spark v2. So all of them take 26GB (1/4 reserved space) in most of Datanodes. And that's only one Spark heavy user. We have lot's of spark heavy user.
BTW, I didn't put the assembly jar to HDFS. It is in $SPARK_HOME/lib and submitted every time. Best regards Alan
