[ https://issues.apache.org/jira/browse/SPARK-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323929#comment-14323929 ]
Theodore Vasiloudis commented on SPARK-5838: -------------------------------------------- My experience is the same as with the email I linked to. So even after changing the SPARK_LOCAL_DIRS in spark-evn.sh and running a new job it keeps using the old SPARK_LOCAL_DIRS. In the documentation it is mentioned that spark-evn.sh "is also sourced when running local Spark applications or submission scripts." So I expected that the value of the SPARK_LOCAL_DIRS env variable would change, and according to the Web UI it does, but shuffle spills still end up in the directories that were previously set. I will try to reproduce the bug next week and post instructions on how to replicate. > Changing SPARK_LOCAL_DIRS option in spark-env.sh does not take effect without > daemon restart > -------------------------------------------------------------------------------------------- > > Key: SPARK-5838 > URL: https://issues.apache.org/jira/browse/SPARK-5838 > Project: Spark > Issue Type: Bug > Components: Deploy, EC2, Spark Submit > Affects Versions: 1.1.1 > Reporter: Theodore Vasiloudis > Priority: Minor > > This issue has already been mentioned in the mailing list here: > http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-local-dir-on-driver-program-doesn-t-take-effect-td11040.html > The problem usually has to do with Spark creating too many files during > shuffles, filling up the small amount of disk space that most EC2 instances > have for root on /mnt2. > The workaround is to set SPARK_LOCAL_DIRS to a larger volume (e.g. to the > /mnt/spark volume only, removing /mnt2). > However for these changes to take effect, the daemons need to be restarted > with sbin/stop-all -> sbin/start-all. > Even more troubling is the fact that the Web UI-> Environment reports that > the spark.local.dir is set to the new path, but Spark still spills to /mnt2 > as well. > To my knowledge this is not mentioned anywhere in the documentation or any > other mailing list reply except for the one I linked. > I guess possible solutions are to either ensure the change does take effect > so that reality agrees with what the Web UI is reporting, or include a > section on the documentation of EC2 for this kind of problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org