[
https://issues.apache.org/jira/browse/SPARK-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322947#comment-14322947
]
Sean Owen commented on SPARK-5838:
----------------------------------
I think it's to be expected that you can't change env variables without
restarting the process. are you suggesting this be changeable on the fly?
Your problem is just that you need to point it at a larger directory, yes. Are
you sure it keeps using the old directory? meaning, are you sure all the
processes were updated, that your driver process is using the new value, that
these aren't just old files, etc.? I don't see how it would recognize the new
value but not use it.
> Changing SPARK_LOCAL_DIRS option in spark-env.sh does not take effect without
> daemon restart
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-5838
> URL: https://issues.apache.org/jira/browse/SPARK-5838
> Project: Spark
> Issue Type: Bug
> Components: Deploy, EC2, Spark Submit
> Affects Versions: 1.1.1
> Reporter: Theodore Vasiloudis
> Priority: Minor
>
> This issue has already been mentioned in the mailing list here:
> http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-local-dir-on-driver-program-doesn-t-take-effect-td11040.html
> The problem usually has to do with Spark creating too many files during
> shuffles, filling up the small amount of disk space that most EC2 instances
> have for root on /mnt2.
> The workaround is to set SPARK_LOCAL_DIRS to a larger volume (e.g. to the
> /mnt/spark volume only, removing /mnt2).
> However for these changes to take effect, the daemons need to be restarted
> with sbin/stop-all -> sbin/start-all.
> Even more troubling is the fact that the Web UI-> Environment reports that
> the spark.local.dir is set to the new path, but Spark still spills to /mnt2
> as well.
> To my knowledge this is not mentioned anywhere in the documentation or any
> other mailing list reply except for the one I linked.
> I guess possible solutions are to either ensure the change does take effect
> so that reality agrees with what the Web UI is reporting, or include a
> section on the documentation of EC2 for this kind of problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
