[ https://issues.apache.org/jira/browse/SPARK-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073938#comment-14073938 ]
Allan Douglas R. de Oliveira commented on SPARK-1044: ----------------------------------------------------- I think it is still a good idea even if the automatic cleanup is implemented. One large job or many small jobs can fill many gigabytes before the cleanup can kick in. > Default spark logs location in EC2 AMI leads to out-of-disk space pretty soon > ----------------------------------------------------------------------------- > > Key: SPARK-1044 > URL: https://issues.apache.org/jira/browse/SPARK-1044 > Project: Spark > Issue Type: Improvement > Components: EC2 > Reporter: Tathagata Das > Priority: Minor > > The default log location is SPARK_HOME/work/ and this leads to disk space > running out pretty quickly. The spark-ec2 scripts should configure the > cluster to automatically set the logging directory to /mnt/spark-work/ or > something like that on the mounted disks. The SPARK_HOME/work may also be > symlinked to that directory to maintain the existing setup. -- This message was sent by Atlassian JIRA (v6.2#6252)