[ https://issues.apache.org/jira/browse/SPARK-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761941#comment-15761941 ]
Marcelo Vanzin commented on SPARK-18716: ---------------------------------------- For posterity, another problem with this feature that I didn't mention in the PR, is that allow users to use the SHS to delete content created by other users. A malicious user can just write a big event log file in the SHS directory and that would eventually make the SHS delete log files from other users. > Restrict the disk usage of spark event log. > -------------------------------------------- > > Key: SPARK-18716 > URL: https://issues.apache.org/jira/browse/SPARK-18716 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.0.2 > Reporter: Genmao Yu > > We've had reports of overfull disk usage of spark event log file. Current > implementation has following drawbacks: > 1. If we did not start Spark HistoryServer or Spark HistoryServer just > failed, there is no chance to do clean work. > 2. Spark HistoryServer is cleaning event log file based on file age only. If > there are abundant applications constantly, the disk usage in every > {{spark.history.fs.cleaner.maxAge}} can still be very large. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org