[jira] [Commented] (SPARK-18716) Restrict the disk usage of spark event log.
[
https://issues.apache.org/jira/browse/SPARK-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761941#comment-15761941
]
Marcelo Vanzin commented on SPARK-18716:
For posterity, another problem with this feature that I didn't mention in the
PR, is that allow users to use the SHS to delete content created by other
users. A malicious user can just write a big event log file in the SHS
directory and that would eventually make the SHS delete log files from other
users.
> Restrict the disk usage of spark event log.
>
>
> Key: SPARK-18716
> URL: https://issues.apache.org/jira/browse/SPARK-18716
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.0.2
>Reporter: Genmao Yu
>
> We've had reports of overfull disk usage of spark event log file. Current
> implementation has following drawbacks:
> 1. If we did not start Spark HistoryServer or Spark HistoryServer just
> failed, there is no chance to do clean work.
> 2. Spark HistoryServer is cleaning event log file based on file age only. If
> there are abundant applications constantly, the disk usage in every
> {{spark.history.fs.cleaner.maxAge}} can still be very large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18716) Restrict the disk usage of spark event log.
[
https://issues.apache.org/jira/browse/SPARK-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721782#comment-15721782
]
Apache Spark commented on SPARK-18716:
--
User 'uncleGen' has created a pull request for this issue:
https://github.com/apache/spark/pull/16142
> Restrict the disk usage of spark event log.
>
>
> Key: SPARK-18716
> URL: https://issues.apache.org/jira/browse/SPARK-18716
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.0.2
>Reporter: Genmao Yu
>
> We've had reports of overfull disk usage of spark event log file. Current
> implementation has following drawbacks:
> 1. If we did not start Spark HistoryServer or Spark HistoryServer just
> failed, there is no chance to do clean work.
> 2. Spark HistoryServer is cleaning event log file based on file age only. If
> there are abundant applications constantly, the disk usage in every
> {{spark.history.fs.cleaner.maxAge}} can still be very large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
