[ https://issues.apache.org/jira/browse/SPARK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739114#comment-14739114 ]
Apache Spark commented on SPARK-9924: ------------------------------------- User 'tgravescs' has created a pull request for this issue: https://github.com/apache/spark/pull/8701 > checkForLogs and cleanLogs are scheduled at fixed rate and can get piled up > --------------------------------------------------------------------------- > > Key: SPARK-9924 > URL: https://issues.apache.org/jira/browse/SPARK-9924 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 1.4.0 > Reporter: Rohit Agarwal > Assignee: Rohit Agarwal > Fix For: 1.6.0 > > > {{checkForLogs}} and {{cleanLogs}} are scheduled using > {{ScheduledThreadPoolExecutor.scheduleAtFixedRate}}. When their execution > takes more time than the interval at which they are scheduled, they get piled > up. > This is a problem on its own but the existence of SPARK-7189 makes it even > worse. Let's say there is an eventLog which takes 15s to parse and which > happens to be the last modified file (that gets reloaded again and again due > to SPARK-7189.) If this file stays the last modified file for, let's say, an > hour, then a lot of executions of that file would have piled up as the > default {{spark.history.fs.update.interval}} is 10s. If there is a new > eventLog file now, it won't show up in the history server ui for a long time. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org