[jira] [Commented] (SPARK-3562) Periodic cleanup event logs
[ https://issues.apache.org/jira/browse/SPARK-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139786#comment-14139786 ] Matthew Farrellee commented on SPARK-3562: -- is logrotate an option for you? > Periodic cleanup event logs > --- > > Key: SPARK-3562 > URL: https://issues.apache.org/jira/browse/SPARK-3562 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: xukun > > If we run spark application frequently, it will write many spark event log > into spark.eventLog.dir. After a long time later, there will be many spark > event log that we do not concern in the spark.eventLog.dir.Periodic cleanups > will ensure that logs older than this duration will be forgotten. It is no > need to clean logs by hands. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3562) Periodic cleanup event logs
[ https://issues.apache.org/jira/browse/SPARK-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14139910#comment-14139910 ] xukun commented on SPARK-3562: -- [~farrellee] logrotate is an option. Clean event logs by historyServer is another option like hadoop‘s jobhistoryserver。 > Periodic cleanup event logs > --- > > Key: SPARK-3562 > URL: https://issues.apache.org/jira/browse/SPARK-3562 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: xukun > > If we run spark application frequently, it will write many spark event log > into spark.eventLog.dir. After a long time later, there will be many spark > event log that we do not concern in the spark.eventLog.dir.Periodic cleanups > will ensure that logs older than this duration will be forgotten. It is no > need to clean logs by hands. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3562) Periodic cleanup event logs
[ https://issues.apache.org/jira/browse/SPARK-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141876#comment-14141876 ] Apache Spark commented on SPARK-3562: - User 'viper-kun' has created a pull request for this issue: https://github.com/apache/spark/pull/2471 > Periodic cleanup event logs > --- > > Key: SPARK-3562 > URL: https://issues.apache.org/jira/browse/SPARK-3562 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: xukun > > If we run spark application frequently, it will write many spark event log > into spark.eventLog.dir. After a long time later, there will be many spark > event log that we do not concern in the spark.eventLog.dir.Periodic cleanups > will ensure that logs older than this duration will be forgotten. It is no > need to clean logs by hands. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3562) Periodic cleanup event logs
[ https://issues.apache.org/jira/browse/SPARK-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292818#comment-14292818 ] Apache Spark commented on SPARK-3562: - User 'viper-kun' has created a pull request for this issue: https://github.com/apache/spark/pull/4214 > Periodic cleanup event logs > --- > > Key: SPARK-3562 > URL: https://issues.apache.org/jira/browse/SPARK-3562 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: xukun > > If we run spark application frequently, it will write many spark event log > into spark.eventLog.dir. After a long time later, there will be many spark > event log that we do not concern in the spark.eventLog.dir.Periodic cleanups > will ensure that logs older than this duration will be forgotten. It is no > need to clean logs by hands. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
