Stephen Levett created SPARK-28594:
--------------------------------------
Summary: Allow event logs for running streaming apps to be rolled
over.
Key: SPARK-28594
URL: https://issues.apache.org/jira/browse/SPARK-28594
Project: Spark
Issue Type: Improvement
Components: Input/Output
Affects Versions: 2.4.0, 2.3.0, 2.2.1, 2.1.0
Environment: This has been reported on 2.0.2.22 but affects all
currently available versions.
Reporter: Stephen Levett
At all current Spark releases when event logging on spark streaming is enabled
the event logs grow massively. The files continue to grow until the
application is stopped or killed.
The Spark history server then has difficulty processing the files.
https://issues.apache.org/jira/browse/SPARK-8617
Addresses .inprogress files but not event log files that are still running.
Identify a mechanism to set a "max file" size so that the file is rolled over
when it reaches this size?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
