HeartSaVioR commented on a change in pull request #25670: [SPARK-28869][CORE]
Roll over event log files
URL: https://github.com/apache/spark/pull/25670#discussion_r334662511
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -179,6 +179,19 @@ package object config {
private[spark] val EVENT_LOG_CALLSITE_LONG_FORM =
ConfigBuilder("spark.eventLog.longForm.enabled").booleanConf.createWithDefault(false)
+ private[spark] val EVENT_LOG_ENABLE_ROLLING =
+ ConfigBuilder("spark.eventLog.logRolling.enabled")
+ .doc("Whether rolling over event log files is enabled. If set to true,
it cuts down " +
+ "each event log file to the configured size.")
+ .booleanConf
+ .createWithDefault(false)
+
+ private[spark] val EVENT_LOG_ROLLED_EVENT_LOG_MAX_FILE_SIZE =
+ ConfigBuilder("spark.eventLog.logRolling.maxFileSize")
+ .doc("The max size of event log file to be rolled over.")
+ .bytesConf(ByteUnit.BYTE)
+ .createWithDefaultString("128m")
Review comment:
Thanks for sharing info!
I'll comment about possible case of desiring to set max file size smaller:
assuming we go with the plan - we would want to set up max number of event log
files to retain, to address issue where storage quota is limited. (I believe I
saw this requirement - limited storage quota - earlier in somewhere but can't
remember.) In that case, "max file size" and "max number of event log files"
work together to forecast the size of event log dir for the app (snapshot files
should be considered so in reality it's going to be bigger than that), and to
control the value finely, we may want to have smaller max file size.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
