holdenk commented on code in PR #58700:
URL: https://github.com/apache/spark/pull/58700#discussion_r4042285017
##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -165,6 +165,19 @@ private[spark] object History {
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("1m")
+ val EVENT_LOG_MAX_LINE_LENGTH =
+ ConfigBuilder("spark.history.fs.eventLog.maxLineLength")
+ .doc("Maximum length of a single event log line during replay. Lines
longer than " +
+ "this are skipped with a warning instead of being read into memory,
bounding the " +
+ "memory replay can use when an event log is corrupt or unexpectedly
large. Setting " +
+ "this to 0 or a negative value disables the limit. " +
+ "Introduced in 4.3.0; also available in 3.5.10, 4.0.5, 4.1.4 and
4.2.1; and in " +
+ "all versions after 4.3.0.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .bytesConf(ByteUnit.BYTE)
+ .createWithDefaultString("512m")
Review Comment:
So I switched it to multiply by two so it matches bytes, also thinking about
this I think it's probably safe to backport with a 512mb default lime limit?
Like it is a behavior change on the history server true, but previously if we
encountered a 512mb line in the history server log files it's not like it would
have worked well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
