L. C. Hsieh created SPARK-60114:
-----------------------------------
Summary: Tolerate an incomplete UTF-8 sequence at the end of a
possibly truncated event log during replay
Key: SPARK-60114
URL: https://issues.apache.org/jira/browse/SPARK-60114
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
{{ReplayListenerBus.replay(InputStream, ...)}} decodes event logs with a strict
UTF-8 decoder. When {{maybeTruncated}} is true (an in-progress log), a partial
last line is meant to be tolerated: a {{JsonParseException}} on the last line
is logged as a warning instead of failing the replay. However, if the writer
has flushed only the first bytes of a multi-byte UTF-8 character at the end of
the file (e.g. CJK SQL text or job descriptions, which Jackson writes as raw
UTF-8), the decoder throws {{MalformedInputException}}. That is an
{{IOException}}, which {{replayEntries}} rethrows regardless of
{{maybeTruncated}}, so the replay fails instead of treating the incomplete last
line as truncated.
This affects both the main replay path (loading an in-progress application UI,
and listing with fast in-progress parsing) and the History Server end-event
reparse with {{spark.history.fs.inProgressOptimization.enabled=false}}. The
previous {{Source.getLines()}}-based reader had the same behavior, so this is
pre-existing.
An incomplete UTF-8 sequence at EOF should be handled like a truncated last
line when {{maybeTruncated}} is set, while malformed input elsewhere in the
file should still fail.
Raised in the review of https://github.com/apache/spark/pull/59327
(SPARK-60109).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
