Github user mallman commented on the pull request:
https://github.com/apache/spark/pull/10700#issuecomment-172443532
I should also state that my original motivation in submitting this patch
was to address the confusing log messages
Application ... is still in progress, it may be terminated abnormally.
which I saw in the Spark master log for app's that actually terminated
normally.
Also, it's just come to mind that this bug may explain another behavior
I've seenâsometimes an app's event log is corrupted if it was configured to
be compressed. If the log is uncompressed then the ability for the history
reader to decode an "in progress" event log allows it to be processed as
expected. However, if the event log is being written through a compressed
output stream and is not properly flushed before it is processed, then the
processing may fail because the file compression was incomplete. (As this just
occurred to me I haven't tested this hypothesis, but it does sound like a
plausible explanation.) If this is the case, then this patch should correct the
problem with corrupt compressed event logs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
