[GitHub] spark pull request #18673: [SPARK-21447][WEB UI] Spark history server fails ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18673 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18673: [SPARK-21447][WEB UI] Spark history server fails ...
Github user ericvandenbergfb commented on a diff in the pull request:
https://github.com/apache/spark/pull/18673#discussion_r128051065
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -107,6 +107,16 @@ private[spark] class ReplayListenerBus extends
SparkListenerBus with Logging {
}
}
} catch {
+ case eofe: EOFException =>
+// If the history event file is compressed and inprogress, the
compressor will throw an
+// EOFException if there is not enough to decompress a proper
frame. This indicates
+// we're at the end of the file so we treat similarly to the
JsonParseException case above.
+if (!maybeTruncated) {
+ throw eofe
+} else {
+ logWarning(s"Got EOFException from log file $sourceName" +
--- End diff --
Agree, the message doesn't add much value since this is expected behavior.
Will remove.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18673: [SPARK-21447][WEB UI] Spark history server fails ...
Github user ericvandenbergfb commented on a diff in the pull request:
https://github.com/apache/spark/pull/18673#discussion_r128050958
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -107,6 +107,16 @@ private[spark] class ReplayListenerBus extends
SparkListenerBus with Logging {
}
}
} catch {
+ case eofe: EOFException =>
+// If the history event file is compressed and inprogress, the
compressor will throw an
+// EOFException if there is not enough to decompress a proper
frame. This indicates
+// we're at the end of the file so we treat similarly to the
JsonParseException case above.
+if (!maybeTruncated) {
--- End diff --
Okay, nice short syntax.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18673: [SPARK-21447][WEB UI] Spark history server fails ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18673#discussion_r128041672
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -107,6 +107,16 @@ private[spark] class ReplayListenerBus extends
SparkListenerBus with Logging {
}
}
} catch {
+ case eofe: EOFException =>
+// If the history event file is compressed and inprogress, the
compressor will throw an
+// EOFException if there is not enough to decompress a proper
frame. This indicates
+// we're at the end of the file so we treat similarly to the
JsonParseException case above.
+if (!maybeTruncated) {
+ throw eofe
+} else {
+ logWarning(s"Got EOFException from log file $sourceName" +
--- End diff --
Since this is sort of expected for in progress apps, warning is a little
too much. The message is also misleading, since it's expected to not be
finished yet.
(I know it's the same message as the existing handling of `maybeTruncated`,
but the same comment applies there.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18673: [SPARK-21447][WEB UI] Spark history server fails ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18673#discussion_r128041077
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -107,6 +107,16 @@ private[spark] class ReplayListenerBus extends
SparkListenerBus with Logging {
}
}
} catch {
+ case eofe: EOFException =>
+// If the history event file is compressed and inprogress, the
compressor will throw an
+// EOFException if there is not enough to decompress a proper
frame. This indicates
+// we're at the end of the file so we treat similarly to the
JsonParseException case above.
+if (!maybeTruncated) {
--- End diff --
You can merge this with the case.
case _: EOFException if !maybeTruncated =>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18673: [SPARK-21447][WEB UI] Spark history server fails ...
GitHub user ericvandenbergfb opened a pull request: https://github.com/apache/spark/pull/18673 [SPARK-21447][WEB UI] Spark history server fails to render compressed inprogress history file in some cases. Add failure handling for EOFException that can be thrown during decompression of an inprogress spark history file, treat same as case where can't parse the last line. ## What changes were proposed in this pull request? Failure handling for case of EOFException thrown within the ReplayListenerBus.replay method to handle the case analogous to json parse fail case. This path can arise in compressed inprogress history files since an incomplete compression block could be read (not flushed by writer on a block boundary). See the stack trace of this occurrence in the jira ticket (https://issues.apache.org/jira/browse/SPARK-21447) ## How was this patch tested? Added a unit test that specifically targets validating the failure handling path appropriately when maybeTruncated is true and false. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ericvandenbergfb/spark fix_inprogress_compr_history_file Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18673.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18673 commit caeac288e2285d16c327a0df6abd6c9fccecdd50 Author: Eric Vandenberg Date: 2017-07-18T16:58:12Z [SPARK-21447][WEB UI] Spark history server fails to render compressed inprogress history file in some cases. Add failure handling for EOFException that can be thrown during decompression of an inprogress spark history file, treat same as case where can't parse the last line. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
