[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-69283439 ok, I'll make new PRs for those old branches 1.0, 1.1, and 1.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-69280965 Yes that would be great. It seems that not all of the changes in this PR are applicable there, however. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-69280380 @andrewor14 , I received an email of your comment about creating other PRs to fix this issue for other older branches, but not found on this page. I think you might have removed that comment, so do I still need to make new PRs or just ignore that message? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-69226898 Hey @liyezhang556520 this doesn't merge cleanly in older branches. Can you make new PRs against those branches? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3824 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-69226708 Ok LGTM I'm merging this into master thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-69137447 [Test build #25199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25199/consoleFull) for PR 3824 at commit [`3c24d98`](https://github.com/apache/spark/commit/3c24d981f7f28b3ce8f56e57c14b48458eb378b2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-69137451 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25199/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-69133658 [Test build #25199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25199/consoleFull) for PR 3824 at commit [`3c24d98`](https://github.com/apache/spark/commit/3c24d981f7f28b3ce8f56e57c14b48458eb378b2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22633623
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event
logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in
$eventLogFile."
-logWarning(msg)
+val exception = URLEncoder.encode(Utils.exceptionString(fnf),
"UTF-8")
+logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
-app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+app.desc.appUiUrl = notFoundBasePath +
s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
ok, I'll revert the change, thanks @andrewor14 , @JoshRosen , @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22626094
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -757,11 +758,12 @@ private[spark] class Master(
case fnf: FileNotFoundException =>
// Event logging is enabled for this application, but no event
logs are found
val title = s"Application history not found (${app.id})"
-var msg = s"No event logs found for application $appName in
$eventLogFile."
-logWarning(msg)
+var msg = s"No event logs found for application $appName in
${app.desc.eventLogDir}."
+val exception = URLEncoder.encode(Utils.exceptionString(fnf),
"UTF-8")
+logWarning(msg, fnf)
--- End diff --
as commented elsewhere, I don't think logging the stack trace here is
particularly useful
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22626035
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event
logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in
$eventLogFile."
-logWarning(msg)
+val exception = URLEncoder.encode(Utils.exceptionString(fnf),
"UTF-8")
+logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
-app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+app.desc.appUiUrl = notFoundBasePath +
s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
I think it's OK to not encode the exception here. This is a file not found
exception so it doesn't convey much extra information.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68622521 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25025/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68622517 [Test build #25025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25025/consoleFull) for PR 3824 at commit [`3c1ac2e`](https://github.com/apache/spark/commit/3c1ac2ef5ea608e00eb45085ee65ae950c21be1b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68620916 [Test build #25025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25025/consoleFull) for PR 3824 at commit [`3c1ac2e`](https://github.com/apache/spark/commit/3c1ac2ef5ea608e00eb45085ee65ae950c21be1b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22336254
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -719,26 +719,28 @@ private[spark] class Master(
def rebuildSparkUI(app: ApplicationInfo): Boolean = {
val appName = app.desc.name
val notFoundBasePath = HistoryServer.UI_PATH_PREFIX + "/not-found"
-val eventLogFile = app.desc.eventLogDir
- .map { dir => EventLoggingListener.getLogPath(dir, app.id) }
- .getOrElse {
-// Event logging is not enabled for this application
-app.desc.appUiUrl = notFoundBasePath
-return false
-}
-val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf)
+var eventLogFile: String = null
--- End diff --
Hi @JoshRosen , `eventLogFile` is also used in `catch` block in this file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68287342 This change seems okay to me overall, aside from one minor nit. Most of the change is just broadening the scope of the `try` block to handle some cases that didn't seem like they could fail. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22323517
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -719,26 +719,28 @@ private[spark] class Master(
def rebuildSparkUI(app: ApplicationInfo): Boolean = {
val appName = app.desc.name
val notFoundBasePath = HistoryServer.UI_PATH_PREFIX + "/not-found"
-val eventLogFile = app.desc.eventLogDir
- .map { dir => EventLoggingListener.getLogPath(dir, app.id) }
- .getOrElse {
-// Event logging is not enabled for this application
-app.desc.appUiUrl = notFoundBasePath
-return false
-}
-val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf)
+var eventLogFile: String = null
--- End diff --
It looks like `eventLogFile` is only read from inside the `try` block on
the following line, so why not move it inside and make it into a `val` instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22323425
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event
logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in
$eventLogFile."
-logWarning(msg)
+val exception = URLEncoder.encode(Utils.exceptionString(fnf),
"UTF-8")
+logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
-app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+app.desc.appUiUrl = notFoundBasePath +
s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
@srowen It looks like this same `exception` URL param is used in other
exception-handling code in this same file (the first instance was added by
@andrewor14 in 6afca2d1079bac6309a595b8e0ffc74ae93fa662).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68248623 [Test build #24861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24861/consoleFull) for PR 3824 at commit [`a49c52f`](https://github.com/apache/spark/commit/a49c52fc995c7ac110d0ab07a4da2f87cf74de2d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68248630 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24861/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68244526 [Test build #24858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24858/consoleFull) for PR 3824 at commit [`12eee85`](https://github.com/apache/spark/commit/12eee8590fb9899c267b29d3a129a169b6cf6ec1). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68244528 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24858/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68243544 [Test build #24861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24861/consoleFull) for PR 3824 at commit [`a49c52f`](https://github.com/apache/spark/commit/a49c52fc995c7ac110d0ab07a4da2f87cf74de2d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22306172
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event
logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in
$eventLogFile."
-logWarning(msg)
+val exception = URLEncoder.encode(Utils.exceptionString(fnf),
"UTF-8")
+logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
-app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+app.desc.appUiUrl = notFoundBasePath +
s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
No
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22306099
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event
logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in
$eventLogFile."
-logWarning(msg)
+val exception = URLEncoder.encode(Utils.exceptionString(fnf),
"UTF-8")
+logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
-app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+app.desc.appUiUrl = notFoundBasePath +
s"?msg=$msg&exception=$exception&title=$title"
false
case e: Exception =>
-// Relay exception message to application UI page
+// Replay exception message to application UI page
--- End diff --
Yes, you are write, relay is correct, replay is not correct, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22306063
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event
logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in
$eventLogFile."
-logWarning(msg)
+val exception = URLEncoder.encode(Utils.exceptionString(fnf),
"UTF-8")
+logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
-app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+app.desc.appUiUrl = notFoundBasePath +
s"?msg=$msg&exception=$exception&title=$title"
--- End diff --
This will likely be too long in general to put in a URL. Did you add this
URL param elsewhere?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22306027
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -758,13 +760,14 @@ private[spark] class Master(
// Event logging is enabled for this application, but no event
logs are found
val title = s"Application history not found (${app.id})"
var msg = s"No event logs found for application $appName in
$eventLogFile."
-logWarning(msg)
+val exception = URLEncoder.encode(Utils.exceptionString(fnf),
"UTF-8")
+logWarning(msg, fnf)
msg += " Did you specify the correct logging directory?"
msg = URLEncoder.encode(msg, "UTF-8")
-app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
+app.desc.appUiUrl = notFoundBasePath +
s"?msg=$msg&exception=$exception&title=$title"
false
case e: Exception =>
-// Relay exception message to application UI page
+// Replay exception message to application UI page
--- End diff --
The word `Relay` was correct here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3824#issuecomment-68239849 [Test build #24858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24858/consoleFull) for PR 3824 at commit [`12eee85`](https://github.com/apache/spark/commit/12eee8590fb9899c267b29d3a129a169b6cf6ec1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
GitHub user liyezhang556520 opened a pull request: https://github.com/apache/spark/pull/3824 [SPARK-4989][CORE] avoid wrong eventlog conf cause cluster down in standalone mode when enabling eventlog in standalone mode, if give the wrong configuration, the standalone cluster will down (cause master restart, lose connection with workers). How to reproduce: just give an invalid value to "spark.eventLog.dir", for example: spark.eventLog.dir=hdfs://tmp/logdir1, hdfs://tmp/logdir2. This will throw illegalArgumentException, which will cause the Master restart. And the whole cluster is not available. You can merge this pull request into a Git repository by running: $ git pull https://github.com/liyezhang556520/spark wrongConf4Cluster Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3824.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3824 commit 5c1fa33799bc503ac1e2d5e9838e8e364bf1f61f Author: Zhang, Liye Date: 2014-12-26T08:23:53Z cache exceptions when eventlog with wrong conf commit 12eee8590fb9899c267b29d3a129a169b6cf6ec1 Author: Zhang, Liye Date: 2014-12-26T08:49:04Z add more message in log and on webUI --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
