[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/7082#discussion_r33614524
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -572,55 +572,55 @@ private[ui] class StagePage(parent: StagesTab)
extends WebUIPage("stage") {
val attempt = taskInfo.attempt
val timelineObject =
s"""
- {
- 'className': 'task task-assignment-timeline-object',
- 'group': '$executorId',
- 'content': '' +
- 'Status: ${taskInfo.status}' +
- 'Launch Time: ${UIUtils.formatDate(new
Date(launchTime))}' +
- '${
+ |{
+ |'className': 'task task-assignment-timeline-object',
+ |'group': '$executorId',
+ |'content': '
+ |Status: ${taskInfo.status}
+ |Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
+ |${
if (!taskInfo.running) {
s"""Finish Time: ${UIUtils.formatDate(new
Date(finishTime))}"""
} else {
""
}
- }' +
- 'Scheduler Delay: $schedulerDelay ms' +
- 'Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}' +
- 'Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}' +
- 'Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}' +
- 'Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}' +
- 'Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}' +
- 'Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '',
- 'start': new Date($launchTime),
- 'end': new Date($finishTime)
- }
- """
+ }
+ |Scheduler Delay: $schedulerDelay ms
+ |Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}
+ |Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}
+ |Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}
+ |Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}
+ |Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}
+ |Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">
+ |
+ |
+ |
+ |
+ |
+ |
+ |
+ |',
+ |'start': new Date($launchTime),
+ |'end': new Date($finishTime)
+ |}
+ |""".stripMargin.replaceAll("\n", " ")
--- End diff --
Thanks I opened a new PR #7133 for the issue you mentioned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user saurfang commented on a diff in the pull request:
https://github.com/apache/spark/pull/7082#discussion_r33610564
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -572,55 +572,55 @@ private[ui] class StagePage(parent: StagesTab)
extends WebUIPage("stage") {
val attempt = taskInfo.attempt
val timelineObject =
s"""
- {
- 'className': 'task task-assignment-timeline-object',
- 'group': '$executorId',
- 'content': '' +
- 'Status: ${taskInfo.status}' +
- 'Launch Time: ${UIUtils.formatDate(new
Date(launchTime))}' +
- '${
+ |{
+ |'className': 'task task-assignment-timeline-object',
+ |'group': '$executorId',
+ |'content': '
+ |Status: ${taskInfo.status}
+ |Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
+ |${
if (!taskInfo.running) {
s"""Finish Time: ${UIUtils.formatDate(new
Date(finishTime))}"""
} else {
""
}
- }' +
- 'Scheduler Delay: $schedulerDelay ms' +
- 'Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}' +
- 'Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}' +
- 'Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}' +
- 'Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}' +
- 'Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}' +
- 'Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '',
- 'start': new Date($launchTime),
- 'end': new Date($finishTime)
- }
- """
+ }
+ |Scheduler Delay: $schedulerDelay ms
+ |Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}
+ |Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}
+ |Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}
+ |Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}
+ |Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}
+ |Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">
+ |
+ |
+ |
+ |
+ |
+ |
+ |
+ |',
+ |'start': new Date($launchTime),
+ |'end': new Date($finishTime)
+ |}
+ |""".stripMargin.replaceAll("\n", " ")
--- End diff --
Right. It complained ILLEGAL character and that string is still shown as
multiline in developer console. I think the `line.separator` wouldn't work at
least for my case where I'm compiling on Windows but deploying on Linux. Don't
ask me why :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/7082#discussion_r33609917
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -572,55 +572,55 @@ private[ui] class StagePage(parent: StagesTab)
extends WebUIPage("stage") {
val attempt = taskInfo.attempt
val timelineObject =
s"""
- {
- 'className': 'task task-assignment-timeline-object',
- 'group': '$executorId',
- 'content': '' +
- 'Status: ${taskInfo.status}' +
- 'Launch Time: ${UIUtils.formatDate(new
Date(launchTime))}' +
- '${
+ |{
+ |'className': 'task task-assignment-timeline-object',
+ |'group': '$executorId',
+ |'content': '
+ |Status: ${taskInfo.status}
+ |Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
+ |${
if (!taskInfo.running) {
s"""Finish Time: ${UIUtils.formatDate(new
Date(finishTime))}"""
} else {
""
}
- }' +
- 'Scheduler Delay: $schedulerDelay ms' +
- 'Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}' +
- 'Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}' +
- 'Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}' +
- 'Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}' +
- 'Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}' +
- 'Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '',
- 'start': new Date($launchTime),
- 'end': new Date($finishTime)
- }
- """
+ }
+ |Scheduler Delay: $schedulerDelay ms
+ |Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}
+ |Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}
+ |Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}
+ |Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}
+ |Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}
+ |Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">
+ |
+ |
+ |
+ |
+ |
+ |
+ |
+ |',
+ |'start': new Date($launchTime),
+ |'end': new Date($finishTime)
+ |}
+ |""".stripMargin.replaceAll("\n", " ")
--- End diff --
I see. Did you see it does not work? I think, it's better to replace it
with a system property `line.separator`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user saurfang commented on a diff in the pull request:
https://github.com/apache/spark/pull/7082#discussion_r33608396
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -572,55 +572,55 @@ private[ui] class StagePage(parent: StagesTab)
extends WebUIPage("stage") {
val attempt = taskInfo.attempt
val timelineObject =
s"""
- {
- 'className': 'task task-assignment-timeline-object',
- 'group': '$executorId',
- 'content': '' +
- 'Status: ${taskInfo.status}' +
- 'Launch Time: ${UIUtils.formatDate(new
Date(launchTime))}' +
- '${
+ |{
+ |'className': 'task task-assignment-timeline-object',
+ |'group': '$executorId',
+ |'content': '
+ |Status: ${taskInfo.status}
+ |Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
+ |${
if (!taskInfo.running) {
s"""Finish Time: ${UIUtils.formatDate(new
Date(finishTime))}"""
} else {
""
}
- }' +
- 'Scheduler Delay: $schedulerDelay ms' +
- 'Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}' +
- 'Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}' +
- 'Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}' +
- 'Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}' +
- 'Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}' +
- 'Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '',
- 'start': new Date($launchTime),
- 'end': new Date($finishTime)
- }
- """
+ }
+ |Scheduler Delay: $schedulerDelay ms
+ |Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}
+ |Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}
+ |Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}
+ |Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}
+ |Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}
+ |Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">
+ |
+ |
+ |
+ |
+ |
+ |
+ |
+ |',
+ |'start': new Date($launchTime),
+ |'end': new Date($finishTime)
+ |}
+ |""".stripMargin.replaceAll("\n", " ")
--- End diff --
The `\n` isn't working on Windows if Git decides to check out the file
using `\r\n` line-endings. May want to change it to `[\r\n]` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7082 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116737751 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116737691 [Test build #35985 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35985/console) for PR 7082 at commit [`b29231d`](https://github.com/apache/spark/commit/b29231d5fd1b5e1ff8bcc68d8dd8706dda99ee58). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116682495 > Oh, I see. could you file another one as a separate issue? Opened a JIRA for this issue: https://issues.apache.org/jira/browse/SPARK-8705 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116669695 LGTM. After test "#35985" pass, I'll merge this into `master`. Thanks @zsxwing for your contribution! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116659288 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116659244 [Test build #35980 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35980/console) for PR 7082 at commit [`b29231d`](https://github.com/apache/spark/commit/b29231d5fd1b5e1ff8bcc68d8dd8706dda99ee58). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/7082#discussion_r33462776
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -572,55 +572,55 @@ private[ui] class StagePage(parent: StagesTab)
extends WebUIPage("stage") {
val attempt = taskInfo.attempt
val timelineObject =
s"""
- {
- 'className': 'task task-assignment-timeline-object',
- 'group': '$executorId',
- 'content': '' +
- 'Status: ${taskInfo.status}' +
- 'Launch Time: ${UIUtils.formatDate(new
Date(launchTime))}' +
- '${
+ |{
+ |'className': 'task task-assignment-timeline-object',
+ |'group': '$executorId',
+ |'content': '
+ |Status: ${taskInfo.status}
+ |Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
+ |${
if (!taskInfo.running) {
s"""Finish Time: ${UIUtils.formatDate(new
Date(finishTime))}"""
} else {
""
}
- }' +
- 'Scheduler Delay: $schedulerDelay ms' +
- 'Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}' +
- 'Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}' +
- 'Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}' +
- 'Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}' +
- 'Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}' +
- 'Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '',
- 'start': new Date($launchTime),
- 'end': new Date($finishTime)
- }
- """
+ }
+ |Scheduler Delay: $schedulerDelay ms
+ |Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}
+ |Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}
+ |Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}
+ |Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}
+ |Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}
+ |Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">
+ |
+ |
+ |
+ |
+ |
+ |
+ |
+ |',
+ |'start': new Date($launchTime),
+ |'end': new Date($finishTime)
+ |}
+ |""".stripMargin.replaceAll("\n", " ")
--- End diff --
I misunderstood the effect of "|" and stripMargin. Yes it's needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116658293 [Test build #35985 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35985/consoleFull) for PR 7082 at commit [`b29231d`](https://github.com/apache/spark/commit/b29231d5fd1b5e1ff8bcc68d8dd8706dda99ee58). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116657944 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116657924 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116657416 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/7082#discussion_r33462112
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -572,55 +572,55 @@ private[ui] class StagePage(parent: StagesTab)
extends WebUIPage("stage") {
val attempt = taskInfo.attempt
val timelineObject =
s"""
- {
- 'className': 'task task-assignment-timeline-object',
- 'group': '$executorId',
- 'content': '' +
- 'Status: ${taskInfo.status}' +
- 'Launch Time: ${UIUtils.formatDate(new
Date(launchTime))}' +
- '${
+ |{
+ |'className': 'task task-assignment-timeline-object',
+ |'group': '$executorId',
+ |'content': '
+ |Status: ${taskInfo.status}
+ |Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
+ |${
if (!taskInfo.running) {
s"""Finish Time: ${UIUtils.formatDate(new
Date(finishTime))}"""
} else {
""
}
- }' +
- 'Scheduler Delay: $schedulerDelay ms' +
- 'Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}' +
- 'Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}' +
- 'Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}' +
- 'Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}' +
- 'Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}' +
- 'Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '',
- 'start': new Date($launchTime),
- 'end': new Date($finishTime)
- }
- """
+ }
+ |Scheduler Delay: $schedulerDelay ms
+ |Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}
+ |Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}
+ |Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}
+ |Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}
+ |Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}
+ |Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">
+ |
+ |
+ |
+ |
+ |
+ |
+ |
+ |',
+ |'start': new Date($launchTime),
+ |'end': new Date($finishTime)
+ |}
+ |""".stripMargin.replaceAll("\n", " ")
--- End diff --
It's necessary. `stripMargin` removes the prefix
"|".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116656454 [Test build #979 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/979/console) for PR 7082 at commit [`b29231d`](https://github.com/apache/spark/commit/b29231d5fd1b5e1ff8bcc68d8dd8706dda99ee58). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/7082#discussion_r33457314
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -572,55 +572,55 @@ private[ui] class StagePage(parent: StagesTab)
extends WebUIPage("stage") {
val attempt = taskInfo.attempt
val timelineObject =
s"""
- {
- 'className': 'task task-assignment-timeline-object',
- 'group': '$executorId',
- 'content': '' +
- 'Status: ${taskInfo.status}' +
- 'Launch Time: ${UIUtils.formatDate(new
Date(launchTime))}' +
- '${
+ |{
+ |'className': 'task task-assignment-timeline-object',
+ |'group': '$executorId',
+ |'content': '
+ |Status: ${taskInfo.status}
+ |Launch Time: ${UIUtils.formatDate(new Date(launchTime))}
+ |${
if (!taskInfo.running) {
s"""Finish Time: ${UIUtils.formatDate(new
Date(finishTime))}"""
} else {
""
}
- }' +
- 'Scheduler Delay: $schedulerDelay ms' +
- 'Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}' +
- 'Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}' +
- 'Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}' +
- 'Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}' +
- 'Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}' +
- 'Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '' +
- '',
- 'start': new Date($launchTime),
- 'end': new Date($finishTime)
- }
- """
+ }
+ |Scheduler Delay: $schedulerDelay ms
+ |Task Deserialization Time:
${UIUtils.formatDuration(deserializationTime)}
+ |Shuffle Read Time:
${UIUtils.formatDuration(shuffleReadTime)}
+ |Executor Computing Time:
${UIUtils.formatDuration(executorComputingTime)}
+ |Shuffle Write Time:
${UIUtils.formatDuration(shuffleWriteTime)}
+ |Result Serialization Time:
${UIUtils.formatDuration(serializationTime)}
+ |Getting Result Time:
${UIUtils.formatDuration(gettingResultTime)}">
+ |
+ |
+ |
+ |
+ |
+ |
+ |
+ |',
+ |'start': new Date($launchTime),
+ |'end': new Date($finishTime)
+ |}
+ |""".stripMargin.replaceAll("\n", " ")
--- End diff --
We don't need `stripMargin` right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116630589 Oh, I see. could you file another one as a separate issue? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116621564 [Test build #979 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/979/consoleFull) for PR 7082 at commit [`b29231d`](https://github.com/apache/spark/commit/b29231d5fd1b5e1ff8bcc68d8dd8706dda99ee58). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116612731 @sarutak I also find another issue in the console. Because `System.currentTimeMillis()` is not accurate for tasks that only need several milliseconds, sometimes `totalExecutionTime` in `makeTimeline` will be 0. If `totalExecutionTime` is 0, there will the following error in the console.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116609513 [Test build #35980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35980/consoleFull) for PR 7082 at commit [`b29231d`](https://github.com/apache/spark/commit/b29231d5fd1b5e1ff8bcc68d8dd8706dda99ee58). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116608379 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/7082 [SPARK-8702][WebUI]Avoid massive concating strings in Javascript When there are massive tasks, such as `sc.parallelize(1 to 10, 1).count()`, the generated JS codes have a lot of string concatenations in the stage page, nearly 40 string concatenations for one task. We can generate the whole string for a task instead of execution string concatenations in the browser. Before this patch, the load time of the page is about 21 seconds.  After this patch, it reduces to about 17 seconds.  One disadvantage is that the generated JS codes become hard to read. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark js-string Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7082.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7082 commit b29231d5fd1b5e1ff8bcc68d8dd8706dda99ee58 Author: zsxwing Date: 2015-06-29T10:42:03Z Avoid massive concating strings in Javascript --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116608318 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8702][WebUI]Avoid massive concating str...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7082#issuecomment-116608173 /cc @sarutak --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
