[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-89199687 [Test build #29649 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29649/consoleFull) for PR 2342 at commit [`d05f2c2`](https://github.com/apache/spark/commit/d05f2c2a5683b0204415c753b401bf7f4f8dccba). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-89199700 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29649/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-89249935 [Test build #29666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29666/consoleFull) for PR 2342 at commit [`9fb522e`](https://github.com/apache/spark/commit/9fb522ed02b5212e87c693943e85ff97634931d8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-89177222 [Test build #29649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29649/consoleFull) for PR 2342 at commit [`d05f2c2`](https://github.com/apache/spark/commit/d05f2c2a5683b0204415c753b401bf7f4f8dccba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88388611 I've updated the description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88388972 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29531/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88388968 [Test build #29531 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29531/consoleFull) for PR 2342 at commit [`082f709`](https://github.com/apache/spark/commit/082f70923f3d6342344b5548bd41ee5bbd3c). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88382376 [Test build #29531 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29531/consoleFull) for PR 2342 at commit [`082f709`](https://github.com/apache/spark/commit/082f70923f3d6342344b5548bd41ee5bbd3c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88409388 [Test build #29535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29535/consoleFull) for PR 2342 at commit [`a76e569`](https://github.com/apache/spark/commit/a76e569c0c922ba755456278051ec6a550120493). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88429222 [Test build #29534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29534/consoleFull) for PR 2342 at commit [`5ce1b21`](https://github.com/apache/spark/commit/5ce1b210fe380c7b0fde8b377a0958dce821fa88). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88429242 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29534/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88400722 [Test build #29534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29534/consoleFull) for PR 2342 at commit [`5ce1b21`](https://github.com/apache/spark/commit/5ce1b210fe380c7b0fde8b377a0958dce821fa88). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88436865 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29535/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88436860 [Test build #29535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29535/consoleFull) for PR 2342 at commit [`a76e569`](https://github.com/apache/spark/commit/a76e569c0c922ba755456278051ec6a550120493). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88709309 [Test build #29583 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29583/consoleFull) for PR 2342 at commit [`e85e9aa`](https://github.com/apache/spark/commit/e85e9aad17e987a5ce5bd32f8581e20af8d3ea0b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88733710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29583/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-88733696 [Test build #29583 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29583/consoleFull) for PR 2342 at commit [`e85e9aa`](https://github.com/apache/spark/commit/e85e9aad17e987a5ce5bd32f8581e20af8d3ea0b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-86337370 This is an image of Task Assignment Timeline view. It's almost done. This view is integrated into existing WebUI, zoomable and scrollable.  Now the code of this feature is a little bit messy but I'll cleanup and show the code soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-85132747 @sarutak it sounds like the plan is to significantly change the implementation; if that's the case, then yes, closing this PR and opening a new one when the new functionality is ready is the right strategy. FYI: there's been some effort towards implementing a much more restricted version of this that uses D3: https://issues.apache.org/jira/browse/SPARK-6418. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-85131296 I apologize for my late reply. I had no time until last weekend. Actually, I'm reconsidering what and how should be visualized and trying implementing. I could show you a concrete implementation in a few weeks. Should I close this PR for now and reopen later? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-85139282 @kayousterhout The basic idea is not changed but I try to use vis.js instead of D3.js because vis.js is easy to build rich timeline view. This is under development version of the new implementation. [Timeline view for an application]  [Timeline view for a stage]  Actually, I talked with Matei at New York last week and showed the implementation, then got some feedbacks. One of the feedbacks is that each square which meaning each task should show it's proportion of duration like you are suggesting at SPARK-6418. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-75155827 @sarutak @kayousterhout @vanzin what is the status of this PR? It is one of the largest out there at the moment and it seems to have mostly gone stale. I think others have had reservations about including d3 inside the Spark repository itself. I wonder if it makes it cleaner to implement all these super cool visualizations outside of the Spark project using the statuses exposed by something like #4435. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-59723719 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21912/consoleFull) for PR 2342 at commit [`35fb0f6`](https://github.com/apache/spark/commit/35fb0f67be7f2f7223e010eca300a7c1ad295c18). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-59723812 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21912/consoleFull) for PR 2342 at commit [`35fb0f6`](https://github.com/apache/spark/commit/35fb0f67be7f2f7223e010eca300a7c1ad295c18). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-59723813 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21912/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-59724241 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21914/consoleFull) for PR 2342 at commit [`afffb05`](https://github.com/apache/spark/commit/afffb05ba1b83564c875ac3bb2aad64339991587). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-59724343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21914/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-59724340 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21914/consoleFull) for PR 2342 at commit [`afffb05`](https://github.com/apache/spark/commit/afffb05ba1b83564c875ac3bb2aad64339991587). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-8998 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20324/consoleFull)** after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user ash211 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2342#discussion_r17521459
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala
---
@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
private[ui] class EnvironmentPage(parent: EnvironmentTab) extends
WebUIPage() {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+val jvmInfoJson =
+ (RUntime Informationf -
listener.jvmInformation.foldLeft(JObject())(_ ~ _))
--- End diff --
Typo: Runtime Information
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-3587 @ash211 Thanks, I've modified. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-3717 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20324/consoleFull) for PR 2342 at commit [`9581660`](https://github.com/apache/spark/commit/95816607b36d68e9a1502db5a5b28dec7e315d59). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-3741 @sarutak yes it represents task ID. The pie chart sounds cool! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-3850 Pie chart I'll make is like the 3rd capture I attached. I think that can be helpful which phase spends long time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-55485203 @kayousterhout Sorry for late reply. I have a question, the Y axis on the graph you showed represents task id right? And I'm planning that if you click the bar, the pie chart which represents the rate of the time for each event (like GC or compute) of whole spent time is appear. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-55248261 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20151/consoleFull) for PR 2342 at commit [`3146bba`](https://github.com/apache/spark/commit/3146bba39066f1155b1ad0ec8a1326c18e10ab30). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-55253964 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20151/consoleFull) for PR 2342 at commit [`3146bba`](https://github.com/apache/spark/commit/3146bba39066f1155b1ad0ec8a1326c18e10ab30). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-55078122 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20080/consoleFull) for PR 2342 at commit [`e660878`](https://github.com/apache/spark/commit/e6608787659d00370d066e83f714c4bb4e6c0d32). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-55156635 The screenshots look great; I had a crude version of something like this I used offline, so I'm glad someone took the time to create a polished version for the web UI. :-) Two questions: - I assume this also works fine when run inside the HistoryServer. - Where's the actual implementation? In the .js files? Those are unreadable... If all the code is indeed in the js files, perhaps it's time to start thinking about leaving the unminimized js files in the repo, and minimizing them as part of the build? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-55159335 @vanzin Thanks for your interest! I assume this also works fine when run inside the HistoryServer. Of course, I'll integrate this feature into existing WebUI / HistoryServer. Where's the actual implementation? In the .js files? Those are unreadable... Sorry, the captures I attached are just only prototype built by html/javascript for sharing images. So, for now, I didn't attach actual code. But, almost all of prototype are completed and I've made how to integrate this feature into existing UI, so I'll show you actual code soon. I'll use d3.js to draw those graphs. As you mentioned, it's worth considering the unminimized js should be left on the repo or minimized. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-55178122 Echoing what @vanzin said, this looks great and I have found a very similar tool to be useful in the past. One feature suggestion: rather than showing a single brown bar for each task, can you show the time breakdown? In an existing (but much more crude!) tool some folks in the AMPLab use, we've found this to a very helpful way of visually diagnosing bottlenecks:  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
GitHub user sarutak opened a pull request: https://github.com/apache/spark/pull/2342 [WIP][SPARK-3468] WebUI Timeline-View feature I sometimes trouble-shoot and analyse the cause of long time spending job. At the time, I find the stages which spends long time or fails, then I find the tasks which spends long time or fails, next I analyse the proportion of each phase in a task. Another case, I find executors which spends long time for running a task and analyse the details of a task. In such situation, I think it's helpful to visualize timeline view of stages / tasks / executors and visualize details of proportion of activity for each task. Now I'm developing prototypes like captures I attached. I'll integrate these viewer into WebUI. [Capture of prototype for Stages timeline view] Each brown bar represents submission time to finish time of a stage.  [Capture of prototype for Task timeline view] Each brown bar represents launch time to finish time of a task.  [Capture of prorotype for Task details view] This shows the proportion of activities in a task.  [Capture of prorotype for Executor timeline view] Each brown bar represents start time to finish time of a task which a executor executes.  You can merge this pull request into a Git repository by running: $ git pull https://github.com/sarutak/spark timeline-viewer-feature Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2342.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2342 commit 4569a8e3ff703c17d125a6a80fff7f6baeb9e1f1 Author: Kousuke Saruta saru...@oss.nttdata.co.jp Date: 2014-09-10T05:09:35Z Added d3.min.js Modified .rat-exclude for d3.min.js Modified UIUtils.scala for d3.min.js --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2342#issuecomment-55073832 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20080/consoleFull) for PR 2342 at commit [`e660878`](https://github.com/apache/spark/commit/e6608787659d00370d066e83f714c4bb4e6c0d32). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
