[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10585 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-178328745 @navis you can close this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-174177590 I'll take this over if there's no progress on this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-173171129 @navis what do you think about following up on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-171944298 I don't feel that strongly but if we're changing this, might as well do it right with interpolation and making the method `private` @navis --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-171443062 Meh, this isn't _that_ hard to read without interpolation, so I'd be okay with merging this if it's fine with @srowen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/10585#discussion_r49654742 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -293,6 +293,15 @@ class StatsReportListener extends SparkListener with Logging { taskInfoMetrics.clear() } + private[spark] def getStatusDetail(stageInfo: StageInfo): String = { --- End diff -- This could even be fully `private`, but not a huge deal. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-170486935 It's looking good though I still sort of think string interpolation would make this clearer. I'll leave open for a bit to see if anyone cares either way on that one --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-170421549 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-170421478 **[Test build #49078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49078/consoleFull)** for PR 10585 at commit [`7875a49`](https://github.com/apache/spark/commit/7875a49a234f8c9055991faccd5a5a9a7f470fa4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-170421550 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49078/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-170333516 I think this still needs the changes I've mentioned above, or at least, that's what I'd prefer to see before merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user navis commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-170411652 @srowen moved it to `StatsReportListener`. sorry for delay. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-170412311 **[Test build #49078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49078/consoleFull)** for PR 10585 at commit [`7875a49`](https://github.com/apache/spark/commit/7875a49a234f8c9055991faccd5a5a9a7f470fa4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/10585#discussion_r48936240 --- Diff: core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala --- @@ -61,6 +61,21 @@ class StageInfo( "running" } } + + private[spark] def getStatusDetail: String = { --- End diff -- Yeah, I mean I wonder if it's worth exposing this pretty specific logging string as an internal API method? you can just create this string in the one place it's used --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/10585#discussion_r48828840 --- Diff: core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala --- @@ -61,6 +61,21 @@ class StageInfo( "running" } } + + private[spark] def getStatusDetail: String = { --- End diff -- Is this used elsewhere or can it just go into `SparkListener`? I think this is confusing as one big format string. I'd make `val`s for the pieces that need some computation and return one big interpolated string. Also this is verbose: ``` foo match { Some(x) => f(x) case _ => "" } ``` compared to just `foo.map(x => f(x)).getOrElse("")` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-169206444 **[Test build #48803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48803/consoleFull)** for PR 10585 at commit [`696f884`](https://github.com/apache/spark/commit/696f884085e61bdc56775c1d65d96bec138a3994). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-169206545 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-169206547 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48803/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-169181495 **[Test build #48803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48803/consoleFull)** for PR 10585 at commit [`696f884`](https://github.com/apache/spark/commit/696f884085e61bdc56775c1d65d96bec138a3994). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user navis commented on a diff in the pull request: https://github.com/apache/spark/pull/10585#discussion_r48915612 --- Diff: core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala --- @@ -61,6 +61,21 @@ class StageInfo( "running" } } + + private[spark] def getStatusDetail: String = { --- End diff -- oh, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user navis commented on a diff in the pull request: https://github.com/apache/spark/pull/10585#discussion_r48916174 --- Diff: core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala --- @@ -61,6 +61,21 @@ class StageInfo( "running" } } + + private[spark] def getStatusDetail: String = { --- End diff -- And, I've searched spark codes but that's the sole case logging StageInfo. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
GitHub user navis opened a pull request: https://github.com/apache/spark/pull/10585 SPARK-12637 Print stage info of finished stages properly Currently it prints hashcode of stage info, which seemed not that useful. ``` INFO scheduler.StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@2eb47d79 ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/navis/spark SPARK-12637 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10585.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10585 commit 868003412e4803b72bd336edae6ec9f1c2de7fbd Author: navis.ryuDate: 2016-01-05T01:49:27Z SPARK-12637 Print stage info of finished stages properly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-168874823 **[Test build #48718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48718/consoleFull)** for PR 10585 at commit [`cdf7027`](https://github.com/apache/spark/commit/cdf7027c870e995b8afd313c8ba5163eabb21f53). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-168896886 **[Test build #48718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48718/consoleFull)** for PR 10585 at commit [`cdf7027`](https://github.com/apache/spark/commit/cdf7027c870e995b8afd313c8ba5163eabb21f53). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-168896989 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48718/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-168896986 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-168868540 **[Test build #48714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48714/consoleFull)** for PR 10585 at commit [`8680034`](https://github.com/apache/spark/commit/868003412e4803b72bd336edae6ec9f1c2de7fbd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-168870006 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-168870008 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48714/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12637 Print stage info of finished stage...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10585#issuecomment-168870004 **[Test build #48714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48714/consoleFull)** for PR 10585 at commit [`8680034`](https://github.com/apache/spark/commit/868003412e4803b72bd336edae6ec9f1c2de7fbd). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org