[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6672 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-121366555 LGTM, merging this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-121072012 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120569858 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120569842 [Test build #37072 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37072/console) for PR 6672 at commit [`df2c1d8`](https://github.com/apache/spark/commit/df2c1d8579e3643e646a1f37534cc4cfb2c7db54). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120562356 [Test build #37072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37072/consoleFull) for PR 6672 at commit [`df2c1d8`](https://github.com/apache/spark/commit/df2c1d8579e3643e646a1f37534cc4cfb2c7db54). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120562210 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120562203 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120562168 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120551937 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120551877 [Test build #37055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37055/console) for PR 6672 at commit [`df2c1d8`](https://github.com/apache/spark/commit/df2c1d8579e3643e646a1f37534cc4cfb2c7db54). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120534629 [Test build #37055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37055/consoleFull) for PR 6672 at commit [`df2c1d8`](https://github.com/apache/spark/commit/df2c1d8579e3643e646a1f37534cc4cfb2c7db54). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120533823 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120533766 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120533135 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120226827 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120221602 [Test build #37009 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37009/console) for PR 6672 at commit [`df2c1d8`](https://github.com/apache/spark/commit/df2c1d8579e3643e646a1f37534cc4cfb2c7db54). * This patch **fails some tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120221598 [Test build #37009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37009/consoleFull) for PR 6672 at commit [`df2c1d8`](https://github.com/apache/spark/commit/df2c1d8579e3643e646a1f37534cc4cfb2c7db54). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120221556 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120221608 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120221566 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120221446 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120218375 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120218297 [Test build #37004 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37004/console) for PR 6672 at commit [`df2c1d8`](https://github.com/apache/spark/commit/df2c1d8579e3643e646a1f37534cc4cfb2c7db54). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120216354 [Test build #37004 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37004/consoleFull) for PR 6672 at commit [`df2c1d8`](https://github.com/apache/spark/commit/df2c1d8579e3643e646a1f37534cc4cfb2c7db54). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120216326 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120216316 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120210380 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120209444 > Screenshot looks great! We can do it separately but maybe we should remove the RDD table if there are no RDD blocks cached. Updated it in this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120208960 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120208972 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120187570 > Everything else looks good, but I just realized that there are no UI tests! There should be tests added to StorageTabSuite, isnt it? I agree that we should do unit tests for UI stuffs. Let me see how to do some unit tests for `StorageTabSuite`. Actually I think many Spark UI tabs lack unit tests. Spark only has some `UISeleniumSuite`s to test UI. But `UISeleniumSuite` is actually an integration test rather than a unit test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120133954 Everything else looks good, but I just realized that there are no UI tests! There should be tests added to StorageTabSuite, isnt it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120117779 Screenshot looks great! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120071821 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120071524 [Test build #36934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36934/console) for PR 6672 at commit [`e29fb53`](https://github.com/apache/spark/commit/e29fb53c3a95f9a89df0d12e854b6bd695934e59). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120016532 [Test build #36934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36934/consoleFull) for PR 6672 at commit [`e29fb53`](https://github.com/apache/spark/commit/e29fb53c3a95f9a89df0d12e854b6bd695934e59). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120015605 New screenshot: https://cloud.githubusercontent.com/assets/1000778/8598561/e483bb78-268d-11e5-8558-313c9b797fd3.png";> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120015796 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-120015905 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r34233775
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+location: String,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long)
+
+/**
+ * The aggregated status of stream blocks in an executor
+ */
+private[spark] case class ExecutorStreamBlockStatus(
+executorId: String,
+location: String,
+blocks: Seq[BlockUIData]) {
+
+ def totalMemSize: Long = blocks.map(_.memSize).sum
+
+ def totalDiskSize: Long = blocks.map(_.diskSize).sum
+
+ def totalExternalBlockStoreSize: Long =
blocks.map(_.externalBlockStoreSize).sum
+
+ def numStreamBlocks: Int = blocks.size
+
+}
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers =
+new mutable.HashMap[BlockManagerId, mutable.HashMap[BlockId,
BlockUIData]]
+ /**
+ * The replication in StorageLevel may be out of date. E.g., when the
first block is added, the
+ * replication is 1. But when the second block with the same ID is
added, the replication should
+ * become 2. To avoid scanning "blockManagers" to modify the replication
number, we maintain
+ * "blockLocations" to get the replication quickly.
+ */
+ private val blockLocations = new mutable.HashMap[BlockId,
mutable.ArrayBuffer[String]]
+
+ override def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated):
Unit = {
+val blockId = blockUpdated.blockUpdatedInfo.blockId
+if (!blockId.isInstanceOf[StreamBlockId]) {
+ // Now we only monitor StreamBlocks
+ return
+}
+val blockManagerId = blockUpdated.blockUpdatedInfo.blockManagerId
+val storageLevel = blockUpdated.blockUpdatedInfo.storageLevel
+val memSize = blockUpdated.blockUpdatedInfo.memSize
+val diskSize = blockUpdated.blockUpdatedInfo.diskSize
+val externalBlockStoreSize =
blockUpdated.blockUpdatedInfo.externalBlockStoreSize
+
+synchronized {
+ // Drop the update info if the block manager is not registered
+ blockManagers.get(blockManagerId).foreach { blocksInBlockManager =>
+if (storageLevel.isValid) {
+ blocksInBlockManager.put(blockId,
+BlockUIData(
+ blockId,
+ blockManagerId.hostPort,
+ storageLevel,
+ memSize,
+ diskSize,
+ externalBlockStoreSize)
+ )
+ val locations = blockLocations.getOrElseUpdate(blockId, new
mutable.ArrayBuffer[String])
+ locations += blockManagerId.hostPort
+} else {
+ // If isValid is not true, it means we should drop the block.
+ blocksInBlockManager -= blockId
+ removeLocationFromBlockLocations(blockId,
blockManagerId.hostPort)
+}
+ }
+}
+ }
+
+ override def onBlockManagerAdded(blockManagerAdded:
SparkListenerBlockManagerAdded): Unit = {
+synchronized {
+ blockManagers.put(blockManagerAdded.blockManagerId,
mutable.HashMap())
+}
+ }
+
+ private def removeLocationFromBlockLocations(blockId: BlockId, location:
String): Unit = {
+synchronized {
+ blockLocations.get(blockId).foreach { locations =>
+locations -= location
+if (locations.isEmpty) {
+ blockLocations -= blockId
+}
+ }
+}
+ }
+
+ override def onBlockManagerRemoved(
+ blockManagerRemoved: SparkListenerBlockManagerR
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119251652 [Test build #36680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36680/console) for PR 6672 at commit [`ccbee07`](https://github.com/apache/spark/commit/ccbee079532bf3c9fadbf58522f08e2444c453fb). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119252068 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119211777 [Test build #36680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36680/consoleFull) for PR 6672 at commit [`ccbee07`](https://github.com/apache/spark/commit/ccbee079532bf3c9fadbf58522f08e2444c453fb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119211547 Updated as per your suggestion: https://cloud.githubusercontent.com/assets/1000778/8547735/5418e1b2-24f3-11e5-9d73-a119e2636ebd.png";> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119210026 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119210065 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119058176 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119058112 [Test build #36627 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36627/console) for PR 6672 at commit [`3de2762`](https://github.com/apache/spark/commit/3de27621844f0b30483f64969a9eda31e1a0af6c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119049255 I see that you have changed the data structures to suit the desired table structure. But since the table structure has to change, i guess the data structures have to change. So I will not take a look at that right now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/6672#discussion_r34002925 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.storage + +import scala.collection.mutable + +import org.apache.spark.scheduler._ + +private[spark] case class BlockUIData( +blockId: BlockId, +storageLevel: StorageLevel, +memSize: Long, +diskSize: Long, +externalBlockStoreSize: Long) + +/** + * The aggregated status of stream blocks in an executor + */ +private[spark] case class ExecutorStreamBlockStatus ( + executorId: String, --- End diff -- incorrect formatting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119048589 On the screenshot - "Aggregated Stream Block Metrics by Executor" --> "Aggregated Metrics by executor" (Stream block part is obvious) - I am not again categorizing the by Executor ID is a good idea. Rather I would like to see input blocks as the first column, sorted alphabetically. Then multiple rows per block ids as there are replicas. That would be easy to visually search through blocks of one input stream (order will show missing blocks). Also will be easy to see if any blocks has less replicas. I think the columns should be Block ID | Replication Level | Location | Storage Level | Size If replication level is 2, there will be two subrow with location + storage level + size each. The Storage Level will be either Memory, or Memory Serialized, or Disk, or External. Does that make sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119037574 [Test build #36627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36627/consoleFull) for PR 6672 at commit [`3de2762`](https://github.com/apache/spark/commit/3de27621844f0b30483f64969a9eda31e1a0af6c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119037008 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119037015 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119036726 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119035347 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119035328 [Test build #36610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36610/console) for PR 6672 at commit [`3de2762`](https://github.com/apache/spark/commit/3de27621844f0b30483f64969a9eda31e1a0af6c). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119017456 [Test build #36610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36610/consoleFull) for PR 6672 at commit [`3de2762`](https://github.com/apache/spark/commit/3de27621844f0b30483f64969a9eda31e1a0af6c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119016947 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119016925 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-119016286 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118969769 Jenkins test this again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118926993 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118926952 [Test build #36581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36581/console) for PR 6672 at commit [`3de2762`](https://github.com/apache/spark/commit/3de27621844f0b30483f64969a9eda31e1a0af6c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-11332 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118885885 [Test build #36581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36581/consoleFull) for PR 6672 at commit [`3de2762`](https://github.com/apache/spark/commit/3de27621844f0b30483f64969a9eda31e1a0af6c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118884923 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118884988 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118884371 New screenshot: https://cloud.githubusercontent.com/assets/1000778/8525454/8bcda46e-2433-11e5-86df-7d6e01b8faa6.png";> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118883543 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-118883560 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33822371
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long,
+locations: Set[String])
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers = new mutable.HashMap[BlockManagerId,
mutable.HashSet[BlockId]]
+ private val blocks = new mutable.HashMap[BlockId, BlockUIData]
+
+ override def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated):
Unit = {
+val blockId = blockUpdated.blockUpdatedInfo.blockId
+if (!blockId.isInstanceOf[StreamBlockId]) {
+ // Now we only monitor StreamBlocks
+ return
+}
+val blockManagerId = blockUpdated.blockUpdatedInfo.blockManagerId
+val storageLevel = blockUpdated.blockUpdatedInfo.storageLevel
+val memSize = blockUpdated.blockUpdatedInfo.memSize
+val diskSize = blockUpdated.blockUpdatedInfo.diskSize
+val externalBlockStoreSize =
blockUpdated.blockUpdatedInfo.externalBlockStoreSize
+
+synchronized {
+ // Drop the update info if the block manager is not registered
+ blockManagers.get(blockManagerId).foreach { blocksInBlockManager =>
+if (storageLevel.isValid) {
+ blocksInBlockManager.add(blockId)
+ val location = blockManagerId.hostPort
+ val newLocations =
+blocks.get(blockId).map(_.locations).getOrElse(Set.empty) +
location
+ val newStorageLevel = StorageLevel(
+useDisk = diskSize > 0,
+useMemory = memSize > 0,
+useOffHeap = externalBlockStoreSize > 0,
--- End diff --
Yikes! Yeah we cant break sorting like that. Hmmm .. we need to think from
scratch. Unlike RDDs, blocks can only be in one level per location. So showing
"size in memory" AND "size in disk" does not make sense. It made sense for RDDs
because some partitions may be in memory and some may be in disk. Let me thinks
whats a better organization.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33782787
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long,
+locations: Set[String])
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers = new mutable.HashMap[BlockManagerId,
mutable.HashSet[BlockId]]
--- End diff --
It would be helpful when a block manager is removed and we need to remove
the blocks on this block manager.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33782490
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long,
+locations: Set[String])
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers = new mutable.HashMap[BlockManagerId,
mutable.HashSet[BlockId]]
+ private val blocks = new mutable.HashMap[BlockId, BlockUIData]
+
+ override def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated):
Unit = {
+val blockId = blockUpdated.blockUpdatedInfo.blockId
+if (!blockId.isInstanceOf[StreamBlockId]) {
+ // Now we only monitor StreamBlocks
+ return
+}
+val blockManagerId = blockUpdated.blockUpdatedInfo.blockManagerId
+val storageLevel = blockUpdated.blockUpdatedInfo.storageLevel
+val memSize = blockUpdated.blockUpdatedInfo.memSize
+val diskSize = blockUpdated.blockUpdatedInfo.diskSize
+val externalBlockStoreSize =
blockUpdated.blockUpdatedInfo.externalBlockStoreSize
+
+synchronized {
+ // Drop the update info if the block manager is not registered
+ blockManagers.get(blockManagerId).foreach { blocksInBlockManager =>
+if (storageLevel.isValid) {
+ blocksInBlockManager.add(blockId)
+ val location = blockManagerId.hostPort
+ val newLocations =
+blocks.get(blockId).map(_.locations).getOrElse(Set.empty) +
location
+ val newStorageLevel = StorageLevel(
+useDisk = diskSize > 0,
+useMemory = memSize > 0,
+useOffHeap = externalBlockStoreSize > 0,
--- End diff --
How about displaying like this:
But it does not support sorting. If I click to sort some column, it will
become like this:
But I think the sorting feature is very important for this table, so I
don't want to disable it. People usually want to find the biggest block.
Thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33738905
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long,
+locations: Set[String])
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers = new mutable.HashMap[BlockManagerId,
mutable.HashSet[BlockId]]
+ private val blocks = new mutable.HashMap[BlockId, BlockUIData]
+
+ override def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated):
Unit = {
+val blockId = blockUpdated.blockUpdatedInfo.blockId
+if (!blockId.isInstanceOf[StreamBlockId]) {
+ // Now we only monitor StreamBlocks
+ return
+}
+val blockManagerId = blockUpdated.blockUpdatedInfo.blockManagerId
+val storageLevel = blockUpdated.blockUpdatedInfo.storageLevel
+val memSize = blockUpdated.blockUpdatedInfo.memSize
+val diskSize = blockUpdated.blockUpdatedInfo.diskSize
+val externalBlockStoreSize =
blockUpdated.blockUpdatedInfo.externalBlockStoreSize
+
+synchronized {
+ // Drop the update info if the block manager is not registered
+ blockManagers.get(blockManagerId).foreach { blocksInBlockManager =>
+if (storageLevel.isValid) {
+ blocksInBlockManager.add(blockId)
+ val location = blockManagerId.hostPort
+ val newLocations =
+blocks.get(blockId).map(_.locations).getOrElse(Set.empty) +
location
+ val newStorageLevel = StorageLevel(
+useDisk = diskSize > 0,
+useMemory = memSize > 0,
+useOffHeap = externalBlockStoreSize > 0,
--- End diff --
Actually, isnt this effectively overwriting the previous storage level?
What happens of a block was stored in memory is exec1 and then it gets an
update that the exec2 adds it in disk. What should be the final storage level
combining these two.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33738805
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long,
+locations: Set[String])
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers = new mutable.HashMap[BlockManagerId,
mutable.HashSet[BlockId]]
--- End diff --
Why do you need to actually keep track of the block managers and
corresponding blockIds in them? If all you need to provide to the UI is all
block information (def allBlocks) then what is the need to keep track of block
managers? You can just add or remove locations from the BlockUIData in the
`blocks` map. Isnt it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33738275
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long,
+locations: Set[String])
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers = new mutable.HashMap[BlockManagerId,
mutable.HashSet[BlockId]]
+ private val blocks = new mutable.HashMap[BlockId, BlockUIData]
+
+ override def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated):
Unit = {
+val blockId = blockUpdated.blockUpdatedInfo.blockId
+if (!blockId.isInstanceOf[StreamBlockId]) {
+ // Now we only monitor StreamBlocks
+ return
+}
+val blockManagerId = blockUpdated.blockUpdatedInfo.blockManagerId
+val storageLevel = blockUpdated.blockUpdatedInfo.storageLevel
+val memSize = blockUpdated.blockUpdatedInfo.memSize
+val diskSize = blockUpdated.blockUpdatedInfo.diskSize
+val externalBlockStoreSize =
blockUpdated.blockUpdatedInfo.externalBlockStoreSize
+
+synchronized {
+ // Drop the update info if the block manager is not registered
+ blockManagers.get(blockManagerId).foreach { blocksInBlockManager =>
+if (storageLevel.isValid) {
+ blocksInBlockManager.add(blockId)
+ val location = blockManagerId.hostPort
+ val newLocations =
+blocks.get(blockId).map(_.locations).getOrElse(Set.empty) +
location
+ val newStorageLevel = StorageLevel(
+useDisk = diskSize > 0,
+useMemory = memSize > 0,
+useOffHeap = externalBlockStoreSize > 0,
+deserialized = storageLevel.deserialized,
+replication = newLocations.size
+ )
+ blocks.put(blockId,
+BlockUIData(
+ blockId,
+ newStorageLevel,
+ memSize,
+ diskSize,
+ externalBlockStoreSize,
+ newLocations))
+} else {
+ // If isValid is not true, it means we should drop the block.
+ blocksInBlockManager -= blockId
+ removeBlockFromBlockManager(blockId, blockManagerId)
+}
+ }
+}
+ }
+
+ override def onBlockManagerAdded(blockManagerAdded:
SparkListenerBlockManagerAdded): Unit = {
+synchronized {
+ blockManagers.put(blockManagerAdded.blockManagerId,
mutable.HashSet())
+}
+ }
+
+ override def onBlockManagerRemoved(
+ blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit = {
+val blockManagerId = blockManagerRemoved.blockManagerId
+synchronized {
+ blockManagers.remove(blockManagerId).foreach { blockIds =>
+for (blockId <- blockIds) {
+ removeBlockFromBlockManager(blockId, blockManagerId)
+}
+ }
+}
+ }
+
+ private def removeBlockFromBlockManager(
+ blockId: BlockId, blockManagerId: BlockManagerId): Unit = {
+val location = blockManagerId.hostPort
+blocks.get(blockId) foreach { blockUIData =>
+ val newLocations = blockUIData.locations - location
+ if (newLocations.isEmpty) {
+// This block is removed from all block managers, so remove it
+blocks -= blockId
+ } else {
+val newStorageLevel = StorageLevel(
+ useDisk = blockUIData.diskSize > 0,
+ useMemory = blockUIData.memSize > 0,
+ useOffHeap = blockUIData
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33738209
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long,
+locations: Set[String])
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers = new mutable.HashMap[BlockManagerId,
mutable.HashSet[BlockId]]
+ private val blocks = new mutable.HashMap[BlockId, BlockUIData]
+
+ override def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated):
Unit = {
+val blockId = blockUpdated.blockUpdatedInfo.blockId
+if (!blockId.isInstanceOf[StreamBlockId]) {
+ // Now we only monitor StreamBlocks
+ return
+}
+val blockManagerId = blockUpdated.blockUpdatedInfo.blockManagerId
+val storageLevel = blockUpdated.blockUpdatedInfo.storageLevel
+val memSize = blockUpdated.blockUpdatedInfo.memSize
+val diskSize = blockUpdated.blockUpdatedInfo.diskSize
+val externalBlockStoreSize =
blockUpdated.blockUpdatedInfo.externalBlockStoreSize
+
+synchronized {
+ // Drop the update info if the block manager is not registered
+ blockManagers.get(blockManagerId).foreach { blocksInBlockManager =>
+if (storageLevel.isValid) {
+ blocksInBlockManager.add(blockId)
+ val location = blockManagerId.hostPort
+ val newLocations =
+blocks.get(blockId).map(_.locations).getOrElse(Set.empty) +
location
+ val newStorageLevel = StorageLevel(
--- End diff --
This set of lines seems duplicated in two methods. May be worth putting
into a method to dedup. Also, it will make this method look less verbose.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33738113
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import scala.collection.mutable
+
+import org.apache.spark.scheduler._
+
+private[spark] case class BlockUIData(
+blockId: BlockId,
+storageLevel: StorageLevel,
+memSize: Long,
+diskSize: Long,
+externalBlockStoreSize: Long,
+locations: Set[String])
+
+private[spark] class BlockStatusListener extends SparkListener {
+
+ private val blockManagers = new mutable.HashMap[BlockManagerId,
mutable.HashSet[BlockId]]
+ private val blocks = new mutable.HashMap[BlockId, BlockUIData]
+
+ override def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated):
Unit = {
+val blockId = blockUpdated.blockUpdatedInfo.blockId
+if (!blockId.isInstanceOf[StreamBlockId]) {
+ // Now we only monitor StreamBlocks
+ return
+}
+val blockManagerId = blockUpdated.blockUpdatedInfo.blockManagerId
+val storageLevel = blockUpdated.blockUpdatedInfo.storageLevel
+val memSize = blockUpdated.blockUpdatedInfo.memSize
+val diskSize = blockUpdated.blockUpdatedInfo.diskSize
+val externalBlockStoreSize =
blockUpdated.blockUpdatedInfo.externalBlockStoreSize
+
+synchronized {
+ // Drop the update info if the block manager is not registered
+ blockManagers.get(blockManagerId).foreach { blocksInBlockManager =>
+if (storageLevel.isValid) {
+ blocksInBlockManager.add(blockId)
+ val location = blockManagerId.hostPort
+ val newLocations =
+blocks.get(blockId).map(_.locations).getOrElse(Set.empty) +
location
+ val newStorageLevel = StorageLevel(
+useDisk = diskSize > 0,
+useMemory = memSize > 0,
+useOffHeap = externalBlockStoreSize > 0,
+deserialized = storageLevel.deserialized,
+replication = newLocations.size
+ )
+ blocks.put(blockId,
+BlockUIData(
+ blockId,
+ newStorageLevel,
+ memSize,
+ diskSize,
+ externalBlockStoreSize,
+ newLocations))
+} else {
+ // If isValid is not true, it means we should drop the block.
+ blocksInBlockManager -= blockId
+ removeBlockFromBlockManager(blockId, blockManagerId)
+}
+ }
+}
+ }
+
+ override def onBlockManagerAdded(blockManagerAdded:
SparkListenerBlockManagerAdded): Unit = {
+synchronized {
+ blockManagers.put(blockManagerAdded.blockManagerId,
mutable.HashSet())
+}
+ }
+
+ override def onBlockManagerRemoved(
+ blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit = {
+val blockManagerId = blockManagerRemoved.blockManagerId
+synchronized {
+ blockManagers.remove(blockManagerId).foreach { blockIds =>
+for (blockId <- blockIds) {
+ removeBlockFromBlockManager(blockId, blockManagerId)
+}
+ }
+}
+ }
+
+ private def removeBlockFromBlockManager(
+ blockId: BlockId, blockManagerId: BlockManagerId): Unit = {
--- End diff --
I think it is worth putting this method inside synchronized so that someone
in future does not accidentally call this method without synchronizing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/6672#discussion_r33737752 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockUpdatedInfo.scala --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.storage + +import org.apache.spark.annotation.DeveloperApi +import org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo + +/** + * :: DeveloperApi :: + * Stores information about a block status in a block manager. + */ +@DeveloperApi +case class BlockUpdatedInfo( +blockManagerId: BlockManagerId, +blockId: BlockId, +storageLevel: StorageLevel, +memSize: Long, +diskSize: Long, +externalBlockStoreSize: Long) + +@DeveloperApi --- End diff -- Does this need to be made public? The user never has to create a BlockUpdateInfo object. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117799361 @JoshRosen Any more thoughts on this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117626128 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117626020 [Test build #36243 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36243/console) for PR 6672 at commit [`ba5d0d1`](https://github.com/apache/spark/commit/ba5d0d16e15a1a436b49f6158095d7f15c5d87d8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117561310 I replaced the screenshot with a new one in the description of this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117561256 [Test build #36243 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36243/consoleFull) for PR 6672 at commit [`ba5d0d1`](https://github.com/apache/spark/commit/ba5d0d16e15a1a436b49f6158095d7f15c5d87d8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117560750 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117560724 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33662209
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockStatusListenerSuite.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.scheduler._
+
+class BlockStatusListenerSuite extends SparkFunSuite {
+
+ test("basic functions") {
+val blockManagerId = BlockManagerId("0", "localhost", 1)
+val listener = new BlockStatusListener()
+listener.onBlockManagerAdded(SparkListenerBlockManagerAdded(0,
blockManagerId, 0))
+listener.onBlockUpdated(SparkListenerBlockUpdated(
+ BlockUpdatedInfo(
+blockManagerId,
+StreamBlockId(0, 100),
+StorageLevel.MEMORY_AND_DISK,
+memSize = 100,
+diskSize = 100,
+externalBlockStoreSize = 0)))
+val blocks = listener.allBlocks
+assert(blocks.size === 1)
--- End diff --
Added more comments and rewrote the asserts for readability.
Breaking into small unit tests needs many duplicate codes to construct the
test cases. So I just added comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117540863 Other than these small comments, LGTM as is. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-117539747 Can you put a final screenshot of what it looks like. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33658277
--- Diff:
core/src/test/scala/org/apache/spark/storage/BlockStatusListenerSuite.scala ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.storage
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.scheduler._
+
+class BlockStatusListenerSuite extends SparkFunSuite {
+
+ test("basic functions") {
+val blockManagerId = BlockManagerId("0", "localhost", 1)
+val listener = new BlockStatusListener()
+listener.onBlockManagerAdded(SparkListenerBlockManagerAdded(0,
blockManagerId, 0))
+listener.onBlockUpdated(SparkListenerBlockUpdated(
+ BlockUpdatedInfo(
+blockManagerId,
+StreamBlockId(0, 100),
+StorageLevel.MEMORY_AND_DISK,
+memSize = 100,
+diskSize = 100,
+externalBlockStoreSize = 0)))
+val blocks = listener.allBlocks
+assert(blocks.size === 1)
--- End diff --
Can you add more comments on what each of these sets of asserts test? Since
it all inside "basic functions" its hard to understand what are actually being
tested. Actually this one unit test " basic functions" is fairly long. If it
doesnt affect efficiency much, I suggest breaking it into multiple unit tests.
TL;DR either add intermediate comments OR break it down into smaller parts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-115201345 [Test build #35769 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35769/console) for PR 6672 at commit [`4bbe341`](https://github.com/apache/spark/commit/4bbe341b10881a58e9a52ab22c86f70ca485b4b1). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-115201468 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-115153937 [Test build #35769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35769/consoleFull) for PR 6672 at commit [`4bbe341`](https://github.com/apache/spark/commit/4bbe341b10881a58e9a52ab22c86f70ca485b4b1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-115153794 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-115153760 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-115152867 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-115131501 [Test build #35754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35754/console) for PR 6672 at commit [`4bbe341`](https://github.com/apache/spark/commit/4bbe341b10881a58e9a52ab22c86f70ca485b4b1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerBlockUpdated(blockUpdatedInfo: BlockUpdatedInfo) extends SparkListenerEvent` * `case class BlockUpdatedInfo(` * `class StorageListener(storageStatusListener: StorageStatusListener) extends BlockStatusListener ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4072][Core]Display Streaming blocks in ...
Github user sujkh85 commented on the pull request: https://github.com/apache/spark/pull/6672#issuecomment-115131648 NAVER - http://www.naver.com/ su...@naver.com ëê» ë³´ë´ì ë©ì¼ ì´ ë¤ìê³¼ ê°ì ì´ì ë¡ ì ì¡ ì¤í¨íìµëë¤. ë°ë ì¬ëì´ íìëì ë©ì¼ì ìì ì°¨ë¨ íììµëë¤. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
