[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
GitHub user KaiXinXiaoLei opened a pull request: https://github.com/apache/spark/pull/7559 [SPARK-9209] Using executor allocation, a executor is removed but it exists in ExecutorsPage of the web ui I set "spark.dynamicAllocation.enabled = trueâ, and run a big job. After some minutes, in driver, a executor is asked to remove. Then it's removed successfully, and the process of this executor is not exist. But it exists in ExecutorPage of the web ui. The log in driver : 2015-07-17 11:48:14,543 | INFO | [sparkDriver-akka.actor.default-dispatcher-3] | Removing block manager BlockManagerId(264, 172.1.1.8, 23811) 2015-07-17 11:48:14,543 | INFO | [dag-scheduler-event-loop] | Removed 264 successfully in removeExecutor 2015-07-17 11:48:21,226 | INFO | [sparkDriver-akka.actor.default-dispatcher-3] | Registering block manager 172.1.1.8:23811 with 10.4 GB RAM, BlockManagerId(264, 172.1.1.8, 23811) 2015-07-17 11:48:21,228 | INFO | [sparkDriver-akka.actor.default-dispatcher-3] | Added broadcast_781_piece0 in memory on 172.1.1.8:23811 (size: 38.6 KB, free: 10.4 GB) 2015-07-17 11:48:35,277 | ERROR | [sparkDriver-akka.actor.default-dispatcher-16] | Lost executor 264 on datasight-195: remote Rpc client disassociated 2015-07-17 11:48:35,277 | WARN | [sparkDriver-akka.actor.default-dispatcher-4] | Association with remote system [akka.tcp://sparkExecutor@datasight-195:23929] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 2015-07-17 11:48:35,277 | INFO | [sparkDriver-akka.actor.default-dispatcher-16] | Re-queueing tasks for 264 from TaskSet 415.0 2015-07-17 11:48:35,804 | INFO | [SparkListenerBus] | Existing executor 264 has been removed (new total is 10) You can merge this pull request into a Git repository by running: $ git pull https://github.com/KaiXinXiaoLei/spark executorpageError Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7559.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7559 commit cc2b0a790fa64c67c647f5ff27f8050acee5c409 Author: KaiXinXiaoLei Date: 2015-07-21T06:16:08Z change file commit f1a20cbb488057e9368285ba6dfdc35647f1369c Author: KaiXinXiaoLei Date: 2015-07-21T06:23:24Z change file --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user KaiXinXiaoLei commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123183603 The log about this executor : 2015-07-17 11:48:20,762 | ERROR | [SIGTERM handler] | RECEIVED SIGNAL 15: SIGTERM 2015-07-17 11:48:20,993 | WARN | [driver-heartbeater] | Told to re-register on heartbeat 2015-07-17 11:48:20,993 | INFO | [driver-heartbeater] | BlockManager re-registering with master 2015-07-17 11:48:21,052 | INFO | [Thread-1] | Shutdown hook called 2015-07-17 11:48:21,224 | INFO | [Thread-1] | Shutdown hook called 2015-07-17 11:48:21,184 | INFO | [driver-heartbeater] | Trying to register BlockManager 2015-07-17 11:48:21,227 | INFO | [driver-heartbeater] | Registered BlockManager 2015-07-17 11:48:21,227 | INFO | [driver-heartbeater] | Reporting 16 blocks to the master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123183920 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123184251 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7559#discussion_r35078099
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
---
@@ -58,7 +58,11 @@ private[ui] class ExecutorsPage(
val diskUsed = storageStatusList.map(_.diskUsed).sum
val execInfo = for (statusId <- 0 until storageStatusList.size) yield
ExecutorsPage.getExecInfo(listener, statusId)
-val execInfoSorted = execInfo.sortBy(_.id)
+val execInfoSorted = execInfo.filter(x => {
--- End diff --
What is this doing? this is inefficient (creates regexes every time) and
has strange syntax. Boolean or is not invoked as a method; extra parens.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7559#discussion_r35078149
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -68,6 +72,7 @@ class ExecutorsListener(storageStatusListener:
StorageStatusListener) extends Sp
override def onExecutorRemoved(
executorRemoved: SparkListenerExecutorRemoved): Unit = synchronized {
val eid = executorRemoved.executorId
+executorIds = executorIds.filter(x => (x != eid))
--- End diff --
I can't make out how this relates to the problem you specified. Do you
mind backing up and explaining in the JIRA first? I would close this PR as it
does not look like the right approach to anything like what you're describing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123397022 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123399124 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123399173 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123399849 [Test build #37960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37960/consoleFull) for PR 7559 at commit [`f1a20cb`](https://github.com/apache/spark/commit/f1a20cbb488057e9368285ba6dfdc35647f1369c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123400328 [Test build #37960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37960/console) for PR 7559 at commit [`f1a20cb`](https://github.com/apache/spark/commit/f1a20cbb488057e9368285ba6dfdc35647f1369c). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-123400334 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-124371932 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-124372643 [Test build #38329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38329/consoleFull) for PR 7559 at commit [`0e973e6`](https://github.com/apache/spark/commit/0e973e6186a121ee567c3d555939ddb860376871). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-124372120 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-124448536 [Test build #38329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38329/console) for PR 7559 at commit [`0e973e6`](https://github.com/apache/spark/commit/0e973e6186a121ee567c3d555939ddb860376871). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-124448628 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user KaiXinXiaoLei commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-124780545 I think my change code is not the reason of this failed test , And in my machine, this test passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-124814662 @KaiXinXiaoLei I think there are still some issues with your code, mostly that you're not explaining the change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user KaiXinXiaoLei commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-125069024 @srowen I am sorry not to make clear. My problem is : a executor is removed successfully but it exists in ExecutorsPage of the web ui. From code, i find the executor id in ExecutorsPage is got from it's blockManagerId: val status = listener.storageStatusList(statusId) val execId = status.blockManagerId.executorId val hostPort = status.blockManagerId.hostPort So i suggest, add a variable to save active executor id, then reading this value of variable to show active executor in ExecutorsPage, For example, i add a variable "executorIds" in ExecutorsTab.scala to save active executor id, and in ExecutorsPage.scala, filter active executor id through "executorIds" . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user KaiXinXiaoLei commented on a diff in the pull request:
https://github.com/apache/spark/pull/7559#discussion_r35503997
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
---
@@ -58,7 +58,11 @@ private[ui] class ExecutorsPage(
val diskUsed = storageStatusList.map(_.diskUsed).sum
val execInfo = for (statusId <- 0 until storageStatusList.size) yield
ExecutorsPage.getExecInfo(listener, statusId)
-val execInfoSorted = execInfo.sortBy(_.id)
+val execInfoSorted = execInfo.filter(x => {
--- End diff --
@srowen
val execInfoSorted = execInfo.filter(x => {
val numPattern = "[0-9]+".r
val noIntId = numPattern.findFirstIn(x.id).isEmpty
listener.getExecutorIds.contains(x.id).||(noIntId)
}).sortBy(_.id)
I intent to filter active executor id. In Executorspage, the type of
executor id is digital or character, eg
So i use regexes .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-125106918 I don't think it makes sense to add yet another piece of bookkeeping to patch over another problem. Update the existing data structures in the listener class. In any event I'm saying this change isn't OK from a style perspective. ... but I'm still not clear from your picture whether the executor is shown in an active state or not? is it not shown to be finished? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user KaiXinXiaoLei commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-125112620 @srowen In my picture, i just want to say the type of executor id is digital or character, eg: the executor id 5 and "driver" in picture. Now, do you understand my problem? If my way to fix this problem is not OK, can you give me a idea to fix this problem? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-125115185 No, that's not what I'm referring to. It is hacky to filter executors based on numeric-or-not name but happens to work now. The problem is that most lines of code in this PR have a problem -- you have a `var` where a `val` would do, if you use collection mutation methods; there's an unnecessary getter; you're using methods like `:+` with a `.`, unnecessary parens, missing whitespace, etc. I think it's best to close this PR and start over, since I don't think the approach is right either. surely the problem, if there is one, is farther upstream? why does the listener have `StorageStatus` for removed executors? In the JIRA, can you propose any easy reproduction of this case? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-126997730 Do you mind closing this PR? I don't think we're making progress in discussing it and I don't think this is a correct change --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user KaiXinXiaoLei closed the pull request at: https://github.com/apache/spark/pull/7559 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9209] Using executor allocation, a exec...
Github user KaiXinXiaoLei commented on the pull request: https://github.com/apache/spark/pull/7559#issuecomment-127095058 ok. i close this. and will find the better way --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
