[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7110#issuecomment-124298551 @SaintBacchus not so far , any idea for this ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8374] [YARN] Job frequently hangs after...
Github user xuchenCN closed the pull request at: https://github.com/apache/spark/pull/7128 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8119] HeartbeatReceiver should replace ...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7107#issuecomment-122153483 +1 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4131][SQL] support writing data into th...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/4380#issuecomment-121565650 LGTM @scwf could you let the test PASSed ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8119] [WIP] HeartbeatReceiver should re...
Github user xuchenCN commented on a diff in the pull request:
https://github.com/apache/spark/pull/7107#discussion_r34648531
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -394,6 +394,27 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
}
/**
+ * Request that the cluster manager kill the specified executor without
adjusting the
+ * application resource requirements.
+ *
+ * The effect is that a new executor will be launched in place of the
one killed by
+ * this request. This assumes the cluster manager will automatically and
eventually
+ * fulfill all missing application resource requests.
+ *
+ * Return whether the request is received.
+ */
+ def killAndReplaceExecutor(executorId: String): Boolean = synchronized {
+logInfo(s"Requesting to replace executor $executorId")
+if (executorDataMap.contains(executorId)) {
+ executorsPendingToRemove += executorId
--- End diff --
@andrewor14 If CoarseGrainedSchedulerBackend.removeExecutor was happened
before HeartbeatReceiver.killExecutorThread
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8374] [YARN] Job frequently hangs after...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7128#issuecomment-121499587 @andrewor14 All right close this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7110#issuecomment-121137474 I follow the code maybe is lazy declare actorRef value ? and akka internal problem? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7110#issuecomment-121117560 @darkcrawler01 I mean this pull request will fix your issue --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7110#issuecomment-120891504 Hi @darkcrawler01 I see the log and this patch looks good to you --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7110#issuecomment-120793051 Hi @darkcrawler01 Could you provide detail log? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8374] [YARN] Job frequently hangs after...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7128#issuecomment-117213892 CC @sryza --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8374] [YARN] Job frequently hangs after...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7083#issuecomment-117213630 PR changes [https://github.com/apache/spark/pull/7128](https://github.com/apache/spark/pull/7128) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8374] [YARN] Job frequently hangs after...
GitHub user xuchenCN opened a pull request: https://github.com/apache/spark/pull/7128 [SPARK-8374] [YARN] Job frequently hangs after YARN preemption Issue description [SPARK-8374](https://issues.apache.org/jira/browse/SPARK-8374) Application starve because YARN scheduler preemption. This PR fix two issues 1. Application starve because CoarseGrainedSchedulerBackend.removeExecutor is happened before HeartbeatReceiver.killExecutorThread 2.SpakrContext.killExecutor less executors than before BUT HeartbeatReceiver just want to expire the executor and expect new one You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchenCN/spark SPARK-8374.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7128.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7128 commit 5c532948b125b6fd9a1e1ca48d1a7ebeb2b5f648 Author: xuchenCN Date: 2015-06-30T13:52:59Z [SPARK-8374] [YARN] Job frequently hangs after YARN preemption commit 316ac2e1e406e5e6401c29117f1ccbe302e64954 Author: xuchenCN Date: 2015-06-30T13:53:55Z [SPARK-8374] [YARN] Job frequently hangs after YARN preemption commit 5357e1363b6e3c192fc9ac64e8e9aca888618d9a Author: xuchenCN Date: 2015-06-30T13:54:21Z [SPARK-8374] [YARN] Job frequently hangs after YARN preemption commit 0238754389ec1830510f4dabc1b033b6364aefb5 Author: xuchenCN Date: 2015-06-30T13:54:36Z [SPARK-8374] [YARN] Job frequently hangs after YARN preemption --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8374] [YARN] Job frequently hangs after...
Github user xuchenCN closed the pull request at: https://github.com/apache/spark/pull/7083 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7110#issuecomment-117052889 Finally build success , thanks @JoshRosen care for this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7077#issuecomment-116934136 PR changes to [https://github.com/apache/spark/pull/7110](https://github.com/apache/spark/pull/7110) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
GitHub user xuchenCN opened a pull request: https://github.com/apache/spark/pull/7110 [SPARK-8592] [CORE] CoarseGrainedExecutorBackend: Cannot register with driver => NPE Look detail of this issue at [SPARK-8592](https://issues.apache.org/jira/browse/SPARK-8592) **CoarseGrainedExecutorBackend** should exit when **RegisterExecutor** failed You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchenCN/spark SPARK-8592 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7110.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7110 commit 71e0077d5c860d2d06b4265de9c21700be661d9b Author: xuchenCN Date: 2015-06-30T03:29:19Z [SPARK-8592] [CORE] CoarseGrainedExecutorBackend: Cannot register with driver => NPE --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7077#issuecomment-116932736 Have no idea for this test ,so I will pull request from master branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
Github user xuchenCN closed the pull request at: https://github.com/apache/spark/pull/7077 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8374] [YARN] Job frequently hangs after...
Github user xuchenCN commented on the pull request: https://github.com/apache/spark/pull/7083#issuecomment-116926849 Thanks @sryza I notice that is mistake, I 'm trying to find way to fix this issue --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-8374] [YARN] Job frequently hangs after ...
GitHub user xuchenCN opened a pull request: https://github.com/apache/spark/pull/7083 SPARK-8374] [YARN] Job frequently hangs after YARN preemption Issue description [SPARK-8374](https://issues.apache.org/jira/browse/SPARK-8374) Application starve because YARN scheduler preemption,we should to keep Running containers number Do not think about the resource ,that is Yarn scheduler's job . You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchenCN/spark SPARK-8374 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7083.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7083 commit f2667f274d1ee3d1c643f19221b9e83141f2137a Author: xuchenCN Date: 2015-06-29T10:53:04Z [SPARK-8374] [YARN] Job frequently hangs after YARN preemption commit 9555dd52c66efc5fa4b748ff9253309ac70b9c0a Author: xuchenCN Date: 2015-06-29T11:04:42Z [SPARK-8374] [YARN] Job frequently hangs after YARN preemption --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8592] [CORE] CoarseGrainedExecutorBacke...
GitHub user xuchenCN opened a pull request: https://github.com/apache/spark/pull/7077 [SPARK-8592] [CORE] CoarseGrainedExecutorBackend: Cannot register with driver => NPE Look detail of this issue at [SPARK-8592](https://issues.apache.org/jira/browse/SPARK-8592) **CoarseGrainedExecutorBackend** should exit when **RegisterExecutor** failed You can merge this pull request into a Git repository by running: $ git pull https://github.com/xuchenCN/spark SPARK-8592 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7077.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7077 commit 6dfdfbc0425985e9a512fb612be94eee0f743152 Author: xuchenCN Date: 2015-06-29T05:43:10Z [SPARK-8592] [CORE] CoarseGrainedExecutorBackend: Cannot register with driver => NPE --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
