Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-45833678
@tgravescs @kayousterhout The PR only work in standalone mode now. But it
provide a abstract method isReady() in SchedulerBackend.scala for all backend
implementations
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-45836171
@tgravescs @mridulm @kayousterhout I add a commit which submit stage after
configured number executor are registered.
# submit stage only after successfully
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-45844273
Thanks @sryza How about spark.scheduler.minRegisteredExecutors?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user li-zhihui commented on a diff in the pull request:
https://github.com/apache/spark/pull/900#discussion_r13692995
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -225,6 +232,17 @@ class
GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/900
submit stage after (configured ratio of) executors have been registered
Because creating TaskSetManager and registering executors are asynchronous,
in most situation, early stages' tasks run
Github user li-zhihui commented on the pull request:
https://github.com/apache/spark/pull/900#issuecomment-44374073
I create a JIRA and attach a file to describe it.
https://issues.apache.org/jira/secure/attachment/12647078/Spark%20Task%20Scheduler%20Optimization%20Proposal.pptx
101 - 106 of 106 matches
Mail list logo
