Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/892#issuecomment-44285670
This is better handled in user code and not in scheduler.
We need a way to surface how many executors have registered - but once that
is available, spinning until some minimum threshold of executors is available
should be done in user code and then start computations.
Ofcourse, if there is interest, this could be abstracted out for common use
in contrib.
Currently, for our jobs, we have an arbitrary sleep introduced which is
based on expectation of cluster load.
One unstated impact of this, which is not mentioned in the PR description,
is that default parallelism is determined (in yarn mode atleast) by the number
of available executors.
So the number of reducers used can be very low resulting for the first
(few) jobs - which might have changed 'later' when reducers actually kick in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
