Thanks Sandy et al, I will try that. I like that I can choose the
minRegisteredResourcesRatio.
On Wed, Jun 24, 2015 at 11:04 AM, Sandy Ryza sandy.r...@cloudera.com
wrote:
Hi Arun,
You can achieve this by
setting spark.scheduler.maxRegisteredResourcesWaitingTime to some really
high number
On 24 Jun 2015, at 05:55, canan chen
ccn...@gmail.commailto:ccn...@gmail.com wrote:
Why do you want it start until all the resources are ready ? Make it start as
early as possible should make it complete earlier and increase the utilization
of resources
On Tue, Jun 23, 2015 at 10:34 PM, Arun
Hi Arun,
You can achieve this by
setting spark.scheduler.maxRegisteredResourcesWaitingTime to some really
high number and spark.scheduler.minRegisteredResourcesRatio to 1.0.
-Sandy
On Wed, Jun 24, 2015 at 2:21 AM, Steve Loughran ste...@hortonworks.com
wrote:
On 24 Jun 2015, at 05:55, canan
Sometimes if my Hortonworks yarn-enabled cluster is fairly busy, Spark (via
spark-submit) will begin its processing even though it apparently did not
get all of the requested resources; it is running very slowly.
Is there a way to force Spark/YARN to only begin when it has the full set
of
Why do you want it start until all the resources are ready ? Make it start
as early as possible should make it complete earlier and increase the
utilization of resources
On Tue, Jun 23, 2015 at 10:34 PM, Arun Luthra arun.lut...@gmail.com wrote:
Sometimes if my Hortonworks yarn-enabled cluster
