Thomas Graves created SPARK-3596:
------------------------------------
Summary: Support changing the yarn client monitor interval
Key: SPARK-3596
URL: https://issues.apache.org/jira/browse/SPARK-3596
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 1.2.0
Reporter: Thomas Graves
Right now spark on yarn has a monitor interval that can be configured by
spark.yarn.report.interval. This is how often the client checks with the RM to
get status on the running application in cluster mode. We should allow users
to set this interval as some may not need to check so often. There is another
jira filed to make it so the client doesn't have to stay around for cluster
mode.
With the changes in https://github.com/apache/spark/pull/2350, it further
extends that to affect client mode.
We may want to add in specific configs for that since the monitorApplication
function is now used in multiple different scenarios it actually might make
sense for it to take the timeout as a parameter. You could want different
timeout for different situations.
for instance how quickly we poll on client side and print information (cluster
mode) vs how quickly we recognize the application quit and we want to terminate
(client mode), I want the latter to happen quickly where as in cluster mode I
might not care as much about how often it is printing updated info to the
screen. I guess its private so we could leave it as is and change if we add
support for that later.
my suggestion for name would be something like
spark.yarn.client.progress.pollinterval. If we were to add separate ones in the
future then they could be something like spark.yarn.app.ready.pollinterval and
spark.yarn.app.completion.pollinterval
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
