[ https://issues.apache.org/jira/browse/SPARK-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486014#comment-14486014 ]
Andrew Or commented on SPARK-3596: ---------------------------------- Transitively fixed by https://github.com/apache/spark/pull/5305 > Support changing the yarn client monitor interval > -------------------------------------------------- > > Key: SPARK-3596 > URL: https://issues.apache.org/jira/browse/SPARK-3596 > Project: Spark > Issue Type: Improvement > Components: YARN > Affects Versions: 1.2.0 > Reporter: Thomas Graves > Fix For: 1.4.0 > > > Right now spark on yarn has a monitor interval that can be configured by > spark.yarn.report.interval. This is how often the client checks with the RM > to get status on the running application in cluster mode. We should allow > users to set this interval as some may not need to check so often. There is > another jira filed to make it so the client doesn't have to stay around for > cluster mode. > With the changes in https://github.com/apache/spark/pull/2350, it further > extends that to affect client mode. > We may want to add in specific configs for that since the monitorApplication > function is now used in multiple different scenarios it actually might make > sense for it to take the timeout as a parameter. You could want different > timeout for different situations. > for instance how quickly we poll on client side and print information > (cluster mode) vs how quickly we recognize the application quit and we want > to terminate (client mode), I want the latter to happen quickly where as in > cluster mode I might not care as much about how often it is printing updated > info to the screen. I guess its private so we could leave it as is and change > if we add support for that later. > my suggestion for name would be something like > spark.yarn.client.progress.pollinterval. If we were to add separate ones in > the future then they could be something like > spark.yarn.app.ready.pollinterval and spark.yarn.app.completion.pollinterval -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org