[jira] [Assigned] (SPARK-3596) Support changing the yarn client monitor interval
[ https://issues.apache.org/jira/browse/SPARK-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-3596: --- Assignee: (was: Apache Spark) > Support changing the yarn client monitor interval > -- > > Key: SPARK-3596 > URL: https://issues.apache.org/jira/browse/SPARK-3596 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 1.2.0 >Reporter: Thomas Graves > > Right now spark on yarn has a monitor interval that can be configured by > spark.yarn.report.interval. This is how often the client checks with the RM > to get status on the running application in cluster mode. We should allow > users to set this interval as some may not need to check so often. There is > another jira filed to make it so the client doesn't have to stay around for > cluster mode. > With the changes in https://github.com/apache/spark/pull/2350, it further > extends that to affect client mode. > We may want to add in specific configs for that since the monitorApplication > function is now used in multiple different scenarios it actually might make > sense for it to take the timeout as a parameter. You could want different > timeout for different situations. > for instance how quickly we poll on client side and print information > (cluster mode) vs how quickly we recognize the application quit and we want > to terminate (client mode), I want the latter to happen quickly where as in > cluster mode I might not care as much about how often it is printing updated > info to the screen. I guess its private so we could leave it as is and change > if we add support for that later. > my suggestion for name would be something like > spark.yarn.client.progress.pollinterval. If we were to add separate ones in > the future then they could be something like > spark.yarn.app.ready.pollinterval and spark.yarn.app.completion.pollinterval -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-3596) Support changing the yarn client monitor interval
[ https://issues.apache.org/jira/browse/SPARK-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-3596: --- Assignee: Apache Spark > Support changing the yarn client monitor interval > -- > > Key: SPARK-3596 > URL: https://issues.apache.org/jira/browse/SPARK-3596 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 1.2.0 >Reporter: Thomas Graves >Assignee: Apache Spark > > Right now spark on yarn has a monitor interval that can be configured by > spark.yarn.report.interval. This is how often the client checks with the RM > to get status on the running application in cluster mode. We should allow > users to set this interval as some may not need to check so often. There is > another jira filed to make it so the client doesn't have to stay around for > cluster mode. > With the changes in https://github.com/apache/spark/pull/2350, it further > extends that to affect client mode. > We may want to add in specific configs for that since the monitorApplication > function is now used in multiple different scenarios it actually might make > sense for it to take the timeout as a parameter. You could want different > timeout for different situations. > for instance how quickly we poll on client side and print information > (cluster mode) vs how quickly we recognize the application quit and we want > to terminate (client mode), I want the latter to happen quickly where as in > cluster mode I might not care as much about how often it is printing updated > info to the screen. I guess its private so we could leave it as is and change > if we add support for that later. > my suggestion for name would be something like > spark.yarn.client.progress.pollinterval. If we were to add separate ones in > the future then they could be something like > spark.yarn.app.ready.pollinterval and spark.yarn.app.completion.pollinterval -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org