Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r222392897 --- Diff: docs/running-on-yarn.md --- @@ -121,6 +121,43 @@ To use a custom metrics.properties for the application master and executors, upd Use lower-case suffixes, e.g. <code>k</code>, <code>m</code>, <code>g</code>, <code>t</code>, and <code>p</code>, for kibi-, mebi-, gibi-, tebi-, and pebibytes, respectively. </td> </tr> +<tr> + <td><code>spark.yarn.am.resource.{resource-type}</code></td> + <td><code>(none)</code></td> + <td> + Amount of resource to use for the YARN Application Master in client mode. + In cluster mode, use <code>spark.yarn.driver.resource.<resource-type></code> instead. + Please note that this feature can be used only with YARN 3.0+ + For reference, see YARN Resource Model documentation: https://hadoop.apache.org/docs/r3.0.1/hadoop-yarn/hadoop-yarn-site/ResourceModel.html + <p/> + Example: + To request GPU resources from YARN, use: <code>spark.yarn.am.resource.yarn.io/gpu</code> + </td> +</tr> +<tr> + <td><code>spark.yarn.driver.resource.{resource-type}</code></td> + <td><code>(none)</code></td> + <td> + Amount of resource to use for the YARN Application Master in cluster mode. + Please note that this feature can be used only with YARN 3.0+ + For reference, see YARN Resource Model documentation: https://hadoop.apache.org/docs/r3.0.1/hadoop-yarn/hadoop-yarn-site/ResourceModel.html + <p/> + Example: + To request GPU resources from YARN, use: <code>spark.yarn.driver.resource.yarn.io/gpu</code> + </td> +</tr> +<tr> + <td><code>spark.yarn.executor.resource.{resource-type}</code></td> --- End diff -- this definitely makes sense for yarn side of things, do you know what mesos/k8s support for this? There was a lot of work with gpu supporting going on. We had configs specific to yarn in the past and over time things move to be more generic so if there are things we know may apply to other resource manager perhaps we should look at generic configs
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org