Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/20761#discussion_r222846574
--- Diff: docs/running-on-yarn.md ---
@@ -121,6 +121,43 @@ To use a custom metrics.properties for the application
master and executors, upd
Use lower-case suffixes, e.g. <code>k</code>, <code>m</code>,
<code>g</code>, <code>t</code>, and <code>p</code>, for kibi-, mebi-, gibi-,
tebi-, and pebibytes, respectively.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.am.resource.{resource-type}</code></td>
+ <td><code>(none)</code></td>
+ <td>
+ Amount of resource to use for the YARN Application Master in client
mode.
+ In cluster mode, use
<code>spark.yarn.driver.resource.<resource-type></code> instead.
+ Please note that this feature can be used only with YARN 3.0+
+ For reference, see YARN Resource Model documentation:
https://hadoop.apache.org/docs/r3.0.1/hadoop-yarn/hadoop-yarn-site/ResourceModel.html
+ <p/>
+ Example:
+ To request GPU resources from YARN, use:
<code>spark.yarn.am.resource.yarn.io/gpu</code>
+ </td>
+</tr>
+<tr>
+ <td><code>spark.yarn.driver.resource.{resource-type}</code></td>
+ <td><code>(none)</code></td>
+ <td>
+ Amount of resource to use for the YARN Application Master in cluster
mode.
+ Please note that this feature can be used only with YARN 3.0+
+ For reference, see YARN Resource Model documentation:
https://hadoop.apache.org/docs/r3.0.1/hadoop-yarn/hadoop-yarn-site/ResourceModel.html
+ <p/>
+ Example:
+ To request GPU resources from YARN, use:
<code>spark.yarn.driver.resource.yarn.io/gpu</code>
+ </td>
+</tr>
+<tr>
+ <td><code>spark.yarn.executor.resource.{resource-type}</code></td>
--- End diff --
The only thing I found for mesos was `spark.mesos.gpus.max`; which is a
single option hardcoded in the mesos backend. There's no generic resource
request mechanism I could find.
k8s might have something but I haven't fully looked at the code. In any
case, I think we can look at merging both separately, if merging makes sense.
There's plenty of time before 3.0 so we can even remove these before they're
out, aside from the AM configs (which only YARN really has).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
