tgravescs commented on a change in pull request #24615: [SPARK-27488][CORE] Driver interface to support GPU resources URL: https://github.com/apache/spark/pull/24615#discussion_r286189602
########## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ########## @@ -360,6 +363,48 @@ class SparkContext(config: SparkConf) extends Logging { Utils.setLogLevel(org.apache.log4j.Level.toLevel(upperCased)) } + /** + * Checks to see if any resources (GPU/FPGA/etc) are available to the driver by looking + * at and processing the spark.driver.resource.resourceName.addresses and + * spark.driver.resource.resourceName.discoveryScript configs. The configs have to be + * present when the driver starts, setting them after startup does not work. + * + * If any resource addresses configs were specified then assume all resources will be specified + * in that way. Otherwise use the discovery scripts to find the resources. Users should + * not really be setting the addresses config directly and should not be mixing methods + * for different types of resources since the addresses config is meant for Standalone mode + * and other cluster managers should use the discovery scripts. + */ + private def setupDriverResources(): Unit = { + // Only call getAllWithPrefix once and filter on those since there could be a lot of spark + // configs. + val allDriverResourceConfs = _conf.getAllWithPrefix(SPARK_DRIVER_RESOURCE_PREFIX) + val resourcesWithAddrsInConfs = + SparkConf.getConfigsWithSuffix(allDriverResourceConfs, SPARK_RESOURCE_ADDRESSES_SUFFIX) + + _resources = if (resourcesWithAddrsInConfs.nonEmpty) { Review comment: also note this matches the way the executor does it as well. For the executor, either the worker in standalone mode passes in a resources file with what it has allocated per executor or it doesn't pass a file and we use the discovery scripts. Yarn and k8s I don't expect to have issues with this as they will always use discovery Script. If we want to be able to support both ways we should update both places. I don't run standalone mode much so if you have a usecase this makes sense we can certainly change it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org