Thomas Graves created SPARK-37208: ------------------------------------- Summary: Support mapping Spark gpu/fpga resource types to custom YARN resource type Key: SPARK-37208 URL: https://issues.apache.org/jira/browse/SPARK-37208 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 3.0.0 Reporter: Thomas Graves
Currently Spark supports gpu/fpga resource scheduling and specifically on YARN it knows how to map gpu/fpga to the YARN resource types yarn.io/gpu and yarn.io/fpga. YARN also supports custom resource types and in Hadoop 3.3.1 made it easier for users to plugin in custom resource types. This means users may create a custom resource type that represents a GPU or FPGAs because they want additional logic that YARN the built in versions don't have. Ideally Spark users still just use the generic "gpu" or "fpga" types in Spark. So we should add the ability to change the Spark internal mappings. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org