[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated SPARK-44134: ---------------------------------- Fix Version/s: 3.4.2 (was: 3.4.1) > Can't set resources (GPU/FPGA) to 0 when they are set to positive value in > spark-defaults.conf > ---------------------------------------------------------------------------------------------- > > Key: SPARK-44134 > URL: https://issues.apache.org/jira/browse/SPARK-44134 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.2.0 > Reporter: Thomas Graves > Assignee: Thomas Graves > Priority: Major > Fix For: 3.3.3, 3.5.0, 3.4.2 > > > With resource aware scheduling, if you specify a default value in the > spark-defaults.conf, a user can't override that to set it to 0. > Meaning spark-defaults.conf has something like: > {{spark.executor.resource.\{resourceName}.amount=1}} > {{spark.task.resource.\{resourceName}.amount}} =1 > If the user tries to override when submitting an application with > {{{}spark.executor.resource.\{resourceName}.amount{}}}=0 and > {{spark.task.resource.\{resourceName}.amount}} =0, it gives the user an error: > > {code:java} > 23/06/21 09:12:57 ERROR Main: Failed to initialize Spark session. > org.apache.spark.SparkException: No executor resource configs were not > specified for the following task configs: gpu > at > org.apache.spark.resource.ResourceProfile.calculateTasksAndLimitingResource(ResourceProfile.scala:206) > at > org.apache.spark.resource.ResourceProfile.$anonfun$limitingResource$1(ResourceProfile.scala:139) > at scala.Option.getOrElse(Option.scala:189) > at > org.apache.spark.resource.ResourceProfile.limitingResource(ResourceProfile.scala:138) > at > org.apache.spark.resource.ResourceProfileManager.addResourceProfile(ResourceProfileManager.scala:95) > at > org.apache.spark.resource.ResourceProfileManager.<init>(ResourceProfileManager.scala:49) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:455) > at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704) > at > org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953){code} > This used to work, my guess is this may have gotten broken with the stage > level scheduling feature. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org