Thomas Graves created SPARK-40524: ------------------------------------- Summary: local mode with resource scheduling should just fail Key: SPARK-40524 URL: https://issues.apache.org/jira/browse/SPARK-40524 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.1.0 Reporter: Thomas Graves
If you try to run spark in local mode and request custom resources like GPU's, Spark will hang. Resource scheduling isn't supported in local mode so just removing the request for resources fixes the issue, but its really confusing to users since it just hangs. ie to reproduce: spark-sql --conf spark.executor.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=1 Run: select 1 result: hangs To fix run: spark-sql spark-sql> select 1; 1 Time taken: 2.853 seconds, Fetched 1 row(s) It would be nice if we just fail to start or threw an exception when using those options in local mode -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org