Thomas Graves created SPARK-40524:
-------------------------------------
Summary: local mode with resource scheduling should just fail
Key: SPARK-40524
URL: https://issues.apache.org/jira/browse/SPARK-40524
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.1.0
Reporter: Thomas Graves
If you try to run spark in local mode and request custom resources like GPU's,
Spark will hang. Resource scheduling isn't supported in local mode so just
removing the request for resources fixes the issue, but its really confusing to
users since it just hangs.
ie to reproduce:
spark-sql --conf spark.executor.resource.gpu.amount=1 --conf
spark.task.resource.gpu.amount=1
Run:
select 1
result: hangs
To fix run:
spark-sql
spark-sql> select 1;
1
Time taken: 2.853 seconds, Fetched 1 row(s)
It would be nice if we just fail to start or threw an exception when using
those options in local mode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
