Vasilii Bulatov created SPARK-50013:
---------------------------------------
Summary: Don't resolve globs in SparkSubmit cluster deploy mode
Key: SPARK-50013
URL: https://issues.apache.org/jira/browse/SPARK-50013
Project: Spark
Issue Type: Improvement
Components: Spark Submit
Affects Versions: 3.5.3, 3.5.2, 3.5.1, 3.5.0
Reporter: Vasilii Bulatov
I was trying to use Spark in cluster deploy mode with K8s Operator and S3, but
stumbled upon an issue
([log|[https://gist.github.com/sdwbgn/c3a8c4e687dc30ad03511d81211abc7e]]) when
Spark Submit tried to authorize to S3 to resolve globs. So it requires having
configured S3 access on Spark Submit.
It seems redundant as the same resolution will also happen on the driver
itself, when job will be submitted. So it seems reasonable to disable on
spark-submit in cluster mode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
