Prashant Sharma created SPARK-32221:
---------------------------------------
Summary: Avoid possible errors due to incorrect file size or type
supplied in spark conf.
Key: SPARK-32221
URL: https://issues.apache.org/jira/browse/SPARK-32221
Project: Spark
Issue Type: Sub-task
Components: Kubernetes
Affects Versions: 3.1.0
Reporter: Prashant Sharma
This would avoid failures, in case the files are a bit large or a user places a
binary file inside the SPARK_CONF_DIR.
Both of which are not supported at the moment.
The reason is, underlying etcd store does limit the size of each entry to only
1 MiB. Once etcd is upgraded in all the popular k8s clusters, then we can hope
to overcome this limitation. e.g.
[https://etcd.io/docs/v3.4.0/dev-guide/limit/] version of etcd allows for
higher limit on each entry.
Even if that does not happen, there are other ways to overcome this limitation,
for example, we can have config files split across multiple configMaps. We need
to discuss, and prioritise, this issue takes the straightforward approach of
skipping files that cannot be accommodated within 1MiB limit and WARNING the
user about the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
