Romain Manni-Bucau created SPARK-55004:
------------------------------------------
Summary: Simplify the creation of ephemeral volumes for spark
operator
Key: SPARK-55004
URL: https://issues.apache.org/jira/browse/SPARK-55004
Project: Spark
Issue Type: Wish
Components: Kubernetes
Affects Versions: 4.1.0
Reporter: Romain Manni-Bucau
Seems it is not inline as most of the desired configuration (without pod
template) to use an ephemeral volume on spark
{code:java}
ephemeral:
volumeClaimTemplate:
metadata:
labels:
spark.app/scope: spark
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "ebs-default" # or default one of the cluster
resources:
requests:
storage: 1Gi {code}
this is important because emptyDir have a lot of cons on the cloud (they can
just shout pods randomly and prevent spark workload to work - it is evicted -
until you accept to pay way more than needed).
Since it exists since v1.23 I guess default can get a new config toggle to
switch back to emptyDir and the new behavior use it instead.
Will avoid to use the host disk in several cases and have full disk errors for
nothing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
