[ https://issues.apache.org/jira/browse/SPARK-32259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156626#comment-17156626 ]
Rob Vesse commented on SPARK-32259: ----------------------------------- [~prakki79] Ideally you'd also include the following in your report: * The full {{spark-submit}} command * The {{kubectl describe pod}} output for the relevant pod(s) * The {{kubectl get pod -o=yaml}} output for the relevant pod(s) bq. I have these config "spark.kubernetes.local.dirs.tmpfs=true", still spark is not pointing its spill data to SPARK_LOCAL_DIRS path. Nothing you have shown so far suggests that this is true, all that configuration setting does is change how Spark configures the relevant {{emptyDir}} volume used for ephemeral storage (and that's assuming you haven't supplied other configuration that explicitly configures local directories). You can exhaust an in-memory volume in exactly the same as you exhaust a disk based volume and get your pod evicted. Note that when using in-memory volumes then you may need to adjust the amount of memory allocated to your pod per the documentation - http://spark.apache.org/docs/latest/running-on-kubernetes.html#using-ram-for-local-storage > tmpfs=true, not pointing to SPARK_LOCAL_DIRS in k8s > --------------------------------------------------- > > Key: SPARK-32259 > URL: https://issues.apache.org/jira/browse/SPARK-32259 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 3.0.0 > Reporter: Prakash Rajendran > Priority: Blocker > Attachments: Capture.PNG > > > In Spark-Submit, I have these config > "{color:#4c9aff}*spark.kubernetes.local.dirs.tmpfs=true*{color}", still spark > is not pointing its spill data to SPARK_LOCAL_DIRS path. > K8s is evicting the pod due to error "{color:#de350b}*Pod ephemeral local > storage usage exceeds the total limit of containers.*{color}" > > We use Spark launcher to do spark submit in k8s. Since it is evicted, the pod > logs for stack trace is not available. we have only pod events given in > attachment > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org