[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152087#comment-17152087 ] Stephen Hopper commented on SPARK-31666: Very well. Thank you for taking the time to consider this, [~dongjoon]. -Stephen > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 > \ > --conf spark.local.dir="/tmp1" \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark > local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 2{code} > Any ideas on what's causing this? > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151363#comment-17151363 ] Dongjoon Hyun commented on SPARK-31666: --- First of all, the following is Spark 2.4 behavior since 2.4.0. It's not a bug. {quote}In Spark 2.4, the `LocalDirsFeatureStep` iterates through the list of paths in `spark.local.dir`. For each one, it creates a Kubernetes volume of mount type `emptyDir` with the name `spark-local-dir-${index}`. {quote} The following is a wrong use case because Spark 2.4.x features are not designed for that. {quote}The issue is that I need my Spark job to use paths from my host machine that are on a mount point that isn't part of the directory which Kubernetes uses to allocate space for `emptyDir` volumes. Therefore, I mount these paths as type `hostPath` and ask Spark to use them as local directory space. {quote} Please note that the error message came from K8s. Apache Spark starts to support your use case in Apache Spark 3.0 by adding new features. I guess you are confused on the issue types in the open source projects. Not only Apache Spark, All Apache Projects distinguishes `New Improvement` and `New Feature` from `Bug`. Many new improvements and features are adding those kind of unsupported stuffs inside old versions. We cannot backport everything into old branches. All committers and developers are already moving to Apache Spark 3.1.0. For the following, you misunderstand again. We didn't kill 2.4.x like 1.6.x. Historically, 1.6 was killed at 1.6.3. For 2.4.x, you can still use Apache Spark 2.4.6 and more. That's the reason why Apache Spark community declared 2.4 as LTS (long term support). [https://spark.apache.org/versioning-policy.html.] We will maintain with critical bug fixes and security fixes like [https://spark.apache.org/security.html] . However, 2.4.7 (or 2.4.8) will be the same with 2.4.0~2.4.6 in terms of the features. That's the community policy. > I feel giving folks 6 months to migrate from one Spark release to the next >is fair, especially now considering how mature Spark is as a project. What are >your thoughts on this? > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf >
[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150622#comment-17150622 ] Dongjoon Hyun commented on SPARK-31666: --- So, "Cannot map hostPath volumes to container" is a wrong claim. It's a fair warning from K8s to prevent duplicated volume names. I'll close this issue. > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 > \ > --conf spark.local.dir="/tmp1" \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark > local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 2{code} > Any ideas on what's causing this? > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150621#comment-17150621 ] Dongjoon Hyun commented on SPARK-31666: --- Hi, [~hopper-signifyd]. I found what is going on there. SPARK-23529 works correctly like the following. {code} # minikube ssh ls /data SPARK-31666.txt {code} {code} export HTTP2_DISABLE=true bin/spark-submit \ --master k8s://$K8S_MASTER \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.kubernetes.driverEnv.HTTP2_DISABLE=true \ --conf spark.executor.instances=1 \ --conf spark.kubernetes.container.image=spark/spark:v2.4.6 \ --conf spark.kubernetes.executor.volumes.hostPath.data.mount.path=/data \ --conf spark.kubernetes.executor.volumes.hostPath.data.options.path=/data \ local:///opt/spark/examples/jars/spark-examples_2.11-2.4.6.jar 1 {code} {code} # k exec po/spark-pi-1593729363998-exec-1 -- ls /data SPARK-31666.txt {code} Please see the error message `Invalid value: "/tmp1": must be unique.`. The error message occurs because `spark-local-dir-x` is already mounted as volume name by Spark. You should not use the same name. {code} 20/07/02 15:38:39 INFO LoggingPodStatusWatcherImpl: State changed, new state: pod name: spark-pi-1593729518015-driver namespace: default labels: spark-app-selector -> spark-74b65a9a61cc46fd8bfc5e03e4b28bb8, spark-role -> driver pod uid: d838532b-eaa9-4b11-8eba-655f66965580 creation time: 2020-07-02T22:38:39Z service account name: default volumes: spark-local-dir-1, spark-conf-volume, default-token-n5wwg node name: N/A start time: N/A container images: N/A phase: Pending status: [] {code} > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 > \ > --conf spark.local.dir="/tmp1" \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark > local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 2{code} > Any ideas on what's causing this? > -- This message was sent by
[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150601#comment-17150601 ] Dongjoon Hyun commented on SPARK-31666: --- I linked SPARK-23529 since `hostPath` is added there. > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 > \ > --conf spark.local.dir="/tmp1" \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark > local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 2{code} > Any ideas on what's causing this? > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150599#comment-17150599 ] Dongjoon Hyun commented on SPARK-31666: --- FYI, Apache Spark 2.4.0 was released at November 2, 2018. It's already over 18 months. Apache Spark community wants to service the users a little longer with critical fixes like security and correctness issues. As a result, Apache Spark 2.4.7 will be released soon again. {quote}Feature release branches will, generally, be maintained with bug fix releases for a period of 18 months {quote} > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 > \ > --conf spark.local.dir="/tmp1" \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark > local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 2{code} > Any ideas on what's causing this? > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150596#comment-17150596 ] Dongjoon Hyun commented on SPARK-31666: --- Apache Spark 3.0.0 is released last month and Apache Spark 3.1.0 is scheduled on December 2020. - https://spark.apache.org/versioning-policy.html > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes, Spark Core >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 > \ > --conf spark.local.dir="/tmp1" \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark > local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 2{code} > Any ideas on what's causing this? > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104947#comment-17104947 ] Stephen Hopper commented on SPARK-31666: I was able to fix the issue by patching Spark. As noted in this issue: [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/828] I applied versions of these two PRs (for Spark 3.0) (with some minor tweaks to make them compatible with 2.4.5): [https://github.com/apache/spark/pull/22323] [https://github.com/apache/spark/pull/24879] I then rebuilt Spark (as well as spark-submit and spark-operator) and it's working now. However, this is still going to be an issue for anyone on 2.4.5 as the docs state that hostPath directories should be useable. Should I open a PR to backport this fix for Spark 2.4.6? When is Spark 3.0 coming out? > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 > \ > --conf spark.local.dir="/tmp1" \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark > local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 2{code} > Any ideas on what's causing this? > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31666) Cannot map hostPath volumes to container
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104391#comment-17104391 ] Hyukjin Kwon commented on SPARK-31666: -- No idea. Something must have been wrong. > Cannot map hostPath volumes to container > > > Key: SPARK-31666 > URL: https://issues.apache.org/jira/browse/SPARK-31666 > Project: Spark > Issue Type: Bug > Components: Kubernetes >Affects Versions: 2.4.5 >Reporter: Stephen Hopper >Priority: Major > > I'm trying to mount additional hostPath directories as seen in a couple of > places: > [https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/] > [https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space] > [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes] > > However, whenever I try to submit my job, I run into this error: > {code:java} > Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │ > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. > Message: Pod "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique. Received status: Status(apiVersion=v1, code=422, > details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, > message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, > additionalProperties={})], group=null, kind=Pod, > name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, > additionalProperties={}), kind=Status, message=Pod > "spark-pi-1588970477877-exec-1" is invalid: > spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be > unique, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Invalid, status=Failure, additionalProperties={}).{code} > > This is my spark-submit command (note: I've used my own build of spark for > kubernetes as well as a few other images that I've seen floating around (such > as this one seedjeffwan/spark:v2.4.5) and they all have this same issue): > {code:java} > bin/spark-submit \ > --master k8s://https://my-k8s-server:443 \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=2 \ > --conf spark.kubernetes.container.image=my-spark-image:my-tag \ > --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \ > --conf spark.kubernetes.namespace=my-spark-ns \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 > \ > --conf > spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 > \ > --conf spark.local.dir="/tmp1" \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark > local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 2{code} > Any ideas on what's causing this? > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org