[jira] [Commented] (SPARK-47010) Kubernetes: support csi driver for volume type
[
https://issues.apache.org/jira/browse/SPARK-47010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839707#comment-17839707
]
Oleg Frenkel commented on SPARK-47010:
--
Posted question on Stackoverflow:
https://stackoverflow.com/questions/78366961/apache-spark-supporting-csi-driver-for-volume-type
> Kubernetes: support csi driver for volume type
> --
>
> Key: SPARK-47010
> URL: https://issues.apache.org/jira/browse/SPARK-47010
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
>Affects Versions: 3.5.0
>Reporter: Oleg Frenkel
>Priority: Major
>
> Today Spark supports the following types of Kubernetes
> [volumes|https://kubernetes.io/docs/concepts/storage/volumes/]: hostPath,
> emptyDir, nfs and persistentVolumeClaim.
> In our case, Kubernetes cluster is multi-tenant and we cannot make
> cluster-wide changes when deploying our application to the Kubernetes
> cluster. Our application requires static shared file system. So, we cannot
> use hostPath (don't have control of hosting VMs) and persistentVolumeClaim
> (requires cluster-wide change when deploying PV). Our security department
> does not allow nfs.
> What would help in our case, is the use of csi driver (taken from here:
> https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/deploy/example/e2e_usage.md#option3-inline-volume):
> {code:java}
> kind: Pod
> apiVersion: v1
> metadata:
> name: nginx-azurefile-inline-volume
> spec:
> nodeSelector:
> "kubernetes.io/os": linux
> containers:
> - image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
> name: nginx-azurefile
> command:
> - "/bin/bash"
> - "-c"
> - set -euo pipefail; while true; do echo $(date) >>
> /mnt/azurefile/outfile; sleep 1; done
> volumeMounts:
> - name: persistent-storage
> mountPath: "/mnt/azurefile"
> readOnly: false
> volumes:
> - name: persistent-storage
> csi:
> driver: file.csi.azure.com
> volumeAttributes:
> shareName: EXISTING_SHARE_NAME # required
> secretName: azure-secret # required
> mountOptions:
> "dir_mode=0777,file_mode=0777,cache=strict,actimeo=30,nosharesock" #
> optional {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47010) Kubernetes: support csi driver for volume type
Oleg Frenkel created SPARK-47010:
Summary: Kubernetes: support csi driver for volume type
Key: SPARK-47010
URL: https://issues.apache.org/jira/browse/SPARK-47010
Project: Spark
Issue Type: New Feature
Components: Kubernetes
Affects Versions: 3.5.0
Reporter: Oleg Frenkel
Today Spark supports the following types of Kubernetes
[volumes|https://kubernetes.io/docs/concepts/storage/volumes/]: hostPath,
emptyDir, nfs and persistentVolumeClaim.
In our case, Kubernetes cluster is multi-tenant and we cannot make cluster-wide
changes when deploying our application to the Kubernetes cluster. Our
application requires static shared file system. So, we cannot use hostPath
(don't have control of hosting VMs) and persistentVolumeClaim (requires
cluster-wide change when deploying PV). Our security department does not allow
nfs.
What would help in our case, is the use of csi driver (taken from here:
https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/deploy/example/e2e_usage.md#option3-inline-volume):
{code:java}
kind: Pod
apiVersion: v1
metadata:
name: nginx-azurefile-inline-volume
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
name: nginx-azurefile
command:
- "/bin/bash"
- "-c"
- set -euo pipefail; while true; do echo $(date) >>
/mnt/azurefile/outfile; sleep 1; done
volumeMounts:
- name: persistent-storage
mountPath: "/mnt/azurefile"
readOnly: false
volumes:
- name: persistent-storage
csi:
driver: file.csi.azure.com
volumeAttributes:
shareName: EXISTING_SHARE_NAME # required
secretName: azure-secret # required
mountOptions:
"dir_mode=0777,file_mode=0777,cache=strict,actimeo=30,nosharesock" # optional
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43342) Spark in Kubernetes mode throws IllegalArgumentException when using static PVC
Oleg Frenkel created SPARK-43342:
Summary: Spark in Kubernetes mode throws IllegalArgumentException
when using static PVC
Key: SPARK-43342
URL: https://issues.apache.org/jira/browse/SPARK-43342
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 3.4.0
Reporter: Oleg Frenkel
When using static PVC with Spark 3.4, spark PI example fails with the error
below. Previous versions of Spark worked well.
{code:java}
23/04/26 13:22:02 INFO ExecutorPodsAllocator: Going to request 5 executors from
Kubernetes for ResourceProfile Id: 0, target: 5, known: 0,
sharedSlotFromPendingPods: 2147483647. 23/04/26 13:22:02 INFO
BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
23/04/26 13:22:02 ERROR ExecutorPodsSnapshotsStoreImpl: Going to stop due to
IllegalArgumentException java.lang.IllegalArgumentException: PVC ClaimName:
a1pvc should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple
executors at
org.apache.spark.deploy.k8s.features.MountVolumesFeatureStep.checkPVCClaimName(MountVolumesFeatureStep.scala:135)
at
org.apache.spark.deploy.k8s.features.MountVolumesFeatureStep.$anonfun$constructVolumes$4(MountVolumesFeatureStep.scala:75)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.Iterator.foreach(Iterator.scala:943) at
scala.collection.Iterator.foreach$(Iterator.scala:943) at
scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at
scala.collection.IterableLike.foreach(IterableLike.scala:74) at
scala.collection.IterableLike.foreach$(IterableLike.scala:73) at
scala.collection.AbstractIterable.foreach(Iterable.scala:56) at
scala.collection.TraversableLike.map(TraversableLike.scala:286) at
scala.collection.TraversableLike.map$(TraversableLike.scala:279) at
scala.collection.AbstractTraversable.map(Traversable.scala:108) at
org.apache.spark.deploy.k8s.features.MountVolumesFeatureStep.constructVolumes(MountVolumesFeatureStep.scala:58)
at
org.apache.spark.deploy.k8s.features.MountVolumesFeatureStep.configurePod(MountVolumesFeatureStep.scala:35)
at
org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBuilder.$anonfun$buildFromFeatures$5(KubernetesExecutorBuilder.scala:83)
at
scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at
scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91) at
org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBuilder.buildFromFeatures(KubernetesExecutorBuilder.scala:82)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:430)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:417)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$36(ExecutorPodsAllocator.scala:370)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$36$adapted(ExecutorPodsAllocator.scala:363)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:363)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:134)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3$adapted(ExecutorPodsAllocator.scala:134)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.org$apache$spark$scheduler$cluster$k8s$ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber$$processSnapshotsInternal(ExecutorPodsSnapshotsStoreImpl.scala:143)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.processSnapshots(ExecutorPodsSnapshotsStoreImpl.scala:131)
at
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl.$anonfun$addSubscriber$1(ExecutorPodsSnapshotsStoreImpl.scala:85)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
[jira] [Commented] (SPARK-24736) --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.
[ https://issues.apache.org/jira/browse/SPARK-24736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757663#comment-16757663 ] Oleg Frenkel commented on SPARK-24736: -- Supporting local files with --py-files would be great. By "local files" I mean files that are coming from the original machine outside of driver pod and not files available in the driver docker container. > --py-files not functional for non local URLs. It appears to pass non-local > URL's into PYTHONPATH directly. > -- > > Key: SPARK-24736 > URL: https://issues.apache.org/jira/browse/SPARK-24736 > Project: Spark > Issue Type: Bug > Components: Kubernetes, PySpark >Affects Versions: 2.4.0 > Environment: Recent 2.4.0 from master branch, submitted on Linux to a > KOPS Kubernetes cluster created on AWS. > >Reporter: Jonathan A Weaver >Priority: Minor > > My spark-submit > bin/spark-submit \ > --master > k8s://[https://internal-api-test-k8s-local-7afed8-796273878.us-east-1.elb.amazonaws.com|https://internal-api-test-k8s-local-7afed8-796273878.us-east-1.elb.amazonaws.com/] > \ > --deploy-mode cluster \ > --name pytest \ > --conf > spark.kubernetes.container.image=[412834075398.dkr.ecr.us-east-1.amazonaws.com/fids/pyspark-k8s:latest|http://412834075398.dkr.ecr.us-east-1.amazonaws.com/fids/pyspark-k8s:latest] > \ > --conf > [spark.kubernetes.driver.pod.name|http://spark.kubernetes.driver.pod.name/]=spark-pi-driver > \ > --conf > spark.kubernetes.authenticate.submission.caCertFile=[cluster.ca|http://cluster.ca/] > \ > --conf spark.kubernetes.authenticate.submission.oauthToken=$TOK \ > --conf spark.kubernetes.authenticate.driver.oauthToken=$TOK \ > --py-files "[https://s3.amazonaws.com/maxar-ids-fids/screw.zip]; \ > [https://s3.amazonaws.com/maxar-ids-fids/it.py] > > *screw.zip is successfully downloaded and placed in SparkFIles.getRootPath()* > 2018-07-01 07:33:43 INFO SparkContext:54 - Added file > [https://s3.amazonaws.com/maxar-ids-fids/screw.zip] at > [https://s3.amazonaws.com/maxar-ids-fids/screw.zip] with timestamp > 1530430423297 > 2018-07-01 07:33:43 INFO Utils:54 - Fetching > [https://s3.amazonaws.com/maxar-ids-fids/screw.zip] to > /var/data/spark-7aba748d-2bba-4015-b388-c2ba9adba81e/spark-0ed5a100-6efa-45ca-ad4c-d1e57af76ffd/userFiles-a053206e-33d9-4245-b587-f8ac26d4c240/fetchFileTemp1549645948768432992.tmp > *I print out the PYTHONPATH and PYSPARK_FILES environment variables from the > driver script:* > PYTHONPATH > /opt/spark/python/lib/pyspark.zip:/opt/spark/python/lib/py4j-0.10.7-src.zip:/opt/spark/jars/spark-core_2.11-2.4.0-SNAPSHOT.jar:/opt/spark/python/lib/pyspark.zip:/opt/spark/python/lib/py4j-*.zip:*[https://s3.amazonaws.com/maxar-ids-fids/screw.zip]* > PYSPARK_FILES [https://s3.amazonaws.com/maxar-ids-fids/screw.zip] > > *I print out sys.path* > ['/tmp/spark-fec3684b-8b63-4f43-91a4-2f2fa41a1914', > u'/var/data/spark-7aba748d-2bba-4015-b388-c2ba9adba81e/spark-0ed5a100-6efa-45ca-ad4c-d1e57af76ffd/userFiles-a053206e-33d9-4245-b587-f8ac26d4c240', > '/opt/spark/python/lib/pyspark.zip', > '/opt/spark/python/lib/py4j-0.10.7-src.zip', > '/opt/spark/jars/spark-core_2.11-2.4.0-SNAPSHOT.jar', > '/opt/spark/python/lib/py4j-*.zip', *'/opt/spark/work-dir/https', > '//[s3.amazonaws.com/maxar-ids-fids/screw.zip|http://s3.amazonaws.com/maxar-ids-fids/screw.zip]',* > '/usr/lib/python27.zip', '/usr/lib/python2.7', > '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', > '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', > '/usr/lib/python2.7/site-packages'] > > *URL from PYTHONFILES gets placed in sys.path verbatim with obvious results.* > > *Dump of spark config from container.* > Spark config dumped: > [(u'spark.master', > u'k8s://[https://internal-api-test-k8s-local-7afed8-796273878.us-east-1.elb.amazonaws.com|https://internal-api-test-k8s-local-7afed8-796273878.us-east-1.elb.amazonaws.com/]'), > (u'spark.kubernetes.authenticate.submission.oauthToken', > u''), > (u'spark.kubernetes.authenticate.driver.oauthToken', > u''), (u'spark.kubernetes.executor.podNamePrefix', > u'pytest-1530430411996'), (u'spark.kubernetes.memoryOverheadFactor', u'0.4'), > (u'spark.driver.blockManager.port', u'7079'), > (u'[spark.app.id|http://spark.app.id/]', u'spark-application-1530430424433'), > (u'[spark.app.name|http://spark.app.name/]', u'pytest'), > (u'[spark.executor.id|http://spark.executor.id/]', u'driver'), > (u'spark.driver.host', u'pytest-1530430411996-driver-svc.default.svc'), > (u'spark.kubernetes.container.image', >
[jira] [Created] (SPARK-26789) [k8s] pyspark needs to upload local resources to driver and executor pods
Oleg Frenkel created SPARK-26789:
Summary: [k8s] pyspark needs to upload local resources to driver
and executor pods
Key: SPARK-26789
URL: https://issues.apache.org/jira/browse/SPARK-26789
Project: Spark
Issue Type: New Feature
Components: Kubernetes, PySpark
Affects Versions: 2.4.0
Reporter: Oleg Frenkel
Kubernetes support provided with [https://github.com/apache-spark-on-k8s/spark]
allows local dependencies to be used with cluster deployment mode.
Specifically, the Resource Staging Server is used in order to upload local
dependencies to Kubernetes so that driver and executor pods can download these
dependencies. It looks that Spark 2.4.0 release does not support local
dependencies.
For example, the following command is expected to automatically upload pi.py
from local machine to the Kubernetes cluster and make it available for both
driver and executor pods:
{{bin/spark-submit --conf spark.app.name=example.python.pi --master
k8s://http://127.0.0.1:8001 --deploy-mode cluster --conf
spark.kubernetes.container.image=spark-py:spark-2.4.0
./examples/src/main/python/pi.py}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster
[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650519#comment-16650519 ] Oleg Frenkel commented on SPARK-18278: -- When is this fork expected to make its way to the official Spark 2.3 release? My company specifically looking for Spark Python support with kubernetes and current implementation in this fork is good enough for us. Thanks. > SPIP: Support native submission of spark jobs to a kubernetes cluster > - > > Key: SPARK-18278 > URL: https://issues.apache.org/jira/browse/SPARK-18278 > Project: Spark > Issue Type: Umbrella > Components: Build, Deploy, Documentation, Kubernetes, Scheduler, > Spark Core >Affects Versions: 2.3.0 >Reporter: Erik Erlandson >Assignee: Anirudh Ramanathan >Priority: Major > Labels: SPIP > Attachments: SPARK-18278 Spark on Kubernetes Design Proposal Revision > 2 (1).pdf > > > A new Apache Spark sub-project that enables native support for submitting > Spark applications to a kubernetes cluster. The submitted application runs > in a driver executing on a kubernetes pod, and executors lifecycles are also > managed as pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
