Hello, I am following the tutorial here ( https://spark.apache.org/docs/latest/running-on-kubernetes.html) to get spark running on a Kubernetes cluster. My Kubernetes cluster is hosted with Digital Ocean's kubernetes cluster manager. I have change the KUBECONFIG environment variable to point to my cluster access credentials, so both Spark and kubectl can speak with the nodes.
I am running into an issue when trying to run the SparkPi example as described in the Spark on Kubernetes tutorials. The command I am running is: ./bin/spark-submit --master k8s://$CLUSTERIP --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --conf spark.kubernetes.container.image=$IMAGEURL --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar where CLUSTERIP contains the ip of my cluster and IMAGEURL contains the URL of the Spark docker image I am using ( https://hub.docker.com/r/stevenstetzler/spark/). This docker image was built and pushed with the script included in the Spark 2.4 distribution. I have created a service account for Spark to ensure that it has proper permissions to create pods etc., which I checked using kubectl auth can-i create pods --as=system:serviceaccount:default:spark When I try to run the SparkPi example using the above command, I get the following output: 2018-12-12 06:26:15 WARN Utils:66 - Your hostname, docker-test resolves to a loopback address: 127.0.1.1; using 10.46.0.6 instead (on interface eth0) 2018-12-12 06:26:15 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state: pod name: spark-pi-1544595975520-driver namespace: default labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 creation time: 2018-12-12T06:26:18Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt node name: N/A start time: N/A container images: N/A phase: Pending status: [] 2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state: pod name: spark-pi-1544595975520-driver namespace: default labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 creation time: 2018-12-12T06:26:18Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt node name: flamboyant-darwin-3rhc start time: N/A container images: N/A phase: Pending status: [] 2018-12-12 06:26:19 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state: pod name: spark-pi-1544595975520-driver namespace: default labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 creation time: 2018-12-12T06:26:18Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt node name: flamboyant-darwin-3rhc start time: 2018-12-12T06:26:18Z container images: docker.io/stevenstetzler/spark:v1 phase: Pending status: [ContainerStatus(containerID=null, image= docker.io/stevenstetzler/spark:v1, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})] 2018-12-12 06:26:19 INFO Client:54 - Waiting for application spark-pi to finish... 2018-12-12 06:26:21 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state: pod name: spark-pi-1544595975520-driver namespace: default labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 creation time: 2018-12-12T06:26:18Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt node name: flamboyant-darwin-3rhc start time: 2018-12-12T06:26:18Z container images: stevenstetzler/spark:v1 phase: Running status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})] 2018-12-12 06:26:24 INFO LoggingPodStatusWatcherImpl:54 - State changed, new state: pod name: spark-pi-1544595975520-driver namespace: default labels: spark-app-selector -> spark-ec5eb54644d348e7a213f8178b8ef61f, spark-role -> driver pod uid: d5d6bdc7-fdd6-11e8-b666-8e815d3815b2 creation time: 2018-12-12T06:26:18Z service account name: spark volumes: spark-local-dir-1, spark-conf-volume, spark-token-qf9dt node name: flamboyant-darwin-3rhc start time: 2018-12-12T06:26:18Z container images: stevenstetzler/spark:v1 phase: Failed status: [ContainerStatus(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, image=stevenstetzler/spark:v1, imageID=docker-pullable://stevenstetzler/spark@sha256:dc4bce1e410ebd7b14a88caa46a4282a61ff058c6374b7cf721b7498829bb041, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://b923c6ff02b93557c8c104c01a4eeb1c05f3d0c0123ec4e5895bfd6be398a03a, exitCode=1, finishedAt=Time(time=2018-12-12T06:26:24Z, additionalProperties={}), message=null, reason=Error, signal=null, startedAt=Time(time=2018-12-12T06:26:21Z, additionalProperties={}), additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})] 2018-12-12 06:26:24 INFO LoggingPodStatusWatcherImpl:54 - Container final statuses: Container name: spark-kubernetes-driver Container image: stevenstetzler/spark:v1 Container state: Terminated Exit code: 1 2018-12-12 06:26:24 INFO Client:54 - Application spark-pi finished. 2018-12-12 06:26:24 INFO ShutdownHookManager:54 - Shutdown hook called 2018-12-12 06:26:24 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-ca66ad4b-6526-4053-9ae3-f9ceb2e41cfd When I check the logs with kubectl logs spark-pi-1544595975520-driver I get the following: ++ id -u + myuid=0 ++ id -g + mygid=0 + set +e ++ getent passwd 0 + uidentry=root:x:0:0:root:/root:/bin/ash + set -e + '[' -z root:x:0:0:root:/root:/bin/ash ']' + SPARK_K8S_CMD=driver + case "$SPARK_K8S_CMD" in + shift 1 + SPARK_CLASSPATH=':/opt/spark/jars/*' + env + grep SPARK_JAVA_OPT_ + sed 's/[^=]*=\(.*\)/\1/g' + sort -t_ -k4 -n + readarray -t SPARK_EXECUTOR_JAVA_OPTS + '[' -n '' ']' + '[' -n '' ']' + PYSPARK_ARGS= + '[' -n '' ']' + R_ARGS= + '[' -n '' ']' + '[' '' == 2 ']' + '[' '' == 3 ']' + case "$SPARK_K8S_CMD" in + CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.96.3 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi spark-internal 2018-12-12 06:26:22 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-12-12 06:26:22 INFO SparkContext:54 - Running Spark version 2.4.0 2018-12-12 06:26:22 INFO SparkContext:54 - Submitted application: Spark Pi 2018-12-12 06:26:22 INFO SecurityManager:54 - Changing view acls to: root 2018-12-12 06:26:22 INFO SecurityManager:54 - Changing modify acls to: root 2018-12-12 06:26:22 INFO SecurityManager:54 - Changing view acls groups to: 2018-12-12 06:26:22 INFO SecurityManager:54 - Changing modify acls groups to: 2018-12-12 06:26:22 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 2018-12-12 06:26:23 INFO Utils:54 - Successfully started service 'sparkDriver' on port 7078. 2018-12-12 06:26:23 INFO SparkEnv:54 - Registering MapOutputTracker 2018-12-12 06:26:23 INFO SparkEnv:54 - Registering BlockManagerMaster 2018-12-12 06:26:23 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 2018-12-12 06:26:23 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up 2018-12-12 06:26:23 INFO DiskBlockManager:54 - Created local directory at /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/blockmgr-63358d1a-1e7e-493e-accd-7992da9b0adf 2018-12-12 06:26:23 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB 2018-12-12 06:26:23 INFO SparkEnv:54 - Registering OutputCommitCoordinator 2018-12-12 06:26:23 INFO log:192 - Logging initialized @2059ms 2018-12-12 06:26:23 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown 2018-12-12 06:26:23 INFO Server:419 - Started @2155ms 2018-12-12 06:26:23 INFO AbstractConnector:278 - Started ServerConnector@7180e701{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 2018-12-12 06:26:23 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040. 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9f6e406{/jobs,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a60c416{/jobs/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60f2e0bd{/jobs/job,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6b410923 {/jobs/job/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@57bd2029{/stages,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@203dd56b{/stages/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@e8fadb0{/stages/stage,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d3e6d34 {/stages/stage/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6eafb10e{/stages/pool,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26a94fa5 {/stages/pool/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@464a4442{/storage,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2873d672{/storage/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3bc735b3{/storage/rdd,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@577f9109 {/storage/rdd/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4303b7f0{/environment,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@757529a4 {/environment/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@779de014{/executors,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c41d037 {/executors/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2234078 {/executors/threadDump,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ec77191 {/executors/threadDump/json,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4642b71d{/static,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7ed9ae94{/,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66908383{/api,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@534ca02b {/jobs/job/kill,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29a23c3d {/stages/stage/kill,null,AVAILABLE,@Spark} 2018-12-12 06:26:23 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://spark-pi-1544595975520-driver-svc.default.svc:4040 2018-12-12 06:26:23 INFO SparkContext:54 - Added JAR file:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar at spark://spark-pi-1544595975520-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.4.0.jar with timestamp 1544595983645 2018-12-12 06:26:23 ERROR SparkContext:91 - Error initializing SparkContext. org.apache.spark.SparkException: External scheduler cannot be instantiated at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) ... 20 more Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) ... 23 more Caused by: java.io.IOException: Empty input at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) ... 29 more 2018-12-12 06:26:23 INFO AbstractConnector:318 - Stopped Spark@7180e701 {HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 2018-12-12 06:26:23 INFO SparkUI:54 - Stopped Spark web UI at http://spark-pi-1544595975520-driver-svc.default.svc:4040 2018-12-12 06:26:23 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped! 2018-12-12 06:26:23 INFO MemoryStore:54 - MemoryStore cleared 2018-12-12 06:26:23 INFO BlockManager:54 - BlockManager stopped 2018-12-12 06:26:23 INFO BlockManagerMaster:54 - BlockManagerMaster stopped 2018-12-12 06:26:23 WARN MetricsSystem:66 - Stopping a MetricsSystem that is not running 2018-12-12 06:26:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped! 2018-12-12 06:26:23 INFO SparkContext:54 - Successfully stopped SparkContext Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2794) at org.apache.spark.SparkContext.<init>(SparkContext.scala:493) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:167) at org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64) at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788) ... 20 more Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93) at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71) at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114) at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) ... 23 more Caused by: java.io.IOException: Empty input at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) ... 29 more 2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Shutdown hook called 2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Deleting directory /var/data/spark-c306f3bc-cc7c-4090-a7f7-22fbc02b72bd/spark-157bde7a-d0e1-4092-9e39-5ab2b727e3e7 2018-12-12 06:26:23 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-07910ccb-c212-4622-9c1b-3e8b772f3faa It looks to me like there is a communication / authentication issue during pod creation, and it looks like there might be a certificate missing that Spark needs to communicate with Kubernetes, which I infer from the "java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input" error message. Any ideas on what I can do to resolve this? Thanks!
