Hello, I have been facing the current issue for some time now and I was wondering if someone might have some inside on how I can resolve the following.
The code (java 11) is working correctly on my local machine but whenever I try to launch the following on K8 I am getting the following error. 21/08/31 07:28:42 ERROR org.apache.spark.SparkContext: Error initializing SparkContext. java.util.ServiceConfigurationError: org.apache.spark.scheduler.ExternalClusterManager: org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a subtype I have a spark that will monitor some directories and handle the data accordingly. That part is working correctly on K8 and the SparkContext has no issue being initialized there. This is the spark-submit for that spark-submit \ --master=k8s://https://url:port \ --deploy-mode cluster \ --name a-name\ --conf spark.driver.userClassPathFirst=true \ --conf spark.kubernetes.file.upload.path=hdfs://upload-path \ --files "application-dev.properties,keystore.jks,truststore.jks" \ --conf spark.kubernetes.container.image=url/spark:spark-submit \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.namespace=spark \ --conf spark.kubernetes.container.image.pullPolicy=Always \ --conf spark.dynamicAllocation.enabled=false \ --driver-memory 525m --executor-memory 525m \ --num-executors 1 --executor-cores 1 \ target/SparkStream.jar continuous-merge My issue comes when I try to launch the service in order to listen to kafka events and store them in HDFS. spark-submit \ --master=k8s://https://url:port \ --deploy-mode cluster \ --name consume-data \ --conf spark.driver.userClassPathFirst=true \ --conf spark.kubernetes.file.upload.path=hdfs://upload-path\ --files "application-dev.properties,keystore.jks,truststore.jks" \ --jars "spark-yarn_2.12-3.1.2.jar,spark-core_2.12-3.1.2.jar,kafka-clients-2.8.0.jar,spark-streaming-kafka-0-10_2.12-3.1.2.jar,spark-token-provider-kafka-0-10_2.12-3.1.2.jar" \ --conf spark.kubernetes.container.image=url/spark:spark-submit \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.authenticate.executor.serviceAccountName=spark \ --conf spark.kubernetes.namespace=spark \ --conf spark.kubernetes.container.image.pullPolicy=Always \ --conf spark.dynamicAllocation.enabled=false \ --driver-memory 1g --executor-memory 1g \ --num-executors 1 --executor-cores 1 \ target/SparkStream.jar consume It could be that I am launching the application wrongly or perhaps that my K8 is not configured correctly ? I have stripped down my code and left it barebone and will end up with the following issue : 21/08/31 07:28:42 ERROR org.apache.spark.SparkContext: Error initializing SparkContext. java.util.ServiceConfigurationError: org.apache.spark.scheduler.ExternalClusterManager: org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a subtype at java.base/java.util.ServiceLoader.fail(Unknown Source) at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source) at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source) at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source) at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source) at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala: 21/08/31 07:28:42 WARN org.springframework.context.annotation.AnnotationConfigApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'mainApplication': Unsatisfied dependency expressed through field 'streamAllKafkaData'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'streamAllKafkaData': Unsatisfied dependency expressed through field 'javaSparkContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'javaSparkContext' defined in class path resource [com/configuration/SparkConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.api.java.JavaSparkContext]: Factory method 'javaSparkContext' threw exception; nested exception is java.util.ServiceConfigurationError: org.apache.spark.scheduler.ExternalClusterManager: org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a subtype 21/08/31 07:28:42 ERROR org.springframework.boot.SpringApplication: Application run failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'mainApplication': Unsatisfied dependency expressed through field 'streamAllKafkaData'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'streamAllKafkaData': Unsatisfied dependency expressed through field 'javaSparkContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'javaSparkContext' defined in class path resource [com/configuration/SparkConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.api.java.JavaSparkContext]: Factory method 'javaSparkContext' threw exception; nested exception is java.util.ServiceConfigurationError: org.apache.spark.scheduler.ExternalClusterManager: org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a subtype It could be that i am launching the application for Kafka wrongly with all the extra jars added ? Just that those seem to be needed or i am getting other errors when not including those. Any help will be greatly appreciated. Cheers, Stelios
