Hi Stelios, I've never seen this error before, but a couple of things caught my attention that I would look at closer to chase the root cause of the issue.
"org.springframework.context.annotation.AnnotationConfigApplicationContext:" and "21/08/31 07:28:42 ERROR org.springframework.boot.SpringApplication: Application run failed" seem to indicate that you're using Spring Boot (that I know almost nothing about so take the following with a pinch of salt :)) Spring Boot manages the classpath by itself and together with another interesting option in your spark-submit, spark.driver.userClassPathFirst=true, makes me wonder how much this exception: > org.apache.spark.scheduler.ExternalClusterManager: org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a subtype could be due to casting compatible types from two different classloaders? Just a thought but wanted to share as I think it's worth investigating. Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Tue, Aug 31, 2021 at 9:44 AM Stelios Philippou <stevo...@gmail.com> wrote: > Hello, > > I have been facing the current issue for some time now and I was wondering > if someone might have some inside on how I can resolve the following. > > The code (java 11) is working correctly on my local machine but whenever I > try to launch the following on K8 I am getting the following error. > > 21/08/31 07:28:42 ERROR org.apache.spark.SparkContext: Error > initializing SparkContext. > > java.util.ServiceConfigurationError: > org.apache.spark.scheduler.ExternalClusterManager: > org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a > subtype > > > > I have a spark that will monitor some directories and handle the data > accordingly. > > That part is working correctly on K8 and the SparkContext has no issue > being initialized there. > > > This is the spark-submit for that > > > spark-submit \ > --master=k8s://https://url:port \ > --deploy-mode cluster \ > --name a-name\ > --conf spark.driver.userClassPathFirst=true \ > --conf spark.kubernetes.file.upload.path=hdfs://upload-path \ > --files "application-dev.properties,keystore.jks,truststore.jks" \ > --conf spark.kubernetes.container.image=url/spark:spark-submit \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ > --conf spark.kubernetes.namespace=spark \ > --conf spark.kubernetes.container.image.pullPolicy=Always \ > --conf spark.dynamicAllocation.enabled=false \ > --driver-memory 525m --executor-memory 525m \ > --num-executors 1 --executor-cores 1 \ > target/SparkStream.jar continuous-merge > > > My issue comes when I try to launch the service in order to listen to > kafka events and store them in HDFS. > > > spark-submit \ > --master=k8s://https://url:port \ > --deploy-mode cluster \ > --name consume-data \ > --conf spark.driver.userClassPathFirst=true \ > --conf spark.kubernetes.file.upload.path=hdfs://upload-path\ > --files "application-dev.properties,keystore.jks,truststore.jks" \ > --jars > "spark-yarn_2.12-3.1.2.jar,spark-core_2.12-3.1.2.jar,kafka-clients-2.8.0.jar,spark-streaming-kafka-0-10_2.12-3.1.2.jar,spark-token-provider-kafka-0-10_2.12-3.1.2.jar" > \ > --conf spark.kubernetes.container.image=url/spark:spark-submit \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ > --conf spark.kubernetes.authenticate.executor.serviceAccountName=spark \ > --conf spark.kubernetes.namespace=spark \ > --conf spark.kubernetes.container.image.pullPolicy=Always \ > --conf spark.dynamicAllocation.enabled=false \ > --driver-memory 1g --executor-memory 1g \ > --num-executors 1 --executor-cores 1 \ > target/SparkStream.jar consume > > > It could be that I am launching the application wrongly or perhaps that my > K8 is not configured correctly ? > > > > I have stripped down my code and left it barebone and will end up with the > following issue : > > > 21/08/31 07:28:42 ERROR org.apache.spark.SparkContext: Error > initializing SparkContext. > > java.util.ServiceConfigurationError: > org.apache.spark.scheduler.ExternalClusterManager: > org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a > subtype > > at java.base/java.util.ServiceLoader.fail(Unknown Source) > > at > java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown > Source) > > at > java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown > Source) > > at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source) > > at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source) > > at > scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala: > > > 21/08/31 07:28:42 WARN > org.springframework.context.annotation.AnnotationConfigApplicationContext: > Exception encountered during context initialization - cancelling refresh > attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: > Error creating bean with name 'mainApplication': Unsatisfied dependency > expressed through field 'streamAllKafkaData'; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name 'streamAllKafkaData': Unsatisfied dependency > expressed through field 'javaSparkContext'; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'javaSparkContext' defined in class path resource > [com/configuration/SparkConfiguration.class]: Bean instantiation via > factory method failed; nested exception is > org.springframework.beans.BeanInstantiationException: Failed to instantiate > [org.apache.spark.api.java.JavaSparkContext]: Factory method > 'javaSparkContext' threw exception; nested exception is > java.util.ServiceConfigurationError: > org.apache.spark.scheduler.ExternalClusterManager: > org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a > subtype > > 21/08/31 07:28:42 ERROR org.springframework.boot.SpringApplication: > Application run failed > > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name 'mainApplication': Unsatisfied dependency expressed > through field 'streamAllKafkaData'; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name 'streamAllKafkaData': Unsatisfied dependency > expressed through field 'javaSparkContext'; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'javaSparkContext' defined in class path resource > [com/configuration/SparkConfiguration.class]: Bean instantiation via > factory method failed; nested exception is > org.springframework.beans.BeanInstantiationException: Failed to instantiate > [org.apache.spark.api.java.JavaSparkContext]: Factory method > 'javaSparkContext' threw exception; nested exception is > java.util.ServiceConfigurationError: > org.apache.spark.scheduler.ExternalClusterManager: > org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a > subtype > > > > > It could be that i am launching the application for Kafka wrongly with all > the extra jars added ? > > Just that those seem to be needed or i am getting other errors when not > including those. > > > > > Any help will be greatly appreciated. > > > > Cheers, > > Stelios > > > >
