[
https://issues.apache.org/jira/browse/SPARK-40814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiangjian updated SPARK-40814:
------------------------------
Attachment: Dockerfile-2
> Exception in thread "main" java.lang.NoClassDefFoundError:
> io/fabric8/kubernetes/client/KubernetesClient
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-40814
> URL: https://issues.apache.org/jira/browse/SPARK-40814
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Spark Submit
> Affects Versions: 2.4.0
> Environment: k8s version: v1.18.9
> spark version: v2.4.0
> kubernetes-client:v6.1.1
> Reporter: jiangjian
> Priority: Major
> Attachments: Dockerfile, Dockerfile-1, Dockerfile-2, spark-error.log
>
>
> After I change the user in the Spark image, the running program reports an
> error. What is the problem
> ++ id -u
> + myuid=2023
> ++ id -g
> + mygid=2023
> + set +e
> ++ getent passwd 2023
> + uidentry=zndw:x:2023:2023::/home/zndw:/bin/sh
> + set -e
> + '[' -z zndw:x:2023:2023::/home/zndw:/bin/sh ']'
> + SPARK_K8S_CMD=driver
> + case "$SPARK_K8S_CMD" in
> + shift 1
> + SPARK_CLASSPATH=':/opt/spark/jars/*'
> + env
> + grep SPARK_JAVA_OPT_
> + sort -t_ -k4 -n
> + sed 's/[^=]*=\(.*\)/\1/g'
> + readarray -t SPARK_EXECUTOR_JAVA_OPTS
> + '[' -n '' ']'
> + '[' -n '' ']'
> + PYSPARK_ARGS=
> + '[' -n '' ']'
> + R_ARGS=
> + '[' -n '' ']'
> + '[' '' == 2 ']'
> + '[' '' == 3 ']'
> + case "$SPARK_K8S_CMD" in
> + CMD=("$SPARK_HOME/bin/spark-submit" --conf
> "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client
> "$@")
> + exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf
> spark.driver.bindAddress=10.1.1.11 --deploy-mode client --properties-file
> /opt/spark/conf/spark.properties --class
> com.frontier.pueedas.computer.batchTool.etl.EtlScheduler
> 'http://26.47.128.120:18000/spark/spark/raw/master/computer-batch-etl-hadoop-basic.jar?inline=false'
> configMode=HDFS metaMode=HDFS platformConfigMode=NACOS storeConfigMode=NACOS
> startDate=2022-08-02 endDate=2022-08-03
> _file=/user/config/YC2/TEST/config/computer/business/opc/EMpHpReadCurveHourData.xml
> runMode=TEST
> 2022-10-14 06:52:21 WARN NativeCodeLoader:62 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2022-10-14 06:52:29 INFO SparkContext:54 - Running Spark version 2.4.0
> 2022-10-14 06:52:29 INFO SparkContext:54 - Submitted application:
> [TEST]ETL[2022-08-02 00:00:00,2022-08-03
> 00:00:00]\{/user/config/YC2/TEST/config/computer/business/opc/EMpHpReadCurveHourData.xml}
> 2022-10-14 06:52:29 INFO SecurityManager:54 - Changing view acls to:
> zndw,root
> 2022-10-14 06:52:29 INFO SecurityManager:54 - Changing modify acls to:
> zndw,root
> 2022-10-14 06:52:29 INFO SecurityManager:54 - Changing view acls groups to:
> 2022-10-14 06:52:29 INFO SecurityManager:54 - Changing modify acls groups
> to:
> 2022-10-14 06:52:29 INFO SecurityManager:54 - SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(zndw, root); groups with view permissions: Set(); users with modify
> permissions: Set(zndw, root); groups with modify permissions: Set()
> 2022-10-14 06:52:29 INFO Utils:54 - Successfully started service
> 'sparkDriver' on port 7078.
> 2022-10-14 06:52:29 INFO SparkEnv:54 - Registering MapOutputTracker
> 2022-10-14 06:52:29 INFO SparkEnv:54 - Registering BlockManagerMaster
> 2022-10-14 06:52:29 INFO BlockManagerMasterEndpoint:54 - Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 2022-10-14 06:52:29 INFO BlockManagerMasterEndpoint:54 -
> BlockManagerMasterEndpoint up
> 2022-10-14 06:52:29 INFO DiskBlockManager:54 - Created local directory at
> /var/data/spark-9a270950-7527-4d08-a7bd-d6c1062e8522/blockmgr-79ab0f0d-6f9e-401e-aa90-91baa00a3ff3
> 2022-10-14 06:52:29 INFO MemoryStore:54 - MemoryStore started with capacity
> 912.3 MB
> 2022-10-14 06:52:29 INFO SparkEnv:54 - Registering OutputCommitCoordinator
> 2022-10-14 06:52:30 INFO log:192 - Logging initialized @9926ms
> 2022-10-14 06:52:30 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp:
> unknown, git hash: unknown
> 2022-10-14 06:52:30 INFO Server:419 - Started @10035ms
> 2022-10-14 06:52:30 INFO AbstractConnector:278 - Started
> ServerConnector@66f0548d\{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
> 2022-10-14 06:52:30 INFO Utils:54 - Successfully started service 'SparkUI'
> on port 4040.
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@59ed3e6c\{/jobs,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@70c53dbe\{/jobs/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@1894e40d\{/jobs/job,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@7342e05d\{/jobs/job/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2a331b46\{/stages,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@15383681\{/stages/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@743e66f7\{/stages/stage,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@761956ac\{/stages/stage/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@71978f46\{/stages/pool,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@304d0259\{/stages/pool/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@1d23ff23\{/storage,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2133661d\{/storage/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@6c9320c2\{/storage/rdd,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3414a8c3\{/storage/rdd/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@36cc9385\{/environment,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@cf518cf\{/environment/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@7915bca3\{/executors,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@68d651f2\{/executors/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@3ad4a7d6\{/executors/threadDump,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@1e43e323\{/executors/threadDump/json,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@4a67b4ec\{/static,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2def7a7a\{/,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@24e83d19\{/api,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@2b03d52f\{/jobs/job/kill,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO ContextHandler:781 - Started
> o.s.j.s.ServletContextHandler@37e0292a\{/stages/stage/kill,null,AVAILABLE,@Spark}
> 2022-10-14 06:52:30 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started
> at
> http://computer-batch-etl-hadoop-basic-5d159383d5437799-driver-svc.spark.svc:4040
> 2022-10-14 06:52:30 INFO SparkContext:54 - Added JAR
> local:///temp/spark_files/computer-package-batch-frame-hadoop_2.11-basic-jar-with-dependencies.jar
> at
> file:/temp/spark_files/computer-package-batch-frame-hadoop_2.11-basic-jar-with-dependencies.jar
> with timestamp 1665730350295
> 2022-10-14 06:52:30 INFO SparkContext:54 - Added JAR
> local:///temp/spark_files/computer-spark-cdh-common_2.11-2.0.0-jar-with-dependencies.jar
> at
> file:/temp/spark_files/computer-spark-cdh-common_2.11-2.0.0-jar-with-dependencies.jar
> with timestamp 1665730350296
> 2022-10-14 06:52:30 INFO SparkContext:54 - Added JAR
> local:///temp/spark_files/computer-spark-cdh-frame_2.11-2.0.0-jar-with-dependencies.jar
> at
> file:/temp/spark_files/computer-spark-cdh-frame_2.11-2.0.0-jar-with-dependencies.jar
> with timestamp 1665730350296
> 2022-10-14 06:52:30 INFO SparkContext:54 - Added JAR
> local:///temp/spark_files/computer-batch-hp-curve-hadoop-2.0.0.jar at
> file:/temp/spark_files/computer-batch-hp-curve-hadoop-2.0.0.jar with
> timestamp 1665730350297
> 2022-10-14 06:52:30 INFO SparkContext:54 - Added JAR
> http://26.47.128.120:18000/spark/spark/raw/master/computer-batch-etl-hadoop-basic.jar?inline=false
> at
> http://26.47.128.120:18000/spark/spark/raw/master/computer-batch-etl-hadoop-basic.jar?inline=false
> with timestamp 1665730350297
> 2022-10-14 06:52:30 WARN SparkContext:66 - The jar
> http://26.47.128.120:18000/spark/spark/raw/master/computer-batch-etl-hadoop-basic.jar?inline=false
> has been added already. Overwriting of added jars is not supported in the
> current version.
> Exception in thread "main" java.lang.NoClassDefFoundError:
> io/fabric8/kubernetes/client/KubernetesClient
> at
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:64)
> at
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2788)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
> at
> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
> at scala.Option.getOrElse(Option.scala:121)
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
> at
> com.frontier.pueedas.computer.common.util.SparkSessionUtil$.getSparkSession(SparkSessionUtil.scala:41)
> at
> com.frontier.pueedas.computer.batchTool.etl.EtlComputer.getSparkSession(EtlComputer.scala:346)
> at
> com.frontier.pueedas.computer.batchTool.etl.EtlComputer.start(EtlComputer.scala:230)
> at
> com.frontier.pueedas.computer.batchTool.etl.EtlScheduler$.main(EtlScheduler.scala:30)
> at
> com.frontier.pueedas.computer.batchTool.etl.EtlScheduler.main(EtlScheduler.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException:
> io.fabric8.kubernetes.client.KubernetesClient
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
> ... 25 more
> 2022-10-14 06:52:30 INFO DiskBlockManager:54 - Shutdown hook called
> 2022-10-14 06:52:30 INFO ShutdownHookManager:54 - Shutdown hook called
> 2022-10-14 06:52:30 INFO ShutdownHookManager:54 - Deleting directory
> /tmp/spark-96c8562a-aa98-4f33-b73a-a09ec99705ac
> 2022-10-14 06:52:30 INFO ShutdownHookManager:54 - Deleting directory
> /var/data/spark-9a270950-7527-4d08-a7bd-d6c1062e8522/spark-828cbff3-6b49-48f0-ada9-854c9891fc86/userFiles-5d6483c6-a1e2-42a0-8aaf-3699aaeba8ff
> 2022-10-14 06:52:30 INFO ShutdownHookManager:54 - Deleting directory
> /var/data/spark-9a270950-7527-4d08-a7bd-d6c1062e8522/spark-828cbff3-6b49-48f0-ada9-854c9891fc86
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
