Yue Peng created SPARK-34684: -------------------------------- Summary: Hadoop config could not be successfully serilized from driver pods to executor pods Key: SPARK-34684 URL: https://issues.apache.org/jira/browse/SPARK-34684 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 3.0.1 Reporter: Yue Peng
I have set HADOOP_CONF_DIR correctly. And I have verified that hadoop configs have been stored into a configmap and mounted to driver. However, spark pi example job keeps failing where executor do not know how to talk to hdfs. I highly suspect that there is a bug causing it, as I manually create a configmap storing hadoop configs and mounted it to executor in template file, which could fix the error. Spark submit command: /opt/spark-3.0/bin/spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master k8s://https://10.***.18.96:6443 --num-executors 1 --conf spark.kubernetes.namespace=test --conf spark.kubernetes.container.image=**** --conf spark.kubernetes.driver.podTemplateFile=/opt/spark-3.0/conf/spark-driver.template --conf spark.kubernetes.executor.podTemplateFile=/opt/spark-3.0/conf/spark-executor.template --conf spark.kubernetes.file.upload.path=/opt/spark-3.0/examples/jars hdfs:///tmp/spark-examples_2.12-3.0.125067.jar 1000 Error log: 21/03/10 06:59:58 INFO TransportClientFactory: Successfully created connection to org-apache-spark-examples-sparkpi-0e58b6781aeef2d5-driver-svc.test.svc/100.64.0.191:7078 after 608 ms (392 ms spent in bootstraps) 21/03/10 06:59:58 INFO SecurityManager: Changing view acls to: root 21/03/10 06:59:58 INFO SecurityManager: Changing modify acls to: root 21/03/10 06:59:58 INFO SecurityManager: Changing view acls groups to: 21/03/10 06:59:58 INFO SecurityManager: Changing modify acls groups to: 21/03/10 06:59:58 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 21/03/10 06:59:59 INFO TransportClientFactory: Successfully created connection to org-apache-spark-examples-sparkpi-0e58b6781aeef2d5-driver-svc.test.svc/100.64.0.191:7078 after 130 ms (104 ms spent in bootstraps) 21/03/10 06:59:59 INFO DiskBlockManager: Created local directory at /var/data/spark-0f541e3d-994f-4c7a-843f-f7dac57dfc13/blockmgr-981cfb62-5b27-4d1a-8fbd-eddb466faf1d 21/03/10 06:59:59 INFO MemoryStore: MemoryStore started with capacity 2047.2 MiB 21/03/10 06:59:59 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://coarsegrainedschedu...@org-apache-spark-examples-sparkpi-0e58b6781aeef2d5-driver-svc.test.svc:7078 21/03/10 06:59:59 INFO ResourceUtils: ============================================================== 21/03/10 06:59:59 INFO ResourceUtils: Resources for spark.executor: 21/03/10 06:59:59 INFO ResourceUtils: ============================================================== 21/03/10 06:59:59 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 21/03/10 06:59:59 INFO Executor: Starting executor ID 1 on host 100.64.0.192 21/03/10 07:00:00 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37956. 21/03/10 07:00:00 INFO NettyBlockTransferService: Server created on 100.64.0.192:37956 21/03/10 07:00:00 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 21/03/10 07:00:00 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, 100.64.0.192, 37956, None) 21/03/10 07:00:00 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, 100.64.0.192, 37956, None) 21/03/10 07:00:00 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, 100.64.0.192, 37956, None) 21/03/10 07:00:01 INFO CoarseGrainedExecutorBackend: Got assigned task 0 21/03/10 07:00:01 INFO CoarseGrainedExecutorBackend: Got assigned task 1 21/03/10 07:00:01 INFO Executor: Running task 1.0 in stage 0.0 (TID 1) 21/03/10 07:00:01 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 21/03/10 07:00:01 INFO Executor: Fetching spark://org-apache-spark-examples-sparkpi-0e58b6781aeef2d5-driver-svc.test.svc:7078/jars/spark-examples_2.12-3.0.125067.jar with timestamp 1615359587432 21/03/10 07:00:01 INFO TransportClientFactory: Successfully created connection to org-apache-spark-examples-sparkpi-0e58b6781aeef2d5-driver-svc.test.svc/100.64.0.191:7078 after 65 ms (58 ms spent in bootstraps) 21/03/10 07:00:01 INFO Utils: Fetching spark://org-apache-spark-examples-sparkpi-0e58b6781aeef2d5-driver-svc.test.svc:7078/jars/spark-examples_2.12-3.0.125067.jar to /var/data/spark-0f541e3d-994f-4c7a-843f-f7dac57dfc13/spark-1b32a101-9bf6-4836-a243-bd853253e85f/fetchFileTemp12837078937383244276.tmp 21/03/10 07:00:01 INFO Utils: Copying /var/data/spark-0f541e3d-994f-4c7a-843f-f7dac57dfc13/spark-1b32a101-9bf6-4836-a243-bd853253e85f/-3355581251615359587432_cache to /opt/spark/work-dir/./spark-examples_2.12-3.0.125067.jar 21/03/10 07:00:01 INFO Executor: Adding file:/opt/spark/work-dir/./spark-examples_2.12-3.0.125067.jar to class loader 21/03/10 07:00:01 INFO Executor: Fetching hdfs:///tmp/spark-examples_2.12-3.0.125067.jar with timestamp 1615359587441 21/03/10 07:00:01 INFO Executor: Fetching hdfs:///tmp/spark-examples_2.12-3.0.125067.jar with timestamp 1615359587441 21/03/10 07:00:01 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1) java.io.IOException: Incomplete HDFS URI, no host: hdfs:///tmp/spark-examples_2.12-3.0.125067.jar at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:170) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:737) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:522) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$7(Executor.scala:871) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$7$adapted(Executor.scala:862) at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877) at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:862) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:406) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 21/03/10 07:00:01 INFO CoarseGrainedExecutorBackend: Got assigned task 2 21/03/10 07:00:01 INFO Executor: Running task 2.0 in stage 0.0 (TID 2) 21/03/10 07:00:01 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.io.IOException: Incomplete HDFS URI, no host: hdfs:///tmp/spark-examples_2.12-3.0.125067.jar at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:170) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:737) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:522) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$7(Executor.scala:871) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$7$adapted(Executor.scala:862) at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877) at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:862) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:406) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 21/03/10 07:00:01 INFO Executor: Fetching hdfs:///tmp/spark-examples_2.12-3.0.125067.jar with timestamp 1615359587441 21/03/10 07:00:01 INFO CoarseGrainedExecutorBackend: Got assigned task 3 21/03/10 07:00:01 INFO Executor: Running task 1.1 in stage 0.0 (TID 3) 21/03/10 07:00:01 ERROR Executor: Exception in task 2.0 in stage 0.0 (TID 2) java.io.IOException: Incomplete HDFS URI, no host: hdfs:///tmp/spark-examples_2.12-3.0.125067.jar at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:170) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:737) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:522) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$7(Executor.scala:871) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$7$adapted(Executor.scala:862) at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877) at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:862) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:406) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) 21/03/10 07:00:01 INFO Executor: Fetching hdfs:///tmp/spark-examples_2.12-3.0.125067.jar with timestamp 1615359587441 21/03/10 07:00:01 INFO CoarseGrainedExecutorBackend: Got assigned task 4 21/03/10 07:00:01 INFO Executor: Running task 0.1 in stage 0.0 (TID 4) 21/03/10 07:00:01 ERROR Executor: Exception in task 1.1 in stage 0.0 (TID 3) java.io.IOException: Incomplete HDFS URI, no host: hdfs:///tmp/spark-examples_2.12-3.0.125067.jar -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org