Hi, I am trying run sparkPi example via Spark on Kubernetes in my cluster. However, it is consistently failing because of executor does not have the correct hadoop configurations. I could fix it by pre-creating a configmap and mounting it into executor by specifying in pod template. But I do see in the official doc that hadoop configuration will be serialized to executor pods.
Did I miss anything? Error message in executor pod: 21/03/10 07:00:01 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1) java.io.IOException: Incomplete HDFS URI, no host: hdfs:///tmp/spark-examples_2.12-3.0.125067.jar at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:170) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:737) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:522) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$7(Executor.scala:871) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$7$adapted(Executor.scala:862) at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877) at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:862) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:406) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) More information: https://issues.apache.org/jira/browse/SPARK-34684 Thanks, Yue
