Hi KhajaAsmath Mohammed Please check the configuration of "spark.speculation.interval", just pass the "30" to it.
''' override def start(): Unit = { backend.start() if (!isLocal && conf.get(SPECULATION_ENABLED)) { logInfo("Starting speculative execution thread") speculationScheduler.scheduleWithFixedDelay( () => Utils.tryOrStopSparkContext(sc) { checkSpeculatableTasks() }, SPECULATION_INTERVAL_MS, SPECULATION_INTERVAL_MS, TimeUnit.MILLISECONDS) } } ''' Sean Owen <sro...@gmail.com> 于2021年4月13日周二 上午3:30写道: > Something is passing this invalid 30s value, yes. Hard to say which > property it is. I'd check if your cluster config sets anything with the > value 30s - whatever is reading this property is not expecting it. > > On Mon, Apr 12, 2021, 2:25 PM KhajaAsmath Mohammed < > mdkhajaasm...@gmail.com> wrote: > >> Hi Sean, >> >> Do you think anything that can cause this with DFS client? >> >> java.lang.NumberFormatException: For input string: "30s" >> at >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> at java.lang.Long.parseLong(Long.java:589) >> at java.lang.Long.parseLong(Long.java:631) >> >> >> >> * at >> org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1429) >> at >> org.apache.hadoop.hdfs.client.impl.DfsClientConf.<init>(DfsClientConf.java:247) >> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:301) >> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:285)* >> at >> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160) >> at >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2859) >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) >> at >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2896) >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2878) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:184) >> at >> org.apache.spark.deploy.yarn.Client$$anonfun$8.apply(Client.scala:137) >> at >> org.apache.spark.deploy.yarn.Client$$anonfun$8.apply(Client.scala:137) >> at scala.Option.getOrElse(Option.scala:121) >> at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:137) >> at >> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56) >> at >> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:183) >> at org.apache.spark.SparkContext.<init>(SparkContext.scala:501) >> at >> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) >> at >> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:936) >> at >> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession >> >> Thanks, >> Asmath >> >> On Mon, Apr 12, 2021 at 2:20 PM KhajaAsmath Mohammed < >> mdkhajaasm...@gmail.com> wrote: >> >>> I am using spark hbase connector provided by hortonwokrs. I was able to >>> run without issues in my local environment and has this issue in emr. >>> >>> Thanks, >>> Asmath >>> >>> On Apr 12, 2021, at 2:15 PM, Sean Owen <sro...@gmail.com> wrote: >>> >>> >>> Somewhere you're passing a property that expects a number, but give it >>> "30s". Is it a time property somewhere that really just wants MS or >>> something? But most time properties (all?) in Spark should accept that type >>> of input anyway. Really depends on what property has a problem and what is >>> setting it. >>> >>> On Mon, Apr 12, 2021 at 1:56 PM KhajaAsmath Mohammed < >>> mdkhajaasm...@gmail.com> wrote: >>> >>>> HI, >>>> >>>> I am getting weird error when running spark job in emr cluster. Same >>>> program runs in my local machine. Is there anything that I need to do to >>>> resolve this? >>>> >>>> 21/04/12 18:48:45 ERROR SparkContext: Error initializing SparkContext. >>>> java.lang.NumberFormatException: For input string: "30s" >>>> >>>> I tried the solution mentioned in the link below but it didn't work for >>>> me. >>>> >>>> >>>> https://hadooptutorials.info/2020/10/11/part-5-using-spark-as-execution-engine-for-hive-2/ >>>> >>>> Thanks, >>>> Asmath >>>> >>>