Hi KhajaAsmath Mohammed
Please check the configuration of "spark.speculation.interval", just pass
the "30" to it.
'''
override def start(): Unit = {
backend.start()
if (!isLocal && conf.get(SPECULATION_ENABLED)) {
logInfo("Starting speculative execution thread")
speculationScheduler.scheduleWithFixedDelay(
() => Utils.tryOrStopSparkContext(sc) { checkSpeculatableTasks() },
SPECULATION_INTERVAL_MS, SPECULATION_INTERVAL_MS, TimeUnit.MILLISECONDS)
}
}
'''
Sean Owen <sro...@gmail.com> 于2021年4月13日周二 上午3:30写道：
> Something is passing this invalid 30s value, yes. Hard to say which
> property it is. I'd check if your cluster config sets anything with the
> value 30s - whatever is reading this property is not expecting it.
>
> On Mon, Apr 12, 2021, 2:25 PM KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi Sean,
>>
>> Do you think anything that can cause this with DFS client?
>>
>> java.lang.NumberFormatException: For input string: "30s"
>> at
>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:589)
>> at java.lang.Long.parseLong(Long.java:631)
>>
>>
>>
>> * at
>> org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1429)
>> at
>> org.apache.hadoop.hdfs.client.impl.DfsClientConf.<init>(DfsClientConf.java:247)
>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:301)
>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:285)*
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2859)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2896)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2878)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:184)
>> at
>> org.apache.spark.deploy.yarn.Client$$anonfun$8.apply(Client.scala:137)
>> at
>> org.apache.spark.deploy.yarn.Client$$anonfun$8.apply(Client.scala:137)
>> at scala.Option.getOrElse(Option.scala:121)
>> at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:137)
>> at
>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
>> at
>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:183)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:501)
>> at
>> org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
>> at
>> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:936)
>> at
>> org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession
>>
>> Thanks,
>> Asmath
>>
>> On Mon, Apr 12, 2021 at 2:20 PM KhajaAsmath Mohammed <
>> mdkhajaasm...@gmail.com> wrote:
>>
>>> I am using spark hbase connector provided by hortonwokrs. I was able to
>>> run without issues in my local environment and has this issue in emr.
>>>
>>> Thanks,
>>> Asmath
>>>
>>> On Apr 12, 2021, at 2:15 PM, Sean Owen <sro...@gmail.com> wrote:
>>>
>>>
>>> Somewhere you're passing a property that expects a number, but give it
>>> "30s". Is it a time property somewhere that really just wants MS or
>>> something? But most time properties (all?) in Spark should accept that type
>>> of input anyway. Really depends on what property has a problem and what is
>>> setting it.
>>>
>>> On Mon, Apr 12, 2021 at 1:56 PM KhajaAsmath Mohammed <
>>> mdkhajaasm...@gmail.com> wrote:
>>>
>>>> HI,
>>>>
>>>> I am getting weird error when running spark job in emr cluster. Same
>>>> program runs in my local machine. Is there anything that I need to do to
>>>> resolve this?
>>>>
>>>> 21/04/12 18:48:45 ERROR SparkContext: Error initializing SparkContext.
>>>> java.lang.NumberFormatException: For input string: "30s"
>>>>
>>>> I tried the solution mentioned in the link below but it didn't work for
>>>> me.
>>>>
>>>>
>>>> https://hadooptutorials.info/2020/10/11/part-5-using-spark-as-execution-engine-for-hive-2/
>>>>
>>>> Thanks,
>>>> Asmath
>>>>
>>>
