[
https://issues.apache.org/jira/browse/HIVE-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238524#comment-14238524
]
Brock Noland commented on HIVE-9040:
------------------------------------
[~jxiang]
do you think we should throw this exception?
https://github.com/apache/hive/blob/spark/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L141
As I only caught this by looking at the logs...
> Spark Memory can be formatted string
> ------------------------------------
>
> Key: HIVE-9040
> URL: https://issues.apache.org/jira/browse/HIVE-9040
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Brock Noland
> Assignee: Jimmy Xiang
> Fix For: spark-branch
>
> Attachments: HIVE-9040.1-spark.patch, HIVE-9040.2-spark.patch
>
>
> Here:
> https://github.com/apache/hive/blob/spark/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java#L72
> we call {{getInt}} on {{spark.executor.memory}} but this is a formatted
> string, example here: http://spark.apache.org/docs/1.0.1/configuration.html
> as such, I get:
> {noformat}
> 2014-12-08 03:04:48,114 WARN [HiveServer2-Handler-Pool: Thread-34]:
> spark.SetSparkReducerParallelism
> (SetSparkReducerParallelism.java:process(141)) - Failed to create spark
> client.
> java.lang.NumberFormatException: For input string: "23000m"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:492)
> at java.lang.Integer.parseInt(Integer.java:527)
> at
> scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
> at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
> at
> org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
> at
> org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
> at scala.Option.map(Option.scala:145)
> at org.apache.spark.SparkConf.getInt(SparkConf.scala:184)
> at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getMemoryAndCores(SparkSessionImpl.java:72)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
