Re: Help with Json array parsing

[SK]

To add to my previous post, the error at runtime is teh following:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 0.0:0 failed 1 times, most recent failure: Exception
failure in TID 0 on host localhost: org.json4s.package$MappingException:
Expected collection but got JInt(20) for root JInt(20) and mapping
int[][int, int]

thanks



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Help-with-Json-array-parsing-tp9807p9820.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Help with Json array parsing

[SK]

Hi,

I have a json file where the object definition in each line includes an
array component "obj" that contains 0 or more elements as shown by the
example below.

 {"name": "16287e9cdf",  "obj": [{"min": 50,"max": 59 }, {"min": 20, "max":
29}]},
 {"name": "17087e9cdf",  "obj": [{"min": 30,"max": 39 }, {"min": 10, "max":
19}, {"min": 60, "max": 69}]},
 {"name": "18287e0cdf"}

I need to extract all the min values from the JSON definition in each line
for further processing. I used the following Spark code to parse the file
and extract the min fields, but I am getting a runtime error. I would like
to know what is the right way to extract the 0 or more min values from the
array above.

val inp = sc.textFile(args(0))
val res = inp.map(line => { parse(line) })
   .map(json =>
  {
 implicit lazy val formats =
org.json4s.DefaultFormats
 val name = (json \ "name").extract[String]
 val min_vals =  (json \ "obj"  \
"min").extract[Array[Int]]
 (name, min_vals)
  }
)


Thanks for  your help.




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Help-with-Json-array-parsing-tp9807.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.