Can anyone provide some suggestions on how to debug this? Using spark
1.3.1. The json itself seems to be valid (other programs can parse it)
and the problem seems to lie in jsonRDD trying to describe & use a
schema.
scala> sqlContext.jsonRDD(rdd).count()
java.util.NoSuchElementException: None.
Can you tell us a bit more about (schema of) your JSON ?
You can find sample JSON
in sql/core/src/test//scala/org/apache/spark/sql/json/TestJsonData.scala
Cheers
On Wed, May 27, 2015 at 12:33 PM, Michael Stone wrote:
> Can anyone provide some suggestions on how to debug this? Using spark
> 1.3
On Wed, May 27, 2015 at 01:13:43PM -0700, Ted Yu wrote:
Can you tell us a bit more about (schema of) your JSON ?
It's fairly simple, consisting of 22 fields with values that are mostly
strings or integers, except that some of the fields are objects
with http header/value pairs. I'd guess it's
Looks like the exception was caused by resolved.get(prefix ++ a) returning
None :
a => StructField(a.head, resolved.get(prefix ++ a).get, nullable =
true)
There are three occurrences of resolved.get() in createSchema() - None
should be better handled in these places.
My two cents.
On Wed
On Wed, May 27, 2015 at 02:06:16PM -0700, Ted Yu wrote:
Looks like the exception was caused by resolved.get(prefix ++ a) returning None
:
a => StructField(a.head, resolved.get(prefix ++ a).get, nullable =
true)
There are three occurrences of resolved.get() in createSchema() - None should
