Hi durin,
I just tried this example (nice data, by the way!), *with each JSON
object on one line*, and it worked fine:
scala rdd.printSchema()
root
|-- entities: org.apache.spark.sql.catalyst.types.StructType$@13b6cdef
||-- friends:
Hi Zongheng Yang,
thanks for your response. Reading your answer, I did some more tests and
realized that analyzing very small parts of the dataset (which is ~130GB in
~4.3M lines) works fine.
The error occurs when I analyze larger parts. Using 5% of the whole data,
the error is the same as
Is it possible you have blank lines in your input? Not that this should be
an error condition, but it may be what's causing it.
On Wed, Jun 25, 2014 at 11:57 AM, durin m...@simon-schaefer.net wrote:
Hi Zongheng Yang,
thanks for your response. Reading your answer, I did some more tests and
Hi Durin,
I guess that blank lines caused the problem (like Aaron said). Right now,
jsonFile does not skip faulty lines. Can you first use sc.textfile to load
the file as RDD[String] and then use filter to filter out those blank lines
(code snippet can be found below)?
val sqlContext = new
Hi Yin an Aaron,
thanks for your help, this was indeed the problem. I've counted 1233 blank
lines using grep, and the code snippet below works with those.
From what you said, I guess that skipping faulty lines will be possible in
later versions?
Kind regards,
Simon
--
View this message in
