Any help? or comments?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-sqlContext-jsonFile-date-type-detection-and-perforormance-tp16881p16939.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Is there any specific issues you are facing?
Thanks,
Yin
On Tue, Oct 21, 2014 at 4:00 PM, tridib tridib.sama...@live.com wrote:
Any help? or comments?
--
View this message in context:
Yes, I am unable to use jsonFile() so that it can detect date type
automatically from json data.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-sqlContext-jsonFile-date-type-detection-and-perforormance-tp16881p16974.html
Sent from the Apache
Add one more thing about question 1. Once you get the SchemaRDD from
jsonFile/jsonRDD, you can use CAST(columnName as DATE) in your query to
cast the column type from the StringType to DateType (the string format
should be -[m]m-[d]d and you need to use hiveContext). Here is the
code snippet
Hi Spark SQL team,
I trying to explore automatic schema detection for json document. I have few
questions:
1. What should be the date format to detect the fields as date type?
2. Is automatic schema infer slower than applying specific schema?
3. At this moment I am parsing json myself using map
