Re: Using Spark to analyze complex JSON

2014-05-25 Thread Michael Armbrust
On Sat, May 24, 2014 at 11:47 PM, Mayur Rustagi mayur.rust...@gmail.comwrote: Is the in-memory columnar store planned as part of SparkSQL ? This has already been ported from Shark, and is used when you run cacheTable. Also will both HiveQL SQLParser be kept updated? Yeah, we need to

Re: Using Spark to analyze complex JSON

2014-05-24 Thread Michael Armbrust
But going back to your presented pattern, I have a question. Say your data does have a fixed structure, but some of the JSON values are lists. How would you map that to a SchemaRDD? (I didn’t notice any list values in the CandyCrush example.) Take the likes field from my original example:

Re: Using Spark to analyze complex JSON

2014-05-24 Thread Mayur Rustagi
Hi Michael, Is the in-memory columnar store planned as part of SparkSQL ? Also will both HiveQL SQLParser be kept updated? Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi https://twitter.com/mayur_rustagi On Sun, May 25, 2014 at 2:44 AM, Michael Armbrust

Re: Using Spark to analyze complex JSON

2014-05-23 Thread Nicholas Chammas
Nested Data. Is there any current or planned functionality in Spark SQL or Shark to enable SQL-like querying of complex JSON? Nick View this message in context: Using Spark to analyze complex JSON Sent from the Apache Spark User List mailing

Re: Using Spark to analyze complex JSON

2014-05-22 Thread Michael Cutler
or Shark to enable SQL-like querying of complex JSON? Nick View this message in context: Using Spark to analyze complex JSON Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Using Spark to analyze complex JSON

2014-05-22 Thread Flavio Pompermaier
. Is there any current or planned functionality in Spark SQL or Shark to enable SQL-like querying of complex JSON? Nick View this message in context: Using Spark to analyze complex JSON Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Using Spark to analyze complex JSON

2014-05-22 Thread Michael Cutler
. Is there any current or planned functionality in Spark SQL or Shark to enable SQL-like querying of complex JSON? Nick View this message in context: Using Spark to analyze complex JSON Sent from the Apache Spark User List mailing list archive

Re: Using Spark to analyze complex JSON

2014-05-21 Thread Nicholas Chammas
to enable SQL-like querying of complex JSON? Nick -- View this message in context: Using Spark to analyze complex JSONhttp://apache-spark-user-list.1001560.n3.nabble.com/Using-Spark-to-analyze-complex-JSON-tp6146.html Sent from the Apache Spark User List mailing

Re: Using Spark to analyze complex JSON

2014-05-21 Thread Tobias Pfeiffer
View this message in context: Using Spark to analyze complex JSON Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Using Spark to analyze complex JSON

2014-05-21 Thread Nicholas Chammas
: The Apache Drill home page has an interesting heading: Liberate Nested Data. Is there any current or planned functionality in Spark SQL or Shark to enable SQL-like querying of complex JSON? Nick View this message in context: Using Spark to analyze

Using Spark to analyze complex JSON

2014-05-20 Thread Nick Chammas
-list.1001560.n3.nabble.com/Using-Spark-to-analyze-complex-JSON-tp6146.html Sent from the Apache Spark User List mailing list archive at Nabble.com.