Re: Reading nested JSON data with Spark SQL

. Pankaj -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reading-nested-JSON-data-with-Spark-SQL-tp19310p20933.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Reading nested JSON data with Spark SQL

, 100, 97, 100, 115, 46, 99, 111, 109, 47, 47, 100, 101, 115, 99, 47, 106, 97, 105, 112, 117, 114, 47, 49, 48, 51, 54, 50, 50, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reading-nested-JSON-data-with-Spark-SQL-tp19310p20935.html Sent from the Apache Spark

Re: Reading nested JSON data with Spark SQL

oops sqlContext.setConf(spark.sql.parquet.binaryAsString, true) thois solved the issue important for everyone -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reading-nested-JSON-data-with-Spark-SQL-tp19310p20936.html Sent from the Apache Spark User

Reading nested JSON data with Spark SQL

I have been using Spark SQL to read in JSON data, like so: val myJsonFile = sqc.jsonFile(args(myLocation)) myJsonFile.registerTempTable(myTable) sqc.sql(mySQLQuery).map { row = myFunction(row) } And then in myFunction(row) I can read the various columns with the Row.getX methods. However, this

Re: Reading nested JSON data with Spark SQL

You can extract the nested fields in sql: SELECT field.nestedField ... If you don't do that then nested fields are represented as rows within rows and can be retrieved as follows: t.getAs[Row](0).getInt(0) Also, I would write t.getAs[Buffer[CharSequence]](12) as t.getAs[Seq[String]](12) since

Re: Reading nested JSON data with Spark SQL

This works great, thank you! Simone Franzini, PhD http://www.linkedin.com/in/simonefranzini On Wed, Nov 19, 2014 at 3:40 PM, Michael Armbrust mich...@databricks.com wrote: You can extract the nested fields in sql: SELECT field.nestedField ... If you don't do that then nested fields are