Re: Spark SQL JSON dataset query nested datastructures

2014-08-10 Thread Michael Armbrust
Sounds like you need to use lateral view with explode https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView, which is supported in Spark SQL's HiveContext. On Sat, Aug 9, 2014 at 6:43 PM, Sathish Kumaran Vairavelu vsathishkuma...@gmail.com wrote: I have a simple JSON

Spark SQL JSON dataset query nested datastructures

2014-08-09 Thread Sathish Kumaran Vairavelu
I have a simple JSON dataset as below. How do I query all parts.lock for the id=1. JSON: { id: 1, name: A green door, price: 12.50, tags: [home, green], parts : [ { lock : One lock, key : single key }, { lock : 2 lock, key : 2 key } ] } Query: select id,name,price,parts.lockfrom product where