I went through complex hierarchal JSON structures and Spark seems to fail in querying them no matter what syntax is used.
Hope this helps, Regards, Alessandro > On Dec 8, 2014, at 6:05 AM, Raghavendra Pandey <raghavendra.pan...@gmail.com> > wrote: > > Yeah, the dot notation works. It works even for arrays. But I am not sure if > it can handle complex hierarchies. > > On Mon Dec 08 2014 at 11:55:19 AM Cheng Lian <lian.cs....@gmail.com > <mailto:lian.cs....@gmail.com>> wrote: > You may access it via something like SELECT filterIp.element > FROM tb, just like Hive. Or if you’re using Spark SQL DSL, you can > use tb.select("filterIp.element".attr). > > On 12/8/14 1:08 PM, Xuelin Cao wrote: > > > >> >> Hi, >> >> I'm generating a Spark SQL table from an offline Json file. >> >> The difficulty is, in the original json file, there is a hierarchical >> structure. And, as a result, this is what I get: >> >> scala> tb.printSchema >> root >> |-- budget: double (nullable = true) >> |-- filterIp: array (nullable = true) >> | |-- element: string (containsNull = false) >> |-- status: integer (nullable = true) >> |-- third_party: integer (nullable = true) >> |-- userId: integer (nullable = true) >> >> As you may have noticed, the table schema is with a hierarchical structure >> ("element" field is a sub-field under the "filterIp" field). Then, my >> question is, how do I access the "element" field with SQL? >> >> > >
