Re: using hiveContext to select a nested Map-data-type from an AVROmodel+parquet file

2015-01-19 Thread BB
I am quoting the reply I got on this - which for some reason did not get posted here. The suggestion in the reply below worked perfectly for me. The error mentioned in the reply is not related (or old). Hope this is helpful to someone. Cheers, BB Hi, BB Ideally you can do the query like:

RE: using hiveContext to select a nested Map-data-type from an AVROmodel+parquet file

2015-01-17 Thread Cheng, Hao
Wow, glad to know that it works well, and sorry, the Jira is another issue, which is not the same case here. From: Bagmeet Behera [mailto:bagme...@gmail.com] Sent: Saturday, January 17, 2015 12:47 AM To: Cheng, Hao Subject: Re: using hiveContext to select a nested Map-data-type from an

using hiveContext to select a nested Map-data-type from an AVROmodel+parquet file

2015-01-15 Thread BB
Hi all, Any help on the following is very much appreciated. = Problem: On a schemaRDD read from a parquet file (data within file uses AVRO model) using the HiveContext: I can't figure out how to 'select' or use 'where' clause, to filter rows on a field that

RE: using hiveContext to select a nested Map-data-type from an AVROmodel+parquet file

2015-01-15 Thread Cheng, Hao
Hi, BB Ideally you can do the query like: select key, value.percent from mytable_data lateral view explode(audiences) f as key, value limit 3; But there is a bug in HiveContext: https://issues.apache.org/jira/browse/SPARK-5237 I am working on it now, hopefully make a patch soon. Cheng