Thanks Cheng for quick clarification.
I'm actually looking into Hive area and want to provide a patch.
Any more context that I can utilize for my parquet+hive effort ?
Also what are the main APIs/config needed. Any example usage in parquet-avro
could be a good pointer to start with.
--Mohammad
On Thursday, October 15, 2015 10:04 AM, Cheng Lian <[email protected]>
wrote:
At its core, Parquet definitely supports reading selected fields of
nested structs, and that's actually one of the initial motivations of
Parquet.
However, not all upper level Parquet data models enable it. For example,
parquet-avro and parquet-thrift work fine, while parquet-hive has to
read out the whole struct.
Cheng
On 10/15/15 9:24 AM, Mohammad Islam wrote:
> Hi,We have data which is few level nested. If we want to read only one column
> from some lower level structure, does Parquet read the complete higher level
> structure? Or it only reads that column and assemble the structure and return.
> For example, my conceptual schema is as follows struct Level1 { struct
> Level21{ x : int; struct level3 {
> ............... } } struct Level22{
> ..... } }
> Now if I need to read only Level.Level21.x, what will parquet do? read
> everything under Level1 or Level2 or only Level1.level2.x?
> If yes, are there any pointers( like example, test class, JIRA or doc) around?
> Thanks,Mohammad
>
