Hi Ryan, this is a very good doc. So hive implementation of 3 layers with " repeated group bag " is the latest, right? I found it was "array_element" in place of "element".
I was working on auto type promotion JIRA ( https://issues.apache.org/jira/browse/HIVE-12080). I was trying to pass the requestedSchema with updated type in different convertors to create those correctly. Current requestedSchema only has the column name - not the requested/table type. As part of that, I found LIST<> schema coming through file schema and from the hive table schema have different internal representations (As mentioned din the original email). It is very hard to compare those two distinct schemas. If you think there is some other alternative, please feel free to share in the JIRA. Regards, Mohammad On Friday, October 16, 2015 11:13 AM, Ryan Blue <[email protected]> wrote: Mohammad, The spec for storing lists in Parquet is here: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists Some older writers use different representations, but we are moving everything to the 3-level structure documented on that page. This is already implemented in Hive, though. What are you need to do? rb On 10/16/2015 11:10 AM, Mohammad Islam wrote: > (sorry for being too specific!) > Hi, > What is the good and consistent way of converting a List into a parquet > schema? > > Let's take an example: > > > My type is : array<struct<latitude:double,longitude:double>>. > Same type from parquet is giving me this schema: > optional group locations (LIST) { > repeated group element { > required double latitude; > required double longitude; > } > } > > But, in Hive, the same is converted to : > optional group locations (LIST) { > repeated group bag { > optional group array_element { > optional double latitude; > optional double longitude; > } > } > } > > Looks like, Hive is adding a new layer called "bag" into it. I want both to > be in the same schema so that it is easier to compare the schema type > evolution. > > My question is : should we modify one (preferably the Hive side)? If yes, do > you have any suggestion? > > > The corresponding code in Hive : > > // An optional group containing a repeated anonymous group "bag", containing > > // 1 anonymous element "array_element" > private static GroupType convertArrayType(final String name, final > ListTypeInfo typeInfo) { > final TypeInfo subType = typeInfo.getListElementTypeInfo(); > return listWrapper(name, OriginalType.LIST, new GroupType(Repetition.REPEATED, > ParquetHiveSerDe.ARRAY.toString(), convertType("array_element", subType))); > } > > Regards, > Mohammad > > Source file: > org.apache.hadoop.hive.ql.io.parquet.convert. > HiveSchemaConverter ( > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java). > -- Ryan Blue Software Engineer Cloudera, Inc.
