[ https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074169#comment-17074169 ]
Ferdinand Xu commented on HIVE-21492: ------------------------------------- Yes, I manually started another job to double check this. > VectorizedParquetRecordReader can't to read parquet file generated using > thrift/custom tool > ------------------------------------------------------------------------------------------- > > Key: HIVE-21492 > URL: https://issues.apache.org/jira/browse/HIVE-21492 > Project: Hive > Issue Type: Bug > Reporter: Ganesha Shreedhara > Assignee: Ganesha Shreedhara > Priority: Major > Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch > > > Taking an example of a parquet table having array of integers as below. > {code:java} > CREATE EXTERNAL TABLE ( list_of_ints` array<int>) > STORED AS PARQUET > LOCATION '{location}'; > {code} > Parquet file generated using hive will have schema for Type as below: > {code:java} > group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n} > {code} > Parquet file generated using thrift or any custom tool (using > org.apache.parquet.io.api.RecordConsumer) > may have schema for Type as below: > {code:java} > required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code} > VectorizedParquetRecordReader handles only parquet file generated using hive. > It throws the following exception when parquet file generated using thrift is > read because of the changes done as part of HIVE-18553 . > {code:java} > Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is > not a group > at org.apache.parquet.schema.Type.asGroupType(Type.java:207) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code} > > I have done a small change to handle the case where the child type of group > type can be PrimitiveType. -- This message was sent by Atlassian Jira (v8.3.4#803005)