[ https://issues.apache.org/jira/browse/HIVE-22670?focusedWorklogId=775370&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-775370 ]
ASF GitHub Bot logged work on HIVE-22670: ----------------------------------------- Author: ASF GitHub Bot Created on: 27/May/22 04:42 Start Date: 27/May/22 04:42 Worklog Time Spent: 10m Work Description: pvary commented on PR #3328: URL: https://github.com/apache/hive/pull/3328#issuecomment-1139271131 @achennagiri: please rebase, to fix the test failure. Thanks, Peter Issue Time Tracking ------------------- Worklog Id: (was: 775370) Time Spent: 20m (was: 10m) > ArrayIndexOutOfBoundsException when vectorized reader is used for reading a > parquet file > ---------------------------------------------------------------------------------------- > > Key: HIVE-22670 > URL: https://issues.apache.org/jira/browse/HIVE-22670 > Project: Hive > Issue Type: Bug > Components: Parquet, Vectorization > Affects Versions: 2.3.6, 3.1.2 > Reporter: Ganesha Shreedhara > Assignee: Ganesha Shreedhara > Priority: Major > Labels: pull-request-available > Attachments: HIVE-22670.1.patch, HIVE-22670.2.patch > > Time Spent: 20m > Remaining Estimate: 0h > > ArrayIndexOutOfBoundsException is getting thrown while decoding dictionaryIds > of a row group in parquet file with vectorization enabled. > *Exception stack trace:* > {code:java} > Caused by: java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainBinaryDictionary.decodeToBinary(PlainValuesDictionary.java:122) > at > org.apache.hadoop.hive.ql.io.parquet.vector.ParquetDataColumnReaderFactory$DefaultParquetDataColumnReader.readString(ParquetDataColumnReaderFactory.java:95) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.decodeDictionaryIds(VectorizedPrimitiveColumnReader.java:467) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.readBatch(VectorizedPrimitiveColumnReader.java:68) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353) > at > org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365) > ... 24 more{code} > > This issue seems to be caused by re-using the same dictionary column vector > while reading consecutive row groups. This looks like one of the corner case > bug which occurs for a certain distribution of dictionary/plain encoded data > while we read/populate the underlying bit packed dictionary data into a > column-vector based data structure. > Similar issue issue was reported in spark (Ref: > https://issues.apache.org/jira/browse/SPARK-16334) -- This message was sent by Atlassian Jira (v8.20.7#820007)