zhaoyudi-creator opened a new issue, #3715:
URL: https://github.com/apache/parquet-java/issues/3715
### Describe the bug, including details regarding any error messages,
version, and platform.
## Description
We intermittently encounter a `ParquetDecodingException` when reading
Parquet files written by Apache Hudi with Spark.
The issue cannot be reproduced consistently for every write, but it has
occurred repeatedly across different write jobs.
The failures have a similar call chain: while reading a dictionary-encoded
data page, parquet-java decodes a dictionary ID that is outside the range of
the current dictionary. `PlainValuesDictionary` then attempts to access the
dictionary array with that ID and throws an `ArrayIndexOutOfBoundsException`.
In the representative case below, the affected column is:
```text
optional int32 chan_second_cate_cd
```
The dictionary contains 13 entries, so the valid dictionary IDs are `0`
through `12`. However, the data page produces dictionary ID `14`, resulting in:
```text
java.lang.ArrayIndexOutOfBoundsException: Index 14 out of bounds for length
13
```
This failure occurs in row group `0`, at column value `292293` out of
`478848`, and at value `12293` out of `20000` in the current data page.
The value `14` is an internal Parquet dictionary ID, not the actual value
stored in the column.
## Environment
| Component | Version |
| --- | --- |
| Apache Hudi | `0.13.1` |
| Write engine | Apache Spark `3.4` |
| parquet-mr | `1.12.3` |
## Representative Stack Trace
<details>
<summary>Click to expand</summary>
```text
org.apache.parquet.io.ParquetDecodingException: Can not read value at 292293
in block 0 in file <sanitized-parquet-file-path>
at
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:264)
at
org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
at
org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at
org.apache.spark.sql.execution.datasources.RecordReaderIterator$$anon$1.hasNext(RecordReaderIterator.scala:61)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:125)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:101)
at org.apache.spark.scheduler.Task.run(Task.scala:140)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:565)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can't read value
in column [chan_second_cate_cd] optional int32 chan_second_cate_cd at value
292293 out of 478848, 12293 out of 20000 in currentPage. repetition level: 0,
definition level: 1
at
org.apache.parquet.column.impl.ColumnReaderBase.readValue(ColumnReaderBase.java:553)
at
org.apache.parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:30)
at
org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:439)
at
org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
at
org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
at
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:234)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 14 out of bounds
for length 13
at
org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary.decodeToInt(PlainValuesDictionary.java:251)
at
org.apache.parquet.column.values.dictionary.DictionaryValuesReader.readInteger(DictionaryValuesReader.java:108)
at
org.apache.parquet.column.impl.ColumnReaderBase$2$3.read(ColumnReaderBase.java:279)
at
org.apache.parquet.column.impl.ColumnReaderBase.readValue(ColumnReaderBase.java:533)
```
</details>
### Component(s)
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
