xiaoh1024 opened a new issue, #3889: URL: https://github.com/apache/parquet-java/issues/3889
Component(s): Core ## Description A Spark read failed with `StackOverflowError` in `org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup()` while reading a Parquet file containing many consecutive empty row groups. The failure investigation recorded 3,263 row groups, 3,199 of them empty, with a longest consecutive run of 2,500 empty groups. The stack trace repeatedly contained `ParquetFileReader.readNextRowGroup`. The expected behavior is to skip empty row groups and return the next non-empty group, or `null` at EOF, without growing the call stack with the number of skipped groups. ## Cause and affected paths In [readNextRowGroup](https://github.com/apache/parquet-java/blob/a449253a458a25a0ad122b250e9964072798527e/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L1137), an empty group raises `ParquetEmptyBlockException`. The catch block advances to the next group and recursively calls the same method: ```java } catch (ParquetEmptyBlockException e) { LOG.warn("Read empty block at index {} from {}", currentBlock, getFile()); advanceToNextBlock(); return readNextRowGroup(); } ``` Each consecutive empty group adds another stack frame. Java does not eliminate these recursive calls, so a sufficiently long run can exhaust the thread stack. The failure threshold depends on the JVM and thread stack size. [readNextFilteredRowGroup](https://github.com/apache/parquet-java/blob/a449253a458a25a0ad122b250e9964072798527e/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L1526) also calls itself recursively when a group is empty or its column-index-filtered row ranges contain no matching rows. The latter path also grows the call stack with each skipped group, even when the groups are non-empty. The recursive empty-group handling is present in the inspected 1.13.1, 1.15.2, 1.16.0, 1.17.1, and 1.18.1 tags and the master revision linked above. ## Suggested fix Replace recursive skipping with iteration in both sequential read methods, preserving group advancement, dictionary-reader updates, and buffer ownership. Regression cases should cover many consecutive empty groups before a non-empty group, all-empty groups, trailing empty groups, and many consecutive groups with no matching column-index-filtered rows. This follows [#2728](https://github.com/apache/parquet-java/issues/2728) (originally PARQUET-2219) and [#1018](https://github.com/apache/parquet-java/pull/1018), which added empty-group skipping. The discussion in #2728 notes that the specification does not prohibit zero-length row groups. The existing `TestParquetReaderEmptyBlock` covers up to two consecutive empty groups. I used an AI agent to help analyze the source and draft this issue. I reviewed the final text. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
