dossett commented on PR #3867: URL: https://github.com/apache/parquet-java/pull/3867#issuecomment-6445182346
Here's how I understand the issue (full disclosure: arrived at with the help of a conversation with codex, but these words are mine). The v1 specification [requires](https://github.com/apache/parquet-java/blob/master/parquet-format-structures/src/main/thrift/parquet.thrift#L712-L716) `definition_level_encoding` and `repetition_level_encoding` but sometimes they aren't actually relevant. This change first looks to the schema to determine if they're relevant and *then* uses them as opposed to using them by default, which can lead to issues like this. Based on that understanding I am +1 (non-binding). --- Here's a minimum reproduction of the issue furnished by codex. ```java @Test public void requiredDoubleIgnoresAbsentRepetitionLevelEncoding() throws Exception { ColumnDescriptor column = MessageTypeParser .parseMessageType("message test { required double foo; }") .getColumns() .get(0); byte[] data = ByteBuffer.allocate(Double.BYTES) .order(ByteOrder.LITTLE_ENDIAN) .putDouble(12.5) .array(); DataPageV1 page = new DataPageV1( BytesInput.from(data), 1, data.length, new DoubleStatistics(), Encoding.PLAIN, // repetition levels: absent despite this label Encoding.RLE, // definition levels: also absent Encoding.PLAIN); ColumnReader reader = new ColumnReaderImpl( column, new MemPageReader(1, List.<DataPage>of(page).iterator(), null), new PrimitiveConverter() {}, VersionParser.parse(Version.FULL_VERSION)); assertThat(reader.getDouble()).isEqualTo(12.5); } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
