dossett commented on PR #3867:
URL: https://github.com/apache/parquet-java/pull/3867#issuecomment-6445182346

   Here's how I understand the issue (full disclosure: arrived at with the help 
of a conversation with codex, but these words are mine).  The v1 specification 
[requires](https://github.com/apache/parquet-java/blob/master/parquet-format-structures/src/main/thrift/parquet.thrift#L712-L716)
 `definition_level_encoding` and `repetition_level_encoding` but sometimes they 
aren't actually relevant.  This change first looks to the schema to determine 
if they're relevant and *then* uses them as opposed to using them by default, 
which can lead to issues like this.
   
   Based on that understanding I am +1 (non-binding).
   
   ---
   
   Here's a minimum reproduction of the issue furnished by codex.
   
   ```java
   @Test
   public void requiredDoubleIgnoresAbsentRepetitionLevelEncoding() throws 
Exception {
     ColumnDescriptor column = MessageTypeParser
         .parseMessageType("message test { required double foo; }")
         .getColumns()
         .get(0);
   
     byte[] data = ByteBuffer.allocate(Double.BYTES)
         .order(ByteOrder.LITTLE_ENDIAN)
         .putDouble(12.5)
         .array();
   
     DataPageV1 page = new DataPageV1(
         BytesInput.from(data),
         1,
         data.length,
         new DoubleStatistics(),
         Encoding.PLAIN, // repetition levels: absent despite this label
         Encoding.RLE,   // definition levels: also absent
         Encoding.PLAIN);
   
     ColumnReader reader = new ColumnReaderImpl(
         column,
         new MemPageReader(1, List.<DataPage>of(page).iterator(), null),
         new PrimitiveConverter() {},
         VersionParser.parse(Version.FULL_VERSION));
   
     assertThat(reader.getDouble()).isEqualTo(12.5);
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to