jaideeppyne commented on code in PR #11136:
URL: https://github.com/apache/arrow-rs/pull/11136#discussion_r4139994399
##########
parquet/src/encodings/decoding.rs:
##########
@@ -2190,6 +2190,35 @@ mod tests {
test_byte_stream_split_decode::<Int64Type>(data, -1);
}
+ #[test]
+ fn test_byte_stream_split_uses_encoded_value_count() {
+ let mut decoder = ByteStreamSplitDecoder::<DoubleType>::new();
+
+ decoder.set_data(Bytes::from(vec![0; 64]), 63).unwrap();
Review Comment:
The decoder now returns an error when `data.len()` is smaller than
`num_values * T::get_type_size()`, instead of ignoring `num_values` and
decoding the encoded count. Commit 87bdd58e18c4e6e94ba21cf6c6f09078dc9f0545.
A short buffer (64 bytes with a declared count of 63 f64 values) now errors.
Exact-size input still decodes. Oversized buffers keep the existing "expected
at most" error.
Data page v1 does not report a non-null count, and BYTE_STREAM_SPLIT stores
only non-null values, so the column reader passes the number of complete values
in the buffer in that case. An explicit count (v2) is still checked and errors
when the buffer is short.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
