yordan-pavlov commented on a change in pull request #692:
URL: https://github.com/apache/arrow-rs/pull/692#discussion_r688980252
##########
File path: parquet/src/arrow/arrow_array_reader.rs
##########
@@ -777,31 +829,118 @@ impl ValueDecoder for LevelValueDecoder {
&mut self,
num_values: usize,
read_bytes: &mut dyn FnMut(&[u8], usize),
- ) -> Result<usize> {
+ ) -> Result<(usize, usize)> {
let value_size = std::mem::size_of::<i16>();
let mut total_values_read = 0;
- while total_values_read < num_values {
- let values_to_read = std::cmp::min(
- num_values - total_values_read,
- self.level_value_buffer.len(),
- );
- let values_read = match self
- .level_decoder
- .get(&mut self.level_value_buffer[..values_to_read])
- {
- Ok(values_read) => values_read,
- Err(e) => return Err(e),
- };
- if values_read > 0 {
- let level_value_bytes =
- &self.level_value_buffer.to_byte_slice()[..values_read *
value_size];
- read_bytes(level_value_bytes, values_read);
- total_values_read += values_read;
- } else {
- break;
+
+ // When reading repetition levels, num_values will likely be less than
the array values
Review comment:
with these changes, how does this ArrowArrayReader work with the
`ListArrayReader`? Shouldn't it be the responsibility of the `ListArrayReader`
to understand the meaning of repetition levels and use them to assemble lists
of values from lower level / primitive arrays?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
