JonSnow1807 commented on code in PR #12122:
URL: https://github.com/apache/arrow-rs/pull/12122#discussion_r4554272569
##########
parquet/src/column/writer/encoder.rs:
##########
@@ -84,7 +84,16 @@ pub trait ColumnValueEncoder {
fn write(&mut self, values: &Self::Values, offset: usize, len: usize) ->
Result<()>;
/// Write the values at the indexes in `indices` to this
[`ColumnValueEncoder`]
- fn write_gather(&mut self, values: &Self::Values, indices: &[usize]) ->
Result<()>;
+ ///
+ /// `num_values_in_batch` is the total number of selected non-null values
in the
+ /// enclosing batch, including any other mini-batches. Arrow callers
provide an
+ /// exact selection with no unused trailing indices.
Review Comment:
`other mini-batches`: the column writer can split one write into several
`write_gather` calls (for example by `write_batch_size` or the page size
limit), so this is the non-null count of the whole write, not of one call.
Unused trailing indices: the only caller passes
`ArrayLevels::non_null_indices()`, which has one entry per non-null value, so
it can’t happen today. If a caller did pass extra indices, they wouldn’t be
written and the count would be too high, so the writer might allocate a cache
it doesn’t need, as before this PR. The written data is the same either way,
since the cache only remembers each value’s interned index.
I reworded the doc comment in f28ba6a.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
