github-actions[bot] commented on code in PR #65674:
URL: https://github.com/apache/doris/pull/65674#discussion_r3592476693
##########
be/src/format_v2/parquet/reader/parquet_leaf_reader.h:
##########
@@ -63,6 +65,23 @@ struct ParquetNestedScalarBatch {
MutableColumnPtr values_column;
bool empty() const { return levels_written == 0; }
+
+ // Reset logical contents without replacing the vectors/Column object. A
reader repeatedly
+ // sees the same physical leaf type, so retaining capacity is safe and
avoids one allocation
+ // family per nested batch. A levels-only read leaves values_column empty
but may keep the
+ // reusable object allocated.
+ void reset() {
+ records_read = 0;
+ levels_written = 0;
+ value_slot_definition_level = 0;
+ value_slot_repetition_level = std::numeric_limits<int16_t>::max();
+ def_levels.clear();
Review Comment:
[P1] Bound all nested scratch after outlier batches
`reset()` clears logical sizes but retains the backing allocations of
`def_levels`, `rep_levels`, `value_indices`, and `values_column`; this PR
likewise makes nested null/conversion vectors and Map level indices Row
Group-lived. For a repeated STRING/BINARY leaf, one parent can produce far more
child entries than the parent-row batch cap, and these vector/PODArray/column
`clear()` calls retain capacity. The reader tree persists until
`reset_current_row_group()`, so staggered outliers in different batches or
leaves accumulate instead of releasing the earlier scratch before later
materialization, which can turn legal input into a memory-limit failure. This
is distinct from the resolved Arrow-chunk thread: those shared owners are
released, but these Doris-owned allocations are not. Please preserve
normal-size reuse while releasing/recreating all outlier nested value, null,
conversion, level, and index scratch above a bounded high-water threshold, with
a test that proves the retained
allocation drops after an oversized repeated-value batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
