matanper opened a new issue, #68912: URL: https://github.com/apache/doris/issues/68912
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, arm64). master is the same: `reserve_rows` in `be/src/core/column/variant_v2/column_variant_v2.cpp:124-131` grows the VARIANT V2 value/offset/id arrays with `PODArray::reserve` (power-of-two growth), and `MemTable::_put_into_output` (`be/src/load/memtable/memtable.cpp:402-416`) copies the whole input block into a sorted output block while the input block is still alive. ### Description Load-path memory for VARIANT V2 columns is higher than it needs to be, in two places: 1. **Power-of-two column growth.** The memtable's V2 column grows one sink batch at a time. `PODArray::reserve` rounds up to the next power of two, so a large column can be up to half empty. The same applies to the scanner and sink copies. 2. **Flush sort copy.** At flush, `_put_into_output` builds the sorted output block with `add_rows` over all columns while the whole input block is still held, so the flush briefly needs about 2× the memtable size. Already-sorted columns are copied too. Measured with real rows (3 VARIANT V2 columns), MB per 4096 rows: memtable 10.9–14.7, plus 10.8 for the flush sort copy; per-load peak 31.2. ### Solution 1. For large V2 arrays (e.g. above 1 MB), allocate exactly when filling an empty column, and grow by 1.25× instead of to the next power of two (needs a `PODArray::reserve_exact`). 2. Build the flush's sorted output one column at a time and release each input column after it's copied; move columns that are already in sorted order instead of copying them. Both can sit behind mutable BE configs. ### How to Reproduce? Stream Load JSON/Arrow data with large VARIANT values (a few KB per row) into a table with VARIANT V2 columns, and track the load's mem tracker and memtable memory per batch and at flush. ### Anything Else? What it gave us (same encoded bytes and output rows), MB per 4096 rows: | Stage | Before | After | |---|---|---| | Scanner block | 10.2 | 8.4 | | Sink copy | 10.2 | 8.4 | | Memtable | 10.9–14.7 | 8.7–10.3 | | Flush sort copy (extra) | 10.8 | 7.5 (0 if already sorted) | | Per-load peak | 31.2 | 28.7 (−8%) | On a 20k-row load the memtable fell 41%; on a 240k-row load the memtable saving is about 5%. The rest of the peak is the V2 shredder at flush, which this doesn't change. The cost is a few more realloc copies (a little extra CPU) from 1.25× growth. The flush change applies to every memtable that sorts, not just V2. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
