xylaaaaa opened a new pull request, #65589:
URL: https://github.com/apache/doris/pull/65589
### What problem does this PR solve?
Issue Number: None
Related PR: #65560
Problem Summary:
FileScannerV2 reads physical columns into a file-local `Block` and then maps
them into the table output `Block`. For a direct mapping, both blocks
temporarily own the same `ColumnPtr`. The old finalization path calls
`IColumn::mutate()` while the file-local owner is still alive, so COW
recursively clones the column tree. For a large MAP this duplicates key, value,
and offset payloads even though no schema conversion is needed.
A normal-read A/B reproduction using `large_string_map.brotli.parquet`
showed that applying only this ownership fix reduced retained RSS from 8.74 GiB
to 6.68 GiB, reduced process HWM from 8.74 GiB to 8.15 GiB, and reduced runtime
from 56.4s to 46.2s.
The fix first materializes immutable output mappings, keeps file-local
columns alive while table virtual columns are evaluated, then releases the
file-local owners before requesting mutable output columns. Direct mappings
therefore transfer their existing payload instead of deep-cloning it.
This PR intentionally does not add COUNT(column) pushdown or change
Parquet/ORC decoding.
### Release note
Reduce memory use when FileScannerV2 finalizes directly mapped external file
columns.
### Check List (For Author)
- Test: Manual test
- A/B normal MAP read verified payload pointer reuse and lower RSS
- Added STRING payload ownership unit test
- Added MAP key/value/offset ownership unit test
- New clean-worktree unit-test build is pending
- Behavior changed: No. Output values are unchanged; direct mappings avoid
an unnecessary COW clone.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
