xylaaaaa opened a new pull request, #65478:
URL: https://github.com/apache/doris/pull/65478
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: ORC file scanner v2 used a thin ORC input stream that
issued reads directly through the underlying file reader. It did not implement
the v1 ORC stripe-level stream collection and small-range merge path, so remote
object storage scans could regress to many small stream reads for wide ORC
files, multi-stripe files, and lazy or predicate-driven scans. This change adds
a v2-native ORC input stream that implements beforeReadStripe(), builds
selected stream ranges, merges adjacent small streams with Doris PrefetchRange
policy, and serves arbitrary repeated or backward stream reads from an
immutable merged-range cache. Large streams and unmerged single streams
continue to read directly. The reader is wired into ORC v2 without changing v1
or falling back to v1.
### Release note
Improve ORC file scanner v2 remote read performance by merging selected
small stripe stream reads.
### Check List (For Author)
- Test: Unit Test / Manual test / Static analysis
- Unit Test: ./run-be-ut.sh --run
--filter='OrcFileInputStreamTest.*:NewOrcReaderTest.StripePrefetchPublishesMergedReadProfile'
-j 8 (passed)
- Manual test: git diff --check (passed)
- Manual test: build-support/check-format.sh (passed)
- Manual test: ninja -C be/ut_build_ASAN
src/format/CMakeFiles/Format.dir/__/format_v2/orc/orc_file_input_stream.cpp.o
test/CMakeFiles/doris_be_test.dir/format_v2/orc/orc_file_input_stream_test.cpp.o
(passed)
- Static analysis: build-support/run-clang-tidy.sh --build-dir
be/ut_build_ASAN --files ... (attempted; own signed/unsigned diagnostic fixed;
remaining failures are pre-existing/toolchain diagnostics in
be/src/core/types.h and be/src/io/fs/buffered_reader.h)
- Behavior changed: Yes. ORC v2 now merges selected small stripe streams and
reads each merged cluster through a cached span instead of issuing one remote
read per small stream request.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
