CalvinKirs opened a new pull request, #67186:
URL: https://github.com/apache/doris/pull/67186
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Hive tables with very large partition counts could issue one HMS RPC per
partition or one unbounded getPartitionsByNames request. The former causes
excessive serial RPC latency, while the latter exposes Thrift message-size, HMS
timeout, and memory-peak risks. Query, MTMV, statistics, and display paths also
lacked shared cancellation and observability semantics.
This PR adds one HMS-owned bounded partition batch loader with:
- configurable chunk size and adaptive halving for classified Thrift
failures;
- strict missing, duplicate, and unexpected-result validation;
- shared deadline and cancellation control;
- physical HMS RPC metrics and Query Profile events;
- per-partition single-flight cache-miss coalescing with bounded FIFO load
slots.
Query, MTMV, statistics, display, and regular write partition-object paths
reuse the shared implementation. This PR does not change split-assignment
timeout or lifecycle behavior.
### Release note
Hive metastore partition-object access is now bounded, cancellable,
observable, and shared across query, MTMV, statistics, display, and write paths.
### Check List (For Author)
- Test: Unit Test
- CachingHmsClientTest: 33 tests passed
- HmsPartitionBatchLoaderTest: 20 tests passed
- ThriftHmsClientPoolControlTest: 6 tests passed
- Hive metadata, transaction, MTMV, MVCC, and connector profile tests
passed
- ./build.sh --fe -j 1 passed
- The 120,000-partition end-to-end benchmark has not yet been rerun
- Behavior changed: Yes. Large Hive partition-object reads use bounded HMS
batches with adaptive fallback and strict integrity checks.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
