924060929 opened a new pull request, #70247: URL: https://github.com/apache/doris/pull/70247
### Background Partitioned Hive Batch Mode resolves partition objects from Hive Metastore (HMS), lists their directories and streams file splits to the existing bounded split queues. Directory work uses small partition groups so scanning can start early and producers can yield fairly. HMS already has a separate catalog property limiting partition names per RPC. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Performance Problem Summary: The partitioned streaming path also used the small directory-task size for HMS requests. A cold scan of 1000 partitions therefore made 63 partition-object RPCs with a directory batch size of 16. This PR decouples metadata acquisition from directory work: one query-local producer fetches a small first HMS batch, then bounded larger batches, while existing split workers claim small directory batches independently. The optional connector SPI and readiness handling keep this coordination out of connector-specific branches in fe-core; most added lines are tests. Existing users get fewer cold HMS requests without changing session defaults or BE code. ### Details - **Change map:** 555 changed lines in production, 886 changed lines in tests, 0 changed lines in generated, 10 changed lines in docs, 4 changed lines in build/API baselines. Hive producer/source integration (394 changed lines) separates batching and owns cancellation; the HMS cache-only probe (30) preserves warm access without task admission; the optional scan SPI (27) and engine adapter/readiness handling (104) release workers while metadata is pending; tests (886) cover the new state transitions and existing fallbacks. The connector guide documents the source-readiness contract. - **Why it was slow:** directory batches of 16 became remote metadata batches of 16. In the cold single full-scan Profile, `TransportInvocations=63`, `TransportItems=1000` and `LargestBatchSize=16`. - **Optimization:** fetch `min(16, hive.hms_partitions_batch_size_per_rpc)` names first, then use that existing positive RPC cap (default 5000). One ready result list is shared by directory readers, each claiming at most the existing directory-task batch size. Claiming the last slice permits the next metadata request, overlapping it with directory work. Cache hits are consumed directly without metadata-task admission. Cold loads use the existing METADATA scheduler; readers use the existing SPLIT scheduler and resume through readiness callbacks, never waiting for children while occupying a worker. The cold single full-scan Profile now reports two partition-object calls, for 16 and 984 names. - **Results:** complete SQL latency is shown below, including warm regressions. Cold single full scan improves by 65.6%, and the cold four-query full-scan wave improves by 78.0%. Cold LIMIT improves by 51.5% for one query and 33.2% for a four-query wave. The full matrix and the separate warm-only control are distinct experiments; no old prototype result is substituted into either. - **Applicability and cost:** this targets partitioned Hive scans with HMS partition-cache misses. It does not make a single directory Listing incremental. Buffered metadata is bounded by one configured HMS batch plus the small slices held by the bounded reader set; existing split-queue backpressure remains in effect. A next metadata batch may already be in flight when a LIMIT query stops. Cancellation drops buffered/late results, removes queued admission and stops subsequent requests; an SDK call already running may finish. The new coordination has bookkeeping/wakeup costs: the warm-only four-query full scan was 6.3% slower, while the complete cold/warm matrix observed 17.2%. A unique cause for the full elapsed-time difference has not been established; this PR does not claim zero warm-cache overhead. - **Behavior change:** partition-object RPC grouping changes, while directory batch size, split queue capacity, split batch size, pruning and query-pinned table handles retain their existing meanings. The optional SPI keeps connectors that do not opt in on their existing paths. Connector API version changes from 29 to 30; deploy the rebuilt matching connector bundle with FE. There is no FE/BE wire-format change. #### Complete SQL performance Same ARM64 Docker host; one engine at a time; each engine has 6 CPUs and 14 GiB, FE heap 4 GiB and BE memory limit 4 GiB. The table has 1000 partitions, 10 small Parquet files per partition and five rows per file: 10000 files and 50000 rows. Both Doris variants use the identical ARM64 Release BE binary. SR is 4.1.4 with its ARM64 Release BE. Both Doris variants use `enable_external_table_batch_mode=true`, `num_partitions_in_batch_mode=1`, `split_batch_size=8`, `split_queue_capacity=1024`, `split_partition_batch_size=16` and `split_producer_parallelism=0`. SQL/result caching and additional statistics collection are disabled. Cold/warm refer to FE metadata/partition/file-listing caches, not JVM or physical object-store/data caches. Cache preparation drains prior work, retains the same connector and clients, clears the relevant FE caches for cold samples and performs a full scan before warm samples. All 30 diagnostic full-scan Profiles verify 10000 footer hits and zero new footer reads/writes or data-page misses/writes. LIMIT can still have a few data-page misses. The order is before/after/SR/SR/after/before. Each cell has 40 single-query samples or 12 synchronized four-query waves. Values are medians with Profile disabled; a wave measures until all four queries finish. Negative change means faster. | Scenario | Before (ms) | After (ms) | SR 4.1.4 (ms) | After vs before | |---|---:|---:|---:|---:| | Single, cold, LIMIT 10 | 62.09 | 30.11 | 62.16 | -51.5% | | Single, cold, full scan | 834.87 | 287.02 | 278.19 | -65.6% | | Single, warm, LIMIT 10 | 8.72 | 10.86 | 15.28 | +24.5% | | Single, warm, full scan | 87.00 | 93.68 | 173.73 | +7.7% | | Four-query wave, cold, LIMIT 10 | 93.76 | 62.63 | 114.22 | -33.2% | | Four-query wave, cold, full scan | 2316.86 | 510.05 | 778.97 | -78.0% | | Four-query wave, warm, LIMIT 10 | 15.08 | 16.01 | 23.74 | +6.1% | | Four-query wave, warm, full scan | 227.91 | 267.04 | 595.46 | +17.2% | Before FE is `0eb5131db62`, which already contains the earlier split optimizations; the measured PR implementation is `330c651e1ab` on `315c2e00331`. This is the recorded comparison from this investigation, rather than a baseline predating all Batch Mode work. It is not an exact parent-commit benchmark: the two FE bases also differ by intervening master changes. Final boundary-test cleanup only changes overflow-safe batch-end arithmetic and removes an unused source field; the batch sizes in this matrix are unaffected. SQLs: ```sql SELECT * FROM cold_hms.cold_compare_20261104.hive_1000_partitions_10_files LIMIT 10; SELECT COUNT(*), SUM(id) FROM cold_hms.cold_compare_20261104.hive_1000_partitions_10_files; ``` #### Separate warm-only verification To check whether alternating cold invalidation was hiding a warm-cache cost, run after/before/before/after without clearing the FE caches. Each measured operation still has a preceding full scan. There are 60 samples per single-query operation and 30 four-query waves per operation and variant; these numbers do not replace the matrix above. | Warm-only scenario | Before (ms) | After (ms) | Change | |---|---:|---:|---:| | Single, LIMIT 10 | 9.30 | 8.96 | -3.6% | | Single, full scan | 88.02 | 87.26 | -0.9% | | Four-query wave, LIMIT 10 | 12.20 | 13.16 | +7.8% | | Four-query wave, full scan | 230.61 | 245.16 | +6.3% | #### Final built-package verification After the overflow-boundary cleanup, an additional package check ran 10 single-query samples and four four-query waves per case. It has no fresh before/SR control, so its timings describe that verification run separately from the controlled matrix. The cold results also show the run-to-run spread and should not be read as a guaranteed latency reduction. | Final-package check | Single LIMIT (ms) | Single full scan (ms) | Four-query LIMIT wave (ms) | Four-query full-scan wave (ms) | |---|---:|---:|---:|---:| | Cold FE caches | 54.09 | 366.93 | 76.06 | 648.72 | | Warm FE caches | 11.31 | 95.40 | 15.74 | 275.32 | ### Testing - Local ARM64 Docker: connector SPI 85 tests, HMS 184 tests, Hive 572 tests and 71 focused fe-core tests; zero failures, errors or skipped tests. - Connector test recipes: `mvn -f fe/pom.xml -pl :fe-connector-spi -am test '-Dtest=org.apache.doris.connector.spi.*Test' -DfailIfNoTests=false`; `mvn -f fe/pom.xml -pl :fe-connector-hive -am package '-Dtest=org.apache.doris.connector.hive.*Test,org.apache.doris.connector.hms.*Test' -DfailIfNoTests=false`. - Engine tests: `./run-fe-ut.sh --run SplitProducerTest,PluginDrivenScanNodePartitionBatchTest,PluginDrivenScanNodeBatchModeTest,PluginDrivenScanNodeHiveStreamingTest`. - Added deterministic coverage for: first-small/subsequent-large batches, the configured cap, one-list metadata backpressure, shared readers without duplicates, warm cache access during an in-flight cache load, invalidation, pinned pruning views, dropped partitions, full-scheduler admission/retry, inline callbacks, admission failure, cancellation/late SDK return, cleanup, exception propagation and engine-level yield/resume. - Self-review reproduced a false-EOF race: metadata could complete between an empty read and its readiness check. `completedMetadataDependencyMustNotLookLikeEndOfInput` fails before retaining the completed dependency and passes after the fix, checking all 17 ranges. The readiness contract is documented in SPI and the connector guide. - `maximumRpcLimitDoesNotOverflowAfterTheFirstBatch` verifies that a valid `Integer.MAX_VALUE` HMS cap still fetches the final partition after the first 16; it fails before the overflow-safe end calculation. - Complete FE build via `./build.sh --fe -j2`, including Checkstyle; build/test containers capped at 2 CPUs. `git diff --check` passes. No BE changes or rebuild. - Manual complete-SQL matrix: 1056 formal SQLs plus independent diagnostics; all full scans check `COUNT(*)=50000` and `SUM(id)=1249975000`, all LIMIT queries check ten rows and derived values. Separate warm-only control adds 720 formal SQLs. Cold/warm timing and Profiles are recorded separately. After the final cleanup, the rebuilt package passes another 104 complete SQLs and 20 independent diagnostic Profiles, covering cold/warm LIMIT/full scan and single/four-query execution. Full-scan Profiles still verify 10000 footer hits, zero new footer reads/data-page misses and two cold partition-object RPCs (zero warm RPCs). ### Notes for Reviewers Start with `HivePartitionBatchLoader` and the shared-source hook in `HiveScanPlanProvider`, then the generic `PluginDrivenScanNode` adapter and `SplitProducer` readiness callback. Source reads and closes remain serial per producer. The loader's short monitor protects cursors, current/ready results and failure state; RPCs, scheduler calls and future completions run outside it. Each reader copies only its claimed reference slice so a slow reader does not retain a fully consumed HMS list. The optional SPI avoids a Hive-specific path in the engine. ### Release note Reduce cold Hive partition planning latency by fetching partition metadata in bounded HMS batches independently of the smaller directory tasks, while retaining early split delivery, cancellation and bounded split queues. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Implementation, tests, self-review and this description. - Human review: The author directed the design and benchmark scenarios. Agent self-review is complete; human line-by-line review is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
