924060929 commented on code in PR #70311:
URL: https://github.com/apache/doris/pull/70311#discussion_r4548006045
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##########
@@ -263,18 +268,54 @@ public void startSplit(int numBackends) throws
UserException {
}
});
}
- if (prunedPartitions.isEmpty()) {
+ int partitionCount = partitionInit ? prunedPartitions.size() : (int)
selectedPartitionNum;
+ if (partitionCount == 0) {
splitAssignment.finishSchedule();
return;
}
HiveExternalMetaCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
.hive(hmsTable.getCatalog().getId());
Executor executor = getSplitTaskExecutor();
- String broker = hmsTable.getCatalog().bindBrokerName();
+ HMSExternalCatalog catalog = (HMSExternalCatalog)
hmsTable.getCatalog();
+ String broker = catalog.bindBrokerName();
// Without the file cache, the eager path listed partitions
concurrently. Keep that overlap.
boolean parallelListing = !fileCountBatch ||
Config.max_external_file_cache_num <= 0;
+ int partitionsPerProducer = partitionInit ? 1 :
sessionVariable.splitPartitionBatchSize;
int producers = parallelListing
- ? Math.min(prunedPartitions.size(),
Config.max_external_cache_loader_thread_pool_size) : 1;
+ ? Math.min((partitionCount - 1) / partitionsPerProducer + 1,
Review Comment:
Fixed in `49c70eddea4b`. Directory-source concurrency now uses the selected
partition count and the existing pool limit independently of the claim size.
Each source receives a captured claim cap of min(configured batch size,
ceil(partitions / producers)), so the first source cannot monopolize a small
selection; sources still claim work dynamically. COUNT lookahead uses the same
corrected producer count and keeps one-partition claims.
Two latch tests fail before the fix for normal and COUNT scans on the real
lazy path. The repaired tests cover 2/8/16/17 partitions, cold/warm metadata
and both policies, require two listings to overlap on a shared two-thread
metadata/split executor, and verify exact-once results. Existing single-worker,
cancellation, backpressure and large-batch tests also pass. The HMS request
batching policy is unchanged.
Validation: 142 tests across nine related classes, zero
failures/errors/skips; standard ARM64 FE build and Checkstyle passed with a
hard 2 CPU / 8 GiB limit. No end-to-end timing claim is made from the latch
tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
