github-actions[bot] commented on code in PR #68726:
URL: https://github.com/apache/doris/pull/68726#discussion_r4236302226
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java:
##########
@@ -234,11 +245,8 @@ public Multimap<Backend, Split>
computeScanRangeAssignment(List<Split> splits) t
Collections.shuffle(splits, new Random(FIXED_SHUFFLE_SEED));
List<Split> remainingSplits;
+ Map<String, List<Backend>> activeBackendMap = activeBackendMap();
Review Comment:
[P2] Avoid rebuilding the backend host map for every streamed split.
`PluginDrivenScanNode.streamSplits` calls `SplitAssignment.addToQueue` with one
split at a time, which invokes this method under `assignLock`. The new
`activeBackendMap()` groups every candidate backend on each call, even for
remotely accessible splits with no hosts that never use the map. A million-file
scan therefore pays for a million additional backend traversals and map/list
allocations; the prior map was built once during initialization. Cache or
refresh it when `replaceBackendOrder` changes the candidates, or build it only
when a split needs host matching.
##########
fe/fe-core/src/main/java/org/apache/doris/load/StreamLoadHandler.java:
##########
@@ -241,9 +241,17 @@ private void setDbAndTable() throws UserException,
MetaNotFoundException {
if (result != null) {
OlapTable olapTable = tables.get(0);
- result.setDbId(db.getId());
- result.setTableId(olapTable.getId());
- result.setBaseSchemaVersion(olapTable.getBaseSchemaVersion());
+ if (!olapTable.tryReadLock(timeoutMs, TimeUnit.MILLISECONDS)) {
+ throw new UserException(
+ "get table read lock timeout, database=" +
request.getDb() + ",table=" + olapTable.getName());
+ }
+ try {
+ result.setDbId(db.getId());
+ result.setTableId(olapTable.getId());
+ result.setBaseSchemaVersion(olapTable.getBaseSchemaVersion());
Review Comment:
[P2] Cover the `load_sql` stream-load path with the same schema-version
lock. `FrontendServiceImpl.streamLoadPut` bypasses `setDbAndTable()` for
requests with `load_sql` and calls `httpStreamPutImpl`, which reads
`httpStreamParams.getTable().getBaseSchemaVersion()` after the insert planner
has released its table read lock. During `SchemaChangeJobV2.onFinished`, the
old base-index metadata is removed before `baseIndexId` is switched under the
table write lock, so that unlocked read can still dereference null and fail the
load. Capture the HTTP result metadata while holding the table read lock too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
