github-actions[bot] commented on code in PR #66089:
URL: https://github.com/apache/doris/pull/66089#discussion_r3663568078
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/kafka/KafkaUtil.java:
##########
@@ -329,6 +311,53 @@ private static InternalService.PProxyResult
getInfoRequest(InternalService.PProx
}
}
+ static List<Long> getAvailableBackendIdsForMetaRequest(
+ List<Long> candidateBackendIds, Set<Long> failedBeIds) {
+ List<Long> backendIds = new ArrayList<>();
+ for (Long beId : candidateBackendIds) {
+ Backend backend = Env.getCurrentSystemInfo().getBackend(beId);
+ if (isBackendAvailableForMetaRequest(backend)
+ && !failedBeIds.contains(beId)
+ &&
!Env.getCurrentEnv().getRoutineLoadManager().isInBlacklist(beId)) {
+ backendIds.add(beId);
+ }
+ }
+ // If there are no available backends, utilize the blacklist.
+ // Special scenarios include:
+ // 1. A specific job that connects to Kafka may time out for topic
config or network error,
+ // leaving only one backend operational.
+ // 2. If that sole backend is decommissioned, the aliveBackends list
becomes empty.
+ // Hence, in such cases, it's essential to rely on the blacklist to
obtain meta information.
+ if (backendIds.isEmpty()) {
+ Map<Long, Long> blacklist =
Env.getCurrentEnv().getRoutineLoadManager().getBlacklist();
+ for (Long beId : new ArrayList<>(blacklist.keySet())) {
+ Backend backend = Env.getCurrentSystemInfo().getBackend(beId);
Review Comment:
[P2] Keep fallback work scoped to this compute group
Now that fallback selection is compute-group scoped, this still scans every
entry in the global routine-load blacklist and resolves foreign-group backends
before checking membership. More importantly, the added
`candidateBackendIds.contains(beId)` is linear because the candidates are a
`List`, turning the inherited `O(B)` scan into `O(B*C)` for `B` global
blacklist entries and `C` backends in this group; retries and concurrent job
refreshes can repeat that work. Iterate candidate IDs and query blacklist
membership (or build a candidate set and check it before `getBackend`) so
selection is `O(C)` or `O(B+C)`; keep any required stale-entry cleanup
separate, and update the test not to require a foreign-group lookup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
