zhangshenghang commented on issue #10123: URL: https://github.com/apache/seatunnel/issues/10123#issuecomment-3587609186
> FYI, if you have multiple BE nodes in your SR cluster, you may encounter load imbalance issues if you only configure one BE node here. For instance, if the specified BE does not have a Tablet for this table, then the BE needs to fetch data from other BE, which will result in duplicate network transfers and disk I/O. For instance, if the BE node you manually specified goes down, the entire data reading will fail and it will not BE possible to automatically switch to other healthy BE nodes. > > Currently, Tablet routing information is dynamically obtained via the `_query_plan` API, which ensures optimal data locality and load balancing. Relevant code references: > > [seatunnel/seatunnel-connectors-v2/connector-starrocks/src/main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/client/source/StarRocksQueryPlanReadClient.java](https://github.com/apache/seatunnel/blob/1a73b93859f73d2f1d3bae8c5f781cc402455347/seatunnel-connectors-v2/connector-starrocks/src/main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/client/source/StarRocksQueryPlanReadClient.java#L151-L168) > > Lines 151 to 168 in [1a73b93](/apache/seatunnel/commit/1a73b93859f73d2f1d3bae8c5f781cc402455347) > > for (String feNode : nodeUrls) { > String url = > new StringBuilder("http://";) > .append(feNode) > .append("/api/") > .append(sourceConfig.getDatabase()) > .append("/") > .append(table) > .append("/_query_plan") > .toString(); > try { > respString = > RetryUtils.retryWithException( > () -> httpHelper.doHttpPost(url, getQueryPlanHttpHeader(), body), > retryMaterial); > if (StringUtils.isNoneEmpty(respString)) { > return JsonUtils.parseObject(respString, QueryPlan.class); > } > [seatunnel/seatunnel-connectors-v2/connector-starrocks/src/main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/client/source/StarRocksQueryPlanReadClient.java](https://github.com/apache/seatunnel/blob/1a73b93859f73d2f1d3bae8c5f781cc402455347/seatunnel-connectors-v2/connector-starrocks/src/main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/client/source/StarRocksQueryPlanReadClient.java#L126-L137) > > Lines 126 to 137 in [1a73b93](/apache/seatunnel/commit/1a73b93859f73d2f1d3bae8c5f781cc402455347) > > for (String beNode : routingList.getRoutings()) { > if (!beXTablets.containsKey(beNode)) { > beXTablets.put(beNode, new ArrayList<>()); > candidateBe = beNode; > break; > } > if (beXTablets.get(beNode).size() < tabletCount) { > candidateBe = beNode; > tabletCount = beXTablets.get(beNode).size(); > } > } > beXTablets.get(candidateBe).add(Long.valueOf(tabletId)); +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
