github-actions[bot] commented on code in PR #66530:
URL: https://github.com/apache/doris/pull/66530#discussion_r3978455088
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -501,8 +502,30 @@ public void releaseReadTransaction(String queryId) {
*/
@Override
public List<ConnectorScanRange> planScan(ConnectorSession session,
ConnectorScanRequest request) {
- return planScanInternal(session, request.getTableHandle(),
request.getColumns(),
- request.getFilter(), request.getLimit(),
request.isCountPushdown());
+ PaimonTableHandle paimonHandle = (PaimonTableHandle)
request.getTableHandle();
+ if (session == null || !session.isExternalScanTaskReuseEnabled()) {
+ return planScanInternal(session, request.getTableHandle(),
request.getColumns(),
+ request.getFilter(), request.getLimit(),
request.isCountPushdown());
+ }
+ if (paimonHandle.isSystemTable()) {
+ // System tables resolve their snapshot on the BE and carry
deferred side effects
+ // (authorized file enumeration); never reuse their planned ranges.
+ return planScanInternal(session, request.getTableHandle(),
request.getColumns(),
+ request.getFilter(), request.getLimit(),
request.isCountPushdown());
+ }
+ // Statement-scoped reuse: within one statement the identical scan
(same table, same
+ // branch/options pin, same projection, same filter, same limit, same
COUNT pushdown) plans once and
+ // every duplicated relation shares the result. The scope is NONE for
offline planning and
+ // tests, in which case the loader runs on every call. Session
variables are constant within
+ // a statement and deliberately absent from the key.
+ String memoKey = SCAN_REUSE_NAMESPACE + ":" + session.getCatalogId() +
":" + session.getQueryId();
+ Map<PaimonScanReuseKey, List<ConnectorScanRange>> scanReuse =
session.getStatementScope().computeIfAbsent(
+ memoKey, () -> new ConcurrentHashMap<>());
+ PaimonScanReuseKey reuseKey = new PaimonScanReuseKey(paimonHandle,
request);
+ return scanReuse.computeIfAbsent(reuseKey,
Review Comment:
[P1] Tie Paimon reuse to the resolved table generation. Each scan node
resolves its own transient Paimon `Table`, and `withScanOptions` preserves that
instance, but this key contains no table/schema generation. With a no-cache
catalog, or a refresh/external schema change between two aliases (which this
connector explicitly supports without a new data snapshot), both aliases can
have the same key while the first plans ranges from generation A and the second
independently builds `getScanNodeProperties` from generation B. That sends
generation A splits together with generation B serialized table,
projection/predicate schema, and schema dictionary; reuse-disabled planning
would keep both halves on generation B. Please either resolve one
statement-shared Paimon table generation for both paths or add a stable
generation token to the reuse identity, and cover two equal logical handles
backed by different schema generations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
