github-actions[bot] commented on code in PR #66999:
URL: https://github.com/apache/doris/pull/66999#discussion_r3828471630
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java:
##########
@@ -179,6 +187,18 @@ public List<Split> getSplits(int numBackends) throws
UserException {
"Lance vector search requires a fixed positive dataset
version");
}
+ if (canPushDownCountStar()) {
+ // COUNT(*)/COUNT(1) with no filter is answered from Lance
metadata. Emit a single
+ // whole-dataset split carrying the logical (post-deletion) row
count so BE
+ // synthesizes that many rows instead of opening any fragment
scanner. One split is
+ // enough: the metadata lookup is O(1) and needs no parallelism.
+ long rowCount = metadata.getRowCount();
+ setPushDownCount(rowCount);
+ LanceSplit countSplit =
LanceSplit.wholeDatasetAtLatest(metadata.getDatasetUri());
Review Comment:
[P1] Keep the metadata-count fallback on the planned snapshot
`metadata` is the relation's fixed MVCC snapshot, but this factory
hard-codes version `0` (latest). In a rolling upgrade the base-sha BE ignores
`table_level_row_count` (the removed `_remaining_table_level_count = -1` path)
and opens this no-fragment split as a whole-dataset scan; the current BE can do
the same whenever its metadata-count gate declines the shortcut. A `COUNT(*)
... FOR VERSION AS OF v` can therefore read latest instead of `v` (and an
ordinary statement can drift after a concurrent commit). Please carry
`metadata.getVersion()` in a fixed-version whole-dataset/count split and cover
the fallback/time-travel case.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java:
##########
@@ -179,6 +187,18 @@ public List<Split> getSplits(int numBackends) throws
UserException {
"Lance vector search requires a fixed positive dataset
version");
}
+ if (canPushDownCountStar()) {
+ // COUNT(*)/COUNT(1) with no filter is answered from Lance
metadata. Emit a single
+ // whole-dataset split carrying the logical (post-deletion) row
count so BE
+ // synthesizes that many rows instead of opening any fragment
scanner. One split is
+ // enough: the metadata lookup is O(1) and needs no parallelism.
+ long rowCount = metadata.getRowCount();
+ setPushDownCount(rowCount);
+ LanceSplit countSplit =
LanceSplit.wholeDatasetAtLatest(metadata.getDatasetUri());
+ countSplit.setTableLevelRowCount(rowCount);
+ return Collections.singletonList(countSplit);
Review Comment:
[P2] Preserve parallelism for large metadata counts
`table_level_row_count` is not consumed as one scalar: BE's
`_read_count_batch()` keeps materializing default rows in runtime-sized batches
until all `rowCount` rows have passed through the unchanged upper `COUNT`.
Returning one range also caps scanner concurrency to one, so a large Lance
table moves this O(N) work onto a single scanner even though its former
fragment splits ran in parallel. Iceberg and Paimon distribute counts above
10,000 across `parallelExecInstanceNum * numBackends` carriers for this
execution model. Please preserve parallelism with fallback-safe fixed-version
carriers (not cloned unrestricted whole-dataset ranges, which a fallback BE
would scan repeatedly), or change the execution contract to consume the count
in O(1), and add a large-count test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
