github-actions[bot] commented on code in PR #68227:
URL: https://github.com/apache/doris/pull/68227#discussion_r4056515786
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownFilterIntoSchemaScan.java:
##########
@@ -48,7 +48,7 @@
public class PushDownFilterIntoSchemaScan extends OneRewriteRuleFactory {
public static ImmutableSet<String> SUPPOPRT_FRONTEND_CONJUNCTS_TABLES =
- ImmutableSet.of("view_dependency", "sql_block_rule_status");
+ ImmutableSet.of("view_dependency", "sql_block_rule_status",
"table_stream_consumption");
Review Comment:
[P1] Preserve non-movable expressions when pruning
Separate from the existing transport failure, this opt-in can change the row
domain below a retained `NoneMovableFunction`. With existing stream rows, the
rewrite turns `Filter(assert_true(false, 'bad'), DB_NAME = '__missing__') ->
SchemaScan` into the same filter above `SchemaScan(frontend = DB_NAME =
'__missing__')`; the scan now emits no rows, so the assertion never runs and
its required error is suppressed. Rejecting only the assertion from the
forwarded list is insufficient because the supported sibling causes the loss.
Please fence the entire frontend pushdown when the containing filter has a
non-movable expression, and add an end-to-end error-preservation test.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/stream/TableStreamManager.java:
##########
@@ -450,8 +471,10 @@ private void
fillCloudStreamConsumptionValuesMetadataResult(List<TRow> dataBatch
}
try {
Map<Long, String> partitionNames = new
LinkedHashMap<>();
- baseTable.getPartitions().forEach(partition ->
- partitionNames.put(partition.getId(),
partition.getName()));
+ baseTable.getPartitions().stream()
+ .filter(partition ->
selector.test(db.get().getFullName(), stream.getName(),
+ stream.getId(), partition.getName()))
Review Comment:
[P1] Move selector evaluation out of the table locks
For a pushed `UNIT IN (...)` predicate, this callback does much more than
inspect the partition name: it builds a value map, rewrites the full expression
tree, and `FoldConstantRuleOnFE.visitInPredicate` linearly scans every option.
Because this runs once per partition while both the stream and base-table read
locks are held (the local path has the same nesting), a table with P partitions
and M options performs O(P*M) work while excluding writers. Stream
commit/publish includes the stream table in a timed write-lock set, so a large
metadata query can delay or time out commits as well as partition DDL. Please
snapshot the required stream/partition/offset fields under the locks and
evaluate the selector after releasing them (or otherwise precompile the
predicate so only bounded lookup remains under lock), with a
contention/lock-state test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
