gabotechs opened a new issue, #25623: URL: https://github.com/apache/datafusion/issues/25623
## Describe the bug Replacing a month-key literal with an equivalent scalar subquery changes the filter estimate from 31 to 14,610 rows. Both return the same 29 rows. ## To Reproduce From the repository root, with [PR #25570](https://github.com/apache/datafusion/pull/25570) applied: ```sh cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench repro_dir=$(mktemp -d) mkdir -p "$repro_dir/data" curl --fail --location \ https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/date_dim.parquet \ -o "$repro_dir/data/date_dim.parquet" cat > "$repro_dir/repro.sql" <<'SQL' SET datafusion.execution.target_partitions = 1; SET datafusion.optimizer.enable_dynamic_filter_pushdown = false; SELECT d_date_sk FROM date_dim WHERE d_month_seq = 1201; SELECT d_date_sk FROM date_dim WHERE d_month_seq = ( SELECT MAX(d_month_seq) FROM date_dim WHERE d_year = 2000 AND d_moy = 2 ); SQL target/ci/dfbench statistics \ --path "$repro_dir/data" --query_path "$repro_dir/repro.sql" ``` Observed at [this revision](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67) with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the empty SET reports. | Predicate | FilterExec node | Estimated rows | Actual rows | | ------------------------ | --------------- | -------------: | ----------: | | Literal equality | `0` | 31 | 29 | | Scalar-subquery equality | `0.0` | 14,610 | 29 | ## Expected behavior When the scalar value is unavailable during planning, use available equality-selectivity information, such as the left-hand NDV or a suitable range-based estimate, instead of an unconditional 20%. This should not require executing arbitrary subqueries during planning. ## Additional context Reduced from the scalar month lookup in TPC-DS Q6; MAX replaces its DISTINCT lookup to keep the reproducer scalar by construction. The outer input estimate is correct at 73,049 rows. This is an estimation-policy improvement, not a query-result correctness bug. Part of #25610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
