Weijun-H opened a new pull request, #26879: URL: https://github.com/apache/datafusion/pull/26879
## Which issue does this PR close? - Closes #26878. ## Rationale for this change Planning a projection that reads many fields of one wide struct column, such as flattening a parsed JSON document, rebuilds the same value over and over in `EquivalenceProperties::project`. In the benchmark added here (230 fields read from a 300-field struct), this repeated work is most of the call's time. `projected_orderings` computes the properties of every projection source. For each `Column` it meets, `get_expr_properties` calls `Interval::make_unbounded`, which for a nested type materializes a null array of the whole type. With many sources over the same struct column, that array is rebuilt once per source. ## What changes are included in this PR? - `get_expr_properties` takes a `ColumnRanges` cache of each column index's unbounded range. `projected_orderings` creates one cache per call and shares it across all of that call's sources, so the cache lives for one `projected_orderings` call. The two other callers each pass a fresh cache per call. - Only successful `Column` ranges are cached. Other expressions and errors are computed as before, and the derived orderings are unchanged. - A new `equivalence_properties` benchmark, `project/wide_struct_fields/230_of_300`, projects 230 `get_field` expressions over a 300-field struct plus one ordered column. Locally it went from about 9.7–10.1 ms to about 171 µs per `project` call. ## What is the testing strategy for this PR? New tests in `equivalence/properties/dependency.rs`: - `project_gives_each_source_the_range_of_its_own_column`: sources over two struct columns of different types each receive only their own column's range, and the derived orderings stay the same. - `project_does_not_reuse_a_column_range_across_schemas`: the same column index with different types in two schemas gets each schema's own range. - `project_keeps_dropping_a_derivation_whose_range_type_fails`: a sort key whose type cannot be computed is still dropped from the derivation, while the direct mapping keeps the ordering. These tests pass both before and after the change. A cache shared across columns fails the first test, and a cache kept across calls fails the second. Run locally on this commit: - `cargo fmt --all -- --check` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`: 13321 passed, 0 failed, 8 ignored, including the sqllogictests runner (543 files). ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
