bhollis-dbx opened a new pull request, #58872: URL: https://github.com/apache/spark/pull/58872
### What changes were proposed in this pull request? Use `AttributeMap` to resolve attributes in `DataSourceStrategy.normalizeExprs`. Preserve the first match when duplicate expression IDs are present. Return immediately when there are no expressions to normalize. ### Why are the changes needed? The previous implementation searched all output attributes for every attribute reference, making normalization quadratic for wide projections. This can add substantial planning latency, and normalization may run several times per scan. In a local benchmark with 10,000 attributes, normalization improved from 876 ms to 2 ms for 10,000 references. The empty-input fast path also avoids constructing an attribute map for unfiltered scans. I decided not to complicate the code by adding a heuristic-gated fast path for small numbers of attributes/references: I could avoid the construction of the map in that case and recover some small speed regression and allocations, but I decided that having a single solution that regressed only a bit on small schemas but vastly improved large schemas was the right tradeoff. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added a unit test covering duplicate expression IDs and first-match behavior. Ran: `build/sbt 'sql/testOnly *DataSourceStrategySuite -- -z "normalizeExprs"'` Also ran a local benchmark comparing the previous linear search with `AttributeMap` across 10 to 10,000 attributes and varying reference counts: | Attributes / references | Result | | ----------|-----------------------| | 10 / 1 | 4.2× slower; +0.36 µs | | 1,000 / 10 | 1.1× slower; +6 µs | | 1,000 / 100 | 7.6× faster | | 10,000 / 1 | 5.1× slower; +0.9 ms | | 10,000 / 10 | 2× faster | | 10,000 / 1,000 | 182× faster | | 10,000 / 10,000 | 434× faster: 876 ms → 2 ms | ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
