cloud-fan opened a new pull request, #46465:
URL: https://github.com/apache/spark/pull/46465
### What changes were proposed in this pull request?
We must consistently use normalized plans for cache filling and lookup, or
inconsistency will lead to cache misses.
To guarantee this, this PR makes `CacheManager` the central place to do plan
normalization, so that callers don't need to care about it. Now most APIs in
`CacheManager` take either `Dataset` or `LogicalPlan`. For `Dataset`, we get
the normalized plan directly. For `LogicalPlan`, we normalize it before further
use.
The caller side should pass `Dataset` when invoking `CacheManager`, if it
already creates `Dataset`. This is to reduce the impact, as extra creation of
`Dataset` may have perf issues or introduce unexpected analysis exception.
### Why are the changes needed?
Avoid unnecessary cache misses for users who add custom normalization rules
### Does this PR introduce _any_ user-facing change?
No, perf only
### How was this patch tested?
existing tests
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
