xumingming opened a new pull request, #58269: URL: https://github.com/apache/spark/pull/58269
### What changes were proposed in this pull request? `InlineCTE.validateNoOuterReferencesAcrossCTEBoundary` currently throws `INTERNAL_ERROR` whenever a `forceSkipInline` CTE definition's body contains any `OuterReference` or correlated subquery (with non-empty `outerScopeAttrs`), even when that reference resolves to an operator *inside* the definition body. Such an internal correlation is self-contained and safe to materialize, because after materialization the def body itself still provides the operator the reference resolves against. This PR makes the validator reject a reference only when it genuinely escapes the CTE boundary: it collects every attribute bound anywhere in the definition (including inside nested subquery plans, skipping unresolved outputs), and throws only when an `OuterReference` / `OuterScopeReference`'s exprId resolves to none of those bound exprIds. ### Why are the changes needed? SPARK-58006 added `forceSkipInline` to guarantee a CTE is materialized rather than duplicated. The follow-up validation was too strict: it flagged *any* outer reference in the def, including self-contained ones. Concretely, a view whose correlated subquery references a column produced within its own body could not be materialized as a `forceSkipInline` CTE — analysis failed or the conversion was silently skipped — even though such a correlation has a valid resolution home inside the materialized def. ### Does this PR introduce _any_ user-facing change? No. It affects only internal optimizer behavior for the newly added `forceSkipInline` path: instead of raising `INTERNAL_ERROR` for self-contained correlations, the CTE is now materialized correctly. Truly escaping references still raise the same `INTERNAL_ERROR`. ### How was this patch tested? Updated `InlineCTESuite` under `sql/catalyst`: - regression: an internal `OuterReference` (correlated `EXISTS`) inside a `forceSkipInline` def is materialized (no throw); - regression: an internal `OuterScopeReference` (nested correlated scalar subquery, two scopes out) is materialized (no throw); - negative: an outer reference inside a nested subquery plan that escapes the def still fails with `INTERNAL_ERROR`. Ran: ``` build/sbt "catalyst/Test/compile" build/sbt "catalyst/Test/scalastyle" build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.InlineCTESuite" ``` All 10 tests in the suite pass; scalastyle reports 0 errors. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Pi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
