zhengruifeng opened a new pull request, #59705: URL: https://github.com/apache/spark/pull/59705
### What changes were proposed in this pull request? Backport https://github.com/apache/spark/pull/55582 to `branch-4.1` (upstream commit `266f7674b96a07958ef117194eae325565b53004`). Prefer visible column candidates over hidden natural/USING-join candidates and defer ambiguity checks until all matching candidates have been collected. Include the original PySpark regression test, shared by classic and Connect suites. The Scala fix applies unchanged. Resolve the Python context conflict by adding only the original regression test, with branch-compatible spacing. ### Why are the changes needed? SPARK-55070 is present in 4.1 and can cause `AMBIGUOUS_COLUMN_REFERENCE` when a DataFrame is used directly in a join and also nested under a natural/USING join in the same plan. ### Does this PR introduce _any_ user-facing change? Yes. Affected queries resolve the visible direct-join column instead of raising `AMBIGUOUS_COLUMN_REFERENCE`, while preserving resolution when only hidden candidates are available. ### How was this patch tested? - Carry over `test_select_regular_column_with_reused_dataframe_hidden_in_natural_join` from #55582; it checks the selected values as well as successful resolution. - Verify that the Scala additions/removals and regression test match upstream. - Pass Black 23.12.1 with the branch's configuration and `git diff --check`; check added lines for ASCII and the 100-character limit. - Local compilation and runtime tests were not completed: Catalyst compilation could not download the required SBT 1.11.7 launcher. The Maven Central hostnames were unavailable from this environment. CI will provide build and test validation. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6), for backport preparation and conflict resolution. Original patch: Generated-by: Claude Code (model: claude-opus-4-7). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
