zhengruifeng opened a new pull request, #55531:
URL: https://github.com/apache/spark/pull/55531
### What changes were proposed in this pull request?
Add a new internal SQL config
`spark.sql.analyzer.strictDataFrameColumnResolution` (default `false`) that
controls how `UnresolvedAttribute`s carrying a `PLAN_ID_TAG` (DataFrame columns
coming through Spark Connect) are resolved in `ColumnResolutionHelper`:
- When `true`:
- In `resolveDataFrameColumnRecursively`, if the target plan node is found
but the column cannot be resolved on it, fail immediately with
`CANNOT_RESOLVE_DATAFRAME_COLUMN` instead of delaying the failure for a later
analyzer iteration.
- In the main `resolveExpression` path, allow name-based resolution as a
fallback for tagged attributes as well.
- When `false` (default): preserve the current behavior, i.e. delay the
failure so that later analyzer iterations / rules still have a chance to
resolve the column, and keep skipping name-based resolution for tagged
attributes.
### Why are the changes needed?
Give users an opt-in mode that surfaces DataFrame column resolution failures
earlier, matching the semantics used internally in Databricks Runtime
(databricks-eng/runtime@4452ee5ecb96e2bdc6184c867d4cbada6f2c13fe). The default
preserves existing Apache Spark behavior so there is no user-facing change
unless the flag is flipped.
### Does this PR introduce _any_ user-facing change?
No behavior change by default. A new internal config is added.
### How was this patch tested?
- New unit test in `SparkSessionExtensionSuite` that enables the config and
verifies `CANNOT_RESOLVE_DATAFRAME_COLUMN` is raised immediately, alongside the
existing "inject analyzer rule - hidden column" test which still exercises the
default delayed-failure path.
- `build/sbt 'catalyst/testOnly *AnalysisSuite'`
- `build/sbt 'sql/testOnly *SparkSessionExtensionSuite
*ColumnNodeToExpressionConverterSuite *ResolverGuardSuite *PlanResolutionSuite'`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic), claude-opus-4-7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
