sadikovi opened a new pull request, #59310: URL: https://github.com/apache/spark/pull/59310
### What changes were proposed in this pull request? Spark Connect represents DataFrame subqueries such as `col.isin(df)`, `df.scalar()`, `df.exists()`, and table arguments as `UnresolvedPlanId` expressions (`UnresolvedInSubqueryPlanId`, `UnresolvedScalarSubqueryPlanId`, `UnresolvedExistsPlanId`, `UnresolvedTableArgPlanId`). These expressions point to the subquery plan by plan id. Only `SparkConnectPlanner.transformWithRelations` replaces them with the actual plan, using the enclosing `WithRelations` node. No analyzer rule resolves them. This PR: - Adds a new error condition `CANNOT_RESOLVE_DATAFRAME_SUBQUERY` (SQLSTATE `42704`, the same as `CANNOT_RESOLVE_DATAFRAME_COLUMN`). - Adds an `UnresolvedPlanId` case to the expression-level checks in `CheckAnalysis.checkAnalysis0`. The case throws the new `AnalysisException` and carries the expression's origin. - Adds `QueryCompilationErrors.cannotResolveDataFrameSubquery`. ### Why are the changes needed? If the referenced plan is never attached (for example, when the root relation is not wrapped in a `WithRelations` that contains it), the `UnresolvedPlanId` survives analysis. CheckAnalysis has no specific case for it, so it reaches the generic unresolved-operator check and fails with an internal error: [INTERNAL_ERROR] Found the unresolved operator: ... SQLSTATE: XX000 The cause is an illegal subquery reference, not a Spark bug. It should be reported as a resolution error, the same way `CANNOT_RESOLVE_DATAFRAME_COLUMN` reports an illegal DataFrame column reference. ### Does this PR introduce _any_ user-facing change? Yes, but only for queries that already fail. A Spark Connect DataFrame subquery whose plan id is never attached used to fail with `[INTERNAL_ERROR] ... SQLSTATE: XX000`. It now fails with an `AnalysisException`: [CANNOT_RESOLVE_DATAFRAME_SUBQUERY] Cannot resolve the DataFrame subquery referenced by plan id 42. It's probably because of an illegal subquery reference, such as using a subquery DataFrame that is not part of the current query. SQLSTATE: 42704 Valid queries are not affected. ### How was this patch tested? Added a new test to `SparkConnectProtoSuite`. It: - builds a Connect IN-subquery with an unattached plan id, nested inside `when(...)`, - transforms it without an enclosing `WithRelations`, - checks that the planner leaves an `UnresolvedInSubqueryPlanId` with that plan id and that the plan is unresolved, - checks that analysis now fails with `CANNOT_RESOLVE_DATAFRAME_SUBQUERY` and has the plan id as a parameter. Ran locally: - `build/sbt "connect/testOnly *SparkConnectProtoSuite"`: 68 passed - `build/sbt "core/testOnly *SparkThrowableSuite"`: 38 passed (validates `error-conditions.json`) ### Was this patch authored or co-authored using generative AI tooling? Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
