uros-db commented on code in PR #55986:
URL: https://github.com/apache/spark/pull/55986#discussion_r3275034416
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:
##########
@@ -284,10 +284,19 @@ object ResolveWithCTE extends Rule[LogicalPlan] {
// This is a non-recursive reference to a definition.
case ref: CTERelationRef if !ref.resolved =>
cteDefMap.get(ref.cteId).map { cteDef =>
- // cteDef is certainly resolved, otherwise it would not have been in
the map.
- CTERelationRef(
- cteDef.id, cteDef.resolved, cteDef.output, cteDef.isStreaming,
maxRows = cteDef.maxRows,
+ // Wait for SQLFunctionExpression placeholders inside the CTE body
to be inlined by
+ // ResolveSQLFunctions before snapshotting the schema. The
placeholder hard-codes
+ // nullable = true, so capturing cteDef.output while one is still
present would freeze
+ // incorrect nullability into CTERelationRef.output.
+ if (cteDef.containsPattern(SQL_FUNCTION_EXPRESSION)) {
Review Comment:
How is this handled in single-pass? Do we need to mirror the fix there as
well, or at least confirm with a test that the single-pass path is not
vulnerable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
