jiwen624 opened a new pull request, #58934:
URL: https://github.com/apache/spark/pull/58934
### What changes were proposed in this pull request?
PivotTransformer checks and casts each pivot value against
pivotColumn.dataType. This PR checks and casts against
pivotColumn.dataType.asNullable instead, so a pivot value is matched to the
pivot column by type, ignoring nullability.
### Why are the changes needed?
pivot() on an array column fails at analysis when the array's element type
is non-nullable, with an error that prints the same type on both sides:
```scala
scala> Seq(1.0d).toDF("v").selectExpr("v", "array(v) AS
a").groupBy("v").pivot("a").count()
org.apache.spark.sql.AnalysisException: [PIVOT_VALUE_DATA_TYPE_MISMATCH]
Invalid pivot value '[1.0]':
value data type array<double> does not match pivot column data type
array<double>
```
The pivot values are collected by Spark and turned into literals by
Literal.apply, which builds ArrayType(elementType), i.e. containsNull = true.
Cast.canCast then rejects the cast to the column's containsNull = false type.
Nullability is irrelevant here — the value is a constant that is only compared
against the column — and there is no workaround, since the user does not supply
the literals.
### Does this PR introduce _any_ user-facing change?
Yes. Queries that failed with PIVOT_VALUE_DATA_TYPE_MISMATCH purely because
of a nullability difference now succeed. No previously working query changes
its result: the cast is only loosened on nullability flags, never on base
types, so genuinely mismatched values are still rejected.
### How was this patch tested?
Added UT cases.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
