[
https://issues.apache.org/jira/browse/SPARK-59684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18117612#comment-18117612
]
Eric Yang commented on SPARK-59684:
-----------------------------------
I'm working on the fix.
> pivot() on a struct column fails unless the pivot values are given explicitly
> -----------------------------------------------------------------------------
>
> Key: SPARK-59684
> URL: https://issues.apache.org/jira/browse/SPARK-59684
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Eric Yang
> Priority: Major
>
> {{pivot()}} on a struct column fails when Spark collects the distinct values
> itself:
> {code:java}
> scala> Seq(1.0d).toDF("v").selectExpr("v", "struct(v, v) AS
> s").groupBy("v").pivot("s").count()
> org.apache.spark.SparkRuntimeException: [UNSUPPORTED_FEATURE.PIVOT_TYPE] The
> feature is not supported:
> Pivoting by the value '[1.0,1.0]' of the column data type "STRUCT<v:
> DOUBLE NOT NULL, v: DOUBLE NOT NULL>".
> {code}
> This happens for any struct column, regardless of the field types or their
> nullability. Passing the same values explicitly as columns works:
> {code:java}
> df.groupBy("v").pivot($"s", Seq(struct(lit(1.0d), lit(1.0d)))).count()
> {code}
> so only the {{pivot(pivotColumn)}} overload is affected. Pivoting by an
> array column works in both forms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
