[
https://issues.apache.org/jira/browse/SPARK-31374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098609#comment-17098609
]
Oleksii Kachaiev commented on SPARK-31374:
------------------------------------------
[~hoeze] do you have an example of the code of {{spark_window_overlap}}
function?
> Returning complex types in Pandas UDF
> -------------------------------------
>
> Key: SPARK-31374
> URL: https://issues.apache.org/jira/browse/SPARK-31374
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.0.0
> Reporter: F. H.
> Priority: Major
> Labels: features
>
> I would like to return a complex type in an GROUPED_AGG operation:
> {code:python}
> window_overlap_schema = t.StructType([
> t.StructField("counts", t.ArrayType(t.LongType())),
> t.StructField("starts", t.ArrayType(t.LongType())),
> t.StructField("ends", t.ArrayType(t.LongType())),
> ])
> @f.pandas_udf(window_overlap_schema, f.PandasUDFType.GROUPED_AGG)
> def spark_window_overlap([...]):
> [...]
> {code}
> However, I get the following error when trying to run this:
> {code:python}
> NotImplementedError: Invalid returnType with grouped aggregate Pandas UDFs:
> StructType(List(StructField(counts,ArrayType(LongType,true),true),StructField(starts,ArrayType(LongType,true),true),StructField(ends,ArrayType(LongType,true),true)))
> is not supported
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
