[ 
https://issues.apache.org/jira/browse/SPARK-31943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jefferson V updated SPARK-31943:
--------------------------------
    Description: 
[31500|[https://github.com/apache/spark/pull/28351/files]] introduced 
unimplemented fields of the `Collect` class that cause client extensions of 
that class to fail unless the spark version is pinned to `2.4.5`. Since this 
was a minor version bump, seems like this wasn't desired.

I believe we should be able to at least put default values:

`convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)`

`bufferElementType: DataType = child.dataType`

and restore the `override def eval` in `Collect`, to support compatibility with 
2.4.5 while allowing implementers in that file to override them to fix the bug. 
Since the abstract `Collect` is currently not designed to fix the bug (just 
provide tools that can be implemented to fix it), this change wouldn't 
undermine the bug fix, just add backwards compatibility to 2.4.5.

  was:
[31500|[https://github.com/apache/spark/pull/28351/files]] introduced 
unimplemented fields of the `Collect` class that cause extensions of that class 
to fail unless the spark version is pinned to `2.4.5`. Since this was a minor 
version bump, seems like this wasn't desired.

I believe we should be able to at least put default values:

`convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)`

`bufferElementType: DataType = child.dataType`

and restore the `override def eval` in `Collect`, to support compatibility with 
2.4.5 while allowing implementers in that file to override them to fix the bug. 
Since the abstract `Collect` is currently not designed to fix the bug (just 
provide tools that can be implemented to fix it), this change wouldn't 
undermine the bug fix, just add backwards compatibility to 2.4.5.


> SPARK-31500 introduces breaking changes in 2.4.6
> ------------------------------------------------
>
>                 Key: SPARK-31943
>                 URL: https://issues.apache.org/jira/browse/SPARK-31943
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.6
>            Reporter: Jefferson V
>            Priority: Major
>              Labels: backward-incompatible
>
> [31500|[https://github.com/apache/spark/pull/28351/files]] introduced 
> unimplemented fields of the `Collect` class that cause client extensions of 
> that class to fail unless the spark version is pinned to `2.4.5`. Since this 
> was a minor version bump, seems like this wasn't desired.
> I believe we should be able to at least put default values:
> `convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)`
> `bufferElementType: DataType = child.dataType`
> and restore the `override def eval` in `Collect`, to support compatibility 
> with 2.4.5 while allowing implementers in that file to override them to fix 
> the bug. Since the abstract `Collect` is currently not designed to fix the 
> bug (just provide tools that can be implemented to fix it), this change 
> wouldn't undermine the bug fix, just add backwards compatibility to 2.4.5.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to