[ https://issues.apache.org/jira/browse/SPARK-31943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jefferson V updated SPARK-31943: -------------------------------- Description: [31500|[https://github.com/apache/spark/pull/28351/files]] introduced unimplemented fields of the `Collect` class that cause client extensions of that class to fail unless the spark version is pinned to `2.4.5`. Since this was a minor version bump, seems like this wasn't desired. I believe we should be able to at least put default values: `convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)` `bufferElementType: DataType = child.dataType` and restore the `override def eval` in `Collect`, to support compatibility with 2.4.5 while allowing implementers in that file to override them to fix the bug. Since the abstract `Collect` is currently not designed to fix the bug (just provide tools that can be implemented to fix it), this change wouldn't undermine the bug fix, just add backwards compatibility to 2.4.5. was: [31500|[https://github.com/apache/spark/pull/28351/files]] introduced unimplemented fields of the `Collect` class that cause extensions of that class to fail unless the spark version is pinned to `2.4.5`. Since this was a minor version bump, seems like this wasn't desired. I believe we should be able to at least put default values: `convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)` `bufferElementType: DataType = child.dataType` and restore the `override def eval` in `Collect`, to support compatibility with 2.4.5 while allowing implementers in that file to override them to fix the bug. Since the abstract `Collect` is currently not designed to fix the bug (just provide tools that can be implemented to fix it), this change wouldn't undermine the bug fix, just add backwards compatibility to 2.4.5. > SPARK-31500 introduces breaking changes in 2.4.6 > ------------------------------------------------ > > Key: SPARK-31943 > URL: https://issues.apache.org/jira/browse/SPARK-31943 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.6 > Reporter: Jefferson V > Priority: Major > Labels: backward-incompatible > > [31500|[https://github.com/apache/spark/pull/28351/files]] introduced > unimplemented fields of the `Collect` class that cause client extensions of > that class to fail unless the spark version is pinned to `2.4.5`. Since this > was a minor version bump, seems like this wasn't desired. > I believe we should be able to at least put default values: > `convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)` > `bufferElementType: DataType = child.dataType` > and restore the `override def eval` in `Collect`, to support compatibility > with 2.4.5 while allowing implementers in that file to override them to fix > the bug. Since the abstract `Collect` is currently not designed to fix the > bug (just provide tools that can be implemented to fix it), this change > wouldn't undermine the bug fix, just add backwards compatibility to 2.4.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org