[
https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937715#comment-16937715
]
Alexey Zinoviev edited comment on IGNITE-12054 at 9/26/19 3:27 PM:
-------------------------------------------------------------------
I've started the R&D about 2.4 version, fixed a few minor changes and posted
here a few thoughts about current problems here, in Jira comments.
An ExternalCatalog was refactored here
[https://github.com/apache/spark/commit/f38ea00e83099a5ae8d3afdec2e896e43c2db612]
and all listener properties were inherited in ExternalCatalogWithListener.
Refactoring for Hive is here [https://github.com/apache/spark/pull/21122/files]
Nobody yet inherited from this class on Github, the known implementations are
HiveExternalCatalog and MemoryExternalCatalog (both of them doesn't support
listeners and events)
Also, people in Spark ML couldn't solve the same problem
[http://mail-archives.apache.org/mod_mbox/spark-issues/201812.mbox/%3cjira.13144856.1520975543000.147283.1544598241...@atlassian.jira%3E]
Also this question is considered in paper
[https://www.waitingforcode.com/apache-spark-sql/writing-custom-external-catalog-listeners-apache-spark-sql/read]
Also the semantic of Having support was changed in Spark
[https://github.com/apache/spark/pull/22696/files]
https://issues.apache.org/jira/browse/SPARK-25708
Now Having could be a legal operation without GroupBY
was (Author: zaleslaw):
I've started the R&D about 2.4 version, fixed a few minor changes and posted
here a few thoughts about current problems here, in Jira comments.
An ExternalCatalog was refactored here
[https://github.com/apache/spark/commit/f38ea00e83099a5ae8d3afdec2e896e43c2db612]
and all listener properties were inherited in ExternalCatalogWithListener.
Nobody yet inherited from this class on Github, the known implementations are
HiveExternalCatalog and MemoryExternalCatalog (both of them doesn't support
listeners and events)
Also, people in Spark ML couldn't solve the same problem
[http://mail-archives.apache.org/mod_mbox/spark-issues/201812.mbox/%3cjira.13144856.1520975543000.147283.1544598241...@atlassian.jira%3E]
Also this question is considered in paper
[https://www.waitingforcode.com/apache-spark-sql/writing-custom-external-catalog-listeners-apache-spark-sql/read]
Also the semantic of Having support was changed in Spark
[https://github.com/apache/spark/pull/22696/files]
https://issues.apache.org/jira/browse/SPARK-25708
Now Having could be a legal operation without GroupBY
> Upgrade Spark module to 2.4
> ---------------------------
>
> Key: IGNITE-12054
> URL: https://issues.apache.org/jira/browse/IGNITE-12054
> Project: Ignite
> Issue Type: Task
> Components: spark
> Affects Versions: 2.7.5
> Reporter: Denis A. Magda
> Assignee: Alexey Zinoviev
> Priority: Blocker
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Users can't use APIs that are already available in Spark 2.4:
> https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite
> Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as
> a separate module that can support multiple Spark versions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
