[
https://issues.apache.org/jira/browse/CASSANALYTICS-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17979923#comment-17979923
]
Jack edited comment on CASSANALYTICS-58 at 6/17/25 12:21 PM:
-------------------------------------------------------------
I have hit a blocking issue in relation to the join planner, spark session
extensions require a type of org.apache.spark.sql.SparkSession while the Data
source v2 API used for that optimisation requires a type of
org.apache.spark.sql.classic.SparkSession. I have reached out to the mailing
list and might raise it to spark dev list if the community list is silent for a
while, as this is a real use case and one I suspect will be common.
To anyone else looking at this issue, a general question I have - should we
look at alternatives to supporting only spark classic/datasource v2 for the
join planner to be something we can carry forward?
I still see value in getting _something_ working against spark 4, but pulling
that out will have user impact. Community guidance and input would be
appreciated on this topic, I'll leave this comment to sit for a while in case
anybody comes across it/is watching this, but I think this could be something
we reasonably discuss on the dev mailing list given the number of potentially
impacted users.
was (Author: jackbuggins):
I have hit a blocking issue in relation to the join planner, spark session
extensions require a type of org.apache.spark.sql.SparkSession while the Data
source v2 API used for that optimisation requires a type of
org.apache.spark.sql.classic.SparkSession. I have reached out to the mailing
list and might raise it to dev, as this is a real use case and one I suspect
will be common.
To anyone else looking at this issue, a general question I have - should we
look at alternatives to supporting only spark classic/datasource v2 for the
join planner to be something we can carry forward?
I still see value in getting _something_ working against spark 4, but pulling
that out will have user impact. Community guidance and input would be
appreciated on this topic, I'll leave this comment to sit for a while in case
anybody comes across it/is watching this, but I think this could be something
we reasonably discuss on the dev mailing list given the number of potentially
impacted users.
> Apache Spark 4.0 support for the Apache Spark Connector
> --------------------------------------------------------
>
> Key: CASSANALYTICS-58
> URL: https://issues.apache.org/jira/browse/CASSANALYTICS-58
> Project: Apache Cassandra Analytics
> Issue Type: New Feature
> Reporter: Jack
> Priority: Normal
>
> Spark 4.0 recently passed voting and subsequently a release has been made, in
> order for users to upgrade to Spark 4, we will need to ensure compatibility
> with spark 4 and mark a stable release once work is completed for this.
> At a high level this will include ensuring compatibility at scala level,
> updating dependencies to match those of the official spark distribution, as
> well as making adjustments for other existing spark interfaces implemented by
> the connector.
> This issue does not account for integration with any new interfaces
> introduced during this release - if any changes of this nature are required,
> these can be split to smaller focussed tasks once all existing
> implementations are ported upward.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
