[
https://issues.apache.org/jira/browse/SEDONA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713940#comment-17713940
]
Martin Andersson commented on SEDONA-276:
-----------------------------------------
During runtime, Sedona utilizes the logging functionality provided by either
Spark or Flink. To maintain consistency, it may be beneficial to employ the
same logging framework during testing as well. This would involve transitioning
to slf4j instead of relying on a specific version of log4j, as discussed
previously in the following GitHub pull request:
[https://github.com/apache/sedona/pull/743.] [~kimahriman] do you think this is
a way forward?
> Add support for Spark 3.4
> -------------------------
>
> Key: SEDONA-276
> URL: https://issues.apache.org/jira/browse/SEDONA-276
> Project: Apache Sedona
> Issue Type: Improvement
> Reporter: Martin Andersson
> Priority: Major
>
> Spark 3.4.0 was announced on April 14.
> [https://lists.apache.org/thread/o1o9n1t036rmfp45q1jrjs0zzzx0j156]
> Findings from a quick test build using -Dspark.version=3.4.0:
> # Incompatible logging api versions causes test failures.
> # GeoParquet and Geotiff datasources fails to compile due to api changes in
> Spark.
> Setting log4j and slf4j to the same versions as Spark 3.4.0 uses solves 1. I
> don't know how that will affect other Spark versions.
> To solve 2 we might need to use reflection to handle the api changes in Spark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
