[
https://issues.apache.org/jira/browse/SEDONA-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17458032#comment-17458032
]
Jia Yu commented on SEDONA-28:
------------------------------
[~malka] Yes, this is still relevant. We need to reproduce this issue in the
cluster mode.
Currently, all our tests are running in local mode. Usually, the local mode
should be able to reveal the bugs. But not sure why this was not found.
In Hadoop, they provide something called minicluster which can test Hadoop in a
test cluster. I searched online but didn't find such a method available in Spark
> Add WKB serializer in RDD and SQL API and let the user choose the SerDe
> ------------------------------------------------------------------------
>
> Key: SEDONA-28
> URL: https://issues.apache.org/jira/browse/SEDONA-28
> Project: Apache Sedona
> Issue Type: Task
> Reporter: Netanel Malka
> Assignee: Netanel Malka
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Add WKB serializer as an optional serializer and use the old Serde as the
> default SerDe for both Core and SQL. The user should only use it if they use
> geometries that are currently not supported by the old Serde.
> The user should be able to choose the serializer as follows
> .config("spark.serializer", classOf[KryoSerializer].getName) //
> org.apache.spark.serializer.KryoSerializer
> .config("spark.kryo.registrator", classOf[SedonaKryoRegistrator].getName)
> .config("spark.serializer", classOf[KryoSerializer].getName) //
> org.apache.spark.serializer.KryoSerializer
> .config("spark.kryo.registrator", classOf[SedonaWKBKryoRegistrator].getName)
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
