[
https://issues.apache.org/jira/browse/SEDONA-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17651474#comment-17651474
]
Jia Yu commented on SEDONA-211:
-------------------------------
[~codingcat] Thanks for bringing this up. This issue only exists in ShapeSerde
which is only used when you serialize/deserialize data in RDD. In Sedona Spark
SQL, we use WKB serializer.
We will fix this issue ASAP on Sedona 1.2.X ASAP.
> Enforce release managers to use JDK 8
> -------------------------------------
>
> Key: SEDONA-211
> URL: https://issues.apache.org/jira/browse/SEDONA-211
> Project: Apache Sedona
> Issue Type: Bug
> Affects Versions: 1.3.0
> Reporter: Seth Fitzsimmons
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.3.1
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When deploying an app built with Sedona 1.3.0-incubating to AWS EMR
> Serverless (which only supports Java 8), executors that indirectly use the
> {{ShapeSerde}} (e.g., by Kryo serializing a geometry column) crash with this
> error:
> {{java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;}}
> {{at
> org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.shp.ShapeSerde.putHeader(ShapeSerde.java:163)}}
> {{at
> org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.shp.ShapeSerde.serialize(ShapeSerde.java:149)}}
> {{at
> org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.shp.ShapeSerde.serialize(ShapeSerde.java:73)}}
> {{at
> org.apache.sedona.core.geometryObjects.GeometrySerde.writeGeometry(GeometrySerde.java:104)}}
> {{at
> org.apache.sedona.core.geometryObjects.GeometrySerde.write(GeometrySerde.java:72)}}
> {{at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)}}
> {{at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)}}
> {{at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)}}
> {{at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)}}
> {{at
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)}}
> {{at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)}}
> {{at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)}}
> {{at
> org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:270)}}
> {{at
> org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:145)}}
> {{at
> org.apache.spark.serializer.DefaultSerializedRecordAppender.process(SerializedRecordAppender.scala:44)}}
> {{at
> org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:242)}}
> {{at
> org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:184)}}
> {{at
> org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)}}
> {{at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)}}
> {{at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)}}
> {{at org.apache.spark.scheduler.Task.run(Task.scala:138)}}
> {{at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)}}
> {{at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516)}}
> {{at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)}}
> {{at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
> {{at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
> {{at java.lang.Thread.run(Thread.java:750)}}
> From Spark's environment view:
> Java Home /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.342.b07-1.amzn2.0.1.x86_64/jre
> Java Version 1.8.0_342 (Red Hat, Inc.)
> Scala Version version 2.12.15
> This appears to be related to {{maven-compiler-plugin}} not doing the right
> thing ([{{source}} and {{target}} are both set to
> {{8}}|https://github.com/apache/incubator-sedona/blob/7b9396d6bb3d083bec1fccfdb98d2007fcfe9df9/core/pom.xml#L83-L84]).
> Perhaps
> [{{release}}|https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-release.html]
> should be used instead?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
