adevore3 opened a new issue, #9738:
URL: https://github.com/apache/iceberg/issues/9738
### Apache Iceberg version
None
### Query engine
None
### Please describe the bug 🐞
We were upgrading to spark 3.4.1 when we ran into this issue. Currently
running on spark 3.2.1 which works. We're using the following dependencies:
```
compile "org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.3.1"
compile "org.apache.iceberg:iceberg-spark-extensions-3.4_2.12:1.3.1"
compile "org.apache.iceberg:iceberg-aws:1.3.1"
```
We didn't need `iceberg-spark-extensions` when running on spark 3.2.1 but
added it in just in case. Before that we attempted to use version 1.4.3 for
`iceberg-spark-runtime` & `iceberg-aws`.
Below is the full error & stacktrace.
```
com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException:
java.lang.Integer cannot be cast to java.nio.ByteBuffer
Serialization trace:
lowerBounds (org.apache.iceberg.GenericDataFile)
taskFiles (org.apache.iceberg.spark.source.SparkWrite$TaskCommit)
writerCommitMessage
(org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTaskResult)
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
~[kryo-shaded-4.0.2.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:391)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
~[kryo-shaded-4.0.2.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
~[kryo-shaded-4.0.2.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
~[kryo-shaded-4.0.2.jar:?]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
~[kryo-shaded-4.0.2.jar:?]
at
org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:311)
~[spark-core_2.12-3.4.1.jar:3.4.1]
at
org.apache.spark.serializer.SerializerHelper$.deserializeFromChunkedBuffer(SerializerHelper.scala:52)
~[spark-core_2.12-3.4.1.jar:3.4.1]
at
org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:108)
~[spark-core_2.12-3.4.1.jar:3.4.1]
at
org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1(TaskResultGetter.scala:75)
~[spark-core_2.12-3.4.1.jar:3.4.1]
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[scala-library-2.12.17.jar:?]
at
org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2088)
[spark-core_2.12-3.4.1.jar:3.4.1]
at
org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:63)
[spark-core_2.12-3.4.1.jar:3.4.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_372]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_372]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.nio.ByteBuffer
at
org.apache.iceberg.SerializableByteBufferMap.put(SerializableByteBufferMap.java:31)
~[main.jar:?]
at
com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:162)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
~[kryo-shaded-4.0.2.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
~[kryo-shaded-4.0.2.jar:?]
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
~[kryo-shaded-4.0.2.jar:?]
... 21 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
