This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new ce414f8 [SPARK-37556][SQL] Deser void class fail with Java
serialization
ce414f8 is described below
commit ce414f82eb69a1888f0a166ce8f3bd3f209b15a6
Author: Daniel Dai <j...@pinterest.com>
AuthorDate: Tue Dec 7 08:48:23 2021 -0600
[SPARK-37556][SQL] Deser void class fail with Java serialization
**What changes were proposed in this pull request?**
Change the deserialization mapping for primitive type void.
**Why are the changes needed?**
The void primitive type in Scala should be classOf[Unit] not classOf[Void].
Spark erroneously [map
it](https://github.com/apache/spark/blob/v3.2.0/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala#L80)
differently than all other primitive types. Here is the code:
```
private object JavaDeserializationStream {
val primitiveMappings = Map[String, Class[_]](
"boolean" -> classOf[Boolean],
"byte" -> classOf[Byte],
"char" -> classOf[Char],
"short" -> classOf[Short],
"int" -> classOf[Int],
"long" -> classOf[Long],
"float" -> classOf[Float],
"double" -> classOf[Double],
"void" -> classOf[Void]
)
}
```
Spark code is Here is the demonstration:
```
scala> classOf[Long]
val res0: Class[Long] = long
scala> classOf[Double]
val res1: Class[Double] = double
scala> classOf[Byte]
val res2: Class[Byte] = byte
scala> classOf[Void]
val res3: Class[Void] = class java.lang.Void <--- this is wrong
scala> classOf[Unit]
val res4: Class[Unit] = void <---- this is right
```
It will result in Spark deserialization error if the Spark code contains
void primitive type:
`java.io.InvalidClassException: java.lang.Void; local class name
incompatible with stream class name "void"`
**Does this PR introduce any user-facing change?**
no
**How was this patch tested?**
Changed test, also tested e2e with the code results deserialization error
and it pass now.
Closes #34816 from daijyc/voidtype.
Authored-by: Daniel Dai <j...@pinterest.com>
Signed-off-by: Sean Owen <sro...@gmail.com>
(cherry picked from commit fb40c0e19f84f2de9a3d69d809e9e4031f76ef90)
Signed-off-by: Sean Owen <sro...@gmail.com>
---
core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala | 4 ++--
.../test/scala/org/apache/spark/serializer/JavaSerializerSuite.scala | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala
b/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala
index 077b035..3c13401 100644
--- a/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala
@@ -87,8 +87,8 @@ private object JavaDeserializationStream {
"long" -> classOf[Long],
"float" -> classOf[Float],
"double" -> classOf[Double],
- "void" -> classOf[Void]
- )
+ "void" -> classOf[Unit])
+
}
private[spark] class JavaSerializerInstance(
diff --git
a/core/src/test/scala/org/apache/spark/serializer/JavaSerializerSuite.scala
b/core/src/test/scala/org/apache/spark/serializer/JavaSerializerSuite.scala
index 6a6ea42..03349f8 100644
--- a/core/src/test/scala/org/apache/spark/serializer/JavaSerializerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/serializer/JavaSerializerSuite.scala
@@ -47,5 +47,5 @@ private class ContainsPrimitiveClass extends Serializable {
val floatClass = classOf[Float]
val booleanClass = classOf[Boolean]
val byteClass = classOf[Byte]
- val voidClass = classOf[Void]
+ val voidClass = classOf[Unit]
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
