This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 2fff060 [SPARK-36009][GRAPHX] Add missing GraphX classes to registerKryoClasses util method 2fff060 is described below commit 2fff060b6d04e984352543efe9b8077b3c5d1c06 Author: Matthew Jones <matt...@vitaler.com> AuthorDate: Tue Jul 6 07:25:22 2021 -0500 [SPARK-36009][GRAPHX] Add missing GraphX classes to registerKryoClasses util method ### What changes were proposed in this pull request? Add `ShippableVertexPartition` and `RoutingTablePartition` to the classes registered with Kryo in `GraphXUtils.registerKryoClasses`. ### Why are the changes needed? `VertexRDDImpl` uses an `RDD[ShippableVertexPartition[VD]]` however, `GraphXUtils.registerKryoClasses` does not register `ShippableVertexPartition`. This means when running with `spark.kryo.registrationRequired` set to `true`, we get a "Class is not registered" exception. This is an issue as it prevents other unregistered classes from being discovered using `spark.kryo.registrationRequired` as the first unregistered class found halts the whole job. It also potentially decreases the se [...] ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Tested manually by running with `spark.kryo.registrationRequired` set to `true` and verifying no "Class is not registered" exception was thrown. Closes #32973 from matthewrj/bug/register-graphx-classes. Lead-authored-by: Matthew Jones <matt...@vitaler.com> Co-authored-by: Matthew Jones <mle...@gmail.com> Signed-off-by: Sean Owen <sro...@gmail.com> --- graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala | 2 ++ 1 file changed, 2 insertions(+) diff --git a/graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala b/graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala index 4ff5b02..39822c4 100644 --- a/graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala +++ b/graphx/src/main/scala/org/apache/spark/graphx/GraphXUtils.scala @@ -35,6 +35,8 @@ object GraphXUtils { classOf[Edge[Object]], classOf[(VertexId, Object)], classOf[EdgePartition[Object, Object]], + classOf[ShippableVertexPartition[Object]], + classOf[RoutingTablePartition], classOf[BitSet], classOf[VertexIdToIndexMap], classOf[VertexAttributeBlock[Object]], --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org