[ https://issues.apache.org/jira/browse/SPARK-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankur Dave updated SPARK-1750: ------------------------------ Comment: was deleted (was: Resolved in PR #742: https://github.com/apache/spark/pull/742) > EdgePartition is not serialized properly > ---------------------------------------- > > Key: SPARK-1750 > URL: https://issues.apache.org/jira/browse/SPARK-1750 > Project: Spark > Issue Type: Bug > Components: GraphX > Affects Versions: 0.9.0, 1.0.0, 0.9.1 > Reporter: Ankur Dave > Fix For: 1.0.0 > > > The GraphX design attempts to avoid moving edges across the network, instead > shipping the vertices to the edge partitions. However, Spark sometimes needs > to move the edges, such as for straggler mitigation. > All EdgePartition fields are currently declared transient, so the edges will > not be serialized properly. Even if they are not marked transient, Kryo is > unable to serialize the EdgePartition, failing with the following error: > {code} > java.lang.IllegalArgumentException: Can not set final > org.apache.spark.graphx.util.collection.PrimitiveKeyOpenHashMap field > org.apache.spark.graphx.impl.EdgePartition.index to > scala.collection.immutable.$colon$colon > {code} > A workaround is to discourage Spark from moving the edges by setting > {{spark.locality.wait}} to a high value such as 100000. -- This message was sent by Atlassian JIRA (v6.2#6252)