Hi all,
I am new to spark/graphx and am trying to use partitioning strategies in
graphx on spark 1.0.0
The workaround I saw on the main page seems not to compile.
The code I added was
def partitionBy[ED](edges: RDD[Edge[ED]], partitionStrategy:
PartitionStrategy): RDD[Edge[ED]] = {
val numPartitions = edges.partitions.size
edges.map(e => (partitionStrategy.getPartition(e.srcId, e.dstId,
numPartitions), e))
.partitionBy(new HashPartitioner(numPartitions))
.mapPartitions(_.map(_._2), preservesPartitioning = true)
}
val unpartitionedGraph = GraphLoader.edgeListFile(sc, fname,
minEdgePartitions = numEPart).cache()
val graph = Graph(unpartitionedGraph.vertices,
partitionBy(unpartitionedGraph.edges, PartitionStrategy.EdgePartition2D))
The partition by function is the same as the workarounds described in the
official documentation
I am however getting the following error
error: value partitionBy is not a member of
org.apache.spark.rdd.RDD[(org.apache.spark.graphx.PartitionID,
org.apache.spark.graphx.Edge[ED])]
[INFO] possible cause: maybe a semicolon is missing before `value
partitionBy'?
[INFO] .partitionBy(new HashPartitioner(numPartitions))
[INFO] ^
Please help me in resolving the error. Note: I cant upgrade spark since I
am only a client on the spark cluster.
--
Arpit Kumar
