On 7/26/14, 4:03 AM, Ankur Dave wrote:
Oops, the code should be:
val unpartitionedGraph: Graph[Int, Int] = ...val numPartitions: Int = 128
def getTripletPartition(e: EdgeTriplet[Int, Int]): PartitionID = ...
// Get the triplets using GraphX, then use Spark to repartition
themval partitionedEdges = unpartitionedGraph.triplets
.map(e => (getTripletPartition(e), e))
.partitionBy(new HashPartitioner(numPartitions))
*.map(pair => Edge(pair._2.srcId, pair._2.dstId, pair._2.attr))*
val partitionedGraph = Graph(unpartitionedGraph.vertices, partitionedEdges)
Ankur <http://www.ankurdave.com/>
Hi Ankur,
Thanks for clear advice!
I tested the 4 partitioning algorithm in GraphX, and implemented two others.
And I find EdgePartition2D performs the best.
(the two other algorithms performs only tiny bit better on graphs that
are highly skewed or bipartite)
Larry
