Hi Ankur, Thanks for the answer. However i still have following queries. On Wed, Apr 22, 2015 at 8:39 AM, Ankur Dave <ankurd...@gmail.com> wrote:
> On Tue, Apr 21, 2015 at 10:39 AM, mas <mas.ha...@gmail.com> wrote: > >> How does GraphX stores the routing table? Is it stored on the master node >> or >> chunks of the routing table is send to each partition that maintains the >> record of vertices and edges at that node? >> > > The latter: the routing table is stored alongside the vertices, and for > each vertex it stores the set of edge partitions that reference that > vertex. > > *Then how the master node tracks that where(in which partition) a particular vertex and edge is?* *Further, does it mean that to fetch a particular edge we first have to find its source or destination vertex ?* > If only customized edge partitioning is performed will the corresponding >> vertices be sent to same partition or not ? > > > If I understand correctly, you're asking whether it's possible to colocate > the vertices with the edges so they don't have to move during replication. > It's possible to do this in some cases by partitioning each edge based on a > hash partitioner of its source or destination vertex. GraphX will still do > replication using a shuffle, but most of the shuffle files should be local > in this case. > > I tried this a while ago but didn't find a very big improvement for > PageRank. Ultimately a more general solution would be to unify the vertex > and edge RDDs by designating one replica for each vertex as the master. > This would also reduce the storage cost by a factor of (average degree - > 1)/(average degree). > > *What do you exactly mean here by "desingating one replica for each vertex as the master" ? How can we perform this ?* > Ankur <http://www.ankurdave.com/> > -- Regards, Muhammad Aamir *CONFIDENTIALITY:This email is intended solely for the person(s) named and may be confidential and/or privileged.If you are not the intended recipient,please delete it,notify me and do not copy,use,or disclose its content.*
