Re: graphx - mutable?
Hi I am also working on the same area where the graph evolves over time and the current approach of rebuilding the graph again and again is very slow and memory consuming did you find any workaround? What was your usecase? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/graphx-mutable-tp15777p25057.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: graphx - mutable?
hi again. just want to check in again to see if anyone could advise on how to implement a mutable, growing graph with graphx? we're building a graph is growing over time. it adds more vertices and edges every iteration of our algorithm. it doesn't look like there is an obvious way to add a new vertice a set of edges to an existing graph. what would be the best way to implement this with graphx? thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/graphx-mutable-tp15777p16409.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: graphx - mutable?
On Tue, Oct 14, 2014 at 12:36 PM, ll duy.huynh@gmail.com wrote: hi again. just want to check in again to see if anyone could advise on how to implement a mutable, growing graph with graphx? we're building a graph is growing over time. it adds more vertices and edges every iteration of our algorithm. it doesn't look like there is an obvious way to add a new vertice a set of edges to an existing graph. Currently the only way to do this is to rebuild the graph in each iteration by adding more vertices and edges to the source RDDs, then calling the graph constructor. I'm working on a way to support this more efficiently (SPARK-2365 https://issues.apache.org/jira/browse/SPARK-2365), but GraphX doesn't take advantage of this yet. Ankur http://www.ankurdave.com/
Re: graphx - mutable?
thanks ankur. indexedrdd sounds super helpful! a related question, what is the best way to update the values of existing vertices and edges? On Tue, Oct 14, 2014 at 4:30 PM, Ankur Dave ankurd...@gmail.com wrote: On Tue, Oct 14, 2014 at 12:36 PM, ll duy.huynh@gmail.com wrote: hi again. just want to check in again to see if anyone could advise on how to implement a mutable, growing graph with graphx? we're building a graph is growing over time. it adds more vertices and edges every iteration of our algorithm. it doesn't look like there is an obvious way to add a new vertice a set of edges to an existing graph. Currently the only way to do this is to rebuild the graph in each iteration by adding more vertices and edges to the source RDDs, then calling the graph constructor. I'm working on a way to support this more efficiently (SPARK-2365 https://issues.apache.org/jira/browse/SPARK-2365), but GraphX doesn't take advantage of this yet. Ankur http://www.ankurdave.com/
Re: graphx - mutable?
On Tue, Oct 14, 2014 at 1:57 PM, Duy Huynh duy.huynh@gmail.com wrote:
a related question, what is the best way to update the values of existing
vertices and edges?
Many of the Graph methods deal with updating the existing values in bulk,
including mapVertices, mapEdges, mapTriplets, mapReduceTriplets, and
outerJoinVertices.
To update just a small number of existing values, IndexedRDD would be
ideal, but until it makes it into GraphX the best way is to use one of the
above methods. This will be slower since it's touching all of the vertices,
but it will achieve the same goal.
For example, if you had a graph and wanted to update the value of vertex 1
to a, you could do the following:
val graph = ...
val updates = sc.parallelize(List((1L, a)))
val newGraph = graph.outerJoinVertices(updates) { (id, a, b) = b }
Ankur http://www.ankurdave.com/
Re: graphx - mutable?
great, thanks!
On Tue, Oct 14, 2014 at 5:08 PM, Ankur Dave ankurd...@gmail.com wrote:
On Tue, Oct 14, 2014 at 1:57 PM, Duy Huynh duy.huynh@gmail.com
wrote:
a related question, what is the best way to update the values of existing
vertices and edges?
Many of the Graph methods deal with updating the existing values in bulk,
including mapVertices, mapEdges, mapTriplets, mapReduceTriplets, and
outerJoinVertices.
To update just a small number of existing values, IndexedRDD would be
ideal, but until it makes it into GraphX the best way is to use one of the
above methods. This will be slower since it's touching all of the vertices,
but it will achieve the same goal.
For example, if you had a graph and wanted to update the value of vertex 1
to a, you could do the following:
val graph = ...
val updates = sc.parallelize(List((1L, a)))
val newGraph = graph.outerJoinVertices(updates) { (id, a, b) = b }
Ankur http://www.ankurdave.com/
graphx - mutable?
i understand that graphx is an immutable rdd. i'm working on an algorithm that requires a mutable graph. initially, the graph starts with just a few nodes and edges. then over time, it adds more and more nodes and edges. what would be the best way to implement this growing graph with graphx? thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/graphx-mutable-tp15777.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
