[jira] [Commented] (SPARK-12431) add local checkpointing to GraphX

2016-10-10 Thread Eyal Farago (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561437#comment-15561437
 ] 

Eyal Farago commented on SPARK-12431:
-

I think this heavily relates to SPARK-14804, assuming this is an ongoing effort 
I think it actually depends on 14804 resolution.


> add local checkpointing to GraphX
> -
>
> Key: SPARK-12431
> URL: https://issues.apache.org/jira/browse/SPARK-12431
> Project: Spark
>  Issue Type: Improvement
>  Components: GraphX
>Affects Versions: 1.5.2
>Reporter: Edward Seidl
>
> local checkpointing was added to RDD to speed up iterative spark jobs, but 
> this capability hasn't been added to GraphX.  Adding localCheckpoint to 
> GraphImpl, EdgeRDDImpl, and VertexRDDImpl greatly improved the speed of a 
> k-core algorithm I'm using (at the cost of fault tolerance, of course).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12431) add local checkpointing to GraphX

2016-05-28 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305412#comment-15305412
 ] 

Apache Spark commented on SPARK-12431:
--

User 'adeandrade' has created a pull request for this issue:
https://github.com/apache/spark/pull/13379

> add local checkpointing to GraphX
> -
>
> Key: SPARK-12431
> URL: https://issues.apache.org/jira/browse/SPARK-12431
> Project: Spark
>  Issue Type: Improvement
>  Components: GraphX
>Affects Versions: 1.5.2
>Reporter: Edward Seidl
>
> local checkpointing was added to RDD to speed up iterative spark jobs, but 
> this capability hasn't been added to GraphX.  Adding localCheckpoint to 
> GraphImpl, EdgeRDDImpl, and VertexRDDImpl greatly improved the speed of a 
> k-core algorithm I'm using (at the cost of fault tolerance, of course).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12431) add local checkpointing to GraphX

2016-01-05 Thread David Youd (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083419#comment-15083419
 ] 

David Youd commented on SPARK-12431:


Since localCheckpoint() was partially implemented, but in a way that doesn’t 
work, should this be changed from “Improvement” to “Bug”?

> add local checkpointing to GraphX
> -
>
> Key: SPARK-12431
> URL: https://issues.apache.org/jira/browse/SPARK-12431
> Project: Spark
>  Issue Type: Improvement
>  Components: GraphX
>Affects Versions: 1.5.2
>Reporter: Edward Seidl
>
> local checkpointing was added to RDD to speed up iterative spark jobs, but 
> this capability hasn't been added to GraphX.  Adding localCheckpoint to 
> GraphImpl, EdgeRDDImpl, and VertexRDDImpl greatly improved the speed of a 
> k-core algorithm I'm using (at the cost of fault tolerance, of course).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12431) add local checkpointing to GraphX

2016-01-04 Thread Edward Seidl (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082081#comment-15082081
 ] 

Edward Seidl commented on SPARK-12431:
--

here's a patch demonstrating what I'm on about...

{code}
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/Graph.scala 
b/graphx/src/main/scala/org/apache/spark/graphx/Graph.scala
index 869caa3..10a3d7c 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/Graph.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/Graph.scala
@@ -103,6 +103,7 @@ abstract class Graph[VD: ClassTag, ED: ClassTag] protected 
() extends Serializab
* memory, otherwise saving it on a file will require recomputation.
*/
   def checkpoint(): Unit
+  def localCheckpoint(): Unit
 
   /**
* Return whether this Graph has been checkpointed or not.
diff --git 
a/graphx/src/main/scala/org/apache/spark/graphx/impl/EdgeRDDImpl.scala 
b/graphx/src/main/scala/org/apache/spark/graphx/impl/EdgeRDDImpl.scala
index c88b2f6..65e24e5 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/impl/EdgeRDDImpl.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/impl/EdgeRDDImpl.scala
@@ -76,6 +76,11 @@ class EdgeRDDImpl[ED: ClassTag, VD: ClassTag] 
private[graphx] (
 partitionsRDD.checkpoint()
   }
 
+  override def localCheckpoint() : this.type = {
+  partitionsRDD.localCheckpoint()
+  this
+  }
+
   override def isCheckpointed: Boolean = {
 firstParent[(PartitionID, EdgePartition[ED, VD])].isCheckpointed
   }
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/impl/GraphImpl.scala 
b/graphx/src/main/scala/org/apache/spark/graphx/impl/GraphImpl.scala
index da95314..cc228ef 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/impl/GraphImpl.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/impl/GraphImpl.scala
@@ -70,6 +70,11 @@ class GraphImpl[VD: ClassTag, ED: ClassTag] protected (
 replicatedVertexView.edges.checkpoint()
   }
 
+  override def localCheckpoint(): Unit = {
+  vertices.localCheckpoint()
+  replicatedVertexView.edges.localCheckpoint()
+  }
+
   override def isCheckpointed: Boolean = {
 vertices.isCheckpointed && replicatedVertexView.edges.isCheckpointed
   }
diff --git 
a/graphx/src/main/scala/org/apache/spark/graphx/impl/VertexRDDImpl.scala 
b/graphx/src/main/scala/org/apache/spark/graphx/impl/VertexRDDImpl.scala
index 7f4e7e9..54e8406 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/impl/VertexRDDImpl.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/impl/VertexRDDImpl.scala
@@ -77,6 +77,11 @@ class VertexRDDImpl[VD] private[graphx] (
 partitionsRDD.checkpoint()
   }
 
+  override def localCheckpoint(): this.type = {
+  partitionsRDD.localCheckpoint()
+  this
+  }
+
   override def isCheckpointed: Boolean = {
 firstParent[ShippableVertexPartition[VD]].isCheckpointed
   }
{code}

> add local checkpointing to GraphX
> -
>
> Key: SPARK-12431
> URL: https://issues.apache.org/jira/browse/SPARK-12431
> Project: Spark
>  Issue Type: Improvement
>  Components: GraphX
>Affects Versions: 1.5.2
>Reporter: Edward Seidl
>
> local checkpointing was added to RDD to speed up iterative spark jobs, but 
> this capability hasn't been added to GraphX.  Adding localCheckpoint to 
> GraphImpl, EdgeRDDImpl, and VertexRDDImpl greatly improved the speed of a 
> k-core algorithm I'm using (at the cost of fault tolerance, of course).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org