Request for a working example of using Pregel API in GraphX using Spark Scala

Hello All I am a beginner in Spark, trying to use GraphX for an iterative processing by connecting to Kafka Stream Processing Looking for any git reference to real application example, in Scala Please revert with any reference to it, or if someone is trying to build, I could join them

Re: Shortest path performance in Graphx with Spark

Hi Gerard, How are you starting spark? Are you allocating enough RAM for processing? I think the default is 512mb. Try to doing the following and see if it helps (based on the size of your dataset, you might not need all 8gb). $SPARK_HOME/bin/spark-shell \ --master local[4] \

Shortest path performance in Graphx with Spark

Hello everyone, I am creating a graph from a `gz` compressed `json` file of `edge` and `vertices` type. I have put the files in a dropbox folder [here][1] I load and map these `json` records to create the `vertices` and `edge` types required by `graphx` like this: val vertices_raw =

Re: use GraphX with Spark Streaming

.nabble.com/use-GraphX-with-Spark-Streaming-tp24418p24451.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h

GraphX and Spark

Hi, Can Spark achieve whatever GraphX can? Keeping aside the performance comparison between Spark and GraphX, if I want to implement any graph algorithm and I do not want to use GraphX, can I get the work done with Spark? Than You

Re: GraphX and Spark

GraphX is build on *top* of Spark, so Spark can achieve whatever GraphX can. On Wed, Nov 5, 2014 at 9:41 AM, Deep Pradhan pradhandeep1...@gmail.com wrote: Hi, Can Spark achieve whatever GraphX can? Keeping aside the performance comparison between Spark and GraphX, if I want to implement any

Re: How to set persistence level of graph in GraphX in spark 1.0.0

Hi Arpit, To try this: val graph = GraphLoader.edgeListFile(sc, edgesFile, minEdgePartitions = numPartitions, edgeStorageLevel = StorageLevel.MEMORY_AND_DISK, vertexStorageLevel = StorageLevel.MEMORY_AND_DISK) Best, Yifan LI On 28 Oct 2014, at 11:17, Arpit Kumar arp8...@gmail.com

Re: How to set persistence level of graph in GraphX in spark 1.0.0

Hi Yifan LI, I am currently working on Spark 1.0 in which we can't pass edgeStorageLevel as parameter. It implicitly caches the edges. So I am looking for a workaround. http://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.graphx.GraphLoader$ Regards, Arpit On Tue, Oct 28,

Re: How to set persistence level of graph in GraphX in spark 1.0.0

I am not sure if it can work on Spark 1.0, but give it a try. or, Maybe you can try: 1) to construct the edges and vertices RDDs respectively with desired storage level. 2) then, to obtain a graph by using Graph(verticesRDD, edgesRDD). Best, Yifan LI On 28 Oct 2014, at 12:10, Arpit Kumar

Re: How to set persistence level of graph in GraphX in spark 1.0.0

At 2014-10-25 08:56:34 +0530, Arpit Kumar arp8...@gmail.com wrote: GraphLoader1.scala:49: error: class EdgePartitionBuilder in package impl cannot be accessed in package org.apache.spark.graphx.impl [INFO] val builder = new EdgePartitionBuilder[Int, Int] Here's a workaround: 1. Copy and

How to set persistence level of graph in GraphX in spark 1.0.0

Hi all, I am using the GrpahLoader class to load graphs from edge list files. But then I need to change the storage level of the graph to some other thing than MEMORY_ONLY. val graph = GraphLoader.edgeListFile(sc, fname, minEdgePartitions =

Re: Using GraphX with Spark Streaming?

Arko, It would be useful to know more details on the use case you are trying to solve. As Tobias wrote, Spark Streaming works on DStream, which is a continuous series of RDDs. Do check out performance tuning :

Re: Using GraphX with Spark Streaming?

Arko, On Sat, Oct 4, 2014 at 1:40 AM, Arko Provo Mukherjee arkoprovomukher...@gmail.com wrote: Apologies if this is a stupid question but I am trying to understand why this can or cannot be done. As far as I understand that streaming algorithms need to be different from batch algorithms as

Using GraphX with Spark Streaming?

Hello Spark Gurus, I am trying to learn Spark. I am specially interested in GraphX. Since Spark can used in streaming context as well, I wanted to know whether it is possible to use the Spark Toolkits like GraphX or MLlib in the streaming context? Apologies if this is a stupid question but I am

Re: [GraphX] How spark parameters relate to Pregel implementation

At 2014-08-04 20:52:26 +0800, Bin wubin_phi...@126.com wrote: I wonder how spark parameters, e.g., number of paralellism, affect Pregel performance? Specifically, sendmessage, mergemessage, and vertexprogram? I have tried label propagation on a 300,000 edges graph, and I found that no