Re: configuration needed to run twitter(25GB) dataset

2014-08-01 Thread shijiaxin
://apache-spark-user-list.1001560.n3.nabble.com/configuration-needed-to-run-twitter-25GB-dataset-tp11044p11150.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

configuration needed to run twitter(25GB) dataset

2014-07-31 Thread Jiaxin Shi
We have a 6-nodes cluster , each node has 64GB memory. here is the command: ./bin/spark-submit --class org.apache.spark.examples.graphx.LiveJournalPageRank examples/target/scala-2.10/spark-examples-1.0.1-hadoop1.0.4.jar hdfs://dataset/twitter --tol=0.01 --numEPart=144 --numIter=10 But it ran out