Hi, We are also running Connected Components test with GraphX. We ran experiments using Spark 1.6.1 on machine which have 16 cores with 2-way and run only a single executor per machine. We got this result: Facebook-like graph with 2^24 edges, using 4 executors with 90GB each, it took 100 seconds to find Connected component. It takes 600s when we tried to increase the number of edges to 2^27. We are so interested in how you can get such good results. We will be so appreciated if you could answer my following questions: 1. Which Connected component code did you use? Did you use the default org.apache.spark.graphx.ConnectedComponents lib which implements using Pregel?Have you made any changes? 2.By saying 20 cores with 2-way,did you mean total 40 threads cpu? 3. Addition to the settings you have mentioned,have you made any other changes in files spark-default.conf and spark-env.sh? Could you please just paste the two files so that we can compare? 4.When you mean Parallel GC, could you please give more detail guides on how to optimize this setting?which parameters should we set?
Appreciating for any feedback! Thank you, Yilei On 2016-06-16 09:01 (+0800), Maja Kabiljo wrote: > Hi,> > > We are running some experiments with GraphX in order to compare it with other > systems. There are multiple settings which significantly affect performance, > and we experimented a lot in order to tune them well. I'll share here what > are the best we found so far and which results we got with them, and would > really appreciate if anyone who used GraphX before has any advice on what > else can make it even better, or confirm that these results are as good as it > gets.> > > Algorithms we used are pagerank and connected components. We used Twitter and > UK graphs from the GraphX paper > (https://amplab.cs.berkeley.edu/wp-content/uploads/2014/09/graphx.pdf), and > also generated graphs with properties similar to Facebook social graph with > various number of edges. Apart from performance we tried to see what is the > minimum amount of resources it requires in order to handle graph of some > size.> > > We ran experiments using Spark 1.6.1, on machines which have 20 cores with > 2-way SMT, always fixing number of executors (min=max=initial), giving 40GB > or 80GB per executor, and making sure we run only a single executor per > machine. Additionally we used:> > > * spark.shuffle.manager=hash, spark.shuffle.service.enabled=false> > * Parallel GC> > * PartitionStrategy.EdgePartition2D> > * 8*numberOfExecutors partitions> > > Here are some data points which we got:> > > * Running on Facebook-like graph with 2 billion edges, using 4 executors with > 80GB each it took 451 seconds to do 20 iterations of pagerank and 236 seconds > to find connected components. It failed when we tried to use 2 executors, or > 4 executors with 40GB each.> > * For graph with 10 billion edges we needed 16 executors with 80GB each (it > failed with 8), 1041 seconds for 20 iterations of pagerank and 716 seconds > for connected components.> > * Twitter-2010 graph (1.5 billion edges), 8 executors, 40GB each, pagerank > 473s, connected components 264s. With 4 executors 80GB each it worked but was > struggling (pr 2475s, cc 4499s), with 8 executors 80GB pr 362s, cc 255s.> > > One more thing, we were not able to reproduce what's mentioned in the paper > about fault tolerance (section 5.2). If we kill an executor during first few > iterations it recovers successfully, but if killed in later iterations > reconstruction of each iteration starts taking exponentially longer and > doesn't finish after letting it run for a few hours. Are there some > additional parameters which we need to set in order for this to work?> > > Any feedback would be highly appreciated!> > > Thank you,> > Maja> > ???? iPhone
