Re: Strongly Connected Components

2016-11-13 Thread Nicholas Chammas
.@lynxanalytics.com> wrote: > > Hi Shreya, > GraphFrames just calls the GraphX strongly connected components code. ( > https://github.com/graphframes/graphframes/blob/release-0.2.0/src/main/scala/org/graphframes/lib/StronglyConnectedComponents.scala#L51 > ) > > For choosing

Re: Strongly Connected Components

2016-11-12 Thread Koert Kuipers
s.com> wrote: > >> Hi Shreya, >> GraphFrames just calls the GraphX strongly connected components code. ( >> https://github.com/graphframes/graphframes/blob/release-0. >> 2.0/src/main/scala/org/graphframes/lib/StronglyConnec >> tedComponents.scala#L51) >>

Re: Strongly Connected Components

2016-11-12 Thread Koert Kuipers
the exact same algo as is implemented in graphx i think. or is it not? On Fri, Nov 11, 2016 at 7:58 PM, Daniel Darabos < daniel.dara...@lynxanalytics.com> wrote: > Hi Shreya, > GraphFrames just calls the GraphX strongly connected components code. ( > https://github.com/graphframes/gr

RE: Strongly Connected Components

2016-11-11 Thread Shreya Agarwal
Thanks for the detailed response ☺ I will try the things you mentioned! From: Daniel Darabos [mailto:daniel.dara...@lynxanalytics.com] Sent: Friday, November 11, 2016 4:59 PM To: Shreya Agarwal Cc: Felix Cheung ; user@spark.apache.org; Denny Lee Subject: Re: Strongly Connected Components Hi

Re: Strongly Connected Components

2016-11-11 Thread Daniel Darabos
Hi Shreya, GraphFrames just calls the GraphX strongly connected components code. ( https://github.com/graphframes/graphframes/blob/release-0.2.0/src/main/scala/org/graphframes/lib/StronglyConnectedComponents.scala#L51 ) For choosing the number of iterations: If the number of iterations is less

RE: Strongly Connected Components

2016-11-11 Thread Shreya Agarwal
) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190) From: Shreya Agarwal Sent: Thursday, November 10, 2016 8:16 PM To: 'Felix Cheung' ; user@spark.apache.org Subject: RE: Strongly Connected

RE: Strongly Connected Components

2016-11-10 Thread Shreya Agarwal
e.org; Shreya Agarwal Subject: Re: Strongly Connected Components It is possible it is dead. Could you check the Spark UI to see if there is any progress? _ From: Shreya Agarwal mailto:shrey...@microsoft.com>> Sent: Thursday, November 10, 2016 12:45 AM Subject: R

Re: Strongly Connected Components

2016-11-10 Thread Felix Cheung
It is possible it is dead. Could you check the Spark UI to see if there is any progress? _ From: Shreya Agarwal mailto:shrey...@microsoft.com>> Sent: Thursday, November 10, 2016 12:45 AM Subject: RE: Strongly Connected Components To: mailto:user@spark.apac

RE: Strongly Connected Components

2016-11-10 Thread Shreya Agarwal
Bump. Anyone? Its been running for 10 hours now. No results. From: Shreya Agarwal Sent: Tuesday, November 8, 2016 9:05 PM To: user@spark.apache.org Subject: Strongly Connected Components Hi, I am running this on a graph with >5B edges and >3B edges and have 2 questions - 1. What

Strongly Connected Components

2016-11-08 Thread Shreya Agarwal
Hi, I am running this on a graph with >5B edges and >3B edges and have 2 questions - 1. What is the optimal number of iterations? 2. I am running it for 1 iteration right now on a beefy 100 node cluster, with 300 executors each having 30GB RAM and 5 cores. I have persisted the graph to M