We have almost zero node info – just an identifying integer.
John Lilley
From: Alexis Roos [mailto:alexis.r...@gmail.com]
Sent: Friday, March 11, 2016 11:24 AM
To: Alexander Pivovarov
Cc: John Lilley ; Ovidiu-Cristian MARCU
; lihu ; Andrew A
; u...@spark.incubator.apache.org; Geoff Thompson
to run our software on
1bn edges.
John Lilley
From: Alexander Pivovarov [mailto:apivova...@gmail.com]
Sent: Friday, March 11, 2016 11:13 AM
To: John Lilley
Cc: Ovidiu-Cristian MARCU ; lihu
; Andrew A ;
u...@spark.incubator.apache.org; Geoff Thompson
Subject: Re: Graphx
we use it in prod
70
currentGroupSize++;
}
if (currentGroupSize >= groupSize) {
currentGroupSize = 0;
currentEdge += 2;
} else {
currentEdge++;
}
}
}
}
John Lilley
Chief Architect, RedPoint Global Inc.
T: +1 303 541 1516 | M: +1 720 938 5761 | F: +1 781-705-2077
Skype: jl
currentGroupSize++;
}
if (currentGroupSize >= groupSize) {
currentGroupSize = 0;
currentEdge += 2;
} else {
currentEdge++;
}
}
}
}
John Lilley
Chief Architect, RedPoint Global Inc.
T: +1 303 541 1516 | M: +1 720 938 5761
degrades gracefully along the O(N^2) curve and
additional memory reduces time.
John Lilley
From: Ovidiu-Cristian MARCU [mailto:ovidiu-cristian.ma...@inria.fr]
Sent: Friday, March 11, 2016 8:14 AM
To: John Lilley
Cc: lihu ; Andrew A ;
u...@spark.incubator.apache.org
Subject: Re: Graphx
Hi,
I
ay, March 11, 2016 7:58 AM
To: John Lilley
Cc: Andrew A ; u...@spark.incubator.apache.org
Subject: Re: Graphx
Hi, John:
I am very intersting in your experiment, How can you get that RDD
serialization cost lots of time, from the log or some other tools?
On Fri, Mar 11, 2016 at 8:46 PM,
would get failures. By
contrast, we have a C++ algorithm that solves 1bn edges using memory+disk on a
single 16GB node in about an hour. I think that a very large cluster will do
better, but we did not explore that.
John Lilley
Chief Architect, RedPoint Global Inc.
T: +1 303 541 1516 | M: +1 720
reached? Are there tuning
parameters that optimize for data all fitting in memory vs. data that must
spill?
Thanks,
John Lilley
From: Igor Berman [mailto:igor.ber...@gmail.com]
Sent: Saturday, October 10, 2015 12:06 PM
To: John Lilley
Cc: user@spark.apache.org; Geoff Thompson
Subject: Re
happens when the data set exceed memory,
does it spill to disk "nicely" or degrade catastrophically?
Thanks,
John Lilley