Our use case is a bit complicated. Essentially our RDD is storing an N x
N matrix A (where each value is a row) and we want to do a computation
which involves another N x N matrix B. However, we have the unfortunate
property that to update the i-th column of A we need to first update
columns 1 to i-1 and we need the i-th row of B.
If B fit on every machine we could simply broadcast it and do a single
map, however that's not the case. So, instead we have a loop from 1 to N
where we broadcast a row of B then use this row in a map to compute a
column of A. However, even much simpler cases of this like the code I'd
posted here result in the master node running out of memory.
We suspect this has something to do with the large lineage A develops.
We're currently trying to checkpoint A after persisting it to break this
lineage up, but this doesn't appear to be working.
Any help here would be greatly appreciated. We'd also love to know if
there's a better way to be doing this, but looking at our dependencies
we believe this is the best we could do.
-Richard
On 05/08/2015 03:26 AM, Akhil Das wrote:
Whats your usecase and what are you trying to achieve? May be there's
a better way of doing it.
Thanks
Best Regards
On Fri, May 8, 2015 at 10:20 AM, Richard Alex Hofer
<[email protected] <mailto:[email protected]>> wrote:
Hi,
I'm working on a project in Spark and am trying to understand
what's going on. Right now to try and understand what's happening
we came up with this snippet of code which very roughly resembles
what we're actually doing. When trying to run this our master node
ends up quickly using up its memory even though all of our RDDs
are very small. Can someone explain what's going on here and how
we can avoid it?
a = sc.parallelize(xrange(100),10)
b = a
for i in xrange(100000):
a = a.map(lambda x: x + 1)
if i % 300 == 0:
# We do this to try and force some of our RDD to evaluate
a.persist()
a.foreachPartition(lambda _: None)
b.unpersist()
b = a
a.collect()
b.unpersist()
-Richard Hofer
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
<mailto:[email protected]>
For additional commands, e-mail: [email protected]
<mailto:[email protected]>
