thank you, it works
after my operation over p, return p.toIterator, because mapPartitions has
iterator return type, is that right?
rdd.mapPartitions{D = {val p = D.toArray; ...; p.toIterator}}
--
View this message in context:
Yeah, should be right
--
Nan Zhu
On Wednesday, April 9, 2014 at 8:54 PM, wxhsdp wrote:
thank you, it works
after my operation over p, return p.toIterator, because mapPartitions has
iterator return type, is that right?
rdd.mapPartitions{D = {val p = D.toArray; ...; p.toIterator}}
so, the data structure looks like:
D consists of D1, D2, D3 (DX is partition)
and
DX consists of d1, d2, d3 (dx is the part in your context)?
what you want to do is to transform
DX to (d1 + d2, d1 + d3, d2 + d3)?
Best,
--
Nan Zhu
On Tuesday, April 8, 2014 at 8:09 AM, wxhsdp wrote:
yes, how can i do this conveniently? i can use filter, but there will be so
many RDDs and it's not concise
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Only-TraversableOnce-tp3873p3875.html
Sent from the Apache Spark User List mailing list archive at
If that’s the case, I think mapPartition is what you need, but it seems that
you have to load the partition into the memory as whole by toArray
rdd.mapPartition{D = {val p = D.toArray; ...}}
--
Nan Zhu
On Tuesday, April 8, 2014 at 8:40 AM, wxhsdp wrote:
yes, how can i do this
thank you for your help! let me have a try
Nan Zhu wrote
If that’s the case, I think mapPartition is what you need, but it seems
that you have to load the partition into the memory as whole by toArray
rdd.mapPartition{D = {val p = D.toArray; ...}}
--
Nan Zhu
On Tuesday, April
