That was helpful!
The conclusion:
(1) The mappers are not executed in parallel when processing independently
the same RDD.
(2) The best way seems to be (if enough memory is available and an action is
applied to d1 and d2 later on)
val d1 = data.map((x,y,z) => (x,y)).cache
val d2 = d1.map((x,y) => (y,x))
- This avoids pipelining the "d1" mapper and "d2" mapper when
computing d2
This is important to write efficient code, toDebugString helps a lot.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Caching-and-Actions-tp22418p22444.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
