If I read the code correctly, in RDD.scala, each rdd keeps track of it's own dependencies, (from Dependency.scala), and has methods to access to it's /ancestors/ dependencies, thus being able to recompute the lineage (see getNarrowAncestors() or getDependencies() in some rdd like UnionRDD).
So it doesn't looks like an RDD knows the whole lineage graph without having to compute it, nor does that an RDD gives more than it's own identity as a parent to a child RDD. As a new user I may be mistaken so any veteran confirmation would be appreciated :) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-does-lineage-get-passed-down-in-RDDs-tp23196p23212.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
