I think I'm getting more confused the longer this thread goes. So rdd1.dependencies provides immediate parents to rdd1. For now i'm going to walk my internal DAG from the root down and see where running the caching of siblings concurrently gets me.
I still like your point, Sean, about trying to do this at the partition level. I'll try something to see where I can get with that next. On Thu, Feb 26, 2015 at 7:46 PM, Sean Owen <so...@cloudera.com> wrote: > I think we already covered that in this thread. You get dependencies > from RDD.dependencies() > > On Fri, Feb 27, 2015 at 12:31 AM, Zhan Zhang <zzh...@hortonworks.com> > wrote: > > Currently in spark, it looks like there is no easy way to know the > > dependencies. It is solved at run time. > > > > Thanks. > > > > Zhan Zhang > > > > On Feb 26, 2015, at 4:20 PM, Corey Nolet <cjno...@gmail.com> wrote: > > > > Ted. That one I know. It was the dependency part I was curious about >
