The mostly likely explanation is that you wanted to put all the partitions in memory and they don't all fit. Unless you asked to persist to memory or disk, some partitions will simply not be cached.
Consider using MEMORY_OR_DISK persistence. This can also happen if blocks were lost due to node failure. On Wed, Feb 18, 2015 at 3:19 PM, shahab <[email protected]> wrote: > Hi, > > I have a cached RDD (I can see in UI that it is cached), but when I use this > RDD , I can see that the RDD is partially recomputed (computed) again. It is > "partially" because I can see in UI that some task are skipped (have a look > at the attached figure). > > Now the question is 1: what causes a cached RDD to be recomputed again? and > why somes tasks are skipped and some not?? > > best, > /Shahab > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
