Github user srowen commented on the issue: https://github.com/apache/spark/pull/14137 I don't think that's true. Each iteration makes a new RDD that depends on the previous iteration's RDD. The execution is actually just a long chain of RDDs if you 'unrolled' it. That's why I think caching actually doesn't do anything here, since no RDD is used more than once. It's also possible I've missed something just reading the code but not seeing it yet. Still if you're saying it doesn't work well with no caching, then there must be some subtle reason the intermediate RDDs do benefit from caching. I'd probably appeal to the author of this code to weigh in in that case.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org