Github user srowen commented on the issue:
https://github.com/apache/spark/pull/14137
I don't think that's true. Each iteration makes a new RDD that depends on
the previous iteration's RDD. The execution is actually just a long chain of
RDDs if you 'unrolled' it. That's why I think caching actually doesn't do
anything here, since no RDD is used more than once. It's also possible I've
missed something just reading the code but not seeing it yet.
Still if you're saying it doesn't work well with no caching, then there
must be some subtle reason the intermediate RDDs do benefit from caching. I'd
probably appeal to the author of this code to weigh in in that case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
