Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/10846
I'm not saying we should fix just one of them. I'm saying we should treat
them as separate issues. I am a little concerned about the workaround for the
soft refs, for example, and that doesn't need to block the fix for the other
issue.
As for the soft ref issue, I'm not sure I understand what you mean: "Due to
softRef they reach till GC threshold and gets cleared up". Are you saying soft
refs don't get cleaned up when the HadoopRDD instances are collected? Or that
they take longer? Can you clarify what you mean?
If there's a problem with using soft refs here, then maybe a more explicit
collection approach (e.g. a new method in `ContextCleaner` to track these)
could be a better work around. But that assumes that your HadoopRDD instances
are being collected, and if they're not, maybe *that's* the problem.
Or if the caching is not bringing any benefits, maybe just remove the cache
altogether. But with the little information you have provided, it's hard to
know what's the case here.
So, as you see, it's better to keep these two as separate issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
