Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1387#issuecomment-49068556
So that's the problem. If ContextCleaner is relying on the gc to clean up
after these references, and leaving those references "uncleaned" causes
problems, then ContextCleaner needs to be fixed to either explicitly clean up
these references when executors fail, or find some other way of doing it. Doing
it through `System.gc()` is not a fix, because it's not guaranteed to work.
But I'm actually not seeing the problem here at all. From your driver log,
I'm not seeing the driver fail because of the OOM. I'm seeing the driver fail
because the same task failed 4 times. Which is to be expected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
