Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1387#issuecomment-49068556 So that's the problem. If ContextCleaner is relying on the gc to clean up after these references, and leaving those references "uncleaned" causes problems, then ContextCleaner needs to be fixed to either explicitly clean up these references when executors fail, or find some other way of doing it. Doing it through `System.gc()` is not a fix, because it's not guaranteed to work. But I'm actually not seeing the problem here at all. From your driver log, I'm not seeing the driver fail because of the OOM. I'm seeing the driver fail because the same task failed 4 times. Which is to be expected.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---