GitHub user mxm opened a pull request:
https://github.com/apache/flink/pull/2499
[FLINK-4485] close and remove user class loader after job completion
Keeping the user class loader around after job completion may lead to
excessive temp space usage because all user jars are kept until the
class loader is garbage collected. Tests showed that garbage collection
can be delayed for a long time after the class loader is not referenced
anymore. Note that for the class loader to not be referenced anymore,
its job has to be removed from the archive.
The fastest way to minimize temp space usage is to close and remove the
URLClassloader after job completion. This requires us to keep a
serializable copy of all data which needs the user class loader after
job completion, e.g. to display data on the web interface.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mxm/flink FLINK-4485
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2499.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2499
----
commit 6ed17b9f5b9c13c80200ccf3db82bbfe727830bb
Author: Maximilian Michels <[email protected]>
Date: 2016-09-15T09:00:58Z
[FLINK-4485] close and remove user class loader after job completion
Keeping the user class loader around after job completion may lead to
excessive temp space usage because all user jars are kept until the
class loader is garbage collected. Tests showed that garbage collection
can be delayed for a long time after the class loader is not referenced
anymore. Note that for the class loader to not be referenced anymore,
its job has to be removed from the archive.
The fastest way to minimize temp space usage is to close and remove the
URLClassloader after job completion. This requires us to keep a
serializable copy of all data which needs the user class loader after
job completion, e.g. to display data on the web interface.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
