[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2499 Merging after tests pass. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2499 This needed another fix because in some tests we use the system class loader instead of a class loader instantiated by the BlobLibraryCacheManager. If we close that one, we cause tests to fail. The solution is to close only `FlinkUserCodeClassLoader`s. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2499 Looks good to me. +1 to merge when tests pass --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2499 @StephanEwen Yes, it is simple. I just pushed a commit. This now releases all temp files after job completion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2499 What would the fix for the TaskManager look like? Simply explicitly closing the UserCodeClassloader, or does it need more? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2499 Thanks! Just a few words to @nielsbasjes who reported the issue. I've tested the fix using the test instructions you provided. Even before this fix, I could get rid of the temp files by forcing a manual garbage collection on the JVM, using `jcmd GC.run`. However, that only worked once the job meta data had been removed from the archive, i.e. it doesn't show up in the web interface anymore. With this fix, the class loader is cleared upon job completion and the files are immediately removed. `lsof | fgrep blob_` didn't show any of these files anymore. Note, that we don't perform any cleanup on the TaskManager side. There we also wind up with some left over files but they don't seem to pile up. It must be that the garbage collector can figure out when to clean much earlier. Plus, we don't keep a reference to old Task instances like we do for the web interface on the JobManager side. @StephanEwen I'm thinking about adding a similar fix for the TaskManager side. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2499 Looks food to me. +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
