[ https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183674#comment-15183674 ]
ASF GitHub Bot commented on FLINK-3322: --------------------------------------- Github user Xazax-hun commented on the pull request: https://github.com/apache/flink/pull/1769#issuecomment-193443590 > I would be curious about the soft reference implementation as in DetaIteration cases I think it is a valid situation that the job needs less and less memory. Please add the licence header to the manual test file. This is my first attempt to use a "soft reference pool": https://github.com/Xazax-hun/flink/commit/2694910e53b2f86412f2a9c3e4d83cf1705e3c65 I could not measure any performance gain compared to the code before this pull request as a baseline. Hopefully I will have some extra time tomorrow, so I can further investigate whether there is something wrong with my first implementation or the approach. > MemoryManager creates too much GC pressure with iterative jobs > -------------------------------------------------------------- > > Key: FLINK-3322 > URL: https://issues.apache.org/jira/browse/FLINK-3322 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime > Affects Versions: 1.0.0 > Reporter: Gabor Gevay > Assignee: Gabor Horvath > Priority: Critical > Fix For: 1.0.0 > > > When taskmanager.memory.preallocate is false (the default), released memory > segments are not added to a pool, but the GC is expected to take care of > them. This puts too much pressure on the GC with iterative jobs, where the > operators reallocate all memory at every superstep. > See the following discussion on the mailing list: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html > Reproducing the issue: > https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc > The class to start is malom.Solver. If you increase the memory given to the > JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. > (It will generate some lookuptables to /tmp on first run for a few minutes.) > (I think the slowdown might also depend somewhat on > taskmanager.memory.fraction, because more unused non-managed memory results > in rarer GCs.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)