[ https://issues.apache.org/jira/browse/FLINK-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402231#comment-15402231 ]
Maximilian Michels commented on FLINK-4094: ------------------------------------------- Thanks for elaborating on your thoughts [~ram_krish]! I was assuming that the memory segments actually would be managed by a pool regardless of preallocation set to false or true. You're right that without preallocation, segments are requested and released as they are needed. This DOES NOT work for the offheap memory segments (at least not how it is implemented now). Two possible fixes that come to my mind: 1) Implement proper clearing of the offheap memory segments and keeping the preallocation:false behavior: http://stackoverflow.com/a/8462690/2225100 2) Use pooling even with preallocation:false which only differs to preallocation:true that the memory is allocated lazily over time. I would say, let's go with 2) which should be easily solvable. {quote} I would rather say that it is better we do internal management of offheap buffers. We should create a pool from which the buffers are allocated and if the pool is of fixed size and we have requests for more buffers than the size of the pool we should allocate them onheap only. (if that is acceptable). {quote} In the case of offheap managed memory, the upper bound of the memory should be respected in the same way as for heap memory. No additional memory should be allocated on the heap instead. The user has to increase the memory size of the job fails. > Off heap memory deallocation might not properly work > ---------------------------------------------------- > > Key: FLINK-4094 > URL: https://issues.apache.org/jira/browse/FLINK-4094 > Project: Flink > Issue Type: Bug > Components: Local Runtime > Affects Versions: 1.1.0 > Reporter: Till Rohrmann > Assignee: ramkrishna.s.vasudevan > Priority: Critical > Fix For: 1.1.0 > > > A user reported that off-heap memory is not properly deallocated when setting > {{taskmanager.memory.preallocate:false}} (per default) [1]. This can cause > the TaskManager process being killed by the OS. > It should be possible to execute multiple batch jobs with preallocation > turned off. No longer used direct memory buffers should be properly garbage > collected so that the JVM process does not exceed it's maximum memory bounds. > [1] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/offheap-memory-allocation-and-memory-leak-bug-td12154.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)