When talking about the "off-heap" in your most recent message, are you still referring to the task's off-heap configuration value? AFAIK, the HybridMemorySegment shouldn't be directly related to the off-heap parameter.
The HybridMemorySegment can be used as a wrapper around any kind of memory, i.e. byte[]. It can be either used for heap memory but also DirectByteBuffers (located in JVM's direct memory pool which is not part of the JVM's heap) or memory allocated through Unsafe's allocation methods (so-called native memory which is also not part of the JVM's heap). The HybridMemorySegments are utilized within the MemoryManager class. The MemoryManager instances are responsible for maintaining the managed memory used in each of the TaskSlots. Managed Memory is used in different settings (e.g. for the RocksDB state backend in streaming applications). It can be configured using taskmanager.memory.managed.size (or the corresponding *.fraction parameter) [1]. See more details on that in [2]. I'm going to pull in Andrey as he has worked on that topic recently. Best, Matthias [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#taskmanager-memory-managed-size [2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#managed-memory On Wed, Nov 11, 2020 at 12:00 PM Jack Kolokasis <koloka...@ics.forth.gr> wrote: > Hi Matthias, > > Thank you for your reply and useful information. I find that the off-heap > is used when Flink uses HybridMemorySegments. Well, how the Flink knows > when to use these HybridMemorySegments and in which operations this is > happened? > > Best, > Iacovos > On 11/11/20 11:41 π.μ., Matthias Pohl wrote: > > Hi Iacovos, > The task's off-heap configuration value is used when spinning up > TaskManager containers in a clustered environment. It will contribute to > the overall memory reserved for a TaskManager container during deployment. > This parameter can be used to influence the amount of memory allocated if > the user code relies on DirectByteBuffers and/or native memory allocation. > There is no active memory pool management beyond that from Flink's side. > The configuration parameter is ignored if you run a Flink cluster locally. > > Besides this, Flink also utilizes the JVM's using DirectByteBuffers (for > network buffers) and native memory (through Flink's internally used managed > memory) internally. > > You can find a more detailed description of Flink's memory model in [1]. I > hope that helps. > > Best, > Matthias > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#detailed-memory-model > > On Tue, Nov 10, 2020 at 3:57 AM Jack Kolokasis <koloka...@ics.forth.gr> > wrote: > >> Thank you Xuannan for the reply. >> >> Also I want to ask about how Flink uses the off-heap memory. If I set >> taskmanager.memory.task.off-heap.size then which data does Flink allocate >> off-heap? This is handle by the programmer? >> >> Best, >> Iacovos >> On 10/11/20 4:42 π.μ., Xuannan Su wrote: >> >> Hi Jack, >> >> At the moment, Flink doesn't support caching the intermediate result. >> However, there is some ongoing effort to support caching in Flink. >> FLIP-36[1] propose to add the caching mechanism at the Table API. And it >> is planned for 1.13. >> >> Best, >> Xuannan >> >> On Nov 10, 2020, 4:29 AM +0800, Jack Kolokasis <koloka...@ics.forth.gr>, >> wrote: >> >> Hello all, >> >> I am new to Flink and I want to ask if the Flink supports a caching >> mechanism to store intermediate results in memory for machine learning >> workloads. >> >> If yes, how can I enable it and how can I use it? >> >> Thank you, >> Iacovos >> >>
