Hi Matthias,
Thank you for your reply and useful information. I find that the
off-heap is used when Flink uses HybridMemorySegments. Well, how the
Flink knows when to use these HybridMemorySegments and in which
operations this is happened?
Best,
Iacovos
On 11/11/20 11:41 π.μ., Matthias Pohl wrote:
Hi Iacovos,
The task's off-heap configuration value is used when spinning up
TaskManager containers in a clustered environment. It will contribute
to the overall memory reserved for a TaskManager container during
deployment. This parameter can be used to influence the amount of
memory allocated if the user code relies on DirectByteBuffers and/or
native memory allocation. There is no active memory pool management
beyond that from Flink's side. The configuration parameter is ignored
if you run a Flink cluster locally.
Besides this, Flink also utilizes the JVM's using DirectByteBuffers
(for network buffers) and native memory (through Flink's internally
used managed memory) internally.
You can find a more detailed description of Flink's memory model in
[1]. I hope that helps.
Best,
Matthias
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html#detailed-memory-model
On Tue, Nov 10, 2020 at 3:57 AM Jack Kolokasis <koloka...@ics.forth.gr
<mailto:koloka...@ics.forth.gr>> wrote:
Thank you Xuannan for the reply.
Also I want to ask about how Flink uses the off-heap memory. If I
set taskmanager.memory.task.off-heap.size then which data does
Flink allocate off-heap? This is handle by the programmer?
Best,
Iacovos
On 10/11/20 4:42 π.μ., Xuannan Su wrote:
Hi Jack,
At the moment, Flink doesn't support caching the intermediate
result. However, there is some ongoing effort to support caching
in Flink.
FLIP-36[1] propose to add the caching mechanism at the Table API.
And it is planned for 1.13.
Best,
Xuannan
On Nov 10, 2020, 4:29 AM +0800, Jack Kolokasis
<koloka...@ics.forth.gr <mailto:koloka...@ics.forth.gr>>, wrote:
Hello all,
I am new to Flink and I want to ask if the Flink supports a caching
mechanism to store intermediate results in memory for machine
learning
workloads.
If yes, how can I enable it and how can I use it?
Thank you,
Iacovos
