Hi Donatien,
The managed memory in Flink actually locates off-heap and stays as native
memory, e.g memory consumed by RocksDB, python. In other words, JVM cannot know
how much the memory that third-party software used.
Thus, Flink will just set the upper limit for the managed memory and let
third-party software to consume. That's why you can see the managed memory is
always allocated full at the beginning.
And if you want to know the memory used by RocksDB, you should use jeprof +
jemalloc to detect the malloced memory from RocksDB side, or refer to RocksDB
reported block cache usage [1]. Please note that all RocksDB instances within
same slot would share the same block cache, they will report same usage.
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#state-backend-rocksdb-metrics-block-cache-usage
Best
Yun Tang
From: Donatien Schmitz
Sent: Tuesday, March 15, 2022 19:45
To: user@flink.apache.org
Subject: RocksDB metrics for effective memory consumption
Hi,
I am working on the analysis of the memory consumption of RocksDB state backend
for simple DAGs. I would like to check fine-grained memory utilization of
RocksDB with the native metrics (reported on Prometheus+Grafana). RocksDB uses
Managed memory allocated to each TaskManager but this value peaks at the
beginning of the job. Is the managed memory always allocated at full even if it
would not be necessary?
For my experiments I am using a simple DAG consisting of Source (FS) -> Map ->
DiscardSink. The Map does not process anything but stores the latest value of
the KeyedStream keys (with predicted amount of keys in the dataset and constant
value size (1024 bytes)).
I anyone has some more insights on the memory utilization of RocksDB at Flink's
level, I would appreciate.
Best,
Donatien Schmitz
PhD Student
