[
https://issues.apache.org/jira/browse/FLINK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101695#comment-18101695
]
Yordan Pavlov commented on FLINK-40327:
---------------------------------------
I can work and this and provide a PR, thanks.
> ForSt async multiGet leaks a native ReadOptions per read batch
> --------------------------------------------------------------
>
> Key: FLINK-40327
> URL: https://issues.apache.org/jira/browse/FLINK-40327
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
> Reporter: Yordan Pavlov
> Priority: Critical
> Attachments: after.png, before.png
>
>
> h4. Problem observed:
> The problem I observed unlimited memory increase of a Flink Kubernetes pod.
> Observed in production (Flink 2.3.0, ForSt disaggregated + async State V2,
> high read volume ~25k rec/s). TaskManager RSS grows ~2.6 GiB/h while
> heap/managed/network stay flat. OOMKill after ~10–14 h.
> h4. Investigation:
> jemalloc allocation profiling attributes the live-allocation
> growth to Java_org_forstdb_ReadOptions_newReadOptions (+530 MiB live in a
> 12-minute steady-state window).
>
> ForStGeneralMultiGetOperation.process() creates a new ReadOptions() inside
> the per-split executor lambda and no code path closes it — not the early
> returns, not the exception paths, not normal completion. ReadOptions is a JNI
> wrapper owning a native
> rocksdb::ReadOptions; without close() the native allocation is never freed,
> so memory leaks proportionally to the number of executed async read batches,
> outside all JVM/Flink memory accounting.
> All other ReadOptions/WriteOptions constructions in the module are
> lifecycle-managed (handlesToClose, toClose, closeQuietly in close()); this is
> the only unmanaged one.
> h4. Fix:
> Open the ReadOptions with try-with-resources spanning the lambda body. Safe
> because multiGetAsList is synchronous and returns byte[] copies, so nothing
> referencing the ReadOptions outlives the batch. We run this patched in
> production. Happy to open a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
