Feng Jiajie created FLINK-35737:
-----------------------------------
Summary: Prevent Memory Leak by Closing
MemoryExecutionGraphInfoStore on MiniCluster Shutdown
Key: FLINK-35737
URL: https://issues.apache.org/jira/browse/FLINK-35737
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Affects Versions: 1.18.1
Reporter: Feng Jiajie
Fix For: 1.18.2, 1.20.0, 1.19.2
MemoryExecutionGraphInfoStore registers a ShutdownHook upon construction and
deregisters it within its close() method.
{code:java}
public MemoryExecutionGraphInfoStore(...) {
...
this.shutdownHook = ShutdownHookUtil.addShutdownHook(this,
getClass().getSimpleName(), LOG);
}
@Override
public void close() throws IOException {
...
// Remove shutdown hook to prevent resource leaks
ShutdownHookUtil.removeShutdownHook(shutdownHook,
getClass().getSimpleName(), LOG);
}{code}
Currently, MiniCluster instantiates a MemoryExecutionGraphInfoStore object but
doesn't retain a reference to it, nor does it call close() during its own
shutdown process.
{code:java}
final DispatcherResourceManagerComponent
dispatcherResourceManagerComponent =
dispatcherResourceManagerComponentFactory.create(
...
new MemoryExecutionGraphInfoStore(), // -> new
...); {code}
This behavior leads to an accumulation of ShutdownHooks when running multiple
Flink jobs within the same local JVM. These accumulating hooks, along with
their associated references, contribute to a memory leak.
This patch addresses the issue by ensuring that MemoryExecutionGraphInfoStore's
close() method is invoked during MiniCluster shutdown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
