Ryan Skraba created FLINK-33177:
-----------------------------------
Summary: Memory leak in MockStreamingRuntimeContext
Key: FLINK-33177
URL: https://issues.apache.org/jira/browse/FLINK-33177
Project: Flink
Issue Type: Bug
Components: Tests
Reporter: Ryan Skraba
(I noticed this when fixing FLINK-33018)
The three-argument constructor for
[MockStreamingRuntimeContext|https://github.com/apache/flink/blob/ab26175a82a836da9edfaea6325038541e492a3e/flink-streaming-java/src/test/java/org/apache/flink/streaming/util/MockStreamingRuntimeContext.java#L42]
has a memory leak due to a MockEnvironment being created and never closed.
You can reproduce this by running any test that uses this constructor in
IntelliJ with a mode set to "Repeat until fail". After about 16K runs:
{code:java}
#
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5f814c1000,
16384, 0) failed; error='Not enough space' (errno=12)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16384 bytes for committing
reserved memory.
# An error report file with more information is saved as:
#
/home/ryan.skraba/working/apache/flink-connector-gcp-pubsub/flink-connector-gcp-pubsub/hs_err_pid214687.log
[154.974s][warning][os,thread] Failed to start the native thread for
java.lang.Thread "IOManager reader thread #1"
Exception in thread "Thread-48747" java.lang.OutOfMemoryError: unable to create
native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:802)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:97)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:66)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:57)
at
org.apache.flink.runtime.operators.testutils.MockEnvironmentBuilder.build(MockEnvironmentBuilder.java:173)
at
org.apache.flink.streaming.util.MockStreamingRuntimeContext.<init>(MockStreamingRuntimeContext.java:52)
at
org.apache.flink.streaming.connectors.gcp.pubsub.PubSubConsumingTest.lambda$createSourceThread$0(PubSubConsumingTest.java:186)
at java.base/java.lang.Thread.run(Thread.java:833)
[154.977s][warning][os,thread] Attempt to deallocate stack guard pages failed
(0x00007f5f816c1000-0x00007f5f816c5000).
[thread 214689 also had an error]
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f69994e0000,
65536, 0) failed; error='Not enough space' (errno=12)
Disconnected from the target VM, address: '127.0.0.1:40395', transport:
'socket' {code}
or
{code:java}
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f01232ab000,
16384, 0) failed; error='Not enough space' (errno=12)
[thread 330183 also had an error]
[21.295s][warning][os,thread] Failed to start thread "Unknown thread" -
pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k,
detached.
[21.295s][warning][os,thread] Failed to start the native thread for
java.lang.Thread "IOManager reader thread #1"
#
# If you would like to submit a bug report, please visit:
#
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&component=java-17-openjdk-portable&version=37
# {code}
This obviously isn't a big deal, since the tests that use this mock are only
intended to be run once. These errors can be fixed by using the four argument
version of the constructor and explicitly closing the MockEnvironment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
