Hi,
We saw a similar issue in one of our job due to ByteBuffer memory leak[1].
We fixed it using the solution in the article, setting
-Djdk.nio.maxCachedBufferSize
This variable is available for Java > 8u102
Best regards,
Kien
[1]http://www.evanjones.ca/java-bytebuffer-leak.html
On 10/18/2017 4:06 PM, Flavio Pompermaier wrote:
We also faced the same problem, but the number of jobs we can run
before restarting the cluster depends on the volume of the data to
shuffle around the network. We even had problems with a single job and
in order to avoid OOM issues we had to put some configuration to limit
Netty memory usage, i.e.:
- Add to flink.yaml -> env.java.opts:
-Dio.netty.recycler.maxCapacity.default=1
- Edit taskmanager.sh and change TM_MAX_OFFHEAP_SIZE from 8388607T to 5g
At this purpose we wrote a small test to reproduce the problem and we
opened an issue for that [1].
We still don't know if the problems are related however..
I hope that could be helpful,
Flavio
[1] https://issues.apache.org/jira/browse/FLINK-7845
<https://issues.apache.org/jira/browse/FLINK-7845>
On Wed, Oct 18, 2017 at 10:48 AM, Javier Lopez
<javier.lo...@zalando.de <mailto:javier.lo...@zalando.de>> wrote:
Hi Robert,
Sorry to reply this late. We did a lot of tests, trying to
identify if the problem was in our custom sources/sinks. We
figured out that none of our custom components is causing this
problem. We came up with a small test, and realized that the Flink
nodes run out of non-heap JVM memory and crash after deployment of
thousands of jobs.
When rapidly deploying thousands or hundreds of thousands of Flink
jobs - depending on job complexity in terms of resource
consumption - Flink nodes non-heap JVM memory consumption grows
until there is no more memory left on the machine and the Flink
process crashes. Both TaskManagers and JobManager exhibit the same
behavior. The TaskManagers die faster though. The memory
consumption doesn't decrease after stopping the deployment of new
jobs, with the cluster being idle (no running jobs).
We could replicate the behavior by the rapid deployment of the
WordCount Job provided in the Quickstart with a Python script. We
started 24 instances of the deployment script to run in parallel.
The non-heap JVM memory consumption grows faster with more complex
jobs, i.e. reading from Kafka 10K events and printing to STDOUT( *
). Thus less deployed jobs are needed until the
TaskManagers/JobManager dies.
We employ Flink 1.3.2 in standalone mode on AWS EC2 t2.large nodes
with 4GB RAM inside Docker containers. For the test, we used 2
TaskManagers and 1 JobManager.
( * ) a slightly changed Python script was used, which waited
after deployment 15 seconds for the 10K events to be read from
Kafka, then it canceled the freshly deployed job via Flink REST API.
If you want we can provide the Scripts and Jobs we used for this
test. We have a workaround for this, which restarts the Flink
nodes once a memory threshold is reached. But this has lowered the
availability of our services.
Thanks for your help.
On 30 August 2017 at 10:39, Robert Metzger <rmetz...@apache.org
<mailto:rmetz...@apache.org>> wrote:
I just saw that your other email is about the same issue.
Since you've done a heapdump already, did you see any pattern
in the allocated objects? Ideally none of the classes from
your user code should stick around when no job is running.
What's the size of the heap dump? I'm happy to take a look at
it if it's reasonably small.
On Wed, Aug 30, 2017 at 10:27 AM, Robert Metzger
<rmetz...@apache.org <mailto:rmetz...@apache.org>> wrote:
Hi Javier,
I'm not aware of such issues with Flink, but if you could
give us some more details on your setup, I might get some
more ideas on what to look for.
are you using the RocksDBStateBackend? (RocksDB is doing
some JNI allocations, that could potentially leak memory)
Also, are you passing any special garbage collector
options? (Maybe some classes are not unloaded)
Are you using anything else that is special (such as
protobuf or avro formats, or any other big library)?
Regards,
Robert
On Mon, Aug 28, 2017 at 5:04 PM, Javier Lopez
<javier.lo...@zalando.de <mailto:javier.lo...@zalando.de>>
wrote:
Hi all,
we are starting a lot of Flink jobs (streaming), and
after we have started 200 or more jobs we see that the
non-heap memory in the taskmanagers increases a lot,
to the point of killing the instances. We found out
that every time we start a new job, the committed
non-heap memory increases by 5 to 10MB. Is this an
expected behavior? Are there ways to prevent this?
