Hi Xintong Song, I tried using the java options to generate heap dump referring to docs[1] in flink-conf.yaml, however after adding this the task manager containers are not coming up. Note that I am using EMR. Am i doing anything wrong here?
env.java.opts: "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/dump.hprof" Thanks, Hemant On Fri, Mar 5, 2021 at 3:05 PM Xintong Song <tonysong...@gmail.com> wrote: > Hi Hemant, > > This exception generally suggests that JVM is running out of heap memory. > Per the official documentation [1], the amount of live data barely fits > into the Java heap having little free space for new allocations. > > You can try to increase the heap size following these guides [2]. > > If a memory leak is suspected, to further understand where the memory is > consumed, you may need to dump the heap on OOMs and looking for unexpected > memory usages leveraging profiling tools. > > Thank you~ > > Xintong Song > > > [1] > https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html > > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/memory/mem_setup.html > > > > On Fri, Mar 5, 2021 at 4:24 PM bat man <tintin0...@gmail.com> wrote: > >> Hi, >> >> Getting the below OOM but the job failed 4-5 times and recovered from >> there. >> >> j >> >> >> >> >> >> >> >> *ava.lang.Exception: java.lang.OutOfMemoryError: GC overhead limit >> exceeded at >> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:212) >> at >> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.performDefaultAction(SourceStreamTask.java:132) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403) >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) >> at java.lang.Thread.run(Thread.java:748)Caused by: >> java.lang.OutOfMemoryError: GC overhead limit exceeded* >> >> Is there any way I can debug this. since the job after a few re-starts >> started running fine. what could be the reason behind this. >> >> Thanks, >> Hemant >> >
