[controller-dev] Bug 7370 OOM due to suspected memory leak in akka.dispatch.Dispatcher found by hprof

2017-05-29 Thread Michael Vorburger
Hi guys, I just ran MAT([1]) over an HPROF heap dump on OOM in Bug 7370, and it (MAT) raises a "leak suspect" in akka.dispatch.Dispatcher - see the [3] java_pid19570_Leak_Suspects.zip just attached to Bug 7370 ... questions: Is this perhaps something you jump at with an "ah that, we know about it

Re: [controller-dev] [mdsal-dev] Bug 7370 OOM due to suspected memory leak in akka.dispatch.Dispatcher found by hprof

2017-05-29 Thread Tom Pantelis
It looks like the Dispatcher was for data change notifications. I suspect a listener was hung or responding slowly so the actor's mailbox filled up with change notifications. I would suggest getting a thread dump next time. On Mon, May 29, 2017 at 7:52 AM, Michael Vorburger wrote: > Hi guys, > >

Re: [controller-dev] [mdsal-dev] Bug 7370 OOM due to suspected memory leak in akka.dispatch.Dispatcher found by hprof

2017-05-29 Thread Michael Vorburger
+openflowplugin-dev & +ovsdb-dev: Tom, On Mon, May 29, 2017 at 2:57 PM, Tom Pantelis wrote: Thanks a lot for replying, really appreciate it! It looks like the Dispatcher was for data change notifications. I suspect a > listener was hung or responding slowly so the actor's mailbox filled up > w

Re: [controller-dev] [mdsal-dev] Bug 7370 OOM due to suspected memory leak in akka.dispatch.Dispatcher found by hprof

2017-05-29 Thread Tom Pantelis
yeah that looks like an issue. DeviceInitializationUtils is doing a blocking get on a Future which is usually not a good thing. And it occurred via an EOS data change and is blocking an akka Dispatcher thread. On a side note, there's a lot of threads with io.netty.util.concurrent.FastThreadLocalTh

Re: [controller-dev] [mdsal-dev] Bug 7370 OOM due to suspected memory leak in akka.dispatch.Dispatcher found by hprof

2017-05-29 Thread Muthukumaran K
I agree with Tom. Had experienced this earlier due to a laggard listener back before stable carbon cut. Eventually , listener had got fixed (not in connection with this observation but I had just picked up the build spaced well after the original build) and I had not been able to repro the same

Re: [controller-dev] [mdsal-dev] Bug 7370 OOM due to suspected memory leak in akka.dispatch.Dispatcher found by hprof

2017-05-29 Thread Muthukumaran K
Regarding Netty ‘s io.netty.util.concurrent.FastThreadLocalThread observation https://github.com/netty/netty/issues/6565 Issue seems very recent. Looking further what could be the reason Regards Muthu From: controller-dev-boun...@lists.opendaylight.org [mailto:controller-dev-boun...@lists.ope