Hi Giulio Since JobPoller mostly deals with creating a ThreadPool and your description says you seem to have stuck on the poller, it looks like there is a possibility of hung threads resulting in OOM. But, again, this is based on cursory glance and might require further investigation based on thread and heap dump analysis.
Too many hung threads can also resulting in OOM and of course memory leak can play its part. But you will have a better idea if you have thread and heap dump with you. Do you see hug thread stacktrace in the log files? I think it'd help you have JVM properly configured to dump "heap dump" as soon as OOM occurs. With the team of specialists you have, I am hoping you already have that configured. For the thread dump, you can always request JVM (may be with jconsole or jstack or visualvm) to dump "thread dump" as you see spike in the CPU usage. It may also be premature to cast doubt on JobPoller only; for, it is very much possible for other components to play their part. Things can get a bit tricky when you are dealing with such a large amount of data. So, it is best to have thread and heap dump for the necessary analysis and then have them analysed with your choice of tools to determine the root cause. Thanks and Best regards, Girish Vasmatkar HotWax Systems On Thu, Sep 20, 2018 at 10:39 PM Taher Alkhateeb <slidingfilame...@gmail.com> wrote: > This smells like a memory leak. The problem is how to pin down the exact > cause. Maybe something in network setting, or the job poller or something > else. > > Perhaps the first thing to try to do to narrow it down is to run a profiler > against the JVM to see where the leak is happening. If you can narrow down > the class / method then it would be much easier to handle. > > On Thu, Sep 20, 2018, 7:43 PM Giulio Speri - MpStyle Srl < > giulio.sp...@mpstyle.it> wrote: > > > Hello everyone, > > > > hope you're doing good. > > I am writing, because I am struggling with a quite strange problem, over > an > > ofbiz installation, for one of our customers. > > This installation is composed by two instances of OFBiz (v13.07.03), > served > > via an Apache Tomcat webserver, along with a load balancer. > > The database server is MariaDB. > > > > We had the first problems, about 3 weeks ago, when suddenly, the front1 > > (ofbiz instance 1), stopped serving web requests; front2, instead, was > > still working correctly. > > > > Obviously we checked the log files, and we saw that async services were > > failing; the failure was accompanied by this error line: > > > > *Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead > limit > > exceeded* > > > > We analyzed the situation with our system specialists, and they told us > > that the application was highly stressing machine resources (cpu always > at > > or near 100%, RAM usage rapidly increasing), until the jvm run out of > > memory. > > This "resource-high-consumption situation", occurred only when ofbiz1 > > instance was started with the JobPoller enabled; if the JobPoller was not > > enabled, ofbiz run with low resource usage. > > > > We then focused on the db, to check first of all the dimensions; the > result > > was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about > > 18 GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR > > (about 2 GB). > > All the other tables had a size in the order of few MB each. > > > > The first thing we did, was to clear all those tables, reducing > > considerably the db size. > > After the cleaning, we tried to start ofbiz1 again, with the JobPoller > > component enabled; this caused a lot of old scheduled/queued jobs, to > > execute. > > Except than for the start-up time, the resource usage of the machine, > > stabilized around normal to low values (cpu 1-10%). > > Ofbiz seemed to work (web request was served), but we noticed taht the > > JobPoller did not schedule or run jobs, anymore. > > The number of job in "Pending" state in the JobSandbox entity was small > > (about 20); no Queued, no Failed, no jobs in other states. > > In addition to this, unfortunately, after few hours, jvm run out of > memory > > again. > > > > Our jvm has an heap maximum size of 20GB ( we have 32GB on the machine), > > so it's not so small, I think. > > The next step we're going to do is set-up locally the application over > the > > same production db to see what happens. > > > > Now that I explained the situation, I am going to ask if, in your > > opinion/experience: > > > > Could the JobPoller component be the root (and only) cause of the > > OutOfMemory of the jvm? > > > > Could this issue be related to this > > https://issues.apache.org/jira/browse/OFBIZ-5710? > > > > Dumping and analyzing the heap of the jvm could help in some way to > > understand what or who fills the memory or is this operation a waste of > > time? > > > > Is there something that we did not considered or missed during the whole > > process of problem analysis? > > > > > > I really thank you all for your attention and your help; any suggestion > or > > advice would really be greatly appreciated. > > > > Kind regards, > > Giulio > > > > > > > > > > > > > > -- > > Giulio Speri > > > > > > *Mp Styl**e Srl* > > via Antonio Meucci, 37 > > 41019 Limidi di Soliera (MO) > > T 059/684916 > > M 334/3779851 > > > > www.mpstyle.it > > >
