Hello Giulio, You can split your OFBiz instance into two instances[1], one for processing synchronous web requests and another for processing only jobs(async server). In this way, you may narrow down your analyses.
[1]: https://www.hotwaxsystems.com/ofbiz/ofbiz-tutorials/apache-ofbiz-performance/ Kind Regards, Arun Patidar Director of Information Systems *HotWax CommerceReal OmniChannel. Real Results.* m: +91 9827353082 w: www.hotwax.co <https://www.linkedin.com/company/hotwaxcommerce/> <https://www.facebook.com/HotWaxCommerce/> <https://twitter.com/hotwaxcommerce> On Thu, Sep 20, 2018 at 10:39 PM Taher Alkhateeb <slidingfilame...@gmail.com> wrote: > This smells like a memory leak. The problem is how to pin down the exact > cause. Maybe something in network setting, or the job poller or something > else. > > Perhaps the first thing to try to do to narrow it down is to run a profiler > against the JVM to see where the leak is happening. If you can narrow down > the class / method then it would be much easier to handle. > > On Thu, Sep 20, 2018, 7:43 PM Giulio Speri - MpStyle Srl < > giulio.sp...@mpstyle.it> wrote: > > > Hello everyone, > > > > hope you're doing good. > > I am writing, because I am struggling with a quite strange problem, over > an > > ofbiz installation, for one of our customers. > > This installation is composed by two instances of OFBiz (v13.07.03), > served > > via an Apache Tomcat webserver, along with a load balancer. > > The database server is MariaDB. > > > > We had the first problems, about 3 weeks ago, when suddenly, the front1 > > (ofbiz instance 1), stopped serving web requests; front2, instead, was > > still working correctly. > > > > Obviously we checked the log files, and we saw that async services were > > failing; the failure was accompanied by this error line: > > > > *Thread "AsyncAppender-async" java.lang.OutOfMemoryError: GC overhead > limit > > exceeded* > > > > We analyzed the situation with our system specialists, and they told us > > that the application was highly stressing machine resources (cpu always > at > > or near 100%, RAM usage rapidly increasing), until the jvm run out of > > memory. > > This "resource-high-consumption situation", occurred only when ofbiz1 > > instance was started with the JobPoller enabled; if the JobPoller was not > > enabled, ofbiz run with low resource usage. > > > > We then focused on the db, to check first of all the dimensions; the > result > > was disconcerting; 45GB, mainly divided on four tables: SERVER_HIT (about > > 18 GB), VISIT (about 15 GB), ENTITY_SYNC_REMOVE (about 8 GB), VISITOR > > (about 2 GB). > > All the other tables had a size in the order of few MB each. > > > > The first thing we did, was to clear all those tables, reducing > > considerably the db size. > > After the cleaning, we tried to start ofbiz1 again, with the JobPoller > > component enabled; this caused a lot of old scheduled/queued jobs, to > > execute. > > Except than for the start-up time, the resource usage of the machine, > > stabilized around normal to low values (cpu 1-10%). > > Ofbiz seemed to work (web request was served), but we noticed taht the > > JobPoller did not schedule or run jobs, anymore. > > The number of job in "Pending" state in the JobSandbox entity was small > > (about 20); no Queued, no Failed, no jobs in other states. > > In addition to this, unfortunately, after few hours, jvm run out of > memory > > again. > > > > Our jvm has an heap maximum size of 20GB ( we have 32GB on the machine), > > so it's not so small, I think. > > The next step we're going to do is set-up locally the application over > the > > same production db to see what happens. > > > > Now that I explained the situation, I am going to ask if, in your > > opinion/experience: > > > > Could the JobPoller component be the root (and only) cause of the > > OutOfMemory of the jvm? > > > > Could this issue be related to this > > https://issues.apache.org/jira/browse/OFBIZ-5710? > > > > Dumping and analyzing the heap of the jvm could help in some way to > > understand what or who fills the memory or is this operation a waste of > > time? > > > > Is there something that we did not considered or missed during the whole > > process of problem analysis? > > > > > > I really thank you all for your attention and your help; any suggestion > or > > advice would really be greatly appreciated. > > > > Kind regards, > > Giulio > > > > > > > > > > > > > > -- > > Giulio Speri > > > > > > *Mp Styl**e Srl* > > via Antonio Meucci, 37 > > 41019 Limidi di Soliera (MO) > > T 059/684916 > > M 334/3779851 > > > > www.mpstyle.it > > >
