Hello, As on every Mondays, analytics worker runs to almost 400% CPU load. The process was launched with -Xms6G -Xmx6G, anyway I found this kind repeated error in my logs
java.lang.OutOfMemoryError: GC overhead limit exceeded meaning to me the are probably some memory leaks issues with this application, don't you think so ? By the way, I can watch when the java process is running in "normal condition" and responding ok, that the permanent CPU load is almost 100%. That seems to be very big for a server which does almost nothing at all (practically no data sent to it). Regards, Bernard Le 6 mars 2019 à 12:11, Rushmin Fernando <rush...@wso2.com<mailto:rush...@wso2.com>> a écrit : Hi Bernard, Were you able to resolve your issue? If yes, please ignore the diagnosis steps below. When there is a high CPU usage, 1) The thread break down of the CPU usage of that process should be taken 2) A thread dump of the JVM should be taken Then you can identify the thread which is taking the CPU By reading your descriptions it seems there is a memory related issue in the related WSO2 server. [1] has a good explanation on this topic. Specifically the 'Java heap space' might be a reason for this behaviour. When the allocated memory for the JVM (-XmX) is not enough for the application to run, the garbage collector frequently tries to free up the memory. This process is CPU intensive, thus causes high CPU usage. The application can reach the max allocated heap size due to ... 1) The allocated max heap is not enough for the intended operation of the application. In this case, the needed memory should be planned and allocated. 2) There is a memory leak in the application. Both of the reasons can be identified by analyzing the heap dump. A heap dump is auto-generated when there a heap issue. This is the 'heap-dump.hprof' file you observed. Best Regards, Rushmin [1] - https://plumbr.io/outofmemoryerror [2] - https://www.eclipse.org/mat/ On Mon, Mar 4, 2019 at 1:59 PM Bernard Paris <bernard.pa...@uclouvain.be<mailto:bernard.pa...@uclouvain.be>> wrote: Hello, This is java version java version "1.8.0_192-ea" Java(TM) SE Runtime Environment (build 1.8.0_192-ea-b02) Java HotSpot(TM) 64-Bit Server VM (build 25.192-b02, mixed mode) This morning, back from the week-end and as usual after some 3 days, CPU load was 400%. The JVM rans with options -Xms512M -Xmx4G I just restart it with options -Xms2G -Xmx2G (oracle recommands to make both values equals) When the problem arrives, there is a big file "heap-dump.hprof" in the log directory. A the moment the file was created, no errors in carbon.log. Our system (EI+Analytics) waits for this problem to be solved in order to go into production mode. So now we are still testing and there are very poor datas/trafic to this servers: all mySQL DBs are 32,1 Mb, a cron process is querying the EI server every hours with some tens requests… Thanks for you help, Bernard Le 2 mars 2019 à 11:14, Niveathika Rajendran <niveath...@wso2.com<mailto:niveath...@wso2.com>> a écrit : Hi Bernard, We haven't come up against this issue, in our testing. Could you give the following information to narrow down the issue, 1. Is the minimum requirement for SP[1] available in the system? 2. How much TPS are you seeing from EI server? 3. How much data is currently stored in the DB as of now? [1] https://docs.wso2.com/display/SP430/Installation+Prerequisites Best Regards, Niveathika Rajendran, Senior Software Engineer. Mobile : +94 077 903 7536 [http://c.content.wso2.com/signatures/wso2-signature-general.png] On Thu, Feb 28, 2019 at 2:05 PM Bernard Paris <bernard.pa...@uclouvain.be<mailto:bernard.pa...@uclouvain.be>> wrote: Hi, using wso2 EI 6.4 Analytics worker: after about 3 days of running well, the worker process raise almost 400% CPU and the server doesn't respond to requests anymore. I need to kill it then re-run. Since I'm testing this 6.4 SP software I always had this behavior: it never never rans more than 3 days ;-( I though DB connexion problems could explain this, but no, we do not have DB problems since we use mysql. <PastedGraphic-2.tiff> What could explain this ? … purging process ? Bernard _______________________________________________ Dev mailing list Dev@wso2.org<mailto:Dev@wso2.org> http://wso2.org/cgi-bin/mailman/listinfo/dev _______________________________________________ Dev mailing list Dev@wso2.org<mailto:Dev@wso2.org> http://wso2.org/cgi-bin/mailman/listinfo/dev -- Best Regards Rushmin Fernando Technical Lead WSO2 Inc.<http://wso2.com/> - Lean . Enterprise . Middleware mobile : +94775615183
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
