Hi Firzhan The cluster have the following configuration currently. The default value is 256m
Xms512m -Xmx2048m -XX:MaxPermSize=512m Please advice with the recommended value and please note that this configuration is not given in the documentation On Mon, Oct 12, 2015 at 10:56 AM, Firzhan Naqash <[email protected]> wrote: > Hi Chathiruka, > > This seems to be the hazlecast OOM error observed with kernel 4.4.1. Can > you increase heap size and check this scenario again. > > > *TID: [-1] [] [2015-10-09 18:49:36,827] ERROR {org.apache.tomcat.util.net > <http://org.apache.tomcat.util.net>.NioEndpoint$Poller} - > {org.apache.tomcat.util.net > <http://org.apache.tomcat.util.net>.NioEndpoint$Poller}java.lang.OutOfMemoryError: > GC overhead limit exceeded* > > Regards, > Firzhan > > > -- > *Firzhan Naqash* > Senior Software Engineer - Integration Platform Team > WSO2 Inc. http://wso2.com > > email: [email protected] <[email protected]> > mobile: (+94) 77 9785674 <%28%2B94%29%2071%205247551>*| > blog: http://firzhanblogger.blogspot.com/ > <http://firzhanblogger.blogspot.com/> <http://suhothayan.blogspot.com/>* > *twitter: https://twitter.com/firzhan007 > <https://twitter.com/firzhan007> | linked-in: > **https://www.linkedin.com/in/firzhan > <https://www.linkedin.com/in/firzhan>* > > On Mon, Oct 12, 2015 at 10:51 AM, Chathurika De Silva < > [email protected]> wrote: > >> Hi All >> >> We have configured a SchedulerTask for the BPS cluster (2 workers and >> manager fronted by nginx). The task runs every day 5.10 pm and deletes >> FAILED instances. The cluster was up and running in the weekend but was >> idle. Following error was observed. >> >> >> TID: [-1] [] [2015-10-09 18:49:23,157] ERROR >> {org.apache.ode.scheduler.simple.SimpleScheduler} - Error during >> SchedulerTask execution {org.apache.ode.scheduler.simple.SimpleScheduler} >> com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast >> instance is not active! >> at >> com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getService(ProxyRegistry.java:65) >> at >> com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.<init>(ProxyRegistry.java:53) >> at >> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:74) >> at >> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:72) >> at >> com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:51) >> at >> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getOrCreateRegistry(ProxyServiceImpl.java:110) >> at >> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:118) >> at >> com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:390) >> at >> com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:192) >> at >> com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:75) >> at >> org.wso2.carbon.bpel.core.ode.integration.BPELServerImpl$ODEClusterImpl.getKnownNodes(BPELServerImpl.java:1009) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler.updateHeartBeatAndGetStaleNodes(SimpleScheduler.java:848) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler.access$800(SimpleScheduler.java:53) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler$CheckStaleNodes.run(SimpleScheduler.java:1226) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:810) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:807) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> TID: [-1] [] [2015-10-09 18:49:36,827] ERROR >> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >> {org.apache.tomcat.util.net.NioEndpoint$Poller} >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> TID: [-1] [] [2015-10-09 18:49:36,827] ERROR >> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >> {org.apache.tomcat.util.net.NioEndpoint$Poller} >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> TID: [-1] [] [2015-10-09 18:49:41,310] ERROR >> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >> {org.apache.tomcat.util.net.NioEndpoint$Poller} >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> TID: [-1] [] [2015-10-09 18:49:53,704] ERROR >> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >> execution of execution callback: >> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@6adde0a6for >> call Invocation{ serviceName='hz:impl:mapService', >> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >> partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1, >> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >> tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null, >> backupsExpected=0, backupsCompleted=0} >> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> TID: [-1] [] [2015-10-09 18:50:06,800] ERROR >> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >> execution of execution callback: >> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@295de461for >> call Invocation{ serviceName='hz:impl:mapService', >> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >> partitionId=-1, callId=0, invocationTime=1444415365856, waitTimeout=-1, >> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >> tryPauseMillis=0, invokeCount=1, callTimeout=5000, >> target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0} >> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> TID: [-1] [] [2015-10-09 18:50:30,826] ERROR >> {org.wso2.carbon.humantask.core.scheduler.SimpleScheduler} - Error loading >> immediate jobs from database. >> {org.wso2.carbon.humantask.core.scheduler.SimpleScheduler} >> <openjpa-2.2.0-wso2v1-r128166:129108 fatal general error> >> org.apache.openjpa.persistence.PersistenceException: No more data to read >> from socket >> FailedObject: SELECT DISTINCT t FROM >> org.wso2.carbon.humantask.core.dao.jpa.openjpa.model.HumanTaskJob t WHERE >> t.nodeId = ?1 AND t.time < ?2 order by t.time [java.lang.String] >> at >> org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4918) >> at >> org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:4878) >> at >> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:136) >> at >> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:110) >> at >> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:62) >> at >> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:949) >> at >> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:240) >> at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:413) >> at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:382) >> at >> org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94) >> at >> org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:34) >> at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1251) >> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1007) >> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863) >> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794) >> at >> org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542) >> at >> org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:286) >> at >> org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302) >> at >> org.wso2.carbon.humantask.core.dao.jpa.openjpa.HumanTaskDAOConnectionImpl.dequeueImmediate(HumanTaskDAOConnectionImpl.java:248) >> at >> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:344) >> at >> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:342) >> at >> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:879) >> at >> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:828) >> at >> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.doLoadImmediate(SimpleScheduler.java:342) >> at >> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$LoadImmediateTask.run(SimpleScheduler.java:297) >> at >> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:149) >> at >> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:146) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.sql.SQLRecoverableException: No more data to read from >> socket >> at >> oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:456) >> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397) >> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257) >> at >> oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:460) >> at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:594) >> at >> oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666) >> at >> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32) >> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566) >> at >> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278) >> at >> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182) >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701) >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635) >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >> at >> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >> at >> org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110) >> at >> org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87) >> at >> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:960) >> at >> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:945) >> ... 25 more >> TID: [-1] [] [2015-10-09 18:51:57,344] ERROR >> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >> execution of execution callback: >> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@1c5fe7d7for >> call Invocation{ serviceName='hz:impl:mapService', >> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >> partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1, >> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >> tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null, >> backupsExpected=0, backupsCompleted=0} >> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >> TID: [-1] [] [2015-10-09 18:54:53,738] ERROR >> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >> execution of execution callback: >> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@3c90fdbffor >> call Invocation{ serviceName='hz:impl:mapService', >> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >> partitionId=-1, callId=0, invocationTime=1444415373639, waitTimeout=-1, >> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >> tryPauseMillis=0, invokeCount=1, callTimeout=5000, >> target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0} >> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> >> What is the reason for such an error? Please note the error is >> continuously coming in the cluster. Any help will be highly appreciated. >> -- >> *Chathurika Erandi* >> Software Engineer, >> WSO2 Inc. >> >> *E-mail:* [email protected] >> *Mobile: *+94714 328 612 >> *Blog*: chathurikaerandi.blogspot.com >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > -- *Chathurika Erandi* Software Engineer, WSO2 Inc. *E-mail:* [email protected] *Mobile: *+94714 328 612 *Blog*: chathurikaerandi.blogspot.com
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
