Hi Done [1]
[1] https://wso2.org/jira/browse/BPS-907 On Mon, Oct 12, 2015 at 11:35 AM, Firzhan Naqash <firz...@wso2.com> wrote: > Hi, > > Then this seems to be the Hazlecast OOM issue. Please report a jira for > this. > > > Regards, > Firzhan > > > -- > *Firzhan Naqash* > Senior Software Engineer - Integration Platform Team > WSO2 Inc. http://wso2.com > > email: firz...@wso2.com <lasan...@wso2.com> > mobile: (+94) 77 9785674 <%28%2B94%29%2071%205247551>*| > blog: http://firzhanblogger.blogspot.com/ > <http://firzhanblogger.blogspot.com/> <http://suhothayan.blogspot.com/>* > *twitter: https://twitter.com/firzhan007 > <https://twitter.com/firzhan007> | linked-in: > **https://www.linkedin.com/in/firzhan > <https://www.linkedin.com/in/firzhan>* > > On Mon, Oct 12, 2015 at 11:11 AM, Chathurika De Silva < > chathuri...@wso2.com> wrote: > >> Hi Firzhan >> >> The cluster have the following configuration currently. The default value >> is 256m >> >> Xms512m -Xmx2048m -XX:MaxPermSize=512m >> >> Please advice with the recommended value and please note that this >> configuration is not given in the documentation >> >> On Mon, Oct 12, 2015 at 10:56 AM, Firzhan Naqash <firz...@wso2.com> >> wrote: >> >>> Hi Chathiruka, >>> >>> This seems to be the hazlecast OOM error observed with kernel 4.4.1. Can >>> you increase heap size and check this scenario again. >>> >>> >>> *TID: [-1] [] [2015-10-09 18:49:36,827] ERROR >>> {org.apache.tomcat.util.net >>> <http://org.apache.tomcat.util.net>.NioEndpoint$Poller} - >>> {org.apache.tomcat.util.net >>> <http://org.apache.tomcat.util.net>.NioEndpoint$Poller}java.lang.OutOfMemoryError: >>> GC overhead limit exceeded* >>> >>> Regards, >>> Firzhan >>> >>> >>> -- >>> *Firzhan Naqash* >>> Senior Software Engineer - Integration Platform Team >>> WSO2 Inc. http://wso2.com >>> >>> email: firz...@wso2.com <lasan...@wso2.com> >>> mobile: (+94) 77 9785674 <%28%2B94%29%2071%205247551>*| >>> blog: http://firzhanblogger.blogspot.com/ >>> <http://firzhanblogger.blogspot.com/> <http://suhothayan.blogspot.com/>* >>> *twitter: https://twitter.com/firzhan007 >>> <https://twitter.com/firzhan007> | linked-in: >>> **https://www.linkedin.com/in/firzhan >>> <https://www.linkedin.com/in/firzhan>* >>> >>> On Mon, Oct 12, 2015 at 10:51 AM, Chathurika De Silva < >>> chathuri...@wso2.com> wrote: >>> >>>> Hi All >>>> >>>> We have configured a SchedulerTask for the BPS cluster (2 workers and >>>> manager fronted by nginx). The task runs every day 5.10 pm and deletes >>>> FAILED instances. The cluster was up and running in the weekend but was >>>> idle. Following error was observed. >>>> >>>> >>>> TID: [-1] [] [2015-10-09 18:49:23,157] ERROR >>>> {org.apache.ode.scheduler.simple.SimpleScheduler} - Error during >>>> SchedulerTask execution {org.apache.ode.scheduler.simple.SimpleScheduler} >>>> com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast >>>> instance is not active! >>>> at >>>> com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getService(ProxyRegistry.java:65) >>>> at >>>> com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.<init>(ProxyRegistry.java:53) >>>> at >>>> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:74) >>>> at >>>> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:72) >>>> at >>>> com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:51) >>>> at >>>> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getOrCreateRegistry(ProxyServiceImpl.java:110) >>>> at >>>> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:118) >>>> at >>>> com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:390) >>>> at >>>> com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:192) >>>> at >>>> com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:75) >>>> at >>>> org.wso2.carbon.bpel.core.ode.integration.BPELServerImpl$ODEClusterImpl.getKnownNodes(BPELServerImpl.java:1009) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler.updateHeartBeatAndGetStaleNodes(SimpleScheduler.java:848) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler.access$800(SimpleScheduler.java:53) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler$CheckStaleNodes.run(SimpleScheduler.java:1226) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:810) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:807) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:745) >>>> TID: [-1] [] [2015-10-09 18:49:36,827] ERROR >>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> TID: [-1] [] [2015-10-09 18:49:36,827] ERROR >>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> TID: [-1] [] [2015-10-09 18:49:41,310] ERROR >>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} - >>>> {org.apache.tomcat.util.net.NioEndpoint$Poller} >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> TID: [-1] [] [2015-10-09 18:49:53,704] ERROR >>>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >>>> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >>>> execution of execution callback: >>>> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@6adde0a6for >>>> call Invocation{ serviceName='hz:impl:mapService', >>>> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >>>> partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1, >>>> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >>>> tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null, >>>> backupsExpected=0, backupsCompleted=0} >>>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> TID: [-1] [] [2015-10-09 18:50:06,800] ERROR >>>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >>>> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >>>> execution of execution callback: >>>> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@295de461for >>>> call Invocation{ serviceName='hz:impl:mapService', >>>> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >>>> partitionId=-1, callId=0, invocationTime=1444415365856, waitTimeout=-1, >>>> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >>>> tryPauseMillis=0, invokeCount=1, callTimeout=5000, >>>> target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0} >>>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> TID: [-1] [] [2015-10-09 18:50:30,826] ERROR >>>> {org.wso2.carbon.humantask.core.scheduler.SimpleScheduler} - Error loading >>>> immediate jobs from database. >>>> {org.wso2.carbon.humantask.core.scheduler.SimpleScheduler} >>>> <openjpa-2.2.0-wso2v1-r128166:129108 fatal general error> >>>> org.apache.openjpa.persistence.PersistenceException: No more data to read >>>> from socket >>>> FailedObject: SELECT DISTINCT t FROM >>>> org.wso2.carbon.humantask.core.dao.jpa.openjpa.model.HumanTaskJob t WHERE >>>> t.nodeId = ?1 AND t.time < ?2 order by t.time [java.lang.String] >>>> at >>>> org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4918) >>>> at >>>> org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:4878) >>>> at >>>> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:136) >>>> at >>>> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:110) >>>> at >>>> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:62) >>>> at >>>> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:949) >>>> at >>>> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:240) >>>> at >>>> org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:413) >>>> at >>>> org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:382) >>>> at >>>> org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94) >>>> at >>>> org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:34) >>>> at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1251) >>>> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1007) >>>> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863) >>>> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794) >>>> at >>>> org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542) >>>> at >>>> org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:286) >>>> at >>>> org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302) >>>> at >>>> org.wso2.carbon.humantask.core.dao.jpa.openjpa.HumanTaskDAOConnectionImpl.dequeueImmediate(HumanTaskDAOConnectionImpl.java:248) >>>> at >>>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:344) >>>> at >>>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:342) >>>> at >>>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:879) >>>> at >>>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:828) >>>> at >>>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.doLoadImmediate(SimpleScheduler.java:342) >>>> at >>>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$LoadImmediateTask.run(SimpleScheduler.java:297) >>>> at >>>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:149) >>>> at >>>> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:146) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.sql.SQLRecoverableException: No more data to read from >>>> socket >>>> at >>>> oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:456) >>>> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397) >>>> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257) >>>> at >>>> oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:460) >>>> at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:594) >>>> at >>>> oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666) >>>> at >>>> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32) >>>> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566) >>>> at >>>> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278) >>>> at >>>> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182) >>>> at >>>> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701) >>>> at >>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635) >>>> at >>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>> at >>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>> at >>>> org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110) >>>> at >>>> org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87) >>>> at >>>> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:960) >>>> at >>>> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:945) >>>> ... 25 more >>>> TID: [-1] [] [2015-10-09 18:51:57,344] ERROR >>>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >>>> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >>>> execution of execution callback: >>>> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@1c5fe7d7for >>>> call Invocation{ serviceName='hz:impl:mapService', >>>> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >>>> partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1, >>>> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >>>> tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null, >>>> backupsExpected=0, backupsCompleted=0} >>>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >>>> TID: [-1] [] [2015-10-09 18:54:53,738] ERROR >>>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} - >>>> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous >>>> execution of execution callback: >>>> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@3c90fdbffor >>>> call Invocation{ serviceName='hz:impl:mapService', >>>> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService', >>>> partitionId=-1, callId=0, invocationTime=1444415373639, waitTimeout=-1, >>>> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, >>>> tryPauseMillis=0, invokeCount=1, callTimeout=5000, >>>> target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0} >>>> {com.hazelcast.spi.impl.operationservice.impl.Invocation} >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> >>>> >>>> What is the reason for such an error? Please note the error is >>>> continuously coming in the cluster. Any help will be highly appreciated. >>>> -- >>>> *Chathurika Erandi* >>>> Software Engineer, >>>> WSO2 Inc. >>>> >>>> *E-mail:* chathuri...@wso2.com >>>> *Mobile: *+94714 328 612 >>>> *Blog*: chathurikaerandi.blogspot.com >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@wso2.org >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >> >> >> -- >> *Chathurika Erandi* >> Software Engineer, >> WSO2 Inc. >> >> *E-mail:* chathuri...@wso2.com >> *Mobile: *+94714 328 612 >> *Blog*: chathurikaerandi.blogspot.com >> > > -- *Chathurika Erandi* Software Engineer, WSO2 Inc. *E-mail:* chathuri...@wso2.com *Mobile: *+94714 328 612 *Blog*: chathurikaerandi.blogspot.com
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
