Hi Chathiruka,
This seems to be the hazlecast OOM error observed with kernel 4.4.1. Can
you increase heap size and check this scenario again.
*TID: [-1] [] [2015-10-09 18:49:36,827] ERROR {org.apache.tomcat.util.net
<http://org.apache.tomcat.util.net>.NioEndpoint$Poller} -
{org.apache.tomcat.util.net
<http://org.apache.tomcat.util.net>.NioEndpoint$Poller}java.lang.OutOfMemoryError:
GC overhead limit exceeded*
Regards,
Firzhan
--
*Firzhan Naqash*
Senior Software Engineer - Integration Platform Team
WSO2 Inc. http://wso2.com
email: [email protected] <[email protected]>
mobile: (+94) 77 9785674 <%28%2B94%29%2071%205247551>*|
blog: http://firzhanblogger.blogspot.com/
<http://firzhanblogger.blogspot.com/> <http://suhothayan.blogspot.com/>*
*twitter: https://twitter.com/firzhan007 <https://twitter.com/firzhan007> |
linked-in: **https://www.linkedin.com/in/firzhan
<https://www.linkedin.com/in/firzhan>*
On Mon, Oct 12, 2015 at 10:51 AM, Chathurika De Silva <[email protected]>
wrote:
> Hi All
>
> We have configured a SchedulerTask for the BPS cluster (2 workers and
> manager fronted by nginx). The task runs every day 5.10 pm and deletes
> FAILED instances. The cluster was up and running in the weekend but was
> idle. Following error was observed.
>
>
> TID: [-1] [] [2015-10-09 18:49:23,157] ERROR
> {org.apache.ode.scheduler.simple.SimpleScheduler} - Error during
> SchedulerTask execution {org.apache.ode.scheduler.simple.SimpleScheduler}
> com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance
> is not active!
> at
> com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getService(ProxyRegistry.java:65)
> at
> com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.<init>(ProxyRegistry.java:53)
> at
> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:74)
> at
> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:72)
> at
> com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:51)
> at
> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getOrCreateRegistry(ProxyServiceImpl.java:110)
> at
> com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:118)
> at
> com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:390)
> at
> com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:192)
> at
> com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:75)
> at
> org.wso2.carbon.bpel.core.ode.integration.BPELServerImpl$ODEClusterImpl.getKnownNodes(BPELServerImpl.java:1009)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler.updateHeartBeatAndGetStaleNodes(SimpleScheduler.java:848)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler.access$800(SimpleScheduler.java:53)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$CheckStaleNodes.run(SimpleScheduler.java:1226)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:810)
> at
> org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:807)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> TID: [-1] [] [2015-10-09 18:49:36,827] ERROR
> {org.apache.tomcat.util.net.NioEndpoint$Poller} -
> {org.apache.tomcat.util.net.NioEndpoint$Poller}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> TID: [-1] [] [2015-10-09 18:49:36,827] ERROR
> {org.apache.tomcat.util.net.NioEndpoint$Poller} -
> {org.apache.tomcat.util.net.NioEndpoint$Poller}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> TID: [-1] [] [2015-10-09 18:49:41,310] ERROR
> {org.apache.tomcat.util.net.NioEndpoint$Poller} -
> {org.apache.tomcat.util.net.NioEndpoint$Poller}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> TID: [-1] [] [2015-10-09 18:49:53,704] ERROR
> {com.hazelcast.spi.impl.operationservice.impl.Invocation} -
> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous
> execution of execution callback:
> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@6adde0a6for
> call Invocation{ serviceName='hz:impl:mapService',
> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService',
> partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1,
> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0,
> tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null,
> backupsExpected=0, backupsCompleted=0}
> {com.hazelcast.spi.impl.operationservice.impl.Invocation}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> TID: [-1] [] [2015-10-09 18:50:06,800] ERROR
> {com.hazelcast.spi.impl.operationservice.impl.Invocation} -
> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous
> execution of execution callback:
> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@295de461for
> call Invocation{ serviceName='hz:impl:mapService',
> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService',
> partitionId=-1, callId=0, invocationTime=1444415365856, waitTimeout=-1,
> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0,
> tryPauseMillis=0, invokeCount=1, callTimeout=5000,
> target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0}
> {com.hazelcast.spi.impl.operationservice.impl.Invocation}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> TID: [-1] [] [2015-10-09 18:50:30,826] ERROR
> {org.wso2.carbon.humantask.core.scheduler.SimpleScheduler} - Error loading
> immediate jobs from database.
> {org.wso2.carbon.humantask.core.scheduler.SimpleScheduler}
> <openjpa-2.2.0-wso2v1-r128166:129108 fatal general error>
> org.apache.openjpa.persistence.PersistenceException: No more data to read
> from socket
> FailedObject: SELECT DISTINCT t FROM
> org.wso2.carbon.humantask.core.dao.jpa.openjpa.model.HumanTaskJob t WHERE
> t.nodeId = ?1 AND t.time < ?2 order by t.time [java.lang.String]
> at
> org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4918)
> at
> org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:4878)
> at
> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:136)
> at
> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:110)
> at
> org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:62)
> at
> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:949)
> at
> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:240)
> at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:413)
> at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:382)
> at
> org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94)
> at
> org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:34)
> at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1251)
> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1007)
> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
> at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
> at
> org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
> at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:286)
> at
> org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
> at
> org.wso2.carbon.humantask.core.dao.jpa.openjpa.HumanTaskDAOConnectionImpl.dequeueImmediate(HumanTaskDAOConnectionImpl.java:248)
> at
> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:344)
> at
> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:342)
> at
> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:879)
> at
> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:828)
> at
> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.doLoadImmediate(SimpleScheduler.java:342)
> at
> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$LoadImmediateTask.run(SimpleScheduler.java:297)
> at
> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:149)
> at
> org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:146)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.sql.SQLRecoverableException: No more data to read from
> socket
> at
> oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:456)
> at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
> at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
> at
> oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:460)
> at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:594)
> at
> oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
> at
> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
> at
> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
> at
> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
> at
> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
> at
> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
> at
> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
> at
> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
> at
> org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
> at
> org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)
> at
> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:960)
> at
> org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:945)
> ... 25 more
> TID: [-1] [] [2015-10-09 18:51:57,344] ERROR
> {com.hazelcast.spi.impl.operationservice.impl.Invocation} -
> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous
> execution of execution callback:
> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@1c5fe7d7for
> call Invocation{ serviceName='hz:impl:mapService',
> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService',
> partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1,
> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0,
> tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null,
> backupsExpected=0, backupsCompleted=0}
> {com.hazelcast.spi.impl.operationservice.impl.Invocation}
> TID: [-1] [] [2015-10-09 18:54:53,738] ERROR
> {com.hazelcast.spi.impl.operationservice.impl.Invocation} -
> [192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous
> execution of execution callback:
> com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@3c90fdbffor
> call Invocation{ serviceName='hz:impl:mapService',
> op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService',
> partitionId=-1, callId=0, invocationTime=1444415373639, waitTimeout=-1,
> callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0,
> tryPauseMillis=0, invokeCount=1, callTimeout=5000,
> target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0}
> {com.hazelcast.spi.impl.operationservice.impl.Invocation}
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
>
> What is the reason for such an error? Please note the error is
> continuously coming in the cluster. Any help will be highly appreciated.
> --
> *Chathurika Erandi*
> Software Engineer,
> WSO2 Inc.
>
> *E-mail:* [email protected]
> *Mobile: *+94714 328 612
> *Blog*: chathurikaerandi.blogspot.com
>
> _______________________________________________
> Dev mailing list
> [email protected]
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
