Hi All
We have configured a SchedulerTask for the BPS cluster (2 workers and
manager fronted by nginx). The task runs every day 5.10 pm and deletes
FAILED instances. The cluster was up and running in the weekend but was
idle. Following error was observed.
TID: [-1] [] [2015-10-09 18:49:23,157] ERROR
{org.apache.ode.scheduler.simple.SimpleScheduler} - Error during
SchedulerTask execution {org.apache.ode.scheduler.simple.SimpleScheduler}
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance
is not active!
at
com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getService(ProxyRegistry.java:65)
at
com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.<init>(ProxyRegistry.java:53)
at
com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:74)
at
com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:72)
at
com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:51)
at
com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getOrCreateRegistry(ProxyServiceImpl.java:110)
at
com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:118)
at
com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:390)
at
com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:192)
at
com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:75)
at
org.wso2.carbon.bpel.core.ode.integration.BPELServerImpl$ODEClusterImpl.getKnownNodes(BPELServerImpl.java:1009)
at
org.apache.ode.scheduler.simple.SimpleScheduler.updateHeartBeatAndGetStaleNodes(SimpleScheduler.java:848)
at
org.apache.ode.scheduler.simple.SimpleScheduler.access$800(SimpleScheduler.java:53)
at
org.apache.ode.scheduler.simple.SimpleScheduler$CheckStaleNodes.run(SimpleScheduler.java:1226)
at
org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:810)
at
org.apache.ode.scheduler.simple.SimpleScheduler$8.call(SimpleScheduler.java:807)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
TID: [-1] [] [2015-10-09 18:49:36,827] ERROR
{org.apache.tomcat.util.net.NioEndpoint$Poller} -
{org.apache.tomcat.util.net.NioEndpoint$Poller}
java.lang.OutOfMemoryError: GC overhead limit exceeded
TID: [-1] [] [2015-10-09 18:49:36,827] ERROR
{org.apache.tomcat.util.net.NioEndpoint$Poller} -
{org.apache.tomcat.util.net.NioEndpoint$Poller}
java.lang.OutOfMemoryError: GC overhead limit exceeded
TID: [-1] [] [2015-10-09 18:49:41,310] ERROR
{org.apache.tomcat.util.net.NioEndpoint$Poller} -
{org.apache.tomcat.util.net.NioEndpoint$Poller}
java.lang.OutOfMemoryError: GC overhead limit exceeded
TID: [-1] [] [2015-10-09 18:49:53,704] ERROR
{com.hazelcast.spi.impl.operationservice.impl.Invocation} -
[192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous
execution of execution callback:
com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@6adde0a6for
call Invocation{ serviceName='hz:impl:mapService',
op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService',
partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1,
callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0,
tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null,
backupsExpected=0, backupsCompleted=0}
{com.hazelcast.spi.impl.operationservice.impl.Invocation}
java.lang.OutOfMemoryError: GC overhead limit exceeded
TID: [-1] [] [2015-10-09 18:50:06,800] ERROR
{com.hazelcast.spi.impl.operationservice.impl.Invocation} -
[192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous
execution of execution callback:
com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@295de461for
call Invocation{ serviceName='hz:impl:mapService',
op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService',
partitionId=-1, callId=0, invocationTime=1444415365856, waitTimeout=-1,
callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0,
tryPauseMillis=0, invokeCount=1, callTimeout=5000,
target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0}
{com.hazelcast.spi.impl.operationservice.impl.Invocation}
java.lang.OutOfMemoryError: GC overhead limit exceeded
TID: [-1] [] [2015-10-09 18:50:30,826] ERROR
{org.wso2.carbon.humantask.core.scheduler.SimpleScheduler} - Error loading
immediate jobs from database.
{org.wso2.carbon.humantask.core.scheduler.SimpleScheduler}
<openjpa-2.2.0-wso2v1-r128166:129108 fatal general error>
org.apache.openjpa.persistence.PersistenceException: No more data to read
from socket
FailedObject: SELECT DISTINCT t FROM
org.wso2.carbon.humantask.core.dao.jpa.openjpa.model.HumanTaskJob t WHERE
t.nodeId = ?1 AND t.time < ?2 order by t.time [java.lang.String]
at
org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4918)
at
org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:4878)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:136)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:110)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:62)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:949)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:240)
at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:413)
at org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:382)
at
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:94)
at
org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:34)
at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1251)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1007)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
at
org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:286)
at
org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
at
org.wso2.carbon.humantask.core.dao.jpa.openjpa.HumanTaskDAOConnectionImpl.dequeueImmediate(HumanTaskDAOConnectionImpl.java:248)
at
org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:344)
at
org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$2.call(SimpleScheduler.java:342)
at
org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:879)
at
org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.execTransaction(SimpleScheduler.java:828)
at
org.wso2.carbon.humantask.core.scheduler.SimpleScheduler.doLoadImmediate(SimpleScheduler.java:342)
at
org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$LoadImmediateTask.run(SimpleScheduler.java:297)
at
org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:149)
at
org.wso2.carbon.humantask.core.scheduler.SimpleScheduler$1.call(SimpleScheduler.java:146)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLRecoverableException: No more data to read from
socket
at
oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:456)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
at
oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:460)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:594)
at
oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
at
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
at
org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
at
org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
at
org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
at
org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
at
org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
at
org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
at
org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
at
org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:960)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:945)
... 25 more
TID: [-1] [] [2015-10-09 18:51:57,344] ERROR
{com.hazelcast.spi.impl.operationservice.impl.Invocation} -
[192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous
execution of execution callback:
com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@1c5fe7d7for
call Invocation{ serviceName='hz:impl:mapService',
op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService',
partitionId=-1, callId=0, invocationTime=-1, waitTimeout=-1,
callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0,
tryPauseMillis=0, invokeCount=0, callTimeout=5000, target=null,
backupsExpected=0, backupsCompleted=0}
{com.hazelcast.spi.impl.operationservice.impl.Invocation}
TID: [-1] [] [2015-10-09 18:54:53,738] ERROR
{com.hazelcast.spi.impl.operationservice.impl.Invocation} -
[192.168.48.199]:4000 [wso2.bps.qa.domain] [3.5] Failed asynchronous
execution of execution callback:
com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback@3c90fdbffor
call Invocation{ serviceName='hz:impl:mapService',
op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:impl:mapService',
partitionId=-1, callId=0, invocationTime=1444415373639, waitTimeout=-1,
callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0,
tryPauseMillis=0, invokeCount=1, callTimeout=5000,
target=Address[192.168.48.2]:4000, backupsExpected=0, backupsCompleted=0}
{com.hazelcast.spi.impl.operationservice.impl.Invocation}
java.lang.OutOfMemoryError: GC overhead limit exceeded
What is the reason for such an error? Please note the error is continuously
coming in the cluster. Any help will be highly appreciated.
--
*Chathurika Erandi*
Software Engineer,
WSO2 Inc.
*E-mail:* [email protected]
*Mobile: *+94714 328 612
*Blog*: chathurikaerandi.blogspot.com
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
