It looks like a database table lock is not being released until it reaches
the timeout:
Timeout trying to lock table "REG_RESOURCE"; SQL statement:
SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=? AND
REG_TENANT_ID=? [50200-140]
The concern is we do not handle any database locks in Stratos code, they
are handled by the Carbon Registry. Shall we try to narrow down the problem
by running it again without using the Metadata API?
Thanks
On Thu, May 7, 2015 at 12:37 PM, Udara Liyanage <ud...@wso2.com> wrote:
> Hi,
>
> I frequently get this exception when running Stratos for longtime (10+
> minutes). Please note that most failures occurs when metadata service also
> save properties in the registry (not all the time, this occurs when
> metadata is not persisting too).
>
>
> 2015-05-07 07:03:11,725] ERROR
> {org.apache.stratos.cloud.controller.registry.RegistryManager} - * Failed
> to persist resource in registry: /cloud.controller/topology*
> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
> check the existence of the resource
> /_system/governance/cloud.controller/topology. Timeout trying to lock table
> "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
> at
> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
> at
> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
> at
> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
> at
> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
> at
> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
> at
> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
> at
> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
> at
> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
> at java.util.Observable.notifyObservers(Observable.java:159)
> at
> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
> at
> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
> at
> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
> at
> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
> at
> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
> "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
> at org.h2.message.DbException.get(DbException.java:167)
> at org.h2.message.DbException.get(DbException.java:144)
> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
> at org.h2.table.RegularTable.lock(RegularTable.java:404)
> at org.h2.table.TableFilter.lock(TableFilter.java:139)
> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
> at org.h2.command.dml.Query.query(Query.java:241)
> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
> at org.h2.command.Command.executeQuery(Command.java:132)
> at
> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
> ... 23 more
> [2015-05-07 07:03:11,726] FATAL
> {org.apache.stratos.cloud.controller.util.CloudControllerUtil} - Failed to
> persist the Topology in registry.
> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to
> persist resource in registry: /cloud.controller/topology
> at
> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:85)
> at
> org.apache.stratos.cloud.controller.util.CloudControllerUtil.persistTopology(CloudControllerUtil.java:276)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyManager.updateTopology(TopologyManager.java:103)
> at
> org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder.handleMemberStarted(TopologyBuilder.java:555)
> at
> org.apache.stratos.cloud.controller.messaging.receiver.instance.status.InstanceStatusTopicReceiver$2.onEvent(InstanceStatusTopicReceiver.java:75)
> at
> org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
> at java.util.Observable.notifyObservers(Observable.java:159)
> at
> org.apache.stratos.messaging.event.EventObservable.notifyEventListeners(EventObservable.java:51)
> at
> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberStartedMessageProcessor.process(InstanceStatusMemberStartedMessageProcessor.java:49)
> at
> org.apache.stratos.messaging.message.processor.instance.status.InstanceStatusMemberActivatedMessageProcessor.process(InstanceStatusMemberActivatedMessageProcessor.java:53)
> at
> org.apache.stratos.messaging.message.processor.MessageProcessorChain.process(MessageProcessorChain.java:61)
> at
> org.apache.stratos.messaging.message.receiver.instance.status.InstanceStatusEventMessageDelegator.run(InstanceStatusEventMessageDelegator.java:71)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.wso2.carbon.registry.core.exceptions.RegistryException:
> Failed to check the existence of the resource
> /_system/governance/cloud.controller/topology. Timeout trying to lock table
> "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:206)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:147)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:156)
> at
> org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
> at
> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:688)
> at
> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.put(CacheBackedRegistry.java:465)
> at
> org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:657)
> at
> org.apache.stratos.cloud.controller.registry.RegistryManager.persist(RegistryManager.java:73)
> ... 16 more
> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table
> "REG_RESOURCE"; SQL statement:
> SELECT REG_VERSION FROM REG_RESOURCE WHERE REG_PATH_ID=? AND REG_NAME=?
> AND REG_TENANT_ID=? [50200-140]
> at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
> at org.h2.message.DbException.get(DbException.java:167)
> at org.h2.message.DbException.get(DbException.java:144)
> at org.h2.table.RegularTable.doLock(RegularTable.java:466)
> at org.h2.table.RegularTable.lock(RegularTable.java:404)
> at org.h2.table.TableFilter.lock(TableFilter.java:139)
> at org.h2.command.dml.Select.queryWithoutCache(Select.java:554)
> at org.h2.command.dml.Query.query(Query.java:241)
> at org.h2.command.CommandContainer.query(CommandContainer.java:80)
> at org.h2.command.Command.executeQuery(Command.java:132)
> at
> org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
> at
> org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getVersion(JDBCResourceDAO.java:197)
> ... 23 more
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>
--
Imesh Gunaratne
Senior Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
