Hello,
Just tried cloudstack 4.11.2 and got the error due to create network. In
cloudstack logs we got the message: "Timed out on acquiring lock account2
. Waited for 300seconds" (please read the attachment for more details).
Which parameter defines this 300 seconds, where we can increase it? Any
ideas?
Thanks in advance for the help,
Dmitry.
2019-06-10 14:24:26,432 DEBUG [c.c.a.ApiServlet] (qtp895947612-1041:ctx-5f19e1ca) (logid:d1cee349) ===START=== 10.10.12.10 -- GET apikey=xRZ8vfHekftVn43EKETrcVlgJu-Vcd7c47-GjsiEIlGhF-D428h5HruTGaN21uDuNsYTsB-NTW8QCTJM8sz2yQ&name=
redistr-200-49308-mac-change-0-0&networkofferingid=3b96bbc8-1102-48de-867b-a4162988db84&gateway=10.200.1.1&netmask=255.255.255.0&command=createNetwork&displaytext=redistr-200-49308-mac-change-0-0&zoneid=d103e528-c0b0-471e-ad55-2b99
c47c4dee&response=json&signature=QTewi3D%2Bo%2Ft4jfoh9UOTLcLBPhk%3D
2019-06-10 14:24:26,439 DEBUG [c.c.a.ApiServer] (qtp895947612-1041:ctx-5f19e1ca ctx-2416e4bc) (logid:d1cee349) CIDRs from which account 'Acct[d9d375fe-87a1-11e9-a390-0cc47aaa7eb0-admin]' is allowed to perform API calls: 0.0.0.0/0,:
:/0
2019-06-10 14:29:28,229 WARN [c.c.u.d.Merovingian2] (qtp895947612-1041:ctx-5f19e1ca ctx-2416e4bc ctx-da492f0d) (logid:d1cee349) Timed out on acquiring lock account2 . Waited for 300seconds
com.cloud.utils.exception.CloudRuntimeException: Timed out on acquiring lock account2 . Waited for 300seconds
at com.cloud.utils.db.Merovingian2.acquire(Merovingian2.java:151)
at com.cloud.utils.db.TransactionLegacy.lock(TransactionLegacy.java:386)
at com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1048)
at com.cloud.utils.db.GenericDaoBase.acquireInLockTable(GenericDaoBase.java:1039)
at sun.reflect.GeneratedMethodAccessor152.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy35.acquireInLockTable(Unknown Source)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:636)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$10.doInTransaction(NetworkOrchestrator.java:2357)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$10.doInTransaction(NetworkOrchestrator.java:2308)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createGuestNetwork(NetworkOrchestrator.java:2308)
at com.cloud.network.NetworkServiceImpl$4.doInTransaction(NetworkServiceImpl.java:1383)
at com.cloud.network.NetworkServiceImpl$4.doInTransaction(NetworkServiceImpl.java:1331)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.network.NetworkServiceImpl.commitNetwork(NetworkServiceImpl.java:1331)
at com.cloud.network.NetworkServiceImpl.createGuestNetwork(NetworkServiceImpl.java:1294)
at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy229.createGuestNetwork(Unknown Source)
at org.apache.cloudstack.api.command.admin.network.CreateNetworkCmdByAdmin.execute(CreateNetworkCmdByAdmin.java:68)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:732)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:556)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:310)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:130)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:127)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:686)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:530)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
2019-06-10 14:29:28,233 DEBUG [c.c.u.d.T.Transaction] (qtp895947612-1041:ctx-5f19e1ca ctx-2416e4bc ctx-da492f0d) (logid:d1cee349) Rolling back the transaction: Time = 301784 Name = qtp895947612-1041; called by -TransactionLegacy.rollback:890-TransactionLegacy.removeUpTo:833-TransactionLegacy.close:657-Transaction.execute:43-Transaction.execute:47-NetworkOrchestrator.createGuestNetwork:2308-NetworkServiceImpl$4.doInTransaction:1383-NetworkServiceImpl$4.doInTransaction:1331-Transaction.execute:40-NetworkServiceImpl.commitNetwork:1331-NetworkServiceImpl.createGuestNetwork:1294-GeneratedMethodAccessor143.invoke:-1
2019-06-10 14:29:28,661 ERROR [c.c.a.ApiServer] (qtp895947612-1041:ctx-5f19e1ca ctx-2416e4bc ctx-da492f0d) (logid:d1cee349) unhandled exception executing api command: [Ljava.lang.String;@7186287b
com.cloud.exception.ConcurrentOperationException: Unable to acquire lock on Acct[a1285a05-f6f8-4065-abc0-58d324afb3c9-admin]
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:638)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$10.doInTransaction(NetworkOrchestrator.java:2357)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$10.doInTransaction(NetworkOrchestrator.java:2308)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createGuestNetwork(NetworkOrchestrator.java:2308)
at com.cloud.network.NetworkServiceImpl$4.doInTransaction(NetworkServiceImpl.java:1383)
at com.cloud.network.NetworkServiceImpl$4.doInTransaction(NetworkServiceImpl.java:1331)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.network.NetworkServiceImpl.commitNetwork(NetworkServiceImpl.java:1331)
at com.cloud.network.NetworkServiceImpl.createGuestNetwork(NetworkServiceImpl.java:1294)
at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy229.createGuestNetwork(Unknown Source)
at org.apache.cloudstack.api.command.admin.network.CreateNetworkCmdByAdmin.execute(CreateNetworkCmdByAdmin.java:68)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:732)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:556)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:310)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:130)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:127)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:686)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:530)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
