Hi All,
I was able to figure out the root cause for the issue. As Geeth and Rasika
pointed out this is happening due to concurrent insert and update query
executions. The root cause for this is calling pending operations endpoint
more than once from a single android device at a given time. Due to that
concurrent insert and update happen in the DB level. Ideally a device
should not call pending operations endpoint simultaneously. But as device
may receive multiple FCM notifications within fractions of seconds, it is
triggering pending operation calls per notification.
So I'm now implementing a state machine to avoid that raise condition.
Thanks & Regards,
/charithag
On Tue, Jan 23, 2018 at 2:14 PM, Rasika Perera <rasi...@wso2.com> wrote:
> Hi All,
>
> This error has raised since two threads trying to update the DB with
> following requirements;
>
> Thread #1
> waiting to lock PUBLIC.*DM_ENROLMENT_OP_MAPPING* while
> locking PUBLIC.*DM_OPERATION* (exclusive), PUBLIC.*DM_COMMAND_OPERATION*
> (exclusive).
>
> Thread #2
> waiting to lock PUBLIC.DM_OPERATION while
> locking PUBLIC.*DM_ENROLMENT_OP_MAPPING* (exclusive), PUBLIC.
> *DM_DEVICE_OPERATION_RESPONSE* (exclusive).";
>
> I noticed we are already using ";LOCK_TIMEOUT=60000" (60sec) H2 DB
> parameter which would allow sometime to continue the thread and do the DB
> update.
>
> Hence, this might happen when you are holding the debug pointer in
> updateOperation() method or in a state where device responses burst occurs.
> However the latter case is very rare on other DBs(other than embedded H2).
> We can even minimize the chance for a such issue implementing a LOCK for
> calling updateOperation method(might affect the performance). WDYT?
>
> Best Regards,
> ~Rasika
>
> On Tue, Jan 23, 2018 at 12:39 PM, Charitha Goonetilleke <
> charit...@wso2.com> wrote:
>
>> Hi All,
>>
>> I have encountered following exception intermittently on IoTS Update 11,
>> when testing with Android device.
>>
>> [2018-01-23 12:19:51,077] [IoT-Core] ERROR -
>> {org.wso2.carbon.mdm.services.android.services.impl.DeviceManagementServiceImpl}
>> Issue in retrieving operation management service instance
>> org.wso2.carbon.device.mgt.common.operation.mgt.OperationManagementException:
>> Error occurred while updating the operation: 90 status:COMPLETED
>> at org.wso2.carbon.device.mgt.core.operation.mgt.OperationManag
>> erImpl.updateOperation(OperationManagerImpl.java:551)
>> at org.wso2.carbon.device.mgt.core.service.DeviceManagementProv
>> iderServiceImpl.updateOperation(DeviceManagementProviderServ
>> iceImpl.java:1426)
>> at org.wso2.carbon.mdm.services.android.util.AndroidDeviceUtils
>> .updateOperation(AndroidDeviceUtils.java:211)
>> at org.wso2.carbon.mdm.services.android.services.impl.DeviceMan
>> agementServiceImpl.updateOperations(DeviceManagementServiceImpl.java:184)
>> at org.wso2.carbon.mdm.services.android.services.impl.DeviceMan
>> agementServiceImpl.getPendingOperations(DeviceManagementServ
>> iceImpl.java:139)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.cxf.service.invoker.AbstractInvoker.performInvoca
>> tion(AbstractInvoker.java:188)
>> at org.apache.cxf.service.invoker.AbstractInvoker.invoke(Abstra
>> ctInvoker.java:104)
>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:204)
>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:101)
>> at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(S
>> erviceInvokerInterceptor.java:58)
>> at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleM
>> essage(ServiceInvokerInterceptor.java:94)
>> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(Phase
>> InterceptorChain.java:272)
>> at org.apache.cxf.transport.ChainInitiationObserver.onMessage(C
>> hainInitiationObserver.java:121)
>> at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke
>> (AbstractHTTPDestination.java:249)
>> at org.apache.cxf.transport.servlet.ServletController.invokeDes
>> tination(ServletController.java:248)
>> at org.apache.cxf.transport.servlet.ServletController.invoke(Se
>> rvletController.java:222)
>> at org.apache.cxf.transport.servlet.ServletController.invoke(Se
>> rvletController.java:153)
>> at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(
>> CXFNonSpringServlet.java:171)
>> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleR
>> equest(AbstractHTTPServlet.java:289)
>> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPut(A
>> bstractHTTPServlet.java:226)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
>> at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service
>> (AbstractHTTPServlet.java:265)
>> at org.apache.catalina.core.ApplicationFilterChain.internalDoFi
>> lter(ApplicationFilterChain.java:303)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(App
>> licationFilterChain.java:208)
>> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
>> at org.apache.catalina.core.ApplicationFilterChain.internalDoFi
>> lter(ApplicationFilterChain.java:241)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(App
>> licationFilterChain.java:208)
>> at org.wso2.carbon.mdm.services.android.util.ApiOriginFilter.do
>> Filter(ApiOriginFilter.java:33)
>> at org.apache.catalina.core.ApplicationFilterChain.internalDoFi
>> lter(ApplicationFilterChain.java:241)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(App
>> licationFilterChain.java:208)
>> at org.apache.catalina.core.StandardWrapperValve.invoke(Standar
>> dWrapperValve.java:218)
>> at org.apache.catalina.core.StandardContextValve.invoke(Standar
>> dContextValve.java:110)
>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(A
>> uthenticatorBase.java:506)
>> at org.apache.catalina.core.StandardHostValve.invoke(StandardHo
>> stValve.java:169)
>> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorRepo
>> rtValve.java:103)
>> at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInv
>> ocation(CompositeValve.java:99)
>> at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke
>> (CarbonTomcatValve.java:47)
>> at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(Tena
>> ntLazyLoaderValve.java:57)
>> at org.wso2.carbon.webapp.authenticator.framework.WebappAuthent
>> icationValve.processRequest(WebappAuthenticationValve.java:151)
>> at org.wso2.carbon.webapp.authenticator.framework.WebappAuthent
>> icationValve.invoke(WebappAuthenticationValve.java:69)
>> at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invok
>> eValves(TomcatValveContainer.java:47)
>> at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(Comp
>> ositeValve.java:62)
>> at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetection
>> Valve.invoke(CarbonStuckThreadDetectionValve.java:159)
>> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogVa
>> lve.java:962)
>> at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.
>> invoke(CarbonContextCreatorValve.java:57)
>> at org.apache.catalina.core.StandardEngineValve.invoke(Standard
>> EngineValve.java:116)
>> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAd
>> apter.java:445)
>> at org.apache.coyote.http11.AbstractHttp11Processor.process(Abs
>> tractHttp11Processor.java:1115)
>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler
>> .process(AbstractProtocol.java:637)
>> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
>> (NioEndpoint.java:1770)
>> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(N
>> ioEndpoint.java:1729)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.r
>> un(TaskThread.java:61)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by:
>> org.wso2.carbon.device.mgt.core.operation.mgt.dao.OperationManagementDAOException:
>> Error occurred while inserting operation response
>> at org.wso2.carbon.device.mgt.core.operation.mgt.dao.impl.Gener
>> icOperationDAOImpl.addOperationResponse(GenericOperationDAOImpl.java:213)
>> at org.wso2.carbon.device.mgt.core.operation.mgt.OperationManag
>> erImpl.updateOperation(OperationManagerImpl.java:544)
>> ... 58 more
>> Caused by: org.h2.jdbc.JdbcSQLException: Deadlock detected. The current
>> transaction was rolled back. Details: "
>> Session #49 (user: WSO2CARBON) on thread pool-54-thread-1 is waiting to
>> lock PUBLIC.DM_ENROLMENT_OP_MAPPING while locking PUBLIC.DM_OPERATION
>> (exclusive), PUBLIC.DM_COMMAND_OPERATION (exclusive).
>> Session #59 (user: WSO2CARBON) on thread http-nio-9443-exec-27 is waiting
>> to lock PUBLIC.DM_OPERATION while locking PUBLIC.DM_ENROLMENT_OP_MAPPING
>> (exclusive), PUBLIC.DM_DEVICE_OPERATION_RESPONSE (exclusive)."; SQL
>> statement:
>> INSERT INTO DM_DEVICE_OPERATION_RESPONSE(OPERATION_ID, ENROLMENT_ID,
>> EN_OP_MAP_ID, OPERATION_RESPONSE, RECEIVED_TIMESTAMP) VALUES(?, ?, ?, ?, ?)
>> [40001-175]
>> at org.h2.message.DbException.getJdbcSQLException(DbException.java:332)
>> at org.h2.message.DbException.get(DbException.java:172)
>> at org.h2.message.DbException.get(DbException.java:149)
>> at org.h2.table.RegularTable.doLock(RegularTable.java:504)
>> at org.h2.table.RegularTable.lock(RegularTable.java:450)
>> at org.h2.constraint.ConstraintReferential.existsRow(Constraint
>> Referential.java:375)
>> at org.h2.constraint.ConstraintReferential.checkRowOwnTable(Con
>> straintReferential.java:367)
>> at org.h2.constraint.ConstraintReferential.checkRow(ConstraintR
>> eferential.java:310)
>> at org.h2.table.Table.fireConstraints(Table.java:894)
>> at org.h2.table.Table.fireAfterRow(Table.java:911)
>> at org.h2.command.dml.Insert.insertRows(Insert.java:162)
>> at org.h2.command.dml.Insert.update(Insert.java:115)
>> at org.h2.command.CommandContainer.update(CommandContainer.java:79)
>> at org.h2.command.Command.executeUpdate(Command.java:253)
>> at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(Jdbc
>> PreparedStatement.java:154)
>> at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPrepared
>> Statement.java:140)
>> at org.wso2.carbon.device.mgt.core.operation.mgt.dao.impl.Gener
>> icOperationDAOImpl.addOperationResponse(GenericOperationDAOImpl.java:211)
>> ... 59 more
>>
>>
>> Thanks & regards,
>> /charithag
>> --
>> *Charitha Goonetilleke*
>> Senior Software Engineer
>> WSO2 Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> mobile: +94 77 751 3669 <%2B94777513669>
>> Twitter:@CharithaWs <https://twitter.com/CharithaWs>, fb: charithag
>> <https://www.facebook.com/charithag>, linkedin: charithag
>> <http://www.linkedin.com/in/charithag>
>>
>> <http://wso2.com/signature>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "WSO2 IoT Team Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to iot-group+unsubscr...@wso2.com.
>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>
>
>
>
> --
> With Regards,
>
> *Rasika Perera*
> Senior Software Engineer
> LinkedIn: http://lk.linkedin.com/in/rasika90
>
> <http://wso2.com/signature>
>
> WSO2 Inc. www.wso2.com
> lean.enterprise.middleware
>
--
*Charitha Goonetilleke*
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
mobile: +94 77 751 3669 <%2B94777513669>
Twitter:@CharithaWs <https://twitter.com/CharithaWs>, fb: charithag
<https://www.facebook.com/charithag>, linkedin: charithag
<http://www.linkedin.com/in/charithag>
<http://wso2.com/signature>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
