[ https://issues.apache.org/jira/browse/CLOUDSTACK-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194884#comment-14194884 ]
ASF subversion and git services commented on CLOUDSTACK-7832: ------------------------------------------------------------- Commit ffaabdc13fde0f0f7b2667a483006e2a4b805f63 in cloudstack's branch refs/heads/master from [~minchen07] [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=ffaabdc ] CLOUDSTACK-7832: Move some job db update and item purge to completeAsyncJob transaction to avoid MySQL deadlock. > MySQL deadlock occurred in resetting job_executing_msid of the completed vm > worker job causing job queue stuck. > --------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-7832 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7832 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.3.0 > Reporter: Min Chen > Assignee: Min Chen > Priority: Critical > Fix For: 4.5.0 > > > This is mainly happening for an vm operation involving two VM work jobs > consecutively, for example, destroyVM which involved two steps: stopVM and > deleteAllVMSnapshots. In updating job_executing_msid column of the first VM > worker job, the second VM worker job has been scheduled to run, then we will > see such MySQL deadlock exception happening: > 2014-09-17 03:26:40,538 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-164:job-831168/job-831176) Double excep > tion > com.cloud.utils.exception.CloudRuntimeException: DB Exception on: > com.mysql.jdbc.PreparedStatement@42c4a7ea: UPDATE async_ > job SET async_job.job_executing_msid=null WHERE async_job.id = 831176 > Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: > Deadlock found when trying to get lock; try restarting transaction > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > Due to this exception, the queue item is not purged for the first vm worker > job, then all the following operations for the same VM will get stuck. -- This message was sent by Atlassian JIRA (v6.3.4#6332)