Hi Somesh, Sorry for late reply. I ctually deleted all the entries from op_dc_ip_address alloc table as the cpvm was hung in start phase...the issue started after that only..I then again entered a dummy entry with corredponding reservation id and nic id uing nic table and vm_instance table but still the error is same. Can you guide me which files or db dump to attach so that you can help me resolve the issue
Regards Tilak Raj Singh On Fri, Jul 10, 2015 at 9:51 PM, Somesh Naidu <somesh.na...@citrix.com> wrote: > Need more logs (the relevant exception stack to be precise) to locate the > root cause of the error. > > In addition, can you check the integrity of the following table/view (just > perform a select on these): > async_job_view > op_dc_ip_address_alloc > > Regards, > Somesh > > > -----Original Message----- > From: Tilak Raj Singh [mailto:tila...@gmail.com] > Sent: Thursday, July 09, 2015 1:36 PM > To: users@cloudstack.apache.org > Subject: Re: Unable to spawn new System VMs after server restarted > > hi all...can somebody please guide me through this process > > Regards > > On Wed, Jul 8, 2015 at 8:25 PM, Tilak Raj Singh <tila...@gmail.com> wrote: > > > Hi All, > > > > Recently my Server started from backup and last I checked all my > > Management IPs were full and hence Cloudstack was not able to spawn new > > System Vms. > > Trying to solv the issue i cleared all the allocated management ips from > > op_dc_ip_address_alloc table in database. But now I am getting this issue > > of Foreign Key constraint when the management server is trying to > expunge a > > whose entry was there in the database. As a consequence new system vms > are > > not able to spawn up. Can somebody help me with details as to which > entries > > and in which tables should I delete for all this to start working again > or > > any other help in this regard. > > > > Here is the error log > > > > 2015-07-08 20:16:54,880 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Expunging unfinished job > AsyncJobVO > > {id:15864, userId: 1, accountId: 1, instanceType: null, instanceId: null, > > cmd: com.cloud.vm.VmWorkStart, cmdInfo: > > > rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAJMHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > > result: null, initMsid: 91019618482122, completeMsid: null, lastUpdated: > > null, lastPolled: null, created: Tue Jun 30 00:59:18 IST 2015} > > 2015-07-08 20:16:54,882 DEBUG [c.c.u.d.T.Transaction] > > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Rolling back the transaction: > Time = > > 2 Name = AsyncJobMgr-Heartbeat-1; called by > > > -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-TransactionLegacy.close:667-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy226.expunge:-1-AsyncJobManagerImpl$8.doInTransactionWithoutResult:813-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:49 > > 2015-07-08 20:16:54,908 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] > > (AsyncJobMgr-Heartbeat-1:ctx-19bf7bc0) Unexpected exception when trying > to > > execute queue item, > > com.cloud.utils.exception.CloudRuntimeException: DB Exception on: > > com.mysql.jdbc.PreparedStatement@1ea2e3c8: DELETE FROM async_job WHERE > > async_job.id= 15864 > > at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178) > > at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > > at > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > > at > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > > at > > > com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) > > at > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > > at > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > > at > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > > at > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > > at com.sun.proxy.$Proxy226.expunge(Unknown Source) > > at > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$8.doInTransactionWithoutResult(AsyncJobManagerImpl.java:813) > > at > > > com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25) > > at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49) > > at com.cloud.utils.db.Transaction.execute(Transaction.java:37) > > at com.cloud.utils.db.Transaction.execute(Transaction.java:46) > > at > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.expungeAsyncJob(AsyncJobManagerImpl.java:810) > > at > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.reallyRun(AsyncJobManagerImpl.java:773) > > at > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$7.runInContext(AsyncJobManagerImpl.java:749) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: > > > com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: > > Cannot delete or update a parent row: a foreign key constraint fails > > (`cloud`.`async_job_join_map`, CONSTRAINT > > `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) > REFERENCES > > `async_job` (`id`)) > > at sun.reflect.GeneratedConstructorAccessor116.newInstance(Unknown > Source) > > at > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at com.mysql.jdbc.Util.getInstance(Util.java:386) > > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040) > > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597) > > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529) > > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) > > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) > > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625) > > at > > > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119) > > at > > > com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415) > > at > > > com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333) > > at > > > com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318) > > at > > > org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) > > at > > > org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) > > at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169) > > ... 32 more > > > > Regards > > >
