/etc/cloudstack/management/log4j-cloud.xml
more info:
https://image.slidesharecdn.com/cloudstacktroubleshooting-lily-150506063617-conversion-gate02/95/cloud-stack-troubleshooting-12-638.jpg?cb=1430894327
cheers
On Thu, 10 Jan 2019 at 19:17, Ivan X Yue <ivan...@ca.ibm.com> wrote:
> Dag,
>
> That's good idea. How can I update log to debug / trace mode?
>
> Thanks
> Ivan
>
>
>
> From: Dag Sonstebo <dag.sonst...@shapeblue.com>
> To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
> Date: 2019/01/10 11:40 AM
> Subject: Re: Unable to schedule async job
>
>
>
> Hi Ivan,
>
> Probably a good idea to bump your logging up to debug or trace – these
> show the SQL queries being prepared in the logs.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
>
> From: Ivan X Yue <ivan...@ca.ibm.com>
> Reply-To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
> Date: Thursday, 10 January 2019 at 16:18
> To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
> Subject: Re: Unable to schedule async job
>
> Hi, Dag,
>
> Thank for the reply.
>
> Below is my async_job table. Today, I try to delete some VM.
> Interestingly, I can delete some VM and some are failed withthis error:
>
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
>
> Duplicate entry '387' for key 'PRIMARY'
>
> But in the async_job table, I don't see id=387 there.
>
> I have attached part of the Management Server log wrt the delete VM
> action.
>
> One minor correction. I am actually using cloudstack 4.11.1 instead of
> 4.9.2.
>
> MariaDB [cloud]> select id, job_cmd from async_job order by id;
> +-----+-----------------------------------------------------------------+
> | id | job_cmd |
> +-----+-----------------------------------------------------------------+
> | 275 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
> | 276 | NULL |
> | 281 | NULL |
> | 287 | NULL |
> | 288 | NULL |
> | 292 | NULL |
> | 305 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 307 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 309 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 313 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd |
> | 314 | NULL |
> | 315 | org.apache.cloudstack.api.command.admin.router.StartRouterCmd |
> | 316 | com.cloud.vm.VmWorkStart |
> | 317 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 319 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 321 | NULL |
> | 322 | com.cloud.vm.VmWorkStop |
> | 323 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 327 | com.cloud.vm.VmWorkStart |
> | 328 | com.cloud.vm.VmWorkStart |
> | 329 | NULL |
> | 330 | com.cloud.vm.VmWorkStop |
> | 332 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 334 | com.cloud.vm.VmWorkStop |
> | 335 | com.cloud.vm.VmWorkStart |
> | 336 | com.cloud.vm.VmWorkStart |
> | 338 | org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin |
> | 340 | com.cloud.vm.VmWorkStop |
> | 341 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 342 | com.cloud.vm.VmWorkStop |
> | 343 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd |
> | 344 | com.cloud.vm.VmWorkStart |
> | 345 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 346 | com.cloud.vm.VmWorkStop |
> | 347 | org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd |
> | 348 | com.cloud.vm.VmWorkStop |
> | 349 | org.apache.cloudstack.api.command.user.vpc.RestartVPCCmd |
> | 350 | com.cloud.vm.VmWorkStart |
> | 351 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 352 | com.cloud.vm.VmWorkStop |
> | 354 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
> | 355 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 356 | com.cloud.vm.VmWorkStop |
> | 357 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots |
> | 358 | com.cloud.vm.VmWorkStop |
> | 359 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
> | 360 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 361 | com.cloud.vm.VmWorkStop |
> | 362 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots |
> | 363 | com.cloud.vm.VmWorkStop |
> | 364 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 365 | com.cloud.vm.VmWorkStop |
> | 366 | com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots |
> | 367 | com.cloud.vm.VmWorkStop |
> | 368 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
> | 369 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
> | 370 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 372 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 374 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 376 | org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd |
> | 377 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 378 | com.cloud.vm.VmWorkStop |
> | 380 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 381 | com.cloud.vm.VmWorkStop |
> | 383 | org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin |
> | 384 | com.cloud.vm.VmWorkStart |
> | 385 | org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin |
> | 386 | com.cloud.vm.VmWorkStop |
> +-----+-----------------------------------------------------------------+
>
>
>
> Thanks
> ------------------------------------------------------------------------
> Ivan Yue
> Development Manager - CLM SaaS Operations
> Persistent Systems Limited
> Email: ivan...@ca.ibm.com
> Email: ivan_...@persistent.com
> ------------------------------------------------------------------------
>
>
>
> dag.sonst...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
>
>
> From: Dag Sonstebo <dag.sonst...@shapeblue.com>
> To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
> Date: 2019/01/10 05:01 AM
> Subject: Re: Unable to schedule async job
> ________________________________
>
>
>
> What does your async_job table say around id>330?
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
>
> On 10/01/2019, 05:34, "Ivan X Yue" <ivan...@ca.ibm.com> wrote:
>
> Hi,
>
> I am using CloudStack 4.9.2 with KVM hypervisors. Today, I find that
> the
> hypervisor is not responding, and therefore I restart it. After that,
> I
> find that virtual routers are stopped. When I try to start them, I
> keep
> getting "Unable to schedule async job" error.
>
> From the management-server.log, I see some exception related to MySQL:
>
> Caused by:
>
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry '333' for key 'PRIMARY'
>
> Is there any cleanup that I need to do in the database? How can I do
> that?
>
>
> Here is the full stacktrace of the exception that I get:
>
>
> 2019-01-09 12:06:31,417 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-5:ctx-3248f525 job-332 ctx-d797e8bd) (logid:d37b7ec3)
> Unable to schedule async job for command com.cloud.vm.VmWorkStop,
> unexpected exception.
> javax.persistence.EntityExistsException: Entity already exists:
> at
> com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
> at
>
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
> at
>
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
> at
> com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> at
>
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491)
> at
>
>
> com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253)
> at
>
>
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> at
>
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> at
>
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
>
>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at
>
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> at
>
>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy239.destroyRouter(Unknown Source)
> at
>
>
> org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd.execute(DestroyRouterCmd.java:103)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
>
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> at
>
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
>
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
>
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
>
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
>
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
>
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
>
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry '333' for key 'PRIMARY'
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
>
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
>
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> ... 55 more
> 2019-01-09 12:06:31,418 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-5:ctx-3248f525 job-332) (logid:d37b7ec3) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd
> com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
> async
> job for command com.cloud.vm.VmWorkStop, unexpected exception.
> at
>
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:513)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:502)
> at
>
>
> com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:491)
> at
>
>
> com.cloud.network.router.NetworkHelperImpl.destroyRouter(NetworkHelperImpl.java:253)
> at
>
>
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.destroyRouter(VirtualNetworkApplianceManagerImpl.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> at
>
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> at
>
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
>
>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at
>
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> at
>
>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy239.destroyRouter(Unknown Source)
> at
>
>
> org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd.execute(DestroyRouterCmd.java:103)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
>
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> at
>
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
>
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
>
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
>
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
>
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
>
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-01-09 12:06:31,419 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-5:ctx-3248f525 job-332) (logid:d37b7ec3) Complete
> async
> job-332, jobStatus: FAILED, resultCode: 530, result:
>
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
> to schedule async job for command com.cloud.vm.VmWorkStop, unexpected
> exception."}
>
> Thanks for all the helps in advance.
>
>
> Thanks,
> Ivan
>
>
>
>
>
> dag.sonst...@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
>
>
>
>
>
>
>
>
>
--
Andrija Panić
