Hi All, I have a basic zone with ACS 4.8 with xenserver 6.5 , I created a new advanced zone (Untagged for management , tagged for public and guest ) , after creating the advanced zone I tried to create an instance but an error appeared , I know that first vm once created it should be create a virtual router and I checked it and it was stopped and I can't start it , also I tried copy templates from basic zone to advanced zone with no luck ( failed to copy template I checked the logs and the error is connection refused ) .
Below logs from management to create a new VM , can you please advise ? 2016-11-23 12:46:20,877 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:c02108f7) Add job-44041 into job monitoring 2016-11-23 12:46:20,884 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Executing AsyncJobVO {id:44041, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAM63QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144344642087, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed Nov 23 12:46:19 EET 2016} 2016-11-23 12:46:20,885 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Run VM work job: com.cloud.vm.VmWorkStop for VM 3307, job origin: 44040 2016-11-23 12:46:20,886 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":3307,"handlerName":"VirtualMachineManagerImpl"} 2016-11-23 12:46:20,890 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Stopped called on VM[User|i-2-3307-VM] but the state is Error 2016-11-23 12:46:20,890 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":3307,"handlerName":"VirtualMachineManagerImpl"} 2016-11-23 12:46:20,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Complete async job-44041, jobStatus: SUCCEEDED, resultCode: 0, result: null 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Publish async job-44041 complete on message bus 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Wake up jobs related to job-44041 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Update db status for job-44041 2016-11-23 12:46:20,892 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Wake up jobs joined with job-44041 and disjoin all subjobs created from job- 44041 2016-11-23 12:46:20,896 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 3307, job origin: 44040 2016-11-23 12:46:20,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Done executing com.cloud.vm.VmWorkStop for job-44041 2016-11-23 12:46:20,897 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Remove job-44041 from job monitoring 2016-11-23 12:46:19,109 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-35:ctx-110f9706 job-44040) (logid:a12b373b) Add job-44040 into job monitoring 2016-11-23 12:46:19,117 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-10:ctx-cde514ac ctx-618855d1) (logid:7a08842b) submit async job-44040, details: AsyncJobVO {id:44040, userId: 2, accountId: 2, instanceType: VirtualMachine, instanceId: 3307, cmd: org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin, cmdInfo: {"response":"json","id":"8e3862b3-c0cd-40f4-93b6-8bb99daaed3c","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"8e3862b3-c0cd-40f4-93b6-8bb99daaed3c\"}","cmdEventType":"VM.DESTROY","ctxUserId":"2","httpmethod":"GET","_":"1479897977373","uuid":"8e3862b3-c0cd-40f4-93b6-8bb99daaed3c","ctxAccountId":"2","expunge":"true","ctxStartEventId":"86061"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144344642087, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2016-11-23 12:46:19,117 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040) (logid:0cdaace9) Executing AsyncJobVO {id:44040, userId: 2, accountId: 2, instanceType: VirtualMachine, instanceId: 3307, cmd: org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin, cmdInfo: {"response":"json","id":"8e3862b3-c0cd-40f4-93b6-8bb99daaed3c","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"8e3862b3-c0cd-40f4-93b6-8bb99daaed3c\"}","cmdEventType":"VM.DESTROY","ctxUserId":"2","httpmethod":"GET","_":"1479897977373","uuid":"8e3862b3-c0cd-40f4-93b6-8bb99daaed3c","ctxAccountId":"2","expunge":"true","ctxStartEventId":"86061"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144344642087, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2016-11-23 12:46:19,136 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Destroying vm VM[User|i-2-3307-VM] 2016-11-23 12:46:19,143 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Sync job-44041 execution on object VmWorkJobQueue.3307 2016-11-23 12:46:20,877 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:c02108f7) Add job-44041 into job monitoring 2016-11-23 12:46:20,884 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Executing AsyncJobVO {id:44041, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAM63QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144344642087, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed Nov 23 12:46:19 EET 2016} 2016-11-23 12:46:20,885 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Run VM work job: com.cloud.vm.VmWorkStop for VM 3307, job origin: 44040 2016-11-23 12:46:20,886 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":3307,"handlerName":"VirtualMachineManagerImpl"} 2016-11-23 12:46:20,890 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Stopped called on VM[User|i-2-3307-VM] but the state is Error 2016-11-23 12:46:20,890 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":3307,"handlerName":"VirtualMachineManagerImpl"} 2016-11-23 12:46:20,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Complete async job-44041, jobStatus: SUCCEEDED, resultCode: 0, result: null 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Publish async job-44041 complete on message bus 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Wake up jobs related to job-44041 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Update db status for job-44041 2016-11-23 12:46:20,892 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Wake up jobs joined with job-44041 and disjoin all subjobs created from job- 44041 2016-11-23 12:46:20,896 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 3307, job origin: 44040 2016-11-23 12:46:20,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Done executing com.cloud.vm.VmWorkStop for job-44041 2016-11-23 12:46:20,897 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Remove job-44041 from job monitoring 2016-11-23 12:46:20,906 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Sync job-44042 execution on object VmWorkJobQueue.3307 2016-11-23 12:46:22,895 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042) (logid:56caed9e) Add job-44042 into job monitoring 2016-11-23 12:46:22,907 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042) (logid:0cdaace9) Executing AsyncJobVO {id:44042, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots, cmdInfo: rO0ABXNyADBjb20uY2xvdWQudm0uc25hcHNob3QuVm1Xb3JrRGVsZXRlQWxsVk1TbmFwc2hvdHOsl-VRajf8cAIAAUwABHR5cGV0ACdMY29tL2Nsb3VkL3ZtL3NuYXBzaG90L1ZNU25hcHNob3QkVHlwZTt4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAACAAAAAAAADOt0ABVWTVNuYXBzaG90TWFuYWdlckltcGxw, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144344642087, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed Nov 23 12:46:20 EET 2016} 2016-11-23 12:46:22,907 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042) (logid:0cdaace9) Run VM work job: com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots for VM 3307, job origin: 44040 2016-11-23 12:46:22,908 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042 ctx-f30788ef) (logid:0cdaace9) Execute VM work job: com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots{"userId":2,"accountId":2,"vmId":3307,"handlerName":"VMSnapshotManagerImpl"} 2016-11-23 12:46:22,910 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042 ctx-f30788ef) (logid:0cdaace9) Done executing VM work job: com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots{"userId":2,"accountId":2,"vmId":3307,"handlerName":"VMSnapshotManagerImpl"} 2016-11-23 12:46:22,910 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042 ctx-f30788ef) (logid:0cdaace9) Complete async job-44042, jobStatus: SUCCEEDED, resultCode: 0, result: rO0ABXNyABFqYXZhLmxhbmcuQm9vbGVhbs0gcoDVnPruAgABWgAFdmFsdWV4cAE 2016-11-23 12:46:22,911 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042 ctx-f30788ef) (logid:0cdaace9) Publish async job-44042 complete on message bus 2016-11-23 12:46:22,911 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042 ctx-f30788ef) (logid:0cdaace9) Wake up jobs related to job-44042 2016-11-23 12:46:22,911 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042 ctx-f30788ef) (logid:0cdaace9) Update db status for job-44042 2016-11-23 12:46:22,912 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042 ctx-f30788ef) (logid:0cdaace9) Wake up jobs joined with job-44042 and disjoin all subjobs created from job- 44042 2016-11-23 12:46:22,916 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042) (logid:0cdaace9) Done with run of VM work job: com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots for VM 3307, job origin: 44040 2016-11-23 12:46:22,916 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042) (logid:0cdaace9) Done executing com.cloud.vm.snapshot.VmWorkDeleteAllVMSnapshots for job-44042 2016-11-23 12:46:22,917 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-5:ctx-edab2aa5 job-44040/job-44042) (logid:0cdaace9) Remove job-44042 from job monitoring 2016-11-23 12:46:22,928 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) VM state transitted from :Error to Expunging with event: DestroyRequestedvm's original host id: null new host id: null host id before state transition: null 2016-11-23 12:46:22,942 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Sync job-44043 execution on object VmWorkJobQueue.3307 2016-11-23 12:46:24,877 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043) (logid:9d472300) Add job-44043 into job monitoring 2016-11-23 12:46:24,886 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043) (logid:0cdaace9) Executing AsyncJobVO {id:44043, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAM63QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144344642087, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed Nov 23 12:46:22 EET 2016} 2016-11-23 12:46:24,886 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043) (logid:0cdaace9) Run VM work job: com.cloud.vm.VmWorkStop for VM 3307, job origin: 44040 2016-11-23 12:46:24,887 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043 ctx-800cc88d) (logid:0cdaace9) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":3307,"handlerName":"VirtualMachineManagerImpl"} 2016-11-23 12:46:24,890 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043 ctx-800cc88d) (logid:0cdaace9) Stopped called on VM[User|i-2-3307-VM] but the state is Expunging 2016-11-23 12:46:24,890 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043 ctx-800cc88d) (logid:0cdaace9) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":3307,"handlerName":"VirtualMachineManagerImpl"} 2016-11-23 12:46:24,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043 ctx-800cc88d) (logid:0cdaace9) Complete async job-44043, jobStatus: SUCCEEDED, resultCode: 0, result: null 2016-11-23 12:46:24,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043 ctx-800cc88d) (logid:0cdaace9) Publish async job-44043 complete on message bus 2016-11-23 12:46:24,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043 ctx-800cc88d) (logid:0cdaace9) Wake up jobs related to job-44043 2016-11-23 12:46:24,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043 ctx-800cc88d) (logid:0cdaace9) Update db status for job-44043 2016-11-23 12:46:24,892 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043 ctx-800cc88d) (logid:0cdaace9) Wake up jobs joined with job-44043 and disjoin all subjobs created from job- 44043 2016-11-23 12:46:24,896 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043) (logid:0cdaace9) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 3307, job origin: 44040 2016-11-23 12:46:24,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043) (logid:0cdaace9) Done executing com.cloud.vm.VmWorkStop for job-44043 2016-11-23 12:46:24,910 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-4:ctx-b018f78d job-44040/job-44043) (logid:0cdaace9) Remove job-44043 from job monitoring 2016-11-23 12:46:24,910 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) VM state transitted from :Expunging to Expunging with event: ExpungeOperationvm's original host id: null new host id: null host id before state transition: null 2016-11-23 12:46:24,910 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Destroying vm VM[User|i-2-3307-VM] 2016-11-23 12:46:24,910 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Cleaning up NICS 2016-11-23 12:46:24,910 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Cleaning network for vm: 3307 2016-11-23 12:46:24,914 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Service SecurityGroup is not supported in the network id=209 2016-11-23 12:46:24,917 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Asking VirtualRouter to release Nic[3307-3307-null-192.168.114.127] 2016-11-23 12:46:24,923 DEBUG [c.c.n.g.DirectNetworkGuru] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Deallocate network: networkId: 209, ip: 192.168.114.127 2016-11-23 12:46:24,929 DEBUG [c.c.n.g.DirectNetworkGuru] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) remove nic 3307 secondary ip 2016-11-23 12:46:24,932 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Removed nic id=3307 2016-11-23 12:46:24,933 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Revoving nic secondary ip entry ... 2016-11-23 12:46:24,933 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed storage 2016-11-23 12:46:24,934 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Cleaning storage for vm: 3307 2016-11-23 12:46:24,936 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Expunged VM[User|i-2-3307-VM] 2016-11-23 12:46:24,936 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Starting cleaning up vm VM[User|i-2-3307-VM] resources... 2016-11-23 12:46:24,947 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) No firewall rules are found for vm id=3307 2016-11-23 12:46:24,951 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Firewall rules are removed successfully as a part of vm id=3307 expunge 2016-11-23 12:46:24,954 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) No port forwarding rules are found for vm id=3307 2016-11-23 12:46:24,954 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Port forwarding rules are removed successfully as a part of vm id=3307 expunge 2016-11-23 12:46:24,956 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Removed vm id=3307 from all load balancers as a part of expunge process 2016-11-23 12:46:24,957 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Successfully cleaned up vm VM[User|i-2-3307-VM] resources as a part of expunge process 2016-11-23 12:46:24,968 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Complete async job-44040, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/null/{"securitygroup":[],"nic":[],"tags":[],"affinitygroup":[]} 2016-11-23 12:46:24,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Publish async job-44040 complete on message bus 2016-11-23 12:46:24,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Wake up jobs related to job-44040 2016-11-23 12:46:24,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Update db status for job-44040 2016-11-23 12:46:24,977 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Wake up jobs joined with job-44040 and disjoin all subjobs created from job- 44040 2016-11-23 12:46:24,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040) (logid:0cdaace9) Done executing org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin for job-44040 2016-11-23 12:46:24,983 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-35:ctx-110f9706 job-44040) (logid:0cdaace9) Remove job-44040 from job monitoring 2016-11-23 12:46:19,143 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-35:ctx-110f9706 job-44040 ctx-02764119) (logid:0cdaace9) Sync job-44041 execution on object VmWorkJobQueue.3307 2016-11-23 12:46:20,872 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-7534d4fd) (logid:ca865d0b) Schedule queued job-44041 2016-11-23 12:46:20,877 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:c02108f7) Add job-44041 into job monitoring 2016-11-23 12:46:20,884 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Executing AsyncJobVO {id:44041, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAM63QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 144344642087, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed Nov 23 12:46:19 EET 2016} 2016-11-23 12:46:20,885 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Run VM work job: com.cloud.vm.VmWorkStop for VM 3307, job origin: 44040 2016-11-23 12:46:20,886 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":3307,"handlerName":"VirtualMachineManagerImpl"} 2016-11-23 12:46:20,890 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Stopped called on VM[User|i-2-3307-VM] but the state is Error 2016-11-23 12:46:20,890 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":3307,"handlerName":"VirtualMachineManagerImpl"} 2016-11-23 12:46:20,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Complete async job-44041, jobStatus: SUCCEEDED, resultCode: 0, result: null 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Publish async job-44041 complete on message bus 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Wake up jobs related to job-44041 2016-11-23 12:46:20,891 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Update db status for job-44041 2016-11-23 12:46:20,892 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041 ctx-89e12346) (logid:0cdaace9) Wake up jobs joined with job-44041 and disjoin all subjobs created from job- 44041 2016-11-23 12:46:20,896 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 3307, job origin: 44040 2016-11-23 12:46:20,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Done executing com.cloud.vm.VmWorkStop for job-44041 2016-11-23 12:46:20,897 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-6:ctx-8e326052 job-44040/job-44041) (logid:0cdaace9) Remove job-44041 from job monitoring Best Regards, EtQ, Inc. Ghaith Bannoura Senior System Administrator MCT, MCSE (Messaging, Server Infrastructure) MCSA (Windows Server 2008, 2012), MCP * Phone: +9626-537-0923 Ext. 376 * gbanno...@etq.com <mailto:mal-khay...@etq.com> * http://www.etq.com<http://www.etq.com/>