RE: vms hang in expunged state
Valery, Do you still have assigned ip addresses to the vms? Regards, -Original Message- From: Valery Ciareszka [mailto:valery.teres...@gmail.com] Sent: donderdag 25 juli 2013 14:54 To: users Subject: vms hang in expunged state Hi all. I have faced the following problem: virtual machines remain in expunging state forever, and I see java.lang.NullPointerException in management server logs. I use CS 4.1, CentOS 6.4, KVM. What is the proper way to handle this issue ? Management server log: 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl] (UserVm-Scavenger-1:null) Found 2 vms to expunge. 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl] (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but the state is Expunging 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl] (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to Expunging with event: ExpungeOperationvm's original host id: 19 new host id: null host id before state transit ion: null 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1] 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] (UserVm-Scavenger-1:null) Cleaning up NICS 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl] (UserVm-Scavenger-1:null) Cleaning network for vm: 6164 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl] (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl] (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1] 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl] (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1] resources... 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl] (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl] (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a part of vm id=6164 expunge 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl] (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm id=6164 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl] (UserVm-Scavenger-1:null) Port forwarding rules are removed su ccessfully as a part of vm id=6164 expunge 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl] (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers as a part of expunge process 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl] (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of disabling static nat for public IP id=1305 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl] (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl] (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1] java.lang.NullPointerException at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116) at com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79) at com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119) at com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734) at com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194) at com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856) at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787) at com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) -- Regards, Valery http://protocol.by/slayer
Re: vms hang in expunged state
Daan, Yes, static nat was enabled for theese vms. I disabled static nat and vms were successfully expunged. Looks like a bug. On Thu, Aug 1, 2013 at 4:27 PM, Daan Hoogland wrote: > Valery, > > Do you still have assigned ip addresses to the vms? > > Regards, > > -Original Message- > From: Valery Ciareszka [mailto:valery.teres...@gmail.com] > Sent: donderdag 25 juli 2013 14:54 > To: users > Subject: vms hang in expunged state > > Hi all. > > I have faced the following problem: virtual machines remain in expunging > state forever, and I see java.lang.NullPointerException in management > server logs. > I use CS 4.1, CentOS 6.4, KVM. > > What is the proper way to handle this issue ? > > Management server log: > > 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl] > (UserVm-Scavenger-1:null) Found 2 vms to expunge. > 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but the > state is Expunging > 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl] > (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to > Expunging with event: ExpungeOperationvm's original host id: 19 new host > id: null host id before state transit > ion: null > 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1] > 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (UserVm-Scavenger-1:null) Cleaning up NICS > 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl] > (UserVm-Scavenger-1:null) Cleaning network for vm: 6164 > 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl] > (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164 > 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1] > 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl] > (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1] > resources... > 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl] > (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164 > 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl] > (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a > part of vm id=6164 expunge > 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl] > (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm > id=6164 > 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl] > (UserVm-Scavenger-1:null) Port forwarding rules are removed su ccessfully > as a part of vm id=6164 expunge > 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl] > (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers as a > part of expunge process > 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl] > (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of > disabling static nat for public IP id=1305 > 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl] > (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305 > 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl] > (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1] > java.lang.NullPointerException > at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116) > at com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79) > at > > com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47) > at > > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119) > at > > com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734) > at > > com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194) > at > > com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856) > at > com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787) > at > com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > > > -- > Regar
Re: vms hang in expunged state
Guess so, please enter it at issues.apache... thanks, Daan On Wed, Aug 14, 2013 at 5:27 PM, Valery Ciareszka wrote: > Daan, > > Yes, static nat was enabled for theese vms. I disabled static nat and vms > were successfully expunged. > Looks like a bug. > > On Thu, Aug 1, 2013 at 4:27 PM, Daan Hoogland > wrote: > >> Valery, >> >> Do you still have assigned ip addresses to the vms? >> >> Regards, >> >> -Original Message- >> From: Valery Ciareszka [mailto:valery.teres...@gmail.com] >> Sent: donderdag 25 juli 2013 14:54 >> To: users >> Subject: vms hang in expunged state >> >> Hi all. >> >> I have faced the following problem: virtual machines remain in expunging >> state forever, and I see java.lang.NullPointerException in management >> server logs. >> I use CS 4.1, CentOS 6.4, KVM. >> >> What is the proper way to handle this issue ? >> >> Management server log: >> >> 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl] >> (UserVm-Scavenger-1:null) Found 2 vms to expunge. >> 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but the >> state is Expunging >> 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl] >> (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to >> Expunging with event: ExpungeOperationvm's original host id: 19 new host >> id: null host id before state transit >> ion: null >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1] >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (UserVm-Scavenger-1:null) Cleaning up NICS >> 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl] >> (UserVm-Scavenger-1:null) Cleaning network for vm: 6164 >> 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl] >> (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164 >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1] >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl] >> (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1] >> resources... >> 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl] >> (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164 >> 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl] >> (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a >> part of vm id=6164 expunge >> 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl] >> (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm >> id=6164 >> 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl] >> (UserVm-Scavenger-1:null) Port forwarding rules are removed su ccessfully >> as a part of vm id=6164 expunge >> 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl] >> (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers as a >> part of expunge process >> 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl] >> (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of >> disabling static nat for public IP id=1305 >> 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl] >> (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305 >> 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl] >> (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1] >> java.lang.NullPointerException >> at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116) >> at com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79) >> at >> >> com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47) >> at >> >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119) >> at >> >> com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734) >> at >> >> com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194) >> at >> >> com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856) >> at >> com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787) >> at >> com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) >> at >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) >> at >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) >> at >> >> java.u
Re: vms hang in expunged state
https://issues.apache.org/jira/browse/CLOUDSTACK-4349 On Wed, Aug 14, 2013 at 6:35 PM, Daan Hoogland wrote: > Guess so, > > please enter it at issues.apache... > > thanks, > Daan > > On Wed, Aug 14, 2013 at 5:27 PM, Valery Ciareszka > wrote: > > Daan, > > > > Yes, static nat was enabled for theese vms. I disabled static nat and vms > > were successfully expunged. > > Looks like a bug. > > > > On Thu, Aug 1, 2013 at 4:27 PM, Daan Hoogland > > wrote: > > > >> Valery, > >> > >> Do you still have assigned ip addresses to the vms? > >> > >> Regards, > >> > >> -Original Message- > >> From: Valery Ciareszka [mailto:valery.teres...@gmail.com] > >> Sent: donderdag 25 juli 2013 14:54 > >> To: users > >> Subject: vms hang in expunged state > >> > >> Hi all. > >> > >> I have faced the following problem: virtual machines remain in expunging > >> state forever, and I see java.lang.NullPointerException in management > >> server logs. > >> I use CS 4.1, CentOS 6.4, KVM. > >> > >> What is the proper way to handle this issue ? > >> > >> Management server log: > >> > >> 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl] > >> (UserVm-Scavenger-1:null) Found 2 vms to expunge. > >> 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but > the > >> state is Expunging > >> 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl] > >> (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to > >> Expunging with event: ExpungeOperationvm's original host id: 19 new host > >> id: null host id before state transit > >> ion: null > >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1] > >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> (UserVm-Scavenger-1:null) Cleaning up NICS > >> 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl] > >> (UserVm-Scavenger-1:null) Cleaning network for vm: 6164 > >> 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl] > >> (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164 > >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl] > >> (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1] > >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl] > >> (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1] > >> resources... > >> 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl] > >> (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164 > >> 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl] > >> (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a > >> part of vm id=6164 expunge > >> 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl] > >> (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm > >> id=6164 > >> 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl] > >> (UserVm-Scavenger-1:null) Port forwarding rules are removed su > ccessfully > >> as a part of vm id=6164 expunge > >> 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl] > >> (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers > as a > >> part of expunge process > >> 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl] > >> (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of > >> disabling static nat for public IP id=1305 > >> 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl] > >> (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305 > >> 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl] > >> (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1] > >> java.lang.NullPointerException > >> at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116) > >> at > com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79) > >> at > >> > >> > com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47) > >> at > >> > >> > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119) > >> at > >> > >> > com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734) > >> at > >> > >> > com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194) > >> at > >> > >> > com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856) > >> at > >> com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1787) > >> at > >> > com.cloud.vm.UserVmManagerImpl$ExpungeTask.run(UserVmManagerImpl.java:2416) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at > >> > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > >> at > java
Re: vms hang in expunged state
Thanks Valery, I know some colleagues had the issue as well. The work around is to delete ip-addresses first and we are using that now. I will pick it up if it reaches the top off my prio list. regards, Daan On Thu, Aug 15, 2013 at 12:14 PM, Valery Ciareszka wrote: > https://issues.apache.org/jira/browse/CLOUDSTACK-4349 > > > On Wed, Aug 14, 2013 at 6:35 PM, Daan Hoogland wrote: > >> Guess so, >> >> please enter it at issues.apache... >> >> thanks, >> Daan >> >> On Wed, Aug 14, 2013 at 5:27 PM, Valery Ciareszka >> wrote: >> > Daan, >> > >> > Yes, static nat was enabled for theese vms. I disabled static nat and vms >> > were successfully expunged. >> > Looks like a bug. >> > >> > On Thu, Aug 1, 2013 at 4:27 PM, Daan Hoogland >> > wrote: >> > >> >> Valery, >> >> >> >> Do you still have assigned ip addresses to the vms? >> >> >> >> Regards, >> >> >> >> -Original Message- >> >> From: Valery Ciareszka [mailto:valery.teres...@gmail.com] >> >> Sent: donderdag 25 juli 2013 14:54 >> >> To: users >> >> Subject: vms hang in expunged state >> >> >> >> Hi all. >> >> >> >> I have faced the following problem: virtual machines remain in expunging >> >> state forever, and I see java.lang.NullPointerException in management >> >> server logs. >> >> I use CS 4.1, CentOS 6.4, KVM. >> >> >> >> What is the proper way to handle this issue ? >> >> >> >> Management server log: >> >> >> >> 2013-07-25 12:30:17,674 INFO [cloud.vm.UserVmManagerImpl] >> >> (UserVm-Scavenger-1:null) Found 2 vms to expunge. >> >> 2013-07-25 12:30:17,674 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> (UserVm-Scavenger-1:null) Stopped called on VM[User|te st090703-1] but >> the >> >> state is Expunging >> >> 2013-07-25 12:30:17,694 DEBUG [cloud.capacity.CapacityManagerImpl] >> >> (UserVm-Scavenger-1:null) VM state transitted from :Ex punging to >> >> Expunging with event: ExpungeOperationvm's original host id: 19 new host >> >> id: null host id before state transit >> >> ion: null >> >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> (UserVm-Scavenger-1:null) Destroying vm VM[User|test09 0703-1] >> >> 2013-07-25 12:30:17,694 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> (UserVm-Scavenger-1:null) Cleaning up NICS >> >> 2013-07-25 12:30:17,694 DEBUG [cloud.network.NetworkManagerImpl] >> >> (UserVm-Scavenger-1:null) Cleaning network for vm: 6164 >> >> 2013-07-25 12:30:17,696 DEBUG [cloud.storage.StorageManagerImpl] >> >> (UserVm-Scavenger-1:null) Cleaning storage for vm: 6164 >> >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> >> (UserVm-Scavenger-1:null) Expunged VM[User|test090703- 1] >> >> 2013-07-25 12:30:17,697 DEBUG [cloud.vm.UserVmManagerImpl] >> >> (UserVm-Scavenger-1:null) Starting cleaning up vm VM[User|test 090703-1] >> >> resources... >> >> 2013-07-25 12:30:17,705 DEBUG [network.firewall.FirewallManagerImpl] >> >> (UserVm-Scavenger-1:null) No firewall rules are foun d for vm id=6164 >> >> 2013-07-25 12:30:17,706 DEBUG [cloud.vm.UserVmManagerImpl] >> >> (UserVm-Scavenger-1:null) Firewall rules are removed successfu lly as a >> >> part of vm id=6164 expunge >> >> 2013-07-25 12:30:17,709 DEBUG [network.rules.RulesManagerImpl] >> >> (UserVm-Scavenger-1:null) No port forwarding rules are fou nd for vm >> >> id=6164 >> >> 2013-07-25 12:30:17,709 DEBUG [cloud.vm.UserVmManagerImpl] >> >> (UserVm-Scavenger-1:null) Port forwarding rules are removed su >> ccessfully >> >> as a part of vm id=6164 expunge >> >> 2013-07-25 12:30:17,711 DEBUG [cloud.vm.UserVmManagerImpl] >> >> (UserVm-Scavenger-1:null) Removed vm id=6164 from all load bal ancers >> as a >> >> part of expunge process >> >> 2013-07-25 12:30:17,714 DEBUG [network.rules.RulesManagerImpl] >> >> (UserVm-Scavenger-1:null) Revoking all Firewallrules as a part of >> >> disabling static nat for public IP id=1305 >> >> 2013-07-25 12:30:17,716 DEBUG [network.firewall.FirewallManagerImpl] >> >> (UserVm-Scavenger-1:null) Releasing 3 firewall rules for ip id=1305 >> >> 2013-07-25 12:30:17,718 WARN [cloud.vm.UserVmManagerImpl] >> >> (UserVm-Scavenger-1:null) Unable to expunge VM[User|test090703-1] >> >> java.lang.NullPointerException >> >> at com.cloud.event.EventUtils.getDomainId(EventUtils.java:116) >> >> at >> com.cloud.event.EventUtils.saveStartedEvent(EventUtils.java:79) >> >> at >> >> >> >> >> com.cloud.event.ActionEventInterceptor.interceptStart(ActionEventInterceptor.java:47) >> >> at >> >> >> >> >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:119) >> >> at >> >> >> >> >> com.cloud.network.firewall.FirewallManagerImpl.revokeFirewallRulesForIp(FirewallManagerImpl.java:734) >> >> at >> >> >> >> >> com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1194) >> >> at >> >> >> >> >> com.cloud.vm.UserVmManagerImpl.cleanupVmResources(UserVmManagerImpl.java:1856) >> >>