The storage pools are looking good. That particular storage pool is the main pool where all vm volumes reside. I don't think it's an issue with the storage pool as when I try to migrate other vms, the NPE happens after a different storage pool in the log entry. All storage pools are working fine and active:
Name State Autostart ------------------------------------------- 53203b53-6a21-4361-88a7-631c517be1bd active no cf771bc7-8998-354d-8e10-5564585a3c20 active no d8d5ec36-3cb0-39af-8fc6-084a4abd5d28 active no I've been asked in the dev list to take a look at the vlans as there was a bug in the past. They have a suspecioun that the npe happens because of the way vlans are stored in the data base for some vms. I will try to investigate this area today. Andrei ----- Original Message ----- From: "Nux!" <n...@li.nux.ro> To: users@cloudstack.apache.org Sent: Friday, 15 May, 2015 9:30:10 AM Subject: Re: ACS 4.5.1 KVM live migration problem Andrei, "trying to fetch storage pool d8d5ec36-3cb0-39af-8fc6-084a4abd5d28 from libvirt", does that pool actually exist in libvirt? virsh pool-list --all What are the libvirtd logs saying? -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro ----- Original Message ----- > From: "Andrei Mikhailovsky" <and...@arhont.com> > To: users@cloudstack.apache.org > Sent: Friday, 15 May, 2015 01:04:22 > Subject: Re: ACS 4.5.1 KVM live migration problem > Lucian, > > I know and I should really have a test env before doing the live update. > Hopefully will get to it at some point. > > I am running acs 4.5.1 on Ubuntu 14.04 servers with stock qemu and libvirt as > far as I can tell. The versions are: > > libvirt-bin 1.2.2-0ubuntu13.1.10 > qemu-kvm 2.0.0+dfsg-2ubuntu1.10 > > The logs are at the bottom of this email, which i've posted earlier. > > Thanks > > ----- Original Message ----- > > From: "Nux!" <n...@li.nux.ro> > To: users@cloudstack.apache.org > Sent: Friday, 15 May, 2015 12:10:06 AM > Subject: Re: ACS 4.5.1 KVM live migration problem > > Andrei, > > It's good to test the RCs, that way anyone can report problems so they don't > end > up in the stable release. > I am using KVM, alas am using local storage and never test for live > migrations > ... > > Is there any more information you can share? ACS/libvirt/qemu logs etc. > > Lucian > > -- > Sent from the Delta quadrant using Borg technology! > > Nux! > www.nux.ro > > ----- Original Message ----- >> From: "Andrei Mikhailovsky" <and...@arhont.com> >> To: users@cloudstack.apache.org >> Sent: Thursday, 14 May, 2015 23:47:51 >> Subject: Re: ACS 4.5.1 KVM live migration problem > >> Lucian, >> >> I don't know as i've not tested the 4.5 branch prior to 4.5.1. I know for >> sure >> that i've not had any migration issues in 4.4.3, 4.4.2, 4.3.2 and 4.2.1. In >> fact, I don't recall having any migration issues until i've upgraded to >> 4.5.1. >> >> Thanks >> >> Andrei >> >> >> >> ----- Original Message ----- >> >> From: "Nux!" <n...@li.nux.ro> >> To: users@cloudstack.apache.org >> Sent: Thursday, 14 May, 2015 11:09:51 PM >> Subject: Re: ACS 4.5.1 KVM live migration problem >> >> Andrei, >> >> Do you know if the problem was also in the RC releases? >> >> Lucian >> >> -- >> Sent from the Delta quadrant using Borg technology! >> >> Nux! >> www.nux.ro >> >> ----- Original Message ----- >>> From: "Andrei Mikhailovsky" <and...@arhont.com> >>> To: users@cloudstack.apache.org >>> Sent: Thursday, 14 May, 2015 21:37:26 >>> Subject: Re: ACS 4.5.1 KVM live migration problem >> >>> Hi, >>> >>> Does anyone have a clue what is causing the migration issue? >>> >>> Has anyone else using ACS 4.5.1 with KVM and can successfully migrate their >>> instances? >>> >>> I've not had any migration issues prior to upgrading to version 4.5.1. >>> >>> Thanks >>> >>> ----- Original Message ----- >>> >>> From: "Andrei Mikhailovsky" <and...@arhont.com> >>> To: users@cloudstack.apache.org >>> Sent: Wednesday, 13 May, 2015 10:44:29 AM >>> Subject: Re: ACS 4.5.1 KVM live migration problem >>> >>> Hi Rohit, >>> >>> forgot to answer you on the cloud.vlan table. >>> >>> That particular vm has a network with vlan id 1151 as shown when i look at >>> the >>> network details in the acs gui. However, this vlan is not shown in the >>> cloud.vlan table. From what I can see the cloud.vlan table shows only the >>> public and management network vlan interfaces and does not show the guest >>> network vlans. >>> >>> In terms of the public network vlan which is used for routing traffic to >>> the >>> internet from this particular vm, it is: >>> >>> >>> mysql> select * from vlan where id=12; >>> +----+--------------------------------------+-------------+---------------+-----------------+-------------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------+ >>> >>>| id | uuid | vlan_id | vlan_gateway | vlan_netmask | description | >>>vlan_type | >>>| data_center_id | network_id | physical_network_id | ip6_gateway | ip6_cidr >>>| >>>| ip6_range | removed | created | >>> +----+--------------------------------------+-------------+---------------+-----------------+-------------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------+ >>> >>>| 12 | d13ea4b3-2087-4376-9d0a-f54efe2a55af | vlan://2030 | 178.XXX.XXX.1 | >>>| 255.255.255.128 | 178.XXX.XXX.2-178.XXX.XXX.119 | VirtualNetwork | 1 | 200 >>>| >>>| 200 | NULL | NULL | NULL | NULL | NULL | >>> +----+--------------------------------------+-------------+---------------+-----------------+-------------------------------+----------------+----------------+------------+---------------------+-------------+----------+-----------+---------+---------+ >>> >>> 1 row in set (0.00 sec) >>> >>> >>> Hope that helps >>> >>> Andrei >>> ----- Original Message ----- >>> >>> From: "Rohit Yadav" <rohit.ya...@shapeblue.com> >>> To: users@cloudstack.apache.org >>> Sent: Wednesday, 13 May, 2015 8:55:55 AM >>> Subject: Re: ACS 4.5.1 KVM live migration problem >>> >>> Hi Andrei, >>> >>> This looks like an issue similar to >>> https://issues.apache.org/jira/browse/CLOUDSTACK-6893 >>> Can share the row from your cloud.vlan table and value of “select >>> cache_mode >>> from volume_view where vm_id=<put the vm id here>\G;" for the VM causing >>> the >>> NPE? >>> >>>> On 12-May-2015, at 10:51 pm, Andrei Mikhailovsky <and...@arhont.com> >>>> wrote: >>>> >>>> >>>> >>>> It seems that the problem is worse than i've initially thought. In fact, I >>>> can't >>>> migrate most of my vms apart from a handful and I can't determine a >>>> correlation >>>> between the migrateable vms and once that produce exception. >>>> >>>> Thanks for any help. >>>> >>>> Andrei >>>> >>>> ----- Original Message ----- >>>> >>>> From: "Andrei Mikhailovsky" <and...@arhont.com> >>>> To: users@cloudstack.apache.org >>>> Sent: Tuesday, 12 May, 2015 8:53:16 PM >>>> Subject: ACS 4.5.1 KVM live migration problem >>>> >>>> Hi, >>>> >>>> I am having an issue migrating some of vms after recently upgrading to ACS >>>> 4.5.1. I am running Ubuntu 14.04 on both host and management servers. Here >>>> is >>>> the output from the log file on a client agent : >>>> >>>> >>>> 2015-05-12 20:42:34,154 DEBUG [kvm.resource.LibvirtComputingResource] >>>> (agentRequest-Handler-1:null) Preparing host for migrating >>>> com.cloud.agent.api.to.VirtualMachineTO@21a038ac >>>> 2015-05-12 20:42:34,157 DEBUG [kvm.resource.LibvirtConnection] >>>> (agentRequest-Handler-1:null) can't find connection: KVM, for vm: >>>> i-9-1162-VM, >>>> continue >>>> 2015-05-12 20:42:34,159 DEBUG [kvm.resource.LibvirtConnection] >>>> (agentRequest-Handler-1:null) can't find connection: LXC, for vm: >>>> i-9-1162-VM, >>>> continue >>>> 2015-05-12 20:42:34,159 DEBUG [kvm.resource.LibvirtConnection] >>>> (agentRequest-Handler-1:null) can't find which hypervisor the vm used , >>>> then >>>> use the default hypervisor >>>> 2015-05-12 20:42:34,160 DEBUG [kvm.resource.BridgeVifDriver] >>>> (agentRequest-Handler-1:null) nic=[Nic:Guest-178.248.108.205-vlan://2014] >>>> 2015-05-12 20:42:34,160 DEBUG [kvm.resource.BridgeVifDriver] >>>> (agentRequest-Handler-1:null) creating a vNet dev and bridge for guest >>>> traffic >>>> per traffic label cloudstackbr0 >>>> 2015-05-12 20:42:34,160 DEBUG [kvm.resource.BridgeVifDriver] >>>> (agentRequest-Handler-1:null) Executing: >>>> /usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 2014 >>>> -p >>>> bond0 -b brbond0-2014 -o add >>>> 2015-05-12 20:42:34,211 DEBUG [kvm.resource.BridgeVifDriver] >>>> (agentRequest-Handler-1:null) Execution is successful. >>>> 2015-05-12 20:42:34,211 DEBUG [kvm.resource.BridgeVifDriver] >>>> (agentRequest-Handler-1:null) nic=[Nic:Guest-10.1.1.66-null] >>>> 2015-05-12 20:42:34,212 DEBUG [kvm.storage.KVMStoragePoolManager] >>>> (agentRequest-Handler-1:null) Disconnecting disk >>>> 23add201-e4ee-447b-a448-ecd152aea4ad >>>> 2015-05-12 20:42:34,212 DEBUG [kvm.storage.LibvirtStorageAdaptor] >>>> (agentRequest-Handler-1:null) Trying to fetch storage pool >>>> cf771bc7-8998-354d-8e10-5564585a3c20 from libvirt >>>> 2015-05-12 20:42:34,223 DEBUG [kvm.storage.KVMStoragePoolManager] >>>> (agentRequest-Handler-1:null) Disconnecting disk >>>> 55100d25-410e-4fa3-a38b-7717f74d2afe >>>> 2015-05-12 20:42:34,223 DEBUG [kvm.storage.LibvirtStorageAdaptor] >>>> (agentRequest-Handler-1:null) Trying to fetch storage pool >>>> cf771bc7-8998-354d-8e10-5564585a3c20 from libvirt >>>> 2015-05-12 20:42:34,232 DEBUG [kvm.storage.KVMStoragePoolManager] >>>> (agentRequest-Handler-1:null) Disconnecting disk >>>> 2db59d16-d17f-49a1-b913-7fbe4025a549 >>>> 2015-05-12 20:42:34,233 DEBUG [kvm.storage.LibvirtStorageAdaptor] >>>> (agentRequest-Handler-1:null) Trying to fetch storage pool >>>> cf771bc7-8998-354d-8e10-5564585a3c20 from libvirt >>>> 2015-05-12 20:42:34,243 DEBUG [kvm.storage.KVMStoragePoolManager] >>>> (agentRequest-Handler-1:null) Disconnecting disk >>>> 17afbf31-ac89-46f7-a2c8-f8aed796e4c6 >>>> 2015-05-12 20:42:34,243 DEBUG [kvm.storage.LibvirtStorageAdaptor] >>>> (agentRequest-Handler-1:null) Trying to fetch storage pool >>>> d8d5ec36-3cb0-39af-8fc6-084a4abd5d28 from libvirt >>>> 2015-05-12 20:42:34,254 WARN [cloud.agent.Agent] >>>> (agentRequest-Handler-1:null) >>>> Caught: >>>> java.lang.NullPointerException >>>> at >>>> com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:172) >>>> >>>> at >>>> com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:226) >>>> at >>>> com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:105) >>>> >>>> at >>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3230) >>>> >>>> at >>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1307) >>>> >>>> at com.cloud.agent.Agent.processRequest(Agent.java:503) >>>> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) >>>> at com.cloud.utils.nio.Task.run(Task.java:84) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> >>>> at java.lang.Thread.run(Thread.java:745) >>>> 2015-05-12 20:42:34,256 DEBUG [cloud.agent.Agent] >>>> (agentRequest-Handler-1:null) >>>> Seq 7-7525233502359390941: { Ans: , MgmtId: 115129173025118, via: 7, Ver: >>>> v1, >>>> Flags: 110, >>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat >>>> >>>> com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:172)\n\tat >>>> >>>> com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:226)\n\tat >>>> >>>> com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:105)\n\tat >>>> >>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3230)\n\tat >>>> >>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1307)\n\tat >>>> >>>> com.cloud.agent.Agent.processRequest(Agent.java:503)\n\tat >>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)\n\tat >>>> com.cloud.utils.nio.Task.run(Task.java:84)\n\tat >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat >>>> >>>> java.lang.Thread.run(Thread.java:745)\n","wait":0}}] } >>>> >>>> >>>> >>>> Any idea how to get this fixed? Not sure why all of a sudden the migration >>>> stopped working for a handful of vms. I can successfully migrate some vms, >>>> but >>>> not others. >>>> >>>> Thanks >>>> >>>> Andrei >>>> >>>> >>> >>> Regards, >>> Rohit Yadav >>> Software Architect, ShapeBlue >>> M. +91 88 262 30892 | rohit.ya...@shapeblue.com >>> Blog: bhaisaab.org | Twitter: @_bhaisaab >>> >>> >>> >>> Find out more about ShapeBlue and our range of CloudStack related services >>> >>> IaaS Cloud Design & >>> Build<http://shapeblue.com/iaas-cloud-design-and-build//> >>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> >>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> >>> CloudStack Software >>> Engineering<http://shapeblue.com/cloudstack-software-engineering/> >>> CloudStack Infrastructure >>> Support<http://shapeblue.com/cloudstack-infrastructure-support/> >>> CloudStack Bootcamp Training >>> Courses<http://shapeblue.com/cloudstack-training/> >>> >>> This email and any attachments to it may be confidential and are intended >>> solely >>> for the use of the individual to whom it is addressed. Any views or >>> opinions >>> expressed are solely those of the author and do not necessarily represent >>> those >>> of Shape Blue Ltd or related companies. If you are not the intended >>> recipient >>> of this email, you must neither take any action based upon its contents, >>> nor >>> copy or show it to anyone. Please contact the sender if you believe you >>> have >>> received this email in error. Shape Blue Ltd is a company incorporated in >>> England & Wales. ShapeBlue Services India LLP is a company incorporated in >>> India and is operated under license from Shape Blue Ltd. Shape Blue Brasil >>> Consultoria Ltda is a company incorporated in Brasil and is operated under >>> license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered >>> by >>> The Republic of South Africa and is traded under license from Shape Blue >>> Ltd. > > > ShapeBlue is a registered trademark.
