Hi, select id, name, traffic_type, broadcast_domain_type, cidr, gateway, mode, state, removed from networks where id = 205; +-----+---------+--------------+-----------------------+----------------+-------------+------+-----------+---------+ | id | name | traffic_type | broadcast_domain_type | cidr | gateway | mode | state | removed | +-----+---------+--------------+-----------------------+----------------+-------------+------+-----------+---------+ | 205 | default | Guest | Vlan | 10.124.99.0/24 | 10.124.99.1 | Dhcp | Allocated | NULL | +-----+---------+--------------+-----------------------+----------------+-------------+------+-----------+---------+
Cheers Kambiz Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: > > No, it doesn’t seem right to me having 2 nics with device id 0. But looks > like they’ve got programmed to correct devices on the backend per your > prev email? > > iface eth0 inet static > address 169.254.1.59 > netmask 255.255.0.0 > > iface eth1 inet static > address 10.193.17.1 > Netmask > > iface eth2 inet static > address 10.193.17.190 > netmask 255.255.255.0 > > > > I can see that only one parameter is missing from the start command, the > second nic (network id=205) doesn’t have the gateway. > From the command/DB, I see that the gateway is missing in the nics table > for the network 205? Can you check gateway information in the networks > table for the id=205 > > > > > On 3/25/14, 1:01 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote: > >>Hi, >> >>select >>id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name >>from nics where instance_id=19; >> >>+----+---------------+---------------+-------------+-----------+---------+ >>------------+--------------------------+ >>| id | ip4_address | netmask | gateway | state | removed >>| network_id | reserver_name | >>+----+---------------+---------------+-------------+-----------+---------+ >>------------+--------------------------+ >>| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL >>| 205 | ExternalGuestNetworkGuru | >>| 30 | NULL | NULL | NULL | Allocated | NULL >>| 202 | ControlNetworkGuru | >>| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL >>| 200 | PublicNetworkGuru | >>+----+---------------+---------------+-------------+-----------+---------+ >>------------+--------------------------+ >> >>and this is the nics element from the StartCmd. Is it normal to have >>two nics with deviceId 0? >> >>"nics":[ >> {"deviceId":2, >> "networkRateMbps":200, >> "defaultNic":true, >> "uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c", >> "ip":"10.193.17.190", >> "netmask":"255.255.255.0", >> "gateway":"10.193.17.1", >> "mac":"06:7e:fe:00:00:bf", >> "dns1":"10.193.17.1", >> "broadcastType":"Vlan", >> "type":"Public", >> "broadcastUri":"vlan://untagged", >> "isolationUri":"vlan://untagged", >> "isSecurityGroupEnabled":false, >> "name":"cloudbr0"}, >> {"deviceId":0, >> "networkRateMbps":200, >> "defaultNic":false, >> "uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8", >> "ip":"10.124.99.1", >> "netmask":"255.255.255.0", >> "mac":"02:00:2a:43:00:0d", >> "dns1":"10.193.17.1", >> "broadcastType":"Vlan", >> "type":"Guest", >> "broadcastUri":"vlan://3949", >> "isolationUri":"vlan://3949", >> "isSecurityGroupEnabled":false, >> "name":"cloudbr1"}, >> {"deviceId":0, >> "networkRateMbps":-1, >> "defaultNic":false, >> "uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d", >> "ip":"169.254.1.59", >> "netmask":"255.255.0.0", >> "gateway":"169.254.0.1", >> "mac":"0e:00:a9:fe:01:3b", >> "broadcastType":"LinkLocal", >> "type":"Control", >> "isSecurityGroupEnabled":false} >>] >> >>Thanks >> >> >>Kambiz >> >>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>> >>> Kambiz, the debug statements below are for the case when eth1 is a >>>control >>> interface as it was in your old command. I’ve looked at the new command, >>> eth1 is not control, its either public or guest >>> >>> eth0: - control >>> >>> iface eth0 inet static >>> address 169.254.1.59 >>> netmask 255.255.0.0 >>> >>> eth1: >>> >>> iface eth1 inet static >>> address 10.193.17.1 >>> Netmask >>> >>> So you need to execute the mysql statements for the traffic type of VR >>>nic >>> eth1 >>> >>> -Alena. >>> >>> >>> >>> >>> On 3/25/14, 9:57 AM, "Alena Prokharchyk" <alena.prokharc...@citrix.com> >>> wrote: >>> >>>>Kambiz, can you please check the following: >>>> >>>> >>>>1) Check if the gateway is set on control network: >>>> >>>>mysql> select gateway, cidr from networks where traffic_type=‘Control’; >>>> >>>>2) For router control nic, check if network/gateway are set. >>>> >>>>Select gateway,netmask from nics where instance_id=<routerid> and >>>>network_id=<control network id> >>>> >>>>-Alena. >>>> >>>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>>> >>>>>Hi, >>>>> >>>>>I looked up the startup command of the old router instance which worked >>>>>correctly: >>>>> >>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l >>>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p >>>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0% >>>>>ga >>>>>t >>>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cl >>>>>ou >>>>>d >>>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0. >>>>>0% >>>>>t >>>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1 >>>>> >>>>>The new command (cf. below) doesn't have the parameters eth1ip and >>>>>eth1mask. >>>>> >>>>>Thanks >>>>> >>>>> >>>>>Kambiz >>>>> >>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>> >>>>>> I don’t think its relevant as the piece we’ve fixed, just eliminated >>>>>> static nat rule programming for non-existing vm. Missing netmask on >>>>>>eth1 >>>>>> doesn’t seem related to the problem (although we have to figure out >>>>>>why >>>>>> its missing), as the connection that fails, happening to link local >>>>>>169.x >>>>>> eth0 interface. >>>>>> >>>>>> Edison, can you please tell us how to debug link local connection >>>>>>failure, >>>>>> on KVM agent? >>>>>> >>>>>> Thank you, >>>>>> Alena. >>>>>> >>>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>>>>> >>>>>>>Hi, >>>>>>> >>>>>>>thank you, the NullPointerException doesn't occur any more, but there >>>>>>>still seems to be a problem during startup of the router. >>>>>>> >>>>>>>When I start the virtual router, it comes up, but in agent.log, there >>>>>>>are lots of 'Could not connect to 169.254.1.x' messages. >>>>>>> >>>>>>>Then I logged into the virtual router to find out that the netmask of >>>>>>>eth1 is missing in the interfaces file: >>>>>>> >>>>>>>root@host:~# virsh console r-19-VM >>>>>>>Connected to domain r-19-VM >>>>>>>Escape character is ^] >>>>>>> >>>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0 >>>>>>> >>>>>>>r-19-VM login: root >>>>>>>... >>>>>>>root@r-19-VM:~# cat /etc/network/interfaces >>>>>>>auto lo eth0 eth1 eth2 >>>>>>>iface lo inet loopback >>>>>>> >>>>>>>iface eth0 inet static >>>>>>> address 169.254.1.59 >>>>>>> netmask 255.255.0.0 >>>>>>>iface eth1 inet static >>>>>>> address 10.193.17.1 >>>>>>> netmask >>>>>>>iface eth2 inet static >>>>>>> address 10.193.17.190 >>>>>>> netmask 255.255.255.0 >>>>>>> >>>>>>>I don't know if it is relevant, but this is the line from agent.log >>>>>>>where the parameters are visible: >>>>>>> >>>>>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource] >>>>>>>(agentRequest-Handler-2:null) Executing: >>>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh >>>>>>>-l >>>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p >>>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255 >>>>>>>.0 >>>>>>>% >>>>>>>ga >>>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs >>>>>>>6c >>>>>>>l >>>>>>>ou >>>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255 >>>>>>>.0 >>>>>>>. >>>>>>>0% >>>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1 >>>>>>> >>>>>>> >>>>>>>Any hint is appreciated. >>>>>>> >>>>>>>Thanks >>>>>>> >>>>>>> >>>>>>>Kambiz >>>>>>> >>>>>>> >>>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>>>> >>>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If >>>>>>>>vm >>>>>>>> id=15 is expunged, we have to clear out the reference to it from >>>>>>>> user_ip_address table. Here is the flow: >>>>>>>> >>>>>>>> 1) Save the db dump. >>>>>>>> 2) Run the query to cleanup the reference: >>>>>>>> >>>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where >>>>>>>> id=<problematic public ip address id> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Let me know how it works. >>>>>>>> >>>>>>>> -Alena. >>>>>>>> >>>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <dar...@m-creations.com> >>>>>>>>wrote: >>>>>>>> >>>>>>>>>Hi, >>>>>>>>> >>>>>>>>>I hope I have understood what you wrote and created the following >>>>>>>>>query >>>>>>>>>correctly: >>>>>>>>> >>>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address, >>>>>>>>> n.state as nic_state, n.removed as nic_removed, >>>>>>>>> vm.state as vm_state, vm.removed as vm_removed >>>>>>>>>from user_ip_address uip >>>>>>>>> join nics n on uip.vm_id = n.instance_id >>>>>>>>> join vm_instance vm on uip.vm_id = vm.id >>>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where >>>>>>>>>fr.network_id=205); >>>>>>>>> >>>>>>>>> >>>>>>>>>+-------+------------+-------------------+--------------+---------- >>>>>>>>>-- >>>>>>>>>- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----+-----------+------------+ >>>>>>>>>| vm_id | network_id | public_ip_address | nic_state | >>>>>>>>>nic_removed >>>>>>>>> | vm_state | vm_removed | >>>>>>>>>+-------+------------+-------------------+--------------+---------- >>>>>>>>>-- >>>>>>>>>- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----+-----------+------------+ >>>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL >>>>>>>>> | Stopped | NULL | >>>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL >>>>>>>>> | Stopped | NULL | >>>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL >>>>>>>>> | Stopped | NULL | >>>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL >>>>>>>>> | Stopped | NULL | >>>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL >>>>>>>>> | Stopped | NULL | >>>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating | >>>>>>>>>2014-03-18 >>>>>>>>>23:00:53 | Expunging | NULL | >>>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL >>>>>>>>> | Stopped | NULL | >>>>>>>>>+-------+------------+-------------------+--------------+---------- >>>>>>>>>-- >>>>>>>>>- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----+-----------+------------+ >>>>>>>>> >>>>>>>>>Is VM id 15 what you are looking for? >>>>>>>>> >>>>>>>>>Thank you >>>>>>>>> >>>>>>>>> >>>>>>>>>Kambiz >>>>>>>>> >>>>>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>>>>>> >>>>>>>>>> Kambiz, can you please try one more thing. >>>>>>>>>> >>>>>>>>>> 1) Locate all the firewall rules for your guest network (205, >>>>>>>>>>right?) >>>>>>>>>> >>>>>>>>>> Select id, ip_address_id from firewall_rules where >>>>>>>>>>network_id=205; >>>>>>>>>> >>>>>>>>>> 2) Now get all static nat enabled ip addresses for those rules: >>>>>>>>>> >>>>>>>>>> Select vm_id, network_id from user_ip_address where id in (Select >>>>>>>>>>id, >>>>>>>>>> ip_address_id from firewall_rules where network_id=205); >>>>>>>>>> >>>>>>>>>> For each vmId/networkId combo, check if there is non-removed nic >>>>>>>>>>and >>>>>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm >>>>>>>>>>reference >>>>>>>>>> referring to vm that is removed already. If you find any, let me >>>>>>>>>>know >>>>>>>>>>and >>>>>>>>>> I will tell you how to clean it up >>>>>>>>>> >>>>>>>>>> -Alena. >>>>>>>>>> >>>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <dar...@m-creations.com> >>>>>>>>>>wrote: >>>>>>>>>> >>>>>>>>>>>Hi Alena, >>>>>>>>>>> >>>>>>>>>>>thank you for your help. >>>>>>>>>>> >>>>>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I >>>>>>>>>>>removed >>>>>>>>>>>the row though to see what happens: a new virtual router was >>>>>>>>>>>created >>>>>>>>>>>which also couldn't be started due to the same NPE. I reverted >>>>>>>>>>>the >>>>>>>>>>>change by restoring from the dump. >>>>>>>>>>> >>>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the >>>>>>>>>>>router >>>>>>>>>>>which >>>>>>>>>>>was used by my instances. I deleted the router using the UI after >>>>>>>>>>>the >>>>>>>>>>>first >>>>>>>>>>>occurrence of the NPE, because a post with a similar problem >>>>>>>>>>>suggested >>>>>>>>>>>that the deleted router would be recreated again (and this >>>>>>>>>>>procedure >>>>>>>>>>>solved the problem). >>>>>>>>>>> >>>>>>>>>>>Below I have attached the state of the two tables. >>>>>>>>>>> >>>>>>>>>>>Anything else I can try? >>>>>>>>>>> >>>>>>>>>>>Thank you >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>Kambiz >>>>>>>>>>> >>>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, >>>>>>>>>>>n.gateway, >>>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, >>>>>>>>>>>i.name, >>>>>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id = >>>>>>>>>>>i.id >>>>>>>>>>>where i.type = 'DomainRouter'; >>>>>>>>>>>+----+---------------------+---------------+---------------+----- >>>>>>>>>>>-- >>>>>>>>>>>- >>>>>>>>>>>-- >>>>>>>>>>>-- >>>>>>>>>>>-+ >>>>>>>>>>>---------+--------------------------+------------+-------------+- >>>>>>>>>>>-- >>>>>>>>>>>- >>>>>>>>>>>-- >>>>>>>>>>>-- >>>>>>>>>>>-+ >>>>>>>>>>>-----------+--------------+ >>>>>>>>>>>| id | removed | ip4_address | netmask | >>>>>>>>>>>gateway >>>>>>>>>>>| ip_type | reserver_name | network_id | instance_id | >>>>>>>>>>>name >>>>>>>>>>>| state | type | >>>>>>>>>>>+----+---------------------+---------------+---------------+----- >>>>>>>>>>>-- >>>>>>>>>>>- >>>>>>>>>>>-- >>>>>>>>>>>-- >>>>>>>>>>>-+ >>>>>>>>>>>---------+--------------------------+------------+-------------+- >>>>>>>>>>>-- >>>>>>>>>>>- >>>>>>>>>>>-- >>>>>>>>>>>-- >>>>>>>>>>>-+ >>>>>>>>>>>-----------+--------------+ >>>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 | >>>>>>>>>>>r-4-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL >>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 4 | >>>>>>>>>>>r-4-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | >>>>>>>>>>>10.193.17.1 >>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 4 | >>>>>>>>>>>r-4-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 | >>>>>>>>>>>r-7-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL >>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 7 | >>>>>>>>>>>r-7-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | >>>>>>>>>>>10.193.17.1 >>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 7 | >>>>>>>>>>>r-7-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 | >>>>>>>>>>>r-18-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL >>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 18 | >>>>>>>>>>>r-18-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | >>>>>>>>>>>10.193.17.1 >>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 18 | >>>>>>>>>>>r-18-VM >>>>>>>>>>>| Expunging | DomainRouter | >>>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 | >>>>>>>>>>>r-19-VM >>>>>>>>>>>| Stopped | DomainRouter | >>>>>>>>>>>| 30 | NULL | NULL | NULL | NULL >>>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 19 | >>>>>>>>>>>r-19-VM >>>>>>>>>>>| Stopped | DomainRouter | >>>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | >>>>>>>>>>>10.193.17.1 >>>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 19 | >>>>>>>>>>>r-19-VM >>>>>>>>>>>| Stopped | DomainRouter | >>>>>>>>>>>+----+---------------------+---------------+---------------+----- >>>>>>>>>>>-- >>>>>>>>>>>- >>>>>>>>>>>-- >>>>>>>>>>>-- >>>>>>>>>>>-+ >>>>>>>>>>>---------+--------------------------+------------+-------------+- >>>>>>>>>>>-- >>>>>>>>>>>- >>>>>>>>>>>-- >>>>>>>>>>>-- >>>>>>>>>>>-+ >>>>>>>>>>>-----------+--------------+ >>>>>>>>>>> >>>>>>>>>>>mysql> select * from router_network_ref; >>>>>>>>>>>+----+-----------+------------+------------+ >>>>>>>>>>>| id | router_id | network_id | guest_type | >>>>>>>>>>>+----+-----------+------------+------------+ >>>>>>>>>>>| 1 | 4 | 204 | Isolated | >>>>>>>>>>>| 2 | 7 | 205 | Isolated | >>>>>>>>>>>| 3 | 18 | 205 | Isolated | >>>>>>>>>>>| 4 | 19 | 205 | Isolated | >>>>>>>>>>>+----+-----------+------------+------------+ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> The error happens not because Ip is null, but because the nic >>>>>>>>>>>>in >>>>>>>>>>>>a >>>>>>>>>>>>certain >>>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic >>>>>>>>>>>> plug/unplug for Guest networks process. >>>>>>>>>>>> >>>>>>>>>>>> Kambiz, please do the following to fix it: >>>>>>>>>>>> >>>>>>>>>>>> 1) Stop the MS >>>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert >>>>>>>>>>>>back. >>>>>>>>>>>> 3) Run the query: >>>>>>>>>>>> >>>>>>>>>>>> select * from router_network_ref where router_id=<id of your >>>>>>>>>>>>VR) >>>>>>>>>>>>and >>>>>>>>>>>> network_id not in (select network_id from nics where >>>>>>>>>>>>instance_id=<ID >>>>>>>>>>>>of >>>>>>>>>>>> your VR> and removed is null); >>>>>>>>>>>> >>>>>>>>>>>> It will give you the list of networks refs that somehow weren¹t >>>>>>>>>>>>cleaned >>>>>>>>>>>> during the nic detach. Remove the entry returned from >>>>>>>>>>>>router_network_ref >>>>>>>>>>>> table. >>>>>>>>>>>> >>>>>>>>>>>> Let me know how it works. >>>>>>>>>>>> >>>>>>>>>>>> -Alena. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <dar...@m-creations.com> >>>>>>>>>>>>wrote: >>>>>>>>>>>> >>>>>>>>>>>>>Hello, >>>>>>>>>>>>> >>>>>>>>>>>>>as this is my first post to the list, I would like to thank all >>>>>>>>>>>>>contributors for Cloudstack which I use since last fall without >>>>>>>>>>>>>any >>>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking. >>>>>>>>>>>>> >>>>>>>>>>>>>After a restart of the management server (stopping and starting >>>>>>>>>>>>>the >>>>>>>>>>>>>java >>>>>>>>>>>>>process), the virtual domain router doesn't start and >>>>>>>>>>>>>management-server.log shows a NullPointerException in >>>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below). >>>>>>>>>>>>> >>>>>>>>>>>>>By putting the server in debug mode and remote debugging, I >>>>>>>>>>>>>found >>>>>>>>>>>>>out >>>>>>>>>>>>>that the reason is a row in the table nics which has NULL in ip >>>>>>>>>>>>>(cf. >>>>>>>>>>>>>row >>>>>>>>>>>>>with id 30 in the result of the select statement below). >>>>>>>>>>>>> >>>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers or >>>>>>>>>>>>>suggestions >>>>>>>>>>>>>are appreciated as the system is currently unusable. >>>>>>>>>>>>> >>>>>>>>>>>>>Thank you for your help >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>Kambiz >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>management-server.log: >>>>>>>>>>>>> >>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to >>>>>>>>>>>>>prepare >>>>>>>>>>>>>for >>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>>>>2014-03-18 10:03:27,151 WARN >>>>>>>>>>>>>[network.element.VpcVirtualRouterElement] >>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not >>>>>>>>>>>>>associated >>>>>>>>>>>>>with >>>>>>>>>>>>>any VPC >>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG >>>>>>>>>>>>>[network.element.NiciraNvpElement] >>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can >>>>>>>>>>>>>handle >>>>>>>>>>>>>service >>>>>>>>>>>>>Connectivity on network net1 >>>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported >>>>>>>>>>>>>in >>>>>>>>>>>>>the >>>>>>>>>>>>>network id=205 >>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 >>>>>>>>>>>>>as a >>>>>>>>>>>>>part >>>>>>>>>>>>>of >>>>>>>>>>>>>network implement >>>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented >>>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 >>>>>>>>>>>>>as a >>>>>>>>>>>>>part >>>>>>>>>>>>>of >>>>>>>>>>>>>network implement >>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to >>>>>>>>>>>>>prepare >>>>>>>>>>>>>for >>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>>>>2014-03-18 10:03:27,187 WARN >>>>>>>>>>>>>[network.element.VpcVirtualRouterElement] >>>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not >>>>>>>>>>>>>associated >>>>>>>>>>>>>with any VPC >>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG >>>>>>>>>>>>>[cloud.network.NetworkManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG >>>>>>>>>>>>>[network.element.NiciraNvpElement] >>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can >>>>>>>>>>>>>handle >>>>>>>>>>>>>service >>>>>>>>>>>>>Connectivity on network null >>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG >>>>>>>>>>>>>[cloud.storage.StorageManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 >>>>>>>>>>>>>volumes >>>>>>>>>>>>>for >>>>>>>>>>>>>VM[DomainRouter|r-19-VM] >>>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG >>>>>>>>>>>>>[cloud.storage.StorageManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume: >>>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, >>>>>>>>>>>>>adding >>>>>>>>>>>>>disk to VM >>>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG >>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for >>>>>>>>>>>>>VM[DomainRouter|r-19-VM]: >>>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190 >>>>>>>>>>>>>eth2mask=255.255.255.0 >>>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0 >>>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 >>>>>>>>>>>>>eth0ip=169.254.3.99 >>>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true >>>>>>>>>>>>>dns1=10.193.17.1 >>>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG >>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of >>>>>>>>>>>>>domR >>>>>>>>>>>>>VM[DomainRouter|r-19-VM] start. >>>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG >>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, >>>>>>>>>>>>>load >>>>>>>>>>>>>balancing rules as a part of Virtual router start >>>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG >>>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to >>>>>>>>>>>>>apply >>>>>>>>>>>>>as >>>>>>>>>>>>>a >>>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start. >>>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR >>>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl] >>>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance >>>>>>>>>>>>>VM[DomainRouter|r-19-VM] >>>>>>>>>>>>>java.lang.NullPointerException >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelI >>>>>>>>>>>>>mp >>>>>>>>>>>>>l >>>>>>>>>>>>>.j >>>>>>>>>>>>>av >>>>>>>>>>>>>a: >>>>>>>>>>>>>76 >>>>>>>>>>>>>3) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin >>>>>>>>>>>>>al >>>>>>>>>>>>>i >>>>>>>>>>>>>ze >>>>>>>>>>>>>Ne >>>>>>>>>>>>>tw >>>>>>>>>>>>>or >>>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl. >>>>>>>>>>>>>fi >>>>>>>>>>>>>n >>>>>>>>>>>>>al >>>>>>>>>>>>>iz >>>>>>>>>>>>>eN >>>>>>>>>>>>>et >>>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java: >>>>>>>>>>>>>92 >>>>>>>>>>>>>8 >>>>>>>>>>>>>) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin >>>>>>>>>>>>>al >>>>>>>>>>>>>i >>>>>>>>>>>>>ze >>>>>>>>>>>>>Co >>>>>>>>>>>>>mm >>>>>>>>>>>>>an >>>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl. >>>>>>>>>>>>>fi >>>>>>>>>>>>>n >>>>>>>>>>>>>al >>>>>>>>>>>>>iz >>>>>>>>>>>>>eC >>>>>>>>>>>>>om >>>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.fin >>>>>>>>>>>>>al >>>>>>>>>>>>>i >>>>>>>>>>>>>ze >>>>>>>>>>>>>De >>>>>>>>>>>>>pl >>>>>>>>>>>>>oy >>>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMach >>>>>>>>>>>>>in >>>>>>>>>>>>>e >>>>>>>>>>>>>Ma >>>>>>>>>>>>>na >>>>>>>>>>>>>ge >>>>>>>>>>>>>rI >>>>>>>>>>>>>mpl.java:763) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineMana >>>>>>>>>>>>>ge >>>>>>>>>>>>>r >>>>>>>>>>>>>Im >>>>>>>>>>>>>pl >>>>>>>>>>>>>.j >>>>>>>>>>>>>av >>>>>>>>>>>>>a:471) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta >>>>>>>>>>>>>rt >>>>>>>>>>>>>( >>>>>>>>>>>>>Vi >>>>>>>>>>>>>rt >>>>>>>>>>>>>ua >>>>>>>>>>>>>lN >>>>>>>>>>>>>etworkApplianceManagerImpl.java:2616) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta >>>>>>>>>>>>>rt >>>>>>>>>>>>>V >>>>>>>>>>>>>ir >>>>>>>>>>>>>tu >>>>>>>>>>>>>al >>>>>>>>>>>>>Ro >>>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta >>>>>>>>>>>>>rt >>>>>>>>>>>>>R >>>>>>>>>>>>>ou >>>>>>>>>>>>>te >>>>>>>>>>>>>rs >>>>>>>>>>>>>(V >>>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.dep >>>>>>>>>>>>>lo >>>>>>>>>>>>>y >>>>>>>>>>>>>Vi >>>>>>>>>>>>>rt >>>>>>>>>>>>>ua >>>>>>>>>>>>>lR >>>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:190 >>>>>>>>>>>>>2) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(Virtua >>>>>>>>>>>>>lR >>>>>>>>>>>>>o >>>>>>>>>>>>>ut >>>>>>>>>>>>>er >>>>>>>>>>>>>El >>>>>>>>>>>>>em >>>>>>>>>>>>>ent.java:175) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAn >>>>>>>>>>>>>dR >>>>>>>>>>>>>e >>>>>>>>>>>>>so >>>>>>>>>>>>>ur >>>>>>>>>>>>>ce >>>>>>>>>>>>>s( >>>>>>>>>>>>>NetworkManagerImpl.java:1518) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkMa >>>>>>>>>>>>>na >>>>>>>>>>>>>g >>>>>>>>>>>>>er >>>>>>>>>>>>>Im >>>>>>>>>>>>>pl >>>>>>>>>>>>>.j >>>>>>>>>>>>>ava:1434) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$I >>>>>>>>>>>>>nt >>>>>>>>>>>>>e >>>>>>>>>>>>>rc >>>>>>>>>>>>>ep >>>>>>>>>>>>>to >>>>>>>>>>>>>rD >>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:12 >>>>>>>>>>>>>5) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManage >>>>>>>>>>>>>rI >>>>>>>>>>>>>m >>>>>>>>>>>>>pl >>>>>>>>>>>>>.j >>>>>>>>>>>>>av >>>>>>>>>>>>>a: >>>>>>>>>>>>>2435) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta >>>>>>>>>>>>>rt >>>>>>>>>>>>>R >>>>>>>>>>>>>ou >>>>>>>>>>>>>te >>>>>>>>>>>>>r( >>>>>>>>>>>>>Vi >>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sta >>>>>>>>>>>>>rt >>>>>>>>>>>>>R >>>>>>>>>>>>>ou >>>>>>>>>>>>>te >>>>>>>>>>>>>r( >>>>>>>>>>>>>Vi >>>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824) >>>>>>>>>>>>> at >>>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$I >>>>>>>>>>>>>nt >>>>>>>>>>>>>e >>>>>>>>>>>>>rc >>>>>>>>>>>>>ep >>>>>>>>>>>>>to >>>>>>>>>>>>>rD >>>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:12 >>>>>>>>>>>>>5) >>>>>>>>>>>>> at >>>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.e >>>>>>>>>>>>>xe >>>>>>>>>>>>>c >>>>>>>>>>>>>ut >>>>>>>>>>>>>e( >>>>>>>>>>>>>St >>>>>>>>>>>>>ar >>>>>>>>>>>>>tRouterCmd.java:103) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>table nics: >>>>>>>>>>>>> >>>>>>>>>>>>>mysql> select * from nics where reserver_name = >>>>>>>>>>>>>'ControlNetworkGuru'; >>>>>>>>>>>>>+----+--------------------------------------+-------------+---- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----+---------------+-------------+-------------+---------+---- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>+------------+--------+--------------+----------+-------------- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-+ >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----------------------------------+-----------+---------------- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>+- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----------+-------------+-------------+--------------------+--- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>>>>>>| id | uuid | instance_id | >>>>>>>>>>>>>mac_address >>>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type | >>>>>>>>>>>>>broadcast_uri >>>>>>>>>>>>>| network_id | mode | state | strategy | reserver_name >>>>>>>>>>>>> | >>>>>>>>>>>>>reservation_id | device_id | update_time >>>>>>>>>>>>> | >>>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type >>>>>>>>>>>>>| >>>>>>>>>>>>>created >>>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr | >>>>>>>>>>>>>+----+--------------------------------------+-------------+---- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----+---------------+-------------+-------------+---------+---- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>+------------+--------+--------------+----------+-------------- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-+ >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----------------------------------+-----------+---------------- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>+- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----------+-------------+-------------+--------------------+--- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 | >>>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | >>>>>>>>>>>>>Ip4 >>>>>>>>>>>>>| >>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start | >>>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | >>>>>>>>>>>>>0 >>>>>>>>>>>>>| >>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | >>>>>>>>>>>>>0 | >>>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL >>>>>>>>>>>>>| >>>>>>>>>>>>>NULL >>>>>>>>>>>>> | NULL | >>>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 | >>>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | >>>>>>>>>>>>>Ip4 >>>>>>>>>>>>>| >>>>>>>>>>>>>NULL | 202 | Static | Reserved | Start | >>>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | >>>>>>>>>>>>>0 >>>>>>>>>>>>>| >>>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | >>>>>>>>>>>>>0 | >>>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL >>>>>>>>>>>>>| >>>>>>>>>>>>>NULL >>>>>>>>>>>>> | NULL | >>>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | >>>>>>>>>>>>>NULL >>>>>>>>>>>>> | NULL | NULL | NULL | NULL | >>>>>>>>>>>>>NULL >>>>>>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>>>>>ControlNetworkGuru | >>>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 >>>>>>>>>>>>>12:27:58 | >>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter >>>>>>>>>>>>>| >>>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL >>>>>>>>>>>>> | >>>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | >>>>>>>>>>>>>NULL >>>>>>>>>>>>> | NULL | NULL | NULL | NULL | >>>>>>>>>>>>>NULL >>>>>>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>>>>>ControlNetworkGuru | >>>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 >>>>>>>>>>>>>12:27:52 | >>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter >>>>>>>>>>>>>| >>>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL >>>>>>>>>>>>> | >>>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | >>>>>>>>>>>>>NULL >>>>>>>>>>>>> | NULL | NULL | NULL | NULL | >>>>>>>>>>>>>NULL >>>>>>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>>>>>ControlNetworkGuru | >>>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 >>>>>>>>>>>>>09:11:16 | >>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter >>>>>>>>>>>>>| >>>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL >>>>>>>>>>>>> | >>>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | >>>>>>>>>>>>>NULL >>>>>>>>>>>>> | NULL | NULL | NULL | NULL | >>>>>>>>>>>>>NULL >>>>>>>>>>>>>| 202 | Static | Allocated | Start | >>>>>>>>>>>>>ControlNetworkGuru | >>>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 >>>>>>>>>>>>>09:11:44 | >>>>>>>>>>>>>NULL | NULL | 0 | DomainRouter >>>>>>>>>>>>>| >>>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL >>>>>>>>>>>>> | >>>>>>>>>>>>>+----+--------------------------------------+-------------+---- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----+---------------+-------------+-------------+---------+---- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>+------------+--------+--------------+----------+-------------- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-+ >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----------------------------------+-----------+---------------- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>+- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>----------+-------------+-------------+--------------------+--- >>>>>>>>>>>>>-- >>>>>>>>>>>>>- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-- >>>>>>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>