Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm id=15 is expunged, we have to clear out the reference to it from user_ip_address table. Here is the flow:
1) Save the db dump. 2) Run the query to cleanup the reference: Update user_ip_address set one_to_one_nat=0, instance_id=null where id=<problematic public ip address id> Let me know how it works. -Alena. On 3/24/14, 10:55 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >Hi, > >I hope I have understood what you wrote and created the following query >correctly: > >select uip.vm_id, uip.network_id, uip.public_ip_address, > n.state as nic_state, n.removed as nic_removed, > vm.state as vm_state, vm.removed as vm_removed >from user_ip_address uip > join nics n on uip.vm_id = n.instance_id > join vm_instance vm on uip.vm_id = vm.id >where uip.id in (Select ip_address_id from firewall_rules fr where >fr.network_id=205); > > >+-------+------------+-------------------+--------------+----------------- >----+-----------+------------+ >| vm_id | network_id | public_ip_address | nic_state | nic_removed > | vm_state | vm_removed | >+-------+------------+-------------------+--------------+----------------- >----+-----------+------------+ >| 6 | 205 | 10.193.17.169 | Allocated | NULL > | Stopped | NULL | >| 10 | 205 | 10.193.17.136 | Allocated | NULL > | Stopped | NULL | >| 12 | 205 | 10.193.17.140 | Allocated | NULL > | Stopped | NULL | >| 13 | 205 | 10.193.17.141 | Allocated | NULL > | Stopped | NULL | >| 14 | 205 | 10.193.17.142 | Allocated | NULL > | Stopped | NULL | >| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18 >23:00:53 | Expunging | NULL | >| 16 | 205 | 10.193.17.103 | Allocated | NULL > | Stopped | NULL | >+-------+------------+-------------------+--------------+----------------- >----+-----------+------------+ > >Is VM id 15 what you are looking for? > >Thank you > > >Kambiz > >Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >> >> Kambiz, can you please try one more thing. >> >> 1) Locate all the firewall rules for your guest network (205, right?) >> >> Select id, ip_address_id from firewall_rules where network_id=205; >> >> 2) Now get all static nat enabled ip addresses for those rules: >> >> Select vm_id, network_id from user_ip_address where id in (Select id, >> ip_address_id from firewall_rules where network_id=205); >> >> For each vmId/networkId combo, check if there is non-removed nic and >> non-expunged vm. There might be some incorrect static nat ip/vm >>reference >> referring to vm that is removed already. If you find any, let me know >>and >> I will tell you how to clean it up >> >> -Alena. >> >> On 3/22/14, 5:41 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >> >>>Hi Alena, >>> >>>thank you for your help. >>> >>>The query returns no rows, i.e. nics.removed was not null, but I removed >>>the row though to see what happens: a new virtual router was created >>>which also couldn't be started due to the same NPE. I reverted the >>>change by restoring from the dump. >>> >>>I have to mention that prior to the restart, r-7-VM was the router which >>>was used by my instances. I deleted the router using the UI after the >>>first >>>occurrence of the NPE, because a post with a similar problem suggested >>>that the deleted router would be recreated again (and this procedure >>>solved the problem). >>> >>>Below I have attached the state of the two tables. >>> >>>Anything else I can try? >>> >>>Thank you >>> >>> >>>Kambiz >>> >>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway, >>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name, >>>i.state, i.type from vm_instance i join nics n on n.instance_id = i.id >>>where i.type = 'DomainRouter'; >>>+----+---------------------+---------------+---------------+------------ >>>-+ >>>---------+--------------------------+------------+-------------+-------- >>>-+ >>>-----------+--------------+ >>>| id | removed | ip4_address | netmask | gateway >>>| ip_type | reserver_name | network_id | instance_id | name >>>| state | type | >>>+----+---------------------+---------------+---------------+------------ >>>-+ >>>---------+--------------------------+------------+-------------+-------- >>>-+ >>>-----------+--------------+ >>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL >>>| NULL | ExternalGuestNetworkGuru | 204 | 4 | r-4-VM >>>| Expunging | DomainRouter | >>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL >>>| NULL | ControlNetworkGuru | 202 | 4 | r-4-VM >>>| Expunging | DomainRouter | >>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | 10.193.17.1 >>>| NULL | PublicNetworkGuru | 200 | 4 | r-4-VM >>>| Expunging | DomainRouter | >>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL >>>| NULL | ExternalGuestNetworkGuru | 205 | 7 | r-7-VM >>>| Expunging | DomainRouter | >>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL >>>| NULL | ControlNetworkGuru | 202 | 7 | r-7-VM >>>| Expunging | DomainRouter | >>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 >>>| NULL | PublicNetworkGuru | 200 | 7 | r-7-VM >>>| Expunging | DomainRouter | >>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL >>>| NULL | ExternalGuestNetworkGuru | 205 | 18 | >>>r-18-VM >>>| Expunging | DomainRouter | >>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL >>>| NULL | ControlNetworkGuru | 202 | 18 | >>>r-18-VM >>>| Expunging | DomainRouter | >>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 >>>| NULL | PublicNetworkGuru | 200 | 18 | >>>r-18-VM >>>| Expunging | DomainRouter | >>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL >>>| NULL | ExternalGuestNetworkGuru | 205 | 19 | >>>r-19-VM >>>| Stopped | DomainRouter | >>>| 30 | NULL | NULL | NULL | NULL >>>| NULL | ControlNetworkGuru | 202 | 19 | >>>r-19-VM >>>| Stopped | DomainRouter | >>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 >>>| NULL | PublicNetworkGuru | 200 | 19 | >>>r-19-VM >>>| Stopped | DomainRouter | >>>+----+---------------------+---------------+---------------+------------ >>>-+ >>>---------+--------------------------+------------+-------------+-------- >>>-+ >>>-----------+--------------+ >>> >>>mysql> select * from router_network_ref; >>>+----+-----------+------------+------------+ >>>| id | router_id | network_id | guest_type | >>>+----+-----------+------------+------------+ >>>| 1 | 4 | 204 | Isolated | >>>| 2 | 7 | 205 | Isolated | >>>| 3 | 18 | 205 | Isolated | >>>| 4 | 19 | 205 | Isolated | >>>+----+-----------+------------+------------+ >>> >>> >>> >>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>> >>>> The error happens not because Ip is null, but because the nic in a >>>>certain >>>> network can¹t be found. Looks like there is some bug in VPC nic >>>> plug/unplug for Guest networks process. >>>> >>>> Kambiz, please do the following to fix it: >>>> >>>> 1) Stop the MS >>>> 2) Take the DB dump of cloud db in case you have to revert back. >>>> 3) Run the query: >>>> >>>> select * from router_network_ref where router_id=<id of your VR) and >>>> network_id not in (select network_id from nics where instance_id=<ID >>>>of >>>> your VR> and removed is null); >>>> >>>> It will give you the list of networks refs that somehow weren¹t >>>>cleaned >>>> during the nic detach. Remove the entry returned from >>>>router_network_ref >>>> table. >>>> >>>> Let me know how it works. >>>> >>>> -Alena. >>>> >>>> >>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>>> >>>>>Hello, >>>>> >>>>>as this is my first post to the list, I would like to thank all >>>>>contributors for Cloudstack which I use since last fall without any >>>>>problems. I run 4.1.1 with KVM and advanced networking. >>>>> >>>>>After a restart of the management server (stopping and starting the >>>>>java >>>>>process), the virtual domain router doesn't start and >>>>>management-server.log shows a NullPointerException in >>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below). >>>>> >>>>>By putting the server in debug mode and remote debugging, I found out >>>>>that the reason is a row in the table nics which has NULL in ip (cf. >>>>>row >>>>>with id 30 in the result of the select statement below). >>>>> >>>>>What can I do to quickly solve this problem? Any pointers or >>>>>suggestions >>>>>are appreciated as the system is currently unusable. >>>>> >>>>>Thank you for your help >>>>> >>>>> >>>>>Kambiz >>>>> >>>>> >>>>>management-server.log: >>>>> >>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for >>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>2014-03-18 10:03:27,151 WARN >>>>>[network.element.VpcVirtualRouterElement] >>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated >>>>>with >>>>>any VPC >>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement] >>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle >>>>>service >>>>>Connectivity on network net1 >>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl] >>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in the >>>>>network id=205 >>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part >>>>>of >>>>>network implement >>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Network id=202 is already implemented >>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a part >>>>>of >>>>>network implement >>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for >>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>2014-03-18 10:03:27,187 WARN >>>>>[network.element.VpcVirtualRouterElement] >>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated >>>>>with any VPC >>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl] >>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement] >>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle >>>>>service >>>>>Connectivity on network null >>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for >>>>>VM[DomainRouter|r-19-VM] >>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >>>>>(Job-Executor-1:job-176) No need to recreate the volume: >>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding >>>>>disk to VM >>>>>2014-03-18 10:03:27,224 DEBUG >>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]: >>>>>template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0 >>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0 >>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99 >>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true >>>>>dns1=10.193.17.1 >>>>>2014-03-18 10:03:27,343 DEBUG >>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR >>>>>VM[DomainRouter|r-19-VM] start. >>>>>2014-03-18 10:03:27,415 DEBUG >>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load >>>>>balancing rules as a part of Virtual router start >>>>>2014-03-18 10:03:27,499 DEBUG >>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as >>>>>a >>>>>part of domR VM[DomainRouter|r-19-VM] start. >>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl] >>>>>(Job-Executor-1:job-176) Failed to start instance >>>>>VM[DomainRouter|r-19-VM] >>>>>java.lang.NullPointerException >>>>> at >>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.jav >>>>>a: >>>>>76 >>>>>3) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNe >>>>>tw >>>>>or >>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >>>>> at >>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finaliz >>>>>eN >>>>>et >>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCo >>>>>mm >>>>>an >>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >>>>> at >>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finaliz >>>>>eC >>>>>om >>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDe >>>>>pl >>>>>oy >>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205) >>>>> at >>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMana >>>>>ge >>>>>rI >>>>>mpl.java:763) >>>>> at >>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl >>>>>.j >>>>>av >>>>>a:471) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(Virt >>>>>ua >>>>>lN >>>>>etworkApplianceManagerImpl.java:2616) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtu >>>>>al >>>>>Ro >>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute >>>>>rs >>>>>(V >>>>>irtualNetworkApplianceManagerImpl.java:1924) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirt >>>>>ua >>>>>lR >>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >>>>> at >>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRouter >>>>>El >>>>>em >>>>>ent.java:175) >>>>> at >>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResour >>>>>ce >>>>>s( >>>>>NetworkManagerImpl.java:1518) >>>>> at >>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerIm >>>>>pl >>>>>.j >>>>>ava:1434) >>>>> at >>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercep >>>>>to >>>>>rD >>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>> at >>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.j >>>>>av >>>>>a: >>>>>2435) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute >>>>>r( >>>>>Vi >>>>>rtualNetworkApplianceManagerImpl.java:2855) >>>>> at >>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRoute >>>>>r( >>>>>Vi >>>>>rtualNetworkApplianceManagerImpl.java:2824) >>>>> at >>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Intercep >>>>>to >>>>>rD >>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>> at >>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute( >>>>>St >>>>>ar >>>>>tRouterCmd.java:103) >>>>> >>>>> >>>>>table nics: >>>>> >>>>>mysql> select * from nics where reserver_name = 'ControlNetworkGuru'; >>>>>+----+--------------------------------------+-------------+----------- >>>>>-- >>>>>-- >>>>>----+---------------+-------------+-------------+---------+----------- >>>>>-- >>>>>-- >>>>>+------------+--------+--------------+----------+--------------------+ >>>>>-- >>>>>-- >>>>>----------------------------------+-----------+---------------------+- >>>>>-- >>>>>-- >>>>>----------+-------------+-------------+--------------------+---------- >>>>>-- >>>>>-- >>>>>-------+---------------------+-------------+----------+ >>>>>| id | uuid | instance_id | >>>>>mac_address >>>>> | ip4_address | netmask | gateway | ip_type | >>>>>broadcast_uri >>>>>| network_id | mode | state | strategy | reserver_name | >>>>>reservation_id | device_id | update_time >>>>> | >>>>>isolation_uri | ip6_address | default_nic | vm_type | >>>>>created >>>>> | removed | ip6_gateway | ip6_cidr | >>>>>+----+--------------------------------------+-------------+----------- >>>>>-- >>>>>-- >>>>>----+---------------+-------------+-------------+---------+----------- >>>>>-- >>>>>-- >>>>>+------------+--------+--------------+----------+--------------------+ >>>>>-- >>>>>-- >>>>>----------------------------------+-----------+---------------------+- >>>>>-- >>>>>-- >>>>>----------+-------------+-------------+--------------------+---------- >>>>>-- >>>>>-- >>>>>-------+---------------------+-------------+----------+ >>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 | >>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4 >>>>>| >>>>>NULL | 202 | Static | Reserved | Start | >>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0 >>>>>| >>>>>2014-03-17 22:21:38 | NULL | NULL | 0 | >>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL >>>>> | NULL | >>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 | >>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4 >>>>>| >>>>>NULL | 202 | Static | Reserved | Start | >>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0 >>>>>| >>>>>2014-03-17 22:21:38 | NULL | NULL | 0 | >>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL >>>>> | NULL | >>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL >>>>> | NULL | NULL | NULL | NULL | NULL >>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru | >>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 >>>>>12:27:58 | >>>>>NULL | NULL | 0 | DomainRouter | >>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL | >>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL >>>>> | NULL | NULL | NULL | NULL | NULL >>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru | >>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 >>>>>12:27:52 | >>>>>NULL | NULL | 0 | DomainRouter | >>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL | >>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL >>>>> | NULL | NULL | NULL | NULL | NULL >>>>>| 202 | Static | Deallocating | Start | ControlNetworkGuru | >>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 >>>>>09:11:16 | >>>>>NULL | NULL | 0 | DomainRouter | >>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL | >>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL >>>>> | NULL | NULL | NULL | NULL | NULL >>>>>| 202 | Static | Allocated | Start | ControlNetworkGuru | >>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 >>>>>09:11:44 | >>>>>NULL | NULL | 0 | DomainRouter | >>>>>2014-03-18 08:11:32 | NULL | NULL | NULL | >>>>>+----+--------------------------------------+-------------+----------- >>>>>-- >>>>>-- >>>>>----+---------------+-------------+-------------+---------+----------- >>>>>-- >>>>>-- >>>>>+------------+--------+--------------+----------+--------------------+ >>>>>-- >>>>>-- >>>>>----------------------------------+-----------+---------------------+- >>>>>-- >>>>>-- >>>>>----------+-------------+-------------+--------------------+---------- >>>>>-- >>>>>-- >>>>>-------+---------------------+-------------+----------+ >>>>> >>>>>
