Hi,
select id,ip4_address,netmask,gateway,state,removed,network_id,reserver_name
from nics where instance_id=19;
+----+---------------+---------------+-------------+-----------+---------+------------+--------------------------+
| id | ip4_address | netmask | gateway | state | removed |
network_id | reserver_name |
+----+---------------+---------------+-------------+-----------+---------+------------+--------------------------+
| 29 | 10.124.99.1 | 255.255.255.0 | NULL | Allocated | NULL |
205 | ExternalGuestNetworkGuru |
| 30 | NULL | NULL | NULL | Allocated | NULL |
202 | ControlNetworkGuru |
| 31 | 10.193.17.190 | 255.255.255.0 | 10.193.17.1 | Allocated | NULL |
200 | PublicNetworkGuru |
+----+---------------+---------------+-------------+-----------+---------+------------+--------------------------+
and this is the nics element from the StartCmd. Is it normal to have
two nics with deviceId 0?
"nics":[
{"deviceId":2,
"networkRateMbps":200,
"defaultNic":true,
"uuid":"22c19454-fd05-45c8-af6b-5f0ef073f86c",
"ip":"10.193.17.190",
"netmask":"255.255.255.0",
"gateway":"10.193.17.1",
"mac":"06:7e:fe:00:00:bf",
"dns1":"10.193.17.1",
"broadcastType":"Vlan",
"type":"Public",
"broadcastUri":"vlan://untagged",
"isolationUri":"vlan://untagged",
"isSecurityGroupEnabled":false,
"name":"cloudbr0"},
{"deviceId":0,
"networkRateMbps":200,
"defaultNic":false,
"uuid":"6c5a8337-620e-49eb-9309-cdfc7039d4a8",
"ip":"10.124.99.1",
"netmask":"255.255.255.0",
"mac":"02:00:2a:43:00:0d",
"dns1":"10.193.17.1",
"broadcastType":"Vlan",
"type":"Guest",
"broadcastUri":"vlan://3949",
"isolationUri":"vlan://3949",
"isSecurityGroupEnabled":false,
"name":"cloudbr1"},
{"deviceId":0,
"networkRateMbps":-1,
"defaultNic":false,
"uuid":"cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d",
"ip":"169.254.1.59",
"netmask":"255.255.0.0",
"gateway":"169.254.0.1",
"mac":"0e:00:a9:fe:01:3b",
"broadcastType":"LinkLocal",
"type":"Control",
"isSecurityGroupEnabled":false}
]
Thanks
Kambiz
Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>
> Kambiz, the debug statements below are for the case when eth1 is a control
> interface as it was in your old command. I’ve looked at the new command,
> eth1 is not control, its either public or guest
>
> eth0: - control
>
> iface eth0 inet static
> address 169.254.1.59
> netmask 255.255.0.0
>
> eth1:
>
> iface eth1 inet static
> address 10.193.17.1
> Netmask
>
> So you need to execute the mysql statements for the traffic type of VR nic
> eth1
>
> -Alena.
>
>
>
>
> On 3/25/14, 9:57 AM, "Alena Prokharchyk" <alena.prokharc...@citrix.com>
> wrote:
>
>>Kambiz, can you please check the following:
>>
>>
>>1) Check if the gateway is set on control network:
>>
>>mysql> select gateway, cidr from networks where traffic_type=‘Control’;
>>
>>2) For router control nic, check if network/gateway are set.
>>
>>Select gateway,netmask from nics where instance_id=<routerid> and
>>network_id=<control network id>
>>
>>-Alena.
>>
>>On 3/25/14, 5:47 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote:
>>
>>>Hi,
>>>
>>>I looked up the startup command of the old router instance which worked
>>>correctly:
>>>
>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>>>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%ga
>>>t
>>>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6clou
>>>d
>>>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0%
>>>t
>>>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>>>
>>>The new command (cf. below) doesn't have the parameters eth1ip and
>>>eth1mask.
>>>
>>>Thanks
>>>
>>>
>>>Kambiz
>>>
>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>>>>
>>>> I don’t think its relevant as the piece we’ve fixed, just eliminated
>>>> static nat rule programming for non-existing vm. Missing netmask on
>>>>eth1
>>>> doesn’t seem related to the problem (although we have to figure out why
>>>> its missing), as the connection that fails, happening to link local
>>>>169.x
>>>> eth0 interface.
>>>>
>>>> Edison, can you please tell us how to debug link local connection
>>>>failure,
>>>> on KVM agent?
>>>>
>>>> Thank you,
>>>> Alena.
>>>>
>>>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>thank you, the NullPointerException doesn't occur any more, but there
>>>>>still seems to be a problem during startup of the router.
>>>>>
>>>>>When I start the virtual router, it comes up, but in agent.log, there
>>>>>are lots of 'Could not connect to 169.254.1.x' messages.
>>>>>
>>>>>Then I logged into the virtual router to find out that the netmask of
>>>>>eth1 is missing in the interfaces file:
>>>>>
>>>>>root@host:~# virsh console r-19-VM
>>>>>Connected to domain r-19-VM
>>>>>Escape character is ^]
>>>>>
>>>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>>>
>>>>>r-19-VM login: root
>>>>>...
>>>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>>>auto lo eth0 eth1 eth2
>>>>>iface lo inet loopback
>>>>>
>>>>>iface eth0 inet static
>>>>> address 169.254.1.59
>>>>> netmask 255.255.0.0
>>>>>iface eth1 inet static
>>>>> address 10.193.17.1
>>>>> netmask
>>>>>iface eth2 inet static
>>>>> address 10.193.17.190
>>>>> netmask 255.255.255.0
>>>>>
>>>>>I don't know if it is relevant, but this is the line from agent.log
>>>>>where the parameters are visible:
>>>>>
>>>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>>>>>(agentRequest-Handler-2:null) Executing:
>>>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0
>>>>>%
>>>>>ga
>>>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6c
>>>>>l
>>>>>ou
>>>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0
>>>>>.
>>>>>0%
>>>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>>>
>>>>>
>>>>>Any hint is appreciated.
>>>>>
>>>>>Thanks
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>
>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>>>>>>
>>>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If
>>>>>>vm
>>>>>> id=15 is expunged, we have to clear out the reference to it from
>>>>>> user_ip_address table. Here is the flow:
>>>>>>
>>>>>> 1) Save the db dump.
>>>>>> 2) Run the query to cleanup the reference:
>>>>>>
>>>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>>>>>> id=<problematic public ip address id>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Let me know how it works.
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hi,
>>>>>>>
>>>>>>>I hope I have understood what you wrote and created the following
>>>>>>>query
>>>>>>>correctly:
>>>>>>>
>>>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>>> n.state as nic_state, n.removed as nic_removed,
>>>>>>> vm.state as vm_state, vm.removed as vm_removed
>>>>>>>from user_ip_address uip
>>>>>>> join nics n on uip.vm_id = n.instance_id
>>>>>>> join vm_instance vm on uip.vm_id = vm.id
>>>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>>>>>fr.network_id=205);
>>>>>>>
>>>>>>>
>>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>>-
>>>>>>>--
>>>>>>>--
>>>>>>>----+-----------+------------+
>>>>>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed
>>>>>>> | vm_state | vm_removed |
>>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>>-
>>>>>>>--
>>>>>>>--
>>>>>>>----+-----------+------------+
>>>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18
>>>>>>>23:00:53 | Expunging | NULL |
>>>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL
>>>>>>> | Stopped | NULL |
>>>>>>>+-------+------------+-------------------+--------------+------------
>>>>>>>-
>>>>>>>--
>>>>>>>--
>>>>>>>----+-----------+------------+
>>>>>>>
>>>>>>>Is VM id 15 what you are looking for?
>>>>>>>
>>>>>>>Thank you
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>>>>>>>>
>>>>>>>> Kambiz, can you please try one more thing.
>>>>>>>>
>>>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>>>right?)
>>>>>>>>
>>>>>>>> Select id, ip_address_id from firewall_rules where network_id=205;
>>>>>>>>
>>>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>>>
>>>>>>>> Select vm_id, network_id from user_ip_address where id in (Select
>>>>>>>>id,
>>>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>>>
>>>>>>>> For each vmId/networkId combo, check if there is non-removed nic
>>>>>>>>and
>>>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>>>reference
>>>>>>>> referring to vm that is removed already. If you find any, let me
>>>>>>>>know
>>>>>>>>and
>>>>>>>> I will tell you how to clean it up
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <dar...@m-creations.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>>Hi Alena,
>>>>>>>>>
>>>>>>>>>thank you for your help.
>>>>>>>>>
>>>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>>>>>removed
>>>>>>>>>the row though to see what happens: a new virtual router was
>>>>>>>>>created
>>>>>>>>>which also couldn't be started due to the same NPE. I reverted the
>>>>>>>>>change by restoring from the dump.
>>>>>>>>>
>>>>>>>>>I have to mention that prior to the restart, r-7-VM was the router
>>>>>>>>>which
>>>>>>>>>was used by my instances. I deleted the router using the UI after
>>>>>>>>>the
>>>>>>>>>first
>>>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>>>suggested
>>>>>>>>>that the deleted router would be recreated again (and this
>>>>>>>>>procedure
>>>>>>>>>solved the problem).
>>>>>>>>>
>>>>>>>>>Below I have attached the state of the two tables.
>>>>>>>>>
>>>>>>>>>Anything else I can try?
>>>>>>>>>
>>>>>>>>>Thank you
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>>>i.name,
>>>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id =
>>>>>>>>>i.id
>>>>>>>>>where i.type = 'DomainRouter';
>>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>-----------+--------------+
>>>>>>>>>| id | removed | ip4_address | netmask |
>>>>>>>>>gateway
>>>>>>>>>| ip_type | reserver_name | network_id | instance_id |
>>>>>>>>>name
>>>>>>>>>| state | type |
>>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>-----------+--------------+
>>>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 |
>>>>>>>>>r-4-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL
>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 4 |
>>>>>>>>>r-4-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>>>10.193.17.1
>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 4 |
>>>>>>>>>r-4-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 |
>>>>>>>>>r-7-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL
>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 7 |
>>>>>>>>>r-7-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>10.193.17.1
>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 7 |
>>>>>>>>>r-7-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 |
>>>>>>>>>r-18-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL
>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 18 |
>>>>>>>>>r-18-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>10.193.17.1
>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 18 |
>>>>>>>>>r-18-VM
>>>>>>>>>| Expunging | DomainRouter |
>>>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL
>>>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 |
>>>>>>>>>r-19-VM
>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>| 30 | NULL | NULL | NULL | NULL
>>>>>>>>>| NULL | ControlNetworkGuru | 202 | 19 |
>>>>>>>>>r-19-VM
>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 |
>>>>>>>>>10.193.17.1
>>>>>>>>>| NULL | PublicNetworkGuru | 200 | 19 |
>>>>>>>>>r-19-VM
>>>>>>>>>| Stopped | DomainRouter |
>>>>>>>>>+----+---------------------+---------------+---------------+-------
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>---------+--------------------------+------------+-------------+---
>>>>>>>>>-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>-----------+--------------+
>>>>>>>>>
>>>>>>>>>mysql> select * from router_network_ref;
>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>| 1 | 4 | 204 | Isolated |
>>>>>>>>>| 2 | 7 | 205 | Isolated |
>>>>>>>>>| 3 | 18 | 205 | Isolated |
>>>>>>>>>| 4 | 19 | 205 | Isolated |
>>>>>>>>>+----+-----------+------------+------------+
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>>>>>>>>>>
>>>>>>>>>> The error happens not because Ip is null, but because the nic in
>>>>>>>>>>a
>>>>>>>>>>certain
>>>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>>>
>>>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>>>
>>>>>>>>>> 1) Stop the MS
>>>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert back.
>>>>>>>>>> 3) Run the query:
>>>>>>>>>>
>>>>>>>>>> select * from router_network_ref where router_id=<id of your VR)
>>>>>>>>>>and
>>>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>>>instance_id=<ID
>>>>>>>>>>of
>>>>>>>>>> your VR> and removed is null);
>>>>>>>>>>
>>>>>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>>>>>cleaned
>>>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>>>router_network_ref
>>>>>>>>>> table.
>>>>>>>>>>
>>>>>>>>>> Let me know how it works.
>>>>>>>>>>
>>>>>>>>>> -Alena.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <dar...@m-creations.com>
>>>>>>>>>>wrote:
>>>>>>>>>>
>>>>>>>>>>>Hello,
>>>>>>>>>>>
>>>>>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>>>>>contributors for Cloudstack which I use since last fall without
>>>>>>>>>>>any
>>>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>>>
>>>>>>>>>>>After a restart of the management server (stopping and starting
>>>>>>>>>>>the
>>>>>>>>>>>java
>>>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>>>
>>>>>>>>>>>By putting the server in debug mode and remote debugging, I found
>>>>>>>>>>>out
>>>>>>>>>>>that the reason is a row in the table nics which has NULL in ip
>>>>>>>>>>>(cf.
>>>>>>>>>>>row
>>>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>>>
>>>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>>>suggestions
>>>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>>>
>>>>>>>>>>>Thank you for your help
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Kambiz
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>management-server.log:
>>>>>>>>>>>
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>>>for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>>>associated
>>>>>>>>>>>with
>>>>>>>>>>>any VPC
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>>>service
>>>>>>>>>>>Connectivity on network net1
>>>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported
>>>>>>>>>>>in
>>>>>>>>>>>the
>>>>>>>>>>>network id=205
>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a
>>>>>>>>>>>part
>>>>>>>>>>>of
>>>>>>>>>>>network implement
>>>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a
>>>>>>>>>>>part
>>>>>>>>>>>of
>>>>>>>>>>>network implement
>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>>>for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>>>associated
>>>>>>>>>>>with any VPC
>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>>>service
>>>>>>>>>>>Connectivity on network null
>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes
>>>>>>>>>>>for
>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>>>>>adding
>>>>>>>>>>>disk to VM
>>>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1
>>>>>>>>>>>eth0ip=169.254.3.99
>>>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to
>>>>>>>>>>>apply
>>>>>>>>>>>as
>>>>>>>>>>>a
>>>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>>>2014-03-18 10:03:27,593 ERROR
>>>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImp
>>>>>>>>>>>l
>>>>>>>>>>>.j
>>>>>>>>>>>av
>>>>>>>>>>>a:
>>>>>>>>>>>76
>>>>>>>>>>>3)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>>i
>>>>>>>>>>>ze
>>>>>>>>>>>Ne
>>>>>>>>>>>tw
>>>>>>>>>>>or
>>>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fi
>>>>>>>>>>>n
>>>>>>>>>>>al
>>>>>>>>>>>iz
>>>>>>>>>>>eN
>>>>>>>>>>>et
>>>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:92
>>>>>>>>>>>8
>>>>>>>>>>>)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>>i
>>>>>>>>>>>ze
>>>>>>>>>>>Co
>>>>>>>>>>>mm
>>>>>>>>>>>an
>>>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fi
>>>>>>>>>>>n
>>>>>>>>>>>al
>>>>>>>>>>>iz
>>>>>>>>>>>eC
>>>>>>>>>>>om
>>>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final
>>>>>>>>>>>i
>>>>>>>>>>>ze
>>>>>>>>>>>De
>>>>>>>>>>>pl
>>>>>>>>>>>oy
>>>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachin
>>>>>>>>>>>e
>>>>>>>>>>>Ma
>>>>>>>>>>>na
>>>>>>>>>>>ge
>>>>>>>>>>>rI
>>>>>>>>>>>mpl.java:763)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManage
>>>>>>>>>>>r
>>>>>>>>>>>Im
>>>>>>>>>>>pl
>>>>>>>>>>>.j
>>>>>>>>>>>av
>>>>>>>>>>>a:471)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>(
>>>>>>>>>>>Vi
>>>>>>>>>>>rt
>>>>>>>>>>>ua
>>>>>>>>>>>lN
>>>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>V
>>>>>>>>>>>ir
>>>>>>>>>>>tu
>>>>>>>>>>>al
>>>>>>>>>>>Ro
>>>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>R
>>>>>>>>>>>ou
>>>>>>>>>>>te
>>>>>>>>>>>rs
>>>>>>>>>>>(V
>>>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deplo
>>>>>>>>>>>y
>>>>>>>>>>>Vi
>>>>>>>>>>>rt
>>>>>>>>>>>ua
>>>>>>>>>>>lR
>>>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualR
>>>>>>>>>>>o
>>>>>>>>>>>ut
>>>>>>>>>>>er
>>>>>>>>>>>El
>>>>>>>>>>>em
>>>>>>>>>>>ent.java:175)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndR
>>>>>>>>>>>e
>>>>>>>>>>>so
>>>>>>>>>>>ur
>>>>>>>>>>>ce
>>>>>>>>>>>s(
>>>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkMana
>>>>>>>>>>>g
>>>>>>>>>>>er
>>>>>>>>>>>Im
>>>>>>>>>>>pl
>>>>>>>>>>>.j
>>>>>>>>>>>ava:1434)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Int
>>>>>>>>>>>e
>>>>>>>>>>>rc
>>>>>>>>>>>ep
>>>>>>>>>>>to
>>>>>>>>>>>rD
>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerI
>>>>>>>>>>>m
>>>>>>>>>>>pl
>>>>>>>>>>>.j
>>>>>>>>>>>av
>>>>>>>>>>>a:
>>>>>>>>>>>2435)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>R
>>>>>>>>>>>ou
>>>>>>>>>>>te
>>>>>>>>>>>r(
>>>>>>>>>>>Vi
>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start
>>>>>>>>>>>R
>>>>>>>>>>>ou
>>>>>>>>>>>te
>>>>>>>>>>>r(
>>>>>>>>>>>Vi
>>>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>>> at
>>>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Int
>>>>>>>>>>>e
>>>>>>>>>>>rc
>>>>>>>>>>>ep
>>>>>>>>>>>to
>>>>>>>>>>>rD
>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>>> at
>>>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.exe
>>>>>>>>>>>c
>>>>>>>>>>>ut
>>>>>>>>>>>e(
>>>>>>>>>>>St
>>>>>>>>>>>ar
>>>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>table nics:
>>>>>>>>>>>
>>>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>+-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>| id | uuid | instance_id |
>>>>>>>>>>>mac_address
>>>>>>>>>>> | ip4_address | netmask | gateway | ip_type |
>>>>>>>>>>>broadcast_uri
>>>>>>>>>>>| network_id | mode | state | strategy | reserver_name
>>>>>>>>>>> |
>>>>>>>>>>>reservation_id | device_id | update_time
>>>>>>>>>>> |
>>>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type |
>>>>>>>>>>>created
>>>>>>>>>>> | removed | ip6_gateway | ip6_cidr |
>>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>+-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 |
>>>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>>Ip4
>>>>>>>>>>>|
>>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>>>0
>>>>>>>>>>>|
>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL |
>>>>>>>>>>>NULL
>>>>>>>>>>> | NULL |
>>>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 |
>>>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>>>Ip4
>>>>>>>>>>>|
>>>>>>>>>>>NULL | 202 | Static | Reserved | Start |
>>>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>>>0
>>>>>>>>>>>|
>>>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 |
>>>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL |
>>>>>>>>>>>NULL
>>>>>>>>>>> | NULL |
>>>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL
>>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17
>>>>>>>>>>>12:27:58 |
>>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL
>>>>>>>>>>> |
>>>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL
>>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17
>>>>>>>>>>>12:27:52 |
>>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL
>>>>>>>>>>> |
>>>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL
>>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>>| 202 | Static | Deallocating | Start |
>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18
>>>>>>>>>>>09:11:16 |
>>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL
>>>>>>>>>>> |
>>>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL
>>>>>>>>>>> | NULL | NULL | NULL | NULL | NULL
>>>>>>>>>>>| 202 | Static | Allocated | Start |
>>>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18
>>>>>>>>>>>09:11:44 |
>>>>>>>>>>>NULL | NULL | 0 | DomainRouter |
>>>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL
>>>>>>>>>>> |
>>>>>>>>>>>+----+--------------------------------------+-------------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----+---------------+-------------+-------------+---------+------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>+------------+--------+--------------+----------+----------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>-+
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------------------------------+-----------+------------------
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>+-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>----------+-------------+-------------+--------------------+-----
>>>>>>>>>>>-
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>--
>>>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>>>
>>>>>>>>>>>
>>
