Re: [openstack-dev] [Openstack] [qa] How to troubleshoot why a VM at Compute node won't response to ARP request from Neutron router

[Remo Mattei]

By default icmp is not allowed 

Inviato da iPhone ()

> Il giorno 12/ott/2014, alle ore 09:25, Danny Choi (dannchoi) 
>  ha scritto:
> 
> Hi,
> 
> Using devstack to deploy OpenStack, I have Controller + Network running at 
> one physical node and Compute at a separate node.
> 
> I launched a VM at the Compute node with a private address 10.0.0.2 (Neutron 
> router interface is 10.0.0.1).
> 
> At the Controller node, in the qrouter namespace, I could not ping the VM 
> private address 10.0.0.2.
> 
> At the Compute node, tcpdump of the tap interface indicated ARP requests were 
> received.
> 
> However, it did not show any ARP response.
> 
> My understanding is that the VM’s virtual interface is directly connected to 
> this tap interface.  Since the VM is unreachable, I cannot
> launch its console to see if the ARP requests are received at the virtual 
> interface.
> 
> Any suggestions on how to troubleshoot this? 
> 
> localadmin@qa4:~/devstack$ nova show vm1
> +--++
> | Property | Value
>   |
> +--++
> | OS-DCF:diskConfig| MANUAL   
>   |
> | OS-EXT-AZ:availability_zone  | nova 
>   |
> | OS-EXT-STS:power_state   | 1
>   |
> | OS-EXT-STS:task_state| -
>   |
> | OS-EXT-STS:vm_state  | active   
>   |
> | OS-SRV-USG:launched_at   | 2014-10-12T14:25:15.00   
>   |
> | OS-SRV-USG:terminated_at | -
>   |
> | accessIPv4   |  
>   |
> | accessIPv6   |  
>   |
> | config_drive |  
>   |
> | created  | 2014-10-12T14:23:30Z 
>   |
> | flavor   | m1.tiny (1)  
>   |
> | hostId   | 
> 00ac69883737ebd290ad4f38cae979a6e268902333261ba6bfbade44   |
> | id   | 04b5a345-cadf-4dee-9209-5bcf589b6a3c 
>   |
> | image| cirros-0.3.2-x86_64-uec 
> (14a55982-a093-4850-80c8-7b2ae3a7eaba) |
> | key_name | -
>   |
> | metadata | {}   
>   |
> | name | vm1  
>   |
> | os-extended-volumes:volumes_attached | []   
>   |
> | private network  | 10.0.0.2, 172.29.173.5   
>   |
> | progress | 0
>   |
> | security_groups  | default  
>   |
> | status   | ACTIVE   
>   |
> | tenant_id| 90058797dddc49efae4d1f45aa5ab982 
>   |
> | updated  | 2014-10-12T14:23:39Z 
>   |
> | user_id  | 5ab6344540974a1fbda5b539778ebe35 
>   |
> +--++
> localadmin@qa4:~/devstack$ 
> localadmin@qa4:~/devstack$ ip netns
> qdhcp-f55f0683-830f-4523-82cb-46d638f91d47
> qrouter-62aecbdd-d58d-4b33-a743-b16ca38544c5
> localadmin@qa4:~/devstack$ 
> localadmin@qa4:~/devstack$ 
> localadmin@qa4:~/devstack$ sudo ip netns exec 
> qrouter-62aecbdd-d58d-4b33-a743-b16ca38544c5 ping 10.0.0.2
> PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
> From 10.0.0.1 icmp_seq=1 Destination Host Unreachable
> From 10.0.0.1 icmp_seq=2 Destination Host Unreachable
> From 10.0.0.1 icmp_seq=3 Destination Host Unreachable
> From 10.0.0.1 icmp_seq=4 Destination Host Unreachable
> From 10.0.0.1 icmp_seq=5 Destination Host Unreachable
> From 10.0.0.1 icmp_seq=6 Destination Host

Re: [openstack-dev] [Openstack] [qa] How to troubleshoot why a VM at Compute node won't response to ARP request from Neutron router

[Danny Choi (dannchoi)]

I do have security rule configured to allow ICMP.


localadmin@qa4:~/devstack$ nova secgroup-list-rules default

+-+---+-+---+--+

| IP Protocol | From Port | To Port | IP Range  | Source Group |

+-+---+-+---+--+

| tcp | 22| 22  | 0.0.0.0/0 |  |

| |   | |   | default  |

| |   | |   | default  |

| icmp| -1| -1  | 0.0.0.0/0 |  |

+-+---+-+---+———+


Danny

From: Remo Mattei mailto:r...@italy1.com>>
Date: Sunday, October 12, 2014 at 1:00 PM
To: Danny Choi mailto:dannc...@cisco.com>>
Cc: "openst...@lists.openstack.org" 
mailto:openst...@lists.openstack.org>>, 
"openstack-dev@lists.openstack.org" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [Openstack] [qa] How to troubleshoot why a VM at Compute node 
won't response to ARP request from Neutron router

By default icmp is not allowed

Inviato da iPhone ()

Il giorno 12/ott/2014, alle ore 09:25, Danny Choi (dannchoi) 
mailto:dannc...@cisco.com>> ha scritto:

Hi,

Using devstack to deploy OpenStack, I have Controller + Network running at one 
physical node and Compute at a separate node.

I launched a VM at the Compute node with a private address 10.0.0.2 (Neutron 
router interface is 10.0.0.1).

At the Controller node, in the qrouter namespace, I could not ping the VM 
private address 10.0.0.2.

At the Compute node, tcpdump of the tap interface indicated ARP requests were 
received.

However, it did not show any ARP response.

My understanding is that the VM’s virtual interface is directly connected to 
this tap interface.  Since the VM is unreachable, I cannot
launch its console to see if the ARP requests are received at the virtual 
interface.

Any suggestions on how to troubleshoot this?


localadmin@qa4:~/devstack$ nova show vm1

+--++

| Property | Value  
|

+--++

| OS-DCF:diskConfig| MANUAL 
|

| OS-EXT-AZ:availability_zone  | nova   
|

| OS-EXT-STS:power_state   | 1  
|

| OS-EXT-STS:task_state| -  
|

| OS-EXT-STS:vm_state  | active 
|

| OS-SRV-USG:launched_at   | 2014-10-12T14:25:15.00 
|

| OS-SRV-USG:terminated_at | -  
|

| accessIPv4   |
|

| accessIPv6   |
|

| config_drive |
|

| created  | 2014-10-12T14:23:30Z   
|

| flavor   | m1.tiny (1)
|

| hostId   | 
00ac69883737ebd290ad4f38cae979a6e268902333261ba6bfbade44   |

| id   | 04b5a345-cadf-4dee-9209-5bcf589b6a3c   
|

| image| cirros-0.3.2-x86_64-uec 
(14a55982-a093-4850-80c8-7b2ae3a7eaba) |

| key_name | -  
|

| metadata | {} 
|

| name | vm1
|

| os-extended-volumes:volumes_attached | [] 
|

| private network  | 10.0.0.2, 172.29.173.5 
|

| progress | 0  
|

| security_groups  | default
|

| status   | ACTIVE 
|

| tenant_id| 90058797dddc49efae4d1f45aa5ab982   
|

|

Re: [openstack-dev] [Openstack] [qa] How to troubleshoot why a VM at Compute node won't response to ARP request from Neutron router

[Shashank Hegde]

Hi Danny,

You can ssh into your compute node with X forwarding:
ssh -X user@compute

You can then attach to the VM's console using the command:
vncviewer :

You can find the display number using the command:
ps aux | grep kvm
Look for vnc there. Its of the form 127.0.0.1:

~ Shashank

On Sun, Oct 12, 2014 at 10:17 AM, Danny Choi (dannchoi) 
wrote:

>  I do have security rule configured to allow ICMP.
>
>  localadmin@qa4:~/devstack$ nova secgroup-list-rules default
>
> +-+---+-+---+--+
>
> | IP Protocol | From Port | To Port | IP Range  | Source Group |
>
> +-+---+-+---+--+
>
> | tcp | 22| 22  | 0.0.0.0/0 |  |
>
> | |   | |   | default  |
>
> | |   | |   | default  |
>
> | icmp| -1| -1  | 0.0.0.0/0 |  |
>
> +-+---+-+---+———+
>
>
>  Danny
>
>   From: Remo Mattei 
> Date: Sunday, October 12, 2014 at 1:00 PM
> To: Danny Choi 
> Cc: "openst...@lists.openstack.org" , "
> openstack-dev@lists.openstack.org" 
> Subject: Re: [Openstack] [qa] How to troubleshoot why a VM at Compute
> node won't response to ARP request from Neutron router
>
>   By default icmp is not allowed
>
> Inviato da iPhone ()
>
> Il giorno 12/ott/2014, alle ore 09:25, Danny Choi (dannchoi) <
> dannc...@cisco.com> ha scritto:
>
>   Hi,
>
>  Using devstack to deploy OpenStack, I have Controller + Network running
> at one physical node and Compute at a separate node.
>
>  I launched a VM at the Compute node with a private address 10.0.0.2
> (Neutron router interface is 10.0.0.1).
>
>  At the Controller node, in the qrouter namespace, I could not ping the
> VM private address 10.0.0.2.
>
>  At the Compute node, tcpdump of the tap interface indicated ARP requests
> were received.
>
>  However, it did not show any ARP response.
>
>  My understanding is that the VM’s virtual interface is directly
> connected to this tap interface.  Since the VM is unreachable, I cannot
> launch its console to see if the ARP requests are received at the virtual
> interface.
>
>  Any suggestions on how to troubleshoot this?
>
>  localadmin@qa4:~/devstack$ nova show vm1
>
>
> +--++
>
> | Property | Value
>   |
>
>
> +--++
>
> | OS-DCF:diskConfig| MANUAL
>   |
>
> | OS-EXT-AZ:availability_zone  | nova
>   |
>
> | OS-EXT-STS:power_state   | 1
>   |
>
> | OS-EXT-STS:task_state| -
>   |
>
> | OS-EXT-STS:vm_state  | active
>   |
>
> | OS-SRV-USG:launched_at   | 2014-10-12T14:25:15.00
>   |
>
> | OS-SRV-USG:terminated_at | -
>   |
>
> | accessIPv4   |
>   |
>
> | accessIPv6   |
>   |
>
> | config_drive |
>   |
>
> | created  | 2014-10-12T14:23:30Z
>   |
>
> | flavor   | m1.tiny (1)
>   |
>
> | hostId   |
> 00ac69883737ebd290ad4f38cae979a6e268902333261ba6bfbade44   |
>
> | id   |
> 04b5a345-cadf-4dee-9209-5bcf589b6a3c   |
>
> | image| cirros-0.3.2-x86_64-uec
> (14a55982-a093-4850-80c8-7b2ae3a7eaba) |
>
> | key_name | -
>   |
>
> | metadata | {}
>   |
>
> | name | vm1
>   |
>
> | os-extended-volumes:volumes_attached | []
>   |
>
> | private network  | 10.0.0.2, 172.29.173.5
>   |
>
> | progress | 0
>   |
>
> | security_groups  | default
>   |
>
> | status   | ACTIVE
>   |
>
> | tenant_id| 90058797dddc49efae4d1f45aa5ab982
>   |
>
> | updated  | 2014-10-12T14:23:39Z
>   |
>
> | user_id  | 5ab6344540974a1fbda5b539778ebe35
>   

Re: [openstack-dev] [Openstack] [qa] How to troubleshoot why a VM at Compute node won't response to ARP request from Neutron router

[Salvatore Orlando]

If the ARP request reaches the compute node, then you do already know
tunnelling (or whatever transport type you're using) is not your problem.
The security group is also configured properly, so it does not seem
something you need to worry about.

This leaves us with two possible problems:
1) did the dhcp agent give our VM an IP address?
To this aim, check if there is any error in the dhcp agent log. Check if
your VM's MAC address is in the hosts file for the VM's network dnsmasq
instance.
In case you should not be able to find the dnsmasq instance, then this
would be the root cause of your problem.
2) did the ovs agent wire the port for your VM?
Find the ovs port associated with the VM's interface. If you are using ML2
with hybrid plugging the interface name should something like qvo-.
Ensure a local vlan tag is attached to that port. If the vlan tag is 4095
then your port is on the "dead vlan" because the ovs agent recognized it as
"alien". Check for errors in the ovs agent log.

However, the first thing you should do in my opinion is to check the VM
log. It won't be the first time I blame the network and then find out that
the VM image I was using was broken.
You don't need to have connectivity to access the VM console. nova
console-proxy does that for you. Also, if you just want to see the vm
console rather than interact with it, just use the command nova console-log


Hope this helps,
Salvatore

On 12 October 2014 19:26, Shashank Hegde  wrote:

> Hi Danny,
>
> You can ssh into your compute node with X forwarding:
> ssh -X user@compute
>
> You can then attach to the VM's console using the command:
> vncviewer :
>
> You can find the display number using the command:
> ps aux | grep kvm
> Look for vnc there. Its of the form 127.0.0.1:
>
> ~ Shashank
>
> On Sun, Oct 12, 2014 at 10:17 AM, Danny Choi (dannchoi) <
> dannc...@cisco.com> wrote:
>
>>  I do have security rule configured to allow ICMP.
>>
>>  localadmin@qa4:~/devstack$ nova secgroup-list-rules default
>>
>> +-+---+-+---+--+
>>
>> | IP Protocol | From Port | To Port | IP Range  | Source Group |
>>
>> +-+---+-+---+--+
>>
>> | tcp | 22| 22  | 0.0.0.0/0 |  |
>>
>> | |   | |   | default  |
>>
>> | |   | |   | default  |
>>
>> | icmp| -1| -1  | 0.0.0.0/0 |  |
>>
>> +-+---+-+---+———+
>>
>>
>>  Danny
>>
>>   From: Remo Mattei 
>> Date: Sunday, October 12, 2014 at 1:00 PM
>> To: Danny Choi 
>> Cc: "openst...@lists.openstack.org" , "
>> openstack-dev@lists.openstack.org" 
>> Subject: Re: [Openstack] [qa] How to troubleshoot why a VM at Compute
>> node won't response to ARP request from Neutron router
>>
>>   By default icmp is not allowed
>>
>> Inviato da iPhone ()
>>
>> Il giorno 12/ott/2014, alle ore 09:25, Danny Choi (dannchoi) <
>> dannc...@cisco.com> ha scritto:
>>
>>   Hi,
>>
>>  Using devstack to deploy OpenStack, I have Controller + Network running
>> at one physical node and Compute at a separate node.
>>
>>  I launched a VM at the Compute node with a private address 10.0.0.2
>> (Neutron router interface is 10.0.0.1).
>>
>>  At the Controller node, in the qrouter namespace, I could not ping the
>> VM private address 10.0.0.2.
>>
>>  At the Compute node, tcpdump of the tap interface indicated ARP
>> requests were received.
>>
>>  However, it did not show any ARP response.
>>
>>  My understanding is that the VM’s virtual interface is directly
>> connected to this tap interface.  Since the VM is unreachable, I cannot
>> launch its console to see if the ARP requests are received at the virtual
>> interface.
>>
>>  Any suggestions on how to troubleshoot this?
>>
>>  localadmin@qa4:~/devstack$ nova show vm1
>>
>>
>> +--++
>>
>> | Property | Value
>>   |
>>
>>
>> +--++
>>
>> | OS-DCF:diskConfig| MANUAL
>>   |
>>
>> | OS-EXT-AZ:availability_zone  | nova
>>   |
>>
>> | OS-EXT-STS:power_state   | 1
>>   |
>>
>> | OS-EXT-STS:task_state| -
>>   |
>>
>> | OS-EXT-STS:vm_state  | active
>>   |
>>
>> | OS-SRV-USG:launched_at   | 2014-10-12T14:25:15.00
>>   |
>>
>> | OS-SRV-USG:terminated_at | -
>>   |
>>
>> | accessIPv4   |
>>   |
>>
>> | accessIPv6   |
>>