Public bug reported: Hello,
I'm running OpenStack RDO with the following setup. - 1x Controller Node ( Neutron - Nova - glance - Horizon - Keystone - GRE + FlatDHCPManager) - 10x Compute Nodes ( Nova Compute - KVM ) - OS : CentOS 6 - 64bit Suddenly all vm's on the compute nodes can't get new IP address from the controller node. It was working properly hours ago without any modifications from me. I tried to restart neutron-dhcp-agent - qpidd and dnsmasq with no luck. This log from the controller node - **many errors** > http://paste.openstack.org/show/88016/ And this from the compute node > http://paste.openstack.org/show/88017/ Realtime compute node log after restarting vm network service > http://paste.openstack.org/show/88039/ Installed packages on Controller Node : # rpm -qa | grep -i openstack openstack-selinux-0.1.3-2.el6ost.noarch openstack-puppet-modules-2013.2-9.1.el6.noarch openstack-ceilometer-api-2013.2.3-2.el6.noarch openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch openstack-nova-scheduler-2013.2.3-1.el6.noarch openstack-ceilometer-common-2013.2.3-2.el6.noarch python-django-openstack-auth-1.1.2-1.el6.noarch openstack-ceilometer-central-2013.2.3-2.el6.noarch openstack-ceilometer-collector-2013.2.3-2.el6.noarch openstack-neutron-openvswitch-2013.2.3-9.el6.noarch openstack-nova-common-2013.2.3-1.el6.noarch openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch openstack-glance-2013.2.3-2.el6.noarch openstack-nova-conductor-2013.2.3-1.el6.noarch openstack-nova-novncproxy-2013.2.3-1.el6.noarch openstack-nova-cert-2013.2.3-1.el6.noarch openstack-keystone-2013.2.3-3.el6.noarch openstack-neutron-2013.2.3-9.el6.noarch openstack-ceilometer-alarm-2013.2.3-2.el6.noarch openstack-dashboard-2013.2.3-1.el6.noarch openstack-nova-api-2013.2.3-1.el6.noarch openstack-nova-console-2013.2.3-1.el6.noarch openstack-utils-2013.2-2.el6.noarch On Controller Node # ps aux | grep -i dhcp nobody 12639 0.0 0.0 12884 864 ? S Jul19 0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap7139b265-41 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/host --dhcp-optsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal neutron 24884 0.0 0.1 273748 32140 ? S Jul24 0:00 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini # ip netns qrouter-id qdhcp-id # ip netns exec qdhcp-network-id ip a 16: tap7139b265-41: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:2e:18:35 brd ff:ff:ff:ff:ff:ff inet 10.0.0.3/24 brd 10.0.0.255 scope global tap7139b265-41 inet6 fe80::f816:3eff:fe2e:1835/64 scope link valid_lft forever preferred_lft forever 17: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever >> ip netns exec qdhcp-network-id ifconfig lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:4 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2304 (2.2 KiB) TX bytes:2304 (2.2 KiB) tap7139b265-41 Link encap:Ethernet HWaddr FA:16:3E:2E:18:35 inet addr:10.0.0.3 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe2e:1835/64 Scope:Link UP BROADCAST RUNNING MTU:1500 Metric:1 RX packets:91007 errors:0 dropped:0 overruns:0 frame:0 TX packets:727 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:4633459 (4.4 MiB) TX bytes:193767 (189.2 KiB) Regards, ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1348421 Title: VM Can't get DHCP IP Status in OpenStack Neutron (virtual network service): New Bug description: Hello, I'm running OpenStack RDO with the following setup. - 1x Controller Node ( Neutron - Nova - glance - Horizon - Keystone - GRE + FlatDHCPManager) - 10x Compute Nodes ( Nova Compute - KVM ) - OS : CentOS 6 - 64bit Suddenly all vm's on the compute nodes can't get new IP address from the controller node. It was working properly hours ago without any modifications from me. I tried to restart neutron-dhcp-agent - qpidd and dnsmasq with no luck. This log from the controller node - **many errors** > http://paste.openstack.org/show/88016/ And this from the compute node > http://paste.openstack.org/show/88017/ Realtime compute node log after restarting vm network service > http://paste.openstack.org/show/88039/ Installed packages on Controller Node : # rpm -qa | grep -i openstack openstack-selinux-0.1.3-2.el6ost.noarch openstack-puppet-modules-2013.2-9.1.el6.noarch openstack-ceilometer-api-2013.2.3-2.el6.noarch openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch openstack-nova-scheduler-2013.2.3-1.el6.noarch openstack-ceilometer-common-2013.2.3-2.el6.noarch python-django-openstack-auth-1.1.2-1.el6.noarch openstack-ceilometer-central-2013.2.3-2.el6.noarch openstack-ceilometer-collector-2013.2.3-2.el6.noarch openstack-neutron-openvswitch-2013.2.3-9.el6.noarch openstack-nova-common-2013.2.3-1.el6.noarch openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch openstack-glance-2013.2.3-2.el6.noarch openstack-nova-conductor-2013.2.3-1.el6.noarch openstack-nova-novncproxy-2013.2.3-1.el6.noarch openstack-nova-cert-2013.2.3-1.el6.noarch openstack-keystone-2013.2.3-3.el6.noarch openstack-neutron-2013.2.3-9.el6.noarch openstack-ceilometer-alarm-2013.2.3-2.el6.noarch openstack-dashboard-2013.2.3-1.el6.noarch openstack-nova-api-2013.2.3-1.el6.noarch openstack-nova-console-2013.2.3-1.el6.noarch openstack-utils-2013.2-2.el6.noarch On Controller Node # ps aux | grep -i dhcp nobody 12639 0.0 0.0 12884 864 ? S Jul19 0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap7139b265-41 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/host --dhcp-optsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal neutron 24884 0.0 0.1 273748 32140 ? S Jul24 0:00 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini # ip netns qrouter-id qdhcp-id # ip netns exec qdhcp-network-id ip a 16: tap7139b265-41: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:2e:18:35 brd ff:ff:ff:ff:ff:ff inet 10.0.0.3/24 brd 10.0.0.255 scope global tap7139b265-41 inet6 fe80::f816:3eff:fe2e:1835/64 scope link valid_lft forever preferred_lft forever 17: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever >> ip netns exec qdhcp-network-id ifconfig lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:4 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2304 (2.2 KiB) TX bytes:2304 (2.2 KiB) tap7139b265-41 Link encap:Ethernet HWaddr FA:16:3E:2E:18:35 inet addr:10.0.0.3 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe2e:1835/64 Scope:Link UP BROADCAST RUNNING MTU:1500 Metric:1 RX packets:91007 errors:0 dropped:0 overruns:0 frame:0 TX packets:727 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:4633459 (4.4 MiB) TX bytes:193767 (189.2 KiB) Regards, To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1348421/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
