Reviewed: https://review.openstack.org/389058 Committed: https://git.openstack.org/cgit/openstack/neutron-fwaas/commit/?id=7de84412592562e1a1b9a0f67caaaad665c9bc9b Submitter: Jenkins Branch: master
commit 7de84412592562e1a1b9a0f67caaaad665c9bc9b Author: ZhaoBo <zhao...@huawei.com> Date: Thu Oct 20 15:57:07 2016 +0800 Fix fwaas_v2 driver internal error when restart l3 agent The fwaas_v2 driver could not process the input 'apply_list' just include port_id. The expected input should be like [(router_info object, port_id),..] This patch fixes the l3 agent sync process with server. Make the input value meet the required. Change-Id: I7891b6c65524afd9eec89aec06d1e6bf6b0f0fae Closes-Bug: #1635156 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1635156 Title: fwaas v2 hit driver internal error when restart l3 agent Status in neutron: Fix Released Bug description: There is a router serviced by l3 agent. And a firewall group located a port which serviced by the router, such as an interface of an internal subnet. Then I restart l3 agent, it will sync the server side router info and refresh the all related router info, include firewall iptables rules. It hit driver internal error during this period. The trace like: 2016-10-20 15:05:11.587 DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 16 iptables commands were issued from (pid=81933) _apply_synchronized /opt/stack/neutron/neutron/agent/linux/iptables_manager.py:533 2016-10-20 15:05:11.588 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "iptables-qrouter-cc5ab5a3-ef25-4496-87c7-5063cd167ce6" from (pid=81933) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225 2016-10-20 15:05:11.589 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [-] Process router update, router_id: cc5ab5a3-ef25-4496-87c7-5063cd167ce6 tenant: 488da3aab0ff45df9e85e17e7f89fedd. from (pid=81933) _process_router_update /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:239 2016-10-20 15:05:11.589 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'list'] from (pid=81933) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100 2016-10-20 15:05:11.593 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141 2016-10-20 15:05:14.358 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.json from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38 2016-10-20 15:05:14.470 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.json from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584 2016-10-20 15:05:14.471 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/bgpvpn.conf from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38 2016-10-20 15:05:14.484 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/bgpvpn.conf from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584 2016-10-20 15:05:14.484 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/dynamic_routing.conf from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38 2016-10-20 15:05:14.491 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/dynamic_routing.conf from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584 2016-10-20 15:05:14.491 DEBUG oslo_policy._cache_handler [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloading cached file /etc/neutron/policy.d/neutron-fwaas.json from (pid=81933) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:38 2016-10-20 15:05:14.502 DEBUG oslo_policy.policy [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Reloaded policy file: /etc/neutron/policy.d/neutron-fwaas.json from (pid=81933) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:584 2016-10-20 15:05:14.503 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Fetch firewall groups from plugin from (pid=81933) get_firewall_groups_for_project /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:43 2016-10-20 15:05:14.504 DEBUG oslo_messaging._drivers.amqpdriver [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] CALL msg_id: 2d47fe084441434db22bbc5e21861abc exchange 'neutron' topic 'q-firewall-plugin' from (pid=81933) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448 2016-10-20 15:05:14.505 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141 2016-10-20 15:05:14.505 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141 2016-10-20 15:05:14.506 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141 2016-10-20 15:05:14.506 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141 2016-10-20 15:05:14.507 DEBUG neutron.agent.linux.utils [-] Exit code: 0 from (pid=81933) execute /opt/stack/neutron/neutron/agent/linux/utils.py:141 2016-10-20 15:05:14.508 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a1662f95-21a5-494c-b14b-a6249e7179b7', 'ip6tables-save'] from (pid=81933) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100 2016-10-20 15:05:14.525 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2d47fe084441434db22bbc5e21861abc from (pid=81933) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296 2016-10-20 15:05:17.668 DEBUG neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Updating firewall 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5 for tenant 488da3aab0ff45df9e85e17e7f89fedd from (pid=81933) update_firewall_group /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py:131 2016-10-20 15:05:17.668 ERROR neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Failed to update firewall: 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5 2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 Traceback (most recent call last): 2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 139, in update_firewall_group 2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 apply_list, firewall) 2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 304, in _remove_conntrack_new_firewall 2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 routers_list = list(set([apply_info[0] for apply_info in apply_list])) 2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 KeyError: 0 2016-10-20 15:05:17.668 TRACE neutron_fwaas.services.firewall.drivers.linux.iptables_fwaas_v2 2016-10-20 15:05:17.669 ERROR neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] FWaaS driver error on ACTIVE for firewall group: 2932b3d9-3a7b-48a1-a16c-bf9f7b2751a5 2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 Traceback (most recent call last): 2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py", line 192, in _invoke_driver_for_sync_from_plugin 2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 if firewall_group['status'] == n_const.PENDING_DELETE: 2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 File "/opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/drivers/linux/iptables_fwaas_v2.py", line 147, in update_firewall_group 2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 raise fw_ext.FirewallInternalDriverError(driver=FWAAS_DRIVER_NAME) 2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 FirewallInternalDriverError: Fwaas iptables driver: Internal driver error. 2016-10-20 15:05:17.669 TRACE neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 2016-10-20 15:05:17.670 DEBUG neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent_v2 [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] Set firewall groups from plugin from (pid=81933) set_firewall_group_status /opt/stack/neutron-fwaas/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py:67 2016-10-20 15:05:17.671 DEBUG oslo_messaging._drivers.amqpdriver [req-646cbdc6-89d9-4af2-9779-3ed1042154f9 None 488da3aab0ff45df9e85e17e7f89fedd] CALL msg_id: f68ec0a4772941ebaacf410f5187c130 exchange 'neutron' topic 'q-firewall-plugin' from (pid=81933) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1635156/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp