[Yahoo-eng-team] [Bug 2046196] Re: Unicast ICMPv6 Router Advertisement packets to VM's dropped by OVS firewall driver
*** This bug is a duplicate of bug 1958643 *** https://bugs.launchpad.net/bugs/1958643 Thanks @Stanislav for the confirmation, will close it as Duplicate. ** This bug has been marked a duplicate of bug 1958643 Unicast RA messages for a VM are filtered out by ovs rules -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2046196 Title: Unicast ICMPv6 Router Advertisement packets to VM's dropped by OVS firewall driver Status in neutron: Incomplete Bug description: When use Open vSwitch Firewall Driver we noticed that ICMPv6 RA unicast packets are not reaching the VM. =Troubleshooting flow:=== 1) Catch ICMPv6 RA package on physical hypervisor bonding interface: #tcpdump -XXvpnei bond0 -Q in -c 1 "icmp6[0] = 134 and ether host fa:16:3e:68:e8:19" dropped privs to tcpdump tcpdump: listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes 21:56:43.669691 26:28:b0:96:c0:c7 > fa:16:3e:68:e8:19, ethertype 802.1Q (0x8100), length 162: vlan 3254, p 0, ethertype IPv6, (flowlabel 0x7cc6b, hlim 255, next-header ICMPv6 (58) payload length: 104) fe80::2428:b0ff:fe96:c0c7 > fe80::f816:3eff:fe68:e819: [icmp6 sum ok] ICMP6, router advertisement, length 104 hop limit 64, Flags [managed], pref medium, router lifetime 6s, reachable time 0ms, retrans timer 0ms prefix info option (3), length 32 (4): 2a05:fc1:200::/64, Flags [onlink, auto], valid time 2592000s, pref. time 14400s rdnss option (25), length 40 (5): lifetime 2s, addr: 2a05:fc1::2 addr: 2a05:fc1::3 source link-address option (1), length 8 (1): 26:28:b0:96:c0:c7 advertisement interval option (7), length 8 (1): 2000ms 0x: fa16 3e68 e819 2628 b096 c0c7 8100 0cb6 ..>h..&( 0x0010: 86dd 6007 cc6b 0068 3aff fe80 ..`..k.h:... 0x0020: 2428 b0ff fe96 c0c7 fe80 ..$( 0x0030: f816 3eff fe68 e819 8600 10d2 4080 >..h..@. 0x0040: 0006 0304 40c0 0027 @..' 0x0050: 8d00 3840 2a05 0fc1 0200 8@*. 0x0060: 1905 0x0070: 0002 2a05 0fc1 ..*. 0x0080: 0002 2a05 0fc1 ..*. 0x0090: 0003 0101 2628 b096 c0c7 0701 &(.. 0x00a0: 07d0 .. 2) Trace the package using "ofproto/trace": #ovs-appctl ofproto/trace br-int in_port=1 fa163e68e8192628b096c0c781000cb686dd6007cc6b00683afffe802428b0fffe96c0c7fe80f8163efffe68e819860010d24086030440c000278d0038402a050fc10200190500022a050fc100022a050fc1000301012628b096c0c7070107d0 Flow: icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,vlan_tci1=0x,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0 bridge("br-int") 0. in_port=1,dl_vlan=3254, priority 3, cookie 0x4ef408376e507615 set_field:4097->vlan_vid goto_table:60 60. dl_vlan=1,dl_dst=fa:16:3e:68:e8:19, priority 90, cookie 0x4ef408376e507615 set_field:0x9e->reg5 set_field:0x1->reg6 pop_vlan resubmit(,81) 81. ct_state=-trk,ipv6,reg5=0x9e, priority 90, cookie 0x4ef408376e507615 ct(table=82,zone=NXM_NX_REG6[0..15]) drop -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 82. -> Sets the packet to an untracked state, and clears all the conntrack fields. Final flow: icmp6,reg5=0x9e,reg6=0x1,in_port=1,vlan_tci=0x,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6b,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=134,icmp_code=0 Megaflow: recirc_id=0,ct_state=-trk,eth,icmp6,in_port=1,dl_vlan=3254,dl_vlan_pcp=0,dl_dst=fa:16:3e:68:e8:19,nw_frag=no,icmp_type=0x86/0xff Datapath actions: pop_vlan,ct(zone=1),recirc(0xc30495) === recirc(0xc30495) - resume conntrack with default ct_state=trk|new (use --ct-next to customize) === Flow: recirc_id=0xc30495,ct_state=new|trk,ct_zone=1,eth,icmp6,reg5=0x9e,reg6=0x1,in_port=1,vlan_tci=0x,dl_src=26:28:b0:96:c0:c7,dl_dst=fa:16:3e:68:e8:19,ipv6_src=fe80::2428:b0ff:fe96:c0c7,ipv6_dst=fe80::f816:3eff:fe68:e819,ipv6_label=0x7cc6
[Yahoo-eng-team] [Bug 2049488] [NEW] [ovn][source-main] job fails with error: too many arguments to function flow_compose
Public bug reported: Fails as:- 2024-01-16 03:02:53.651183 | controller | gcc -DHAVE_CONFIG_H -I. -I ./include -I ./include -I ./ovn -I ./include -I ./lib -I ./lib -I /opt/stack/ovs/include -I /opt/stack/ovs/include -I /opt/stack/ovs/lib -I /opt/stack/ovs/lib -I /opt/stack/ovs -I /opt/stack/ovs-Wstrict-prototypes -Wall -Wextra -Wno-sign-compare -Wpointer-arith -Wformat -Wformat-security -Wswitch-enum -Wunused-parameter -Wbad-function-cast -Wcast-align -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-field-initializers -fno-strict-aliasing -Wswitch-bool -Wlogical-not-parentheses -Wsizeof-array-argument -Wbool-compare -Wshift-negative-value -Wduplicated-cond -Wshadow -Wmultistatement-macros -Wcast-align=strict -g -O2 -MT controller/ovn-controller.o -MD -MP -MF $depbase.Tpo -c -o controller/ovn-controller.o controller/ovn-controller.c &&\ 2024-01-16 03:02:53.651243 | controller | mv -f $depbase.Tpo $depbase.Po 2024-01-16 03:02:53.651272 | controller | controller/ofctrl.c: In function ‘ofctrl_inject_pkt’: 2024-01-16 03:02:53.651946 | controller | controller/ofctrl.c:3048:5: error: too many arguments to function ‘flow_compose’ 2024-01-16 03:02:53.652293 | controller | 3048 | flow_compose(&packet, &uflow, NULL, 64, false); 2024-01-16 03:02:53.652320 | controller | | ^~~~ 2024-01-16 03:02:53.652342 | controller | In file included from /opt/stack/ovs/lib/dp-packet.h:34, 2024-01-16 03:02:53.682892 | controller | from controller/ofctrl.c:21: 2024-01-16 03:02:53.682967 | controller | /opt/stack/ovs/lib/flow.h:129:6: note: declared here 2024-01-16 03:02:53.682996 | controller | 129 | void flow_compose(struct dp_packet *, const struct flow *, 2024-01-16 03:02:53.683024 | controller | | ^~~~ 2024-01-16 03:02:53.683059 | controller | make[1]: *** [Makefile:2325: controller/ofctrl.o] Error 1 2024-01-16 03:03:02.115087 | controller | make[1]: *** Waiting for unfinished jobs 2024-01-16 03:03:02.115190 | controller | make[1]: Leaving directory '/opt/stack/ovn' 2024-01-16 03:03:02.115584 | controller | make: *** [Makefile:1505: all] Error 2 2024-01-16 03:03:02.120930 | controller | + lib/neutron_plugins/ovn_agent:compile_ovn:1 : exit_trap Example failure:- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c5a/periodic/opendev.org/openstack/neutron/master/neutron- ovn-tempest-ipv6-only-ovs-master/c5a8aa0/job-output.txt builds:- https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ipv6-only-ovs-master&job_name=neutron-ovn-tempest-ovs-master-centos-9-stream&skip=0 Broken since https://github.com/ovn- org/ovn/commit/66ef6709678486f7abf88db10eed15fb72edcc4a, need to update OVS_BRANCH in these jobs. ** Affects: neutron Importance: High Status: Triaged ** Tags: ovn ** Changed in: neutron Status: New => Triaged ** Changed in: neutron Importance: Undecided => High ** Tags added: ovn -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2049488 Title: [ovn][source-main] job fails with error: too many arguments to function flow_compose Status in neutron: Triaged Bug description: Fails as:- 2024-01-16 03:02:53.651183 | controller | gcc -DHAVE_CONFIG_H -I. -I ./include -I ./include -I ./ovn -I ./include -I ./lib -I ./lib -I /opt/stack/ovs/include -I /opt/stack/ovs/include -I /opt/stack/ovs/lib -I /opt/stack/ovs/lib -I /opt/stack/ovs -I /opt/stack/ovs-Wstrict-prototypes -Wall -Wextra -Wno-sign-compare -Wpointer-arith -Wformat -Wformat-security -Wswitch-enum -Wunused-parameter -Wbad-function-cast -Wcast-align -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-field-initializers -fno-strict-aliasing -Wswitch-bool -Wlogical-not-parentheses -Wsizeof-array-argument -Wbool-compare -Wshift-negative-value -Wduplicated-cond -Wshadow -Wmultistatement-macros -Wcast-align=strict -g -O2 -MT controller/ovn-controller.o -MD -MP -MF $depbase.Tpo -c -o controller/ovn-controller.o controller/ovn-controller.c &&\ 2024-01-16 03:02:53.651243 | controller | mv -f $depbase.Tpo $depbase.Po 2024-01-16 03:02:53.651272 | controller | controller/ofctrl.c: In function ‘ofctrl_inject_pkt’: 2024-01-16 03:02:53.651946 | controller | controller/ofctrl.c:3048:5: error: too many arguments to function ‘flow_compose’ 2024-01-16 03:02:53.652293 | controller | 3048 | flow_compose(&packet, &uflow, NULL, 64, false); 2024-01-16 03:02:53.652320 | controller | | ^~~~ 2024-01-16 03:02:53.652342 | controller | In file included from /opt/stack/ovs/lib/dp-packet.h:34, 2024-01-16 03:02:53.682892 | controller | from controller/ofctrl.c:21: 2024-01-16 03:02:53.682967 | controller | /opt/stack/ovs/lib/flow.h:129:6: note: declared here 2024-01-16 03:0
[Yahoo-eng-team] [Bug 2051831] [NEW] [ovn] neutron-ovn-tempest-slow job fail tests relying on FIP
Public bug reported: Example failure:- https://2f4a32f753edcd6fd518-38c49964a79149719549049b602122d6.ssl.cf5.rackcdn.com/906628/1/experimental/neutron- ovn-tempest-slow/1b35fb8/testr_results.html Fails as:- Traceback (most recent call last): File "/opt/stack/tempest/tempest/scenario/test_security_groups_basic_ops.py", line 191, in setUp self._deploy_tenant(self.primary_tenant) File "/opt/stack/tempest/tempest/scenario/test_security_groups_basic_ops.py", line 354, in _deploy_tenant self._set_access_point(tenant) File "/opt/stack/tempest/tempest/scenario/test_security_groups_basic_ops.py", line 321, in _set_access_point self._assign_floating_ips(tenant, server) File "/opt/stack/tempest/tempest/scenario/test_security_groups_basic_ops.py", line 325, in _assign_floating_ips floating_ip = self.create_floating_ip( File "/opt/stack/tempest/tempest/scenario/manager.py", line 1132, in create_floating_ip result = client.create_floatingip(**floatingip_kwargs) File "/opt/stack/tempest/tempest/lib/services/network/floating_ips_client.py", line 30, in create_floatingip return self.create_resource(uri, post_data) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 62, in create_resource resp, body = self.post(req_uri, req_post_data) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 300, in post return self.request('POST', url, extra_headers, headers, body, chunked) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 852, in _error_checker raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'type': 'ExternalGatewayForFloatingIPNotFound', 'message': 'External network 43cd92cd-4957-4770-9945-584e8d4da9e3 is not reachable from subnet e354d5ec-5be0-4536-9348-fd819e3f7464. Therefore, cannot associate Port 63869e58-05b8-4a18-be25-77500966df61 with a Floating IP.', 'detail': ''} neutron-server trace:- Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn [None req-1674312b-7443-4d41-bd9b-5f69f5173083 tempest-TestNetworkAdvancedServerOps-1277687553 tempest-TestNetworkAdvancedServerOps-1277687553-project-member] Unable to add router interface to lrouter 31bdd6ba-45cf-45bd-aa3d-907a217ce2a3. Interface info: {'id': '31bdd6ba-45cf-45bd-aa3d-907a217ce2a3', 'tenant_id': 'f7ebd951642c4987ad034d6180f81784', 'port_id': 'f8606824-f658-4634-b9ce-2f8e2ce0d1c3', 'network_id': '2f74ef7b-8374-4f36-a4bf-fbf225b87c48', 'subnet_id': 'd61fb992-2d87-4198-82f1-a687609c3e7c', 'subnet_ids': ['d61fb992-2d87-4198-82f1-a687609c3e7c']}: neutron_lib.exceptions.SubnetNotFound: Subnet a283b869-2e54-44da-b109-06da46933d06 could not be found. Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn Traceback (most recent call last): Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/services/ovn_l3/service_providers/ovn.py", line 122, in _process_add_router_interface Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self.l3plugin._ovn_client.create_router_port(context, router.id, Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1754, in create_router_port Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self._update_lrouter_port(context, router_port, Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1762, in _update_lrouter_port Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn self._get_nets_and_ipv6_ra_confs_for_router_port(context, port)) Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1236, in _get_nets_and_ipv6_ra_confs_for_router_port Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn subnet = self._plugin.get_subnet(context, subnet_id) Jan 26 15:10:35.544611 np0036547416 neutron-server[63495]: ERROR neutron.services.ovn_l3.service_providers.ovn File "/opt/stack/data/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped Jan 26 15:10:35.544611 np0036547416 neutron-server[63
[Yahoo-eng-team] [Bug 2051845] [NEW] [ovn][source-main] job fails with error: too many arguments to function mcast_snooping_add_group4
Public bug reported: Fails as:- 2024-01-31 03:02:06.467122 | controller | controller/pinctrl.c: In function ‘pinctrl_ip_mcast_handle_igmp’: 2024-01-31 03:02:06.468027 | controller | controller/pinctrl.c:5478:54: error: ‘MCAST_GROUP_IGMPV1’ undeclared (first use in this function) 2024-01-31 03:02:06.468683 | controller | 5478 | port_key_data, MCAST_GROUP_IGMPV1); 2024-01-31 03:02:06.469444 | controller | | ^~ 2024-01-31 03:02:06.469472 | controller | controller/pinctrl.c:5478:54: note: each undeclared identifier is reported only once for each function it appears in 2024-01-31 03:02:06.469493 | controller | controller/pinctrl.c:5477:13: error: too many arguments to function ‘mcast_snooping_add_group4’ 2024-01-31 03:02:06.469774 | controller | 5477 | mcast_snooping_add_group4(ip_ms->ms, ip4, IP_MCAST_VLAN, 2024-01-31 03:02:06.469799 | controller | | ^ 2024-01-31 03:02:06.469821 | controller | In file included from controller/ip-mcast.h:19, 2024-01-31 03:02:06.502161 | controller | from controller/pinctrl.c:64: 2024-01-31 03:02:06.502217 | controller | /opt/stack/ovs/lib/mcast-snooping.h:190:6: note: declared here 2024-01-31 03:02:06.502255 | controller | 190 | bool mcast_snooping_add_group4(struct mcast_snooping *ms, ovs_be32 ip4, 2024-01-31 03:02:06.502286 | controller | | ^ 2024-01-31 03:02:06.502324 | controller | controller/pinctrl.c:5483:54: error: ‘MCAST_GROUP_IGMPV2’ undeclared (first use in this function) 2024-01-31 03:02:06.503464 | controller | 5483 | port_key_data, MCAST_GROUP_IGMPV2); 2024-01-31 03:02:06.503491 | controller | | ^~ 2024-01-31 03:02:06.503513 | controller | depbase=`echo controller/local_data.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\ 2024-01-31 03:02:06.503936 | controller | gcc -DHAVE_CONFIG_H -I. -I ./include -I ./include -I ./ovn -I ./include -I ./lib -I ./lib -I /opt/stack/ovs/include -I /opt/stack/ovs/include -I /opt/stack/ovs/lib -I /opt/stack/ovs/lib -I /opt/stack/ovs -I /opt/stack/ovs-Wstrict-prototypes -Wall -Wextra -Wno-sign-compare -Wpointer-arith -Wformat -Wformat-security -Wswitch-enum -Wunused-parameter -Wbad-function-cast -Wcast-align -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-field-initializers -fno-strict-aliasing -Wswitch-bool -Wlogical-not-parentheses -Wsizeof-array-argument -Wbool-compare -Wshift-negative-value -Wduplicated-cond -Wshadow -Wmultistatement-macros -Wcast-align=strict -g -O2 -MT controller/local_data.o -MD -MP -MF $depbase.Tpo -c -o controller/local_data.o controller/local_data.c &&\ 2024-01-31 03:02:06.503963 | controller | mv -f $depbase.Tpo $depbase.Po 2024-01-31 03:02:06.504000 | controller | controller/pinctrl.c:5482:13: error: too many arguments to function ‘mcast_snooping_add_group4’ 2024-01-31 03:02:06.504288 | controller | 5482 | mcast_snooping_add_group4(ip_ms->ms, ip4, IP_MCAST_VLAN, 2024-01-31 03:02:06.504314 | controller | | ^ 2024-01-31 03:02:06.504335 | controller | In file included from controller/ip-mcast.h:19, Example failure:- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3c5/periodic/opendev.org/openstack/neutron/master/neutron- ovn-tempest-ipv6-only-ovs-master/3c5404e/job-output.txt Failing since https://github.com/ovn-org/ovn/commit/dc34b4d9f7f3efb4e7547f9850f6086a7e1a2338 need to update OVS_BRANCH ** Affects: neutron Importance: High Assignee: yatin (yatinkarel) Status: Triaged ** Changed in: neutron Status: New => Triaged ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2051845 Title: [ovn][source-main] job fails with error: too many arguments to function mcast_snooping_add_group4 Status in neutron: Triaged Bug description: Fails as:- 2024-01-31 03:02:06.467122 | controller | controller/pinctrl.c: In function ‘pinctrl_ip_mcast_handle_igmp’: 2024-01-31 03:02:06.468027 | controller | controller/pinctrl.c:5478:54: error: ‘MCAST_GROUP_IGMPV1’ undeclared (first use in this function) 2024-01-31 03:02:06.468683 | controller | 5478 | port_key_data, MCAST_GROUP_IGMPV1); 2024-01-31 03:02:06.469444 | controller | | ^~ 2024-01-31 03:02:06.469472 | controller | cont
[Yahoo-eng-team] [Bug 2052508] [NEW] [doc][troubleshooting] how to enable ovs vswitchd debug logs
Public bug reported: It was raised while evaluating ovs-vswitchd debug logs in CI[1] that we should also document it in neutron docs on steps enabling/disabling debug logs for ovs-vswitchd. [1] https://review.opendev.org/c/openstack/neutron/+/907037 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2052508 Title: [doc][troubleshooting] how to enable ovs vswitchd debug logs Status in neutron: New Bug description: It was raised while evaluating ovs-vswitchd debug logs in CI[1] that we should also document it in neutron docs on steps enabling/disabling debug logs for ovs-vswitchd. [1] https://review.opendev.org/c/openstack/neutron/+/907037 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2052508/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2052509] [NEW] [CI][ovs/ovn sqlalchemy master] jobs broken with Exception: Not enough arguments given
Public bug reported: Fails like:- 2024-02-06 02:28:34.618579 | controller | ++ inc/python:_setup_package_with_constraints_edit:400 : cd /opt/stack/sqlalchemy 2024-02-06 02:28:34.621804 | controller | ++ inc/python:_setup_package_with_constraints_edit:400 : pwd 2024-02-06 02:28:34.624859 | controller | + inc/python:_setup_package_with_constraints_edit:400 : project_dir=/opt/stack/sqlalchemy 2024-02-06 02:28:34.627646 | controller | + inc/python:_setup_package_with_constraints_edit:402 : '[' -n /opt/stack/requirements ']' 2024-02-06 02:28:34.630146 | controller | + inc/python:_setup_package_with_constraints_edit:406 : local name 2024-02-06 02:28:34.633640 | controller | ++ inc/python:_setup_package_with_constraints_edit:407 : awk '/^name.*=/ {print $3}' /opt/stack/sqlalchemy/setup.cfg 2024-02-06 02:28:34.639338 | controller | + inc/python:_setup_package_with_constraints_edit:407 : name= 2024-02-06 02:28:34.641840 | controller | + inc/python:_setup_package_with_constraints_edit:409 : /opt/stack/requirements/.venv/bin/edit-constraints /opt/stack/requirements/upper-constraints.txt -- 2024-02-06 02:28:34.784286 | controller | Traceback (most recent call last): 2024-02-06 02:28:34.784331 | controller | File "/opt/stack/requirements/.venv/bin/edit-constraints", line 10, in 2024-02-06 02:28:34.784449 | controller | sys.exit(main()) 2024-02-06 02:28:34.784484 | controller | File "/opt/stack/requirements/.venv/lib/python3.10/site-packages/openstack_requirements/cmds/edit_constraint.py", line 66, in main 2024-02-06 02:28:34.784629 | controller | _validate_options(options, args) 2024-02-06 02:28:34.784661 | controller | File "/opt/stack/requirements/.venv/lib/python3.10/site-packages/openstack_requirements/cmds/edit_constraint.py", line 47, in _validate_options 2024-02-06 02:28:34.784783 | controller | raise Exception("Not enough arguments given") 2024-02-06 02:28:34.784821 | controller | Exception: Not enough arguments given 2024-02-06 02:28:34.805546 | controller | + inc/python:_setup_package_with_constraints_edit:1 : exit_trap 2024-02-06 02:28:34.808305 | controller | + ./stack.sh:exit_trap:549 : local r=1 Example log:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_22d/periodic/opendev.org/openstack/neutron/master/neutron- ovs-tempest-with-sqlalchemy-master/22dfd7f/job-output.txt Builds:- https://zuul.openstack.org/builds?job_name=neutron-ovs-tempest- with-sqlalchemy-master&job_name=neutron-ovn-tempest-with-sqlalchemy- master Broken since https://github.com/sqlalchemy/sqlalchemy/commit/a8dbf8763a8fa2ca53cc01033f06681a421bf60b ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2052509 Title: [CI][ovs/ovn sqlalchemy master] jobs broken with Exception: Not enough arguments given Status in neutron: New Bug description: Fails like:- 2024-02-06 02:28:34.618579 | controller | ++ inc/python:_setup_package_with_constraints_edit:400 : cd /opt/stack/sqlalchemy 2024-02-06 02:28:34.621804 | controller | ++ inc/python:_setup_package_with_constraints_edit:400 : pwd 2024-02-06 02:28:34.624859 | controller | + inc/python:_setup_package_with_constraints_edit:400 : project_dir=/opt/stack/sqlalchemy 2024-02-06 02:28:34.627646 | controller | + inc/python:_setup_package_with_constraints_edit:402 : '[' -n /opt/stack/requirements ']' 2024-02-06 02:28:34.630146 | controller | + inc/python:_setup_package_with_constraints_edit:406 : local name 2024-02-06 02:28:34.633640 | controller | ++ inc/python:_setup_package_with_constraints_edit:407 : awk '/^name.*=/ {print $3}' /opt/stack/sqlalchemy/setup.cfg 2024-02-06 02:28:34.639338 | controller | + inc/python:_setup_package_with_constraints_edit:407 : name= 2024-02-06 02:28:34.641840 | controller | + inc/python:_setup_package_with_constraints_edit:409 : /opt/stack/requirements/.venv/bin/edit-constraints /opt/stack/requirements/upper-constraints.txt -- 2024-02-06 02:28:34.784286 | controller | Traceback (most recent call last): 2024-02-06 02:28:34.784331 | controller | File "/opt/stack/requirements/.venv/bin/edit-constraints", line 10, in 2024-02-06 02:28:34.784449 | controller | sys.exit(main()) 2024-02-06 02:28:34.784484 | controller | File "/opt/stack/requirements/.venv/lib/python3.10/site-packages/openstack_requirements/cmds/edit_constraint.py", line 66, in main 2024-02-06 02:28:34.784629 | controller | _validate_options(options, args) 2024-02-06 02:28:34.784661 | controller | File "/opt/stack/requirements/.venv/lib/python3.10/site-packages/openstack_requirements/cmds/edit_constraint.py", line 47, in _validate_options 2024-02-06 02:28:34.784783 | controller | raise Exception("Not enough arguments given")
[Yahoo-eng-team] [Bug 2056276] [NEW] unmaintained/yoga periodic jobs broken
Public bug reported: Couple of jobs failing in unmaintained/yoga periodic pipeline. https://zuul.openstack.org/buildsets?project=openstack%2Fneutron&branch=unmaintained%2Fyoga&branch=stable%2Fyoga&pipeline=periodic&skip=0 Before switching to unmaintained/yoga jobs were green:- https://zuul.openstack.org/buildset/eab6ceae73d7437186cb7432e7ad8897 Currently following jobs broken:- - devstack-tobiko-neutron - neutron-ovn-tempest-slow - neutron-ovs-tempest-slow All fails with:- 2024-03-04 02:30:28.791273 | controller | + ./stack.sh:main:230 : SUPPORTED_DISTROS='bullseye|focal|f35|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel9|openEuler-20.03' 2024-03-04 02:30:28.793551 | controller | + ./stack.sh:main:232 : [[ ! jammy =~ bullseye|focal|f35|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel9|openEuler-20.03 ]] 2024-03-04 02:30:28.795963 | controller | + ./stack.sh:main:233 : echo 'WARNING: this script has not been tested on jammy' builds:- https://zuul.openstack.org/builds?job_name=devstack-tobiko-neutron&job_name=neutron-ovn-tempest-slow&job_name=neutron-ovs-tempest-slow&project=openstack%2Fneutron&branch=%09unmaintained%2Fyoga&skip=0 The bug is to track the fixes and also discuss what to do with similar issues going forward in unmaintained branches? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2056276 Title: unmaintained/yoga periodic jobs broken Status in neutron: New Bug description: Couple of jobs failing in unmaintained/yoga periodic pipeline. https://zuul.openstack.org/buildsets?project=openstack%2Fneutron&branch=unmaintained%2Fyoga&branch=stable%2Fyoga&pipeline=periodic&skip=0 Before switching to unmaintained/yoga jobs were green:- https://zuul.openstack.org/buildset/eab6ceae73d7437186cb7432e7ad8897 Currently following jobs broken:- - devstack-tobiko-neutron - neutron-ovn-tempest-slow - neutron-ovs-tempest-slow All fails with:- 2024-03-04 02:30:28.791273 | controller | + ./stack.sh:main:230 : SUPPORTED_DISTROS='bullseye|focal|f35|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel9|openEuler-20.03' 2024-03-04 02:30:28.793551 | controller | + ./stack.sh:main:232 : [[ ! jammy =~ bullseye|focal|f35|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel9|openEuler-20.03 ]] 2024-03-04 02:30:28.795963 | controller | + ./stack.sh:main:233 : echo 'WARNING: this script has not been tested on jammy' builds:- https://zuul.openstack.org/builds?job_name=devstack-tobiko-neutron&job_name=neutron-ovn-tempest-slow&job_name=neutron-ovs-tempest-slow&project=openstack%2Fneutron&branch=%09unmaintained%2Fyoga&skip=0 The bug is to track the fixes and also discuss what to do with similar issues going forward in unmaintained branches? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2056276/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2057492] [NEW] [devstack-tobiko-neutron] periodic broken running on ubuntu focal
Public bug reported: The job is broken in stable/zed and stable/2023.1 where it's running on ubuntu focal since patch https://review.opendev.org/c/x/tobiko/+/910589 Fails like:- 2024-03-12 02:33:20.806847 | controller | interpreter = self.creator.interpreter 2024-03-12 02:33:20.806858 | controller | File "/usr/local/lib/python3.8/dist-packages/tox/tox_env/python/virtual_env/api.py", line 127, in creator 2024-03-12 02:33:20.806868 | controller | return self.session.creator 2024-03-12 02:33:20.806880 | controller | File "/usr/local/lib/python3.8/dist-packages/tox/tox_env/python/virtual_env/api.py", line 108, in session 2024-03-12 02:33:20.806890 | controller | self._virtualenv_session = session_via_cli(env_dir, options=None, setup_logging=False, env=env) 2024-03-12 02:33:20.806900 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/__init__.py", line 49, in session_via_cli 2024-03-12 02:33:20.806912 | controller | parser, elements = build_parser(args, options, setup_logging, env) 2024-03-12 02:33:20.806922 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/__init__.py", line 82, in build_parser 2024-03-12 02:33:20.806932 | controller | CreatorSelector(interpreter, parser), 2024-03-12 02:33:20.806942 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/plugin/creators.py", line 24, in __init__ 2024-03-12 02:33:20.806953 | controller | creators, self.key_to_meta, self.describe, self.builtin_key = self.for_interpreter(interpreter) 2024-03-12 02:33:20.806963 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/plugin/creators.py", line 31, in for_interpreter 2024-03-12 02:33:20.806973 | controller | for key, creator_class in cls.options("virtualenv.create").items(): 2024-03-12 02:33:20.806991 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/plugin/base.py", line 45, in options 2024-03-12 02:33:20.843299 | controller | cls._OPTIONS = cls.entry_points_for(key) 2024-03-12 02:33:20.843338 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/plugin/base.py", line 24, in entry_points_for 2024-03-12 02:33:20.843350 | controller | return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {})) 2024-03-12 02:33:20.843361 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/plugin/base.py", line 24, in 2024-03-12 02:33:20.843495 | controller | return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {})) 2024-03-12 02:33:20.843511 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 210, in load 2024-03-12 02:33:20.843522 | controller | return functools.reduce(getattr, attrs, module) 2024-03-12 02:33:20.843532 | controller | AttributeError: module 'virtualenv.create.via_global_ref.builtin.cpython.mac_os' has no attribute 'CPython2macOsArmFramework' Example failure:- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2c4/periodic/opendev.org/openstack/neutron/stable/2023.1/devstack-tobiko-neutron/2c413ae/job-output.txt Builds:- https://zuul.openstack.org/builds?job_name=devstack-tobiko- neutron&project=openstack%2Fneutron&branch=stable%2Fzed&branch=stable%2F2023.1&skip=0 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2057492 Title: [devstack-tobiko-neutron] periodic broken running on ubuntu focal Status in neutron: New Bug description: The job is broken in stable/zed and stable/2023.1 where it's running on ubuntu focal since patch https://review.opendev.org/c/x/tobiko/+/910589 Fails like:- 2024-03-12 02:33:20.806847 | controller | interpreter = self.creator.interpreter 2024-03-12 02:33:20.806858 | controller | File "/usr/local/lib/python3.8/dist-packages/tox/tox_env/python/virtual_env/api.py", line 127, in creator 2024-03-12 02:33:20.806868 | controller | return self.session.creator 2024-03-12 02:33:20.806880 | controller | File "/usr/local/lib/python3.8/dist-packages/tox/tox_env/python/virtual_env/api.py", line 108, in session 2024-03-12 02:33:20.806890 | controller | self._virtualenv_session = session_via_cli(env_dir, options=None, setup_logging=False, env=env) 2024-03-12 02:33:20.806900 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/__init__.py", line 49, in session_via_cli 2024-03-12 02:33:20.806912 | controller | parser, elements = build_parser(args, options, setup_logging, env) 2024-03-12 02:33:20.806922 | controller | File "/home/zuul/.local/lib/python3.8/site-packages/virtualenv/run/__init
[Yahoo-eng-team] [Bug 2058378] [NEW] [devstack-tobiko-neutron] stable/2023.2 periodic broken with ERROR: Cannot install tox==4.13.0, tox==4.14.0 and tox==4.14.1
Public bug reported: Fails like:- 2024-03-17 03:08:11.011222 | controller | ++ /opt/stack/devstack-plugin-tobiko/devstack/plugin.sh:install_tobiko_deps:14 : pip_install 'tox>=4.13' 2024-03-17 03:08:11.036778 | controller | Using python 3.10 to install tox>=4.13 2024-03-17 03:08:11.040275 | controller | ++ inc/python:pip_install:216 : env http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= /opt/stack/data/venv/bin/pip install -c /opt/stack/requirements/upper-constraints.txt 'tox>=4.13' 2024-03-17 03:08:11.951193 | controller | Looking in indexes: https://mirror.ca-ymq-1.vexxhost.opendev.org/pypi/simple, https://mirror.ca-ymq-1.vexxhost.opendev.org/wheel/ubuntu-22.04-x86_64 2024-03-17 03:08:12.099362 | controller | Collecting tox>=4.13 2024-03-17 03:08:12.100739 | controller | Using cached https://mirror.ca-ymq-1.vexxhost.opendev.org/pypifiles/packages/ed/4c/4c7daf604fe4b136a1e25d41eb7cb2d644d1d8d4d6694eb6ffa7f7dd60cd/tox-4.14.1-py3-none-any.whl.metadata (5.0 kB) 2024-03-17 03:08:12.162454 | controller | INFO: pip is looking at multiple versions of tox to determine which version is compatible with other requirements. This could take a while. 2024-03-17 03:08:12.175333 | controller | Using cached https://mirror.ca-ymq-1.vexxhost.opendev.org/pypifiles/packages/dd/48/57d7f9338669686e5f60794b2291684eb89e49bbf01f695bad651a14e4b5/tox-4.14.0-py3-none-any.whl.metadata (5.0 kB) 2024-03-17 03:08:12.211821 | controller | Using cached https://mirror.ca-ymq-1.vexxhost.opendev.org/pypifiles/packages/14/1a/19c783781c4638bc076f21c76c1c55d2a7bef7381b7c47911e1e65c6e340/tox-4.13.0-py3-none-any.whl.metadata (5.0 kB) 2024-03-17 03:08:12.237786 | controller | ERROR: Cannot install tox==4.13.0, tox==4.14.0 and tox==4.14.1 because these package versions have conflicting dependencies. 2024-03-17 03:08:12.238263 | controller | 2024-03-17 03:08:12.238298 | controller | The conflict is caused by: 2024-03-17 03:08:12.238312 | controller | tox 4.14.1 depends on cachetools>=5.3.2 2024-03-17 03:08:12.238324 | controller | tox 4.14.0 depends on cachetools>=5.3.2 2024-03-17 03:08:12.238343 | controller | tox 4.13.0 depends on cachetools>=5.3.2 2024-03-17 03:08:12.238355 | controller | The user requested (constraint) cachetools===5.3.0 2024-03-17 03:08:12.238366 | controller | 2024-03-17 03:08:12.238377 | controller | To fix this you could try to: 2024-03-17 03:08:12.238391 | controller | 1. loosen the range of package versions you've specified 2024-03-17 03:08:12.238610 | controller | 2. remove package versions to allow pip attempt to solve the dependency conflict 2024-03-17 03:08:12.238646 | controller | 2024-03-17 03:08:12.238666 | controller | ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts 2024-03-17 03:08:12.313244 | controller | + inc/python:pip_install:1 : exit_trap Example failures:- https://a854d101f2ba9a65b7e6-41ecee2e9c4e7ff71d14a9d7f8c78625.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/2023.2/devstack-tobiko-neutron/46ee578/job-output.txt Builds:- https://zuul.openstack.org/builds?job_name=devstack-tobiko- neutron&project=openstack%2Fneutron&branch=stable%2F2023.2&skip=0 Job works fine in another branches, so need to check what's missing for stable/2023.2 Failing since:- https://review.opendev.org/c/x/devstack-plugin- tobiko/+/911092 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2058378 Title: [devstack-tobiko-neutron] stable/2023.2 periodic broken with ERROR: Cannot install tox==4.13.0, tox==4.14.0 and tox==4.14.1 Status in neutron: New Bug description: Fails like:- 2024-03-17 03:08:11.011222 | controller | ++ /opt/stack/devstack-plugin-tobiko/devstack/plugin.sh:install_tobiko_deps:14 : pip_install 'tox>=4.13' 2024-03-17 03:08:11.036778 | controller | Using python 3.10 to install tox>=4.13 2024-03-17 03:08:11.040275 | controller | ++ inc/python:pip_install:216 : env http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= /opt/stack/data/venv/bin/pip install -c /opt/stack/requirements/upper-constraints.txt 'tox>=4.13' 2024-03-17 03:08:11.951193 | controller | Looking in indexes: https://mirror.ca-ymq-1.vexxhost.opendev.org/pypi/simple, https://mirror.ca-ymq-1.vexxhost.opendev.org/wheel/ubuntu-22.04-x86_64 2024-03-17 03:08:12.099362 | controller | Collecting tox>=4.13 2024-03-17 03:08:12.100739 | controller | Using cached https://mirror.ca-ymq-1.vexxhost.opendev.org/pypifiles/packages/ed/4c/4c7daf604fe4b136a1e25d41eb7cb2d644d1d8d4d6694eb6ffa7f7dd60cd/tox-4.14.1-py3-none-any.whl.metadata (5.0 kB) 2024-03-17 03:08:12.162454 | controller | INFO: pip is looking at multiple versions of tox to determine which vers
[Yahoo-eng-team] [Bug 2060029] [NEW] [devstack-tobiko-neutron] stable/2023.1 periodic failing stateless secgroup tests
Public bug reported: One of the test fails like:- testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/x/tobiko/tobiko/tests/scenario/neutron/test_security_groups.py", line 227, in test_security_group_stateful_to_stateless_switch self._check_sg_rules_in_ovn_nb_db(sg, neutron.STATELESS_OVN_ACTION) File "/home/zuul/src/opendev.org/x/tobiko/tobiko/tests/scenario/neutron/test_security_groups.py", line 139, in _check_sg_rules_in_ovn_nb_db self._check_sg_rule_in_ovn_nb_db(sg_rule_id, expected_action) File "/home/zuul/src/opendev.org/x/tobiko/tobiko/tests/scenario/neutron/test_security_groups.py", line 127, in _check_sg_rule_in_ovn_nb_db self._assert_acl_action(acl_rule, expected_action) File "/home/zuul/src/opendev.org/x/tobiko/tobiko/tests/scenario/neutron/test_security_groups.py", line 131, in _assert_acl_action self.assertEqual( File "/home/zuul/src/opendev.org/x/tobiko/.tox/py3/lib/python3.8/site-packages/testtools/testcase.py", line 393, in assertEqual self.assertThat(observed, matcher, message) File "/home/zuul/src/opendev.org/x/tobiko/.tox/py3/lib/python3.8/site-packages/testtools/testcase.py", line 480, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 'allow-stateless' != 'allow-related' Example failure:- https://606e7073949a555d6ce7-a5a94d16cd1e63fdf610099df3afaf88.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/2023.1/devstack- tobiko- neutron/e20c66f/tobiko_results_02_create_neutron_resources_neutron.html?sort=result Builds:- https://zuul.openstack.org/builds?job_name=devstack-tobiko- neutron&project=openstack%2Fneutron&branch=stable%2F2023.1&skip=0 Issue triggered with hhttps://review.opendev.org/c/x/devstack-plugin- tobiko/+/912588 which enabled those tests. 2023.1 jobs running with ubuntu-focal. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2060029 Title: [devstack-tobiko-neutron] stable/2023.1 periodic failing stateless secgroup tests Status in neutron: New Bug description: One of the test fails like:- testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/x/tobiko/tobiko/tests/scenario/neutron/test_security_groups.py", line 227, in test_security_group_stateful_to_stateless_switch self._check_sg_rules_in_ovn_nb_db(sg, neutron.STATELESS_OVN_ACTION) File "/home/zuul/src/opendev.org/x/tobiko/tobiko/tests/scenario/neutron/test_security_groups.py", line 139, in _check_sg_rules_in_ovn_nb_db self._check_sg_rule_in_ovn_nb_db(sg_rule_id, expected_action) File "/home/zuul/src/opendev.org/x/tobiko/tobiko/tests/scenario/neutron/test_security_groups.py", line 127, in _check_sg_rule_in_ovn_nb_db self._assert_acl_action(acl_rule, expected_action) File "/home/zuul/src/opendev.org/x/tobiko/tobiko/tests/scenario/neutron/test_security_groups.py", line 131, in _assert_acl_action self.assertEqual( File "/home/zuul/src/opendev.org/x/tobiko/.tox/py3/lib/python3.8/site-packages/testtools/testcase.py", line 393, in assertEqual self.assertThat(observed, matcher, message) File "/home/zuul/src/opendev.org/x/tobiko/.tox/py3/lib/python3.8/site-packages/testtools/testcase.py", line 480, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 'allow-stateless' != 'allow-related' Example failure:- https://606e7073949a555d6ce7-a5a94d16cd1e63fdf610099df3afaf88.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/2023.1/devstack- tobiko- neutron/e20c66f/tobiko_results_02_create_neutron_resources_neutron.html?sort=result Builds:- https://zuul.openstack.org/builds?job_name=devstack-tobiko- neutron&project=openstack%2Fneutron&branch=stable%2F2023.1&skip=0 Issue triggered with hhttps://review.opendev.org/c/x/devstack-plugin- tobiko/+/912588 which enabled those tests. 2023.1 jobs running with ubuntu-focal. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2060029/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2075151] [NEW] ovs-master jobs failing with lib/actions.c:4549:9: error: struct ofpact_sample has no member named obs_domain_imm; did you mean obs_domain_id?
Public bug reported: Broken since https://github.com/ovn- org/ovn/commit/4ed71c20f53fb7ed58e8a0f993125a70a19619f4 Fails as below:- 2024-07-30 02:40:39.534572 | controller | mv -f $depbase.Tpo $depbase.Plo 2024-07-30 02:40:39.600783 | controller | lib/actions.c: In function ‘encode_SAMPLE’: 2024-07-30 02:40:39.601802 | controller | lib/actions.c:4549:9: error: ‘struct ofpact_sample’ has no member named ‘obs_domain_imm’; did you mean ‘obs_domain_id’? 2024-07-30 02:40:39.601837 | controller | 4549 | os->obs_domain_imm = 2024-07-30 02:40:39.601860 | controller | | ^~ 2024-07-30 02:40:39.601881 | controller | | obs_domain_id 2024-07-30 02:40:39.603037 | controller | lib/actions.c:4553:13: error: ‘struct ofpact_sample’ has no member named ‘obs_point_imm’; did you mean ‘obs_point_id’? 2024-07-30 02:40:39.603071 | controller | 4553 | os->obs_point_imm = ep->lflow_uuid.parts[0]; 2024-07-30 02:40:39.603093 | controller | | ^ 2024-07-30 02:40:39.603113 | controller | | obs_point_id 2024-07-30 02:40:39.604224 | controller | lib/actions.c:4555:13: error: ‘struct ofpact_sample’ has no member named ‘obs_point_imm’; did you mean ‘obs_point_id’? 2024-07-30 02:40:39.604287 | controller | 4555 | os->obs_point_imm = sample->obs_point_id; 2024-07-30 02:40:39.604340 | controller | | ^ 2024-07-30 02:40:39.604368 | controller | | obs_point_id example failure:- https://c33b3f5b3ef18e057f86-4f1ab06e880eb7499aa5249ca5a98f04.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ipv6-only-ovs-master/eac2338/job-output.txt Builds:- https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest- ipv6-only-ovs-master&project=openstack/neutron need to use OVS_BRANCH=branch3.4 to be compatible with OVN_BRANCH=main ** Affects: neutron Importance: High Assignee: yatin (yatinkarel) Status: In Progress ** Changed in: neutron Status: New => Triaged ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2075151 Title: ovs-master jobs failing with lib/actions.c:4549:9: error: struct ofpact_sample has no member named obs_domain_imm; did you mean obs_domain_id? Status in neutron: In Progress Bug description: Broken since https://github.com/ovn- org/ovn/commit/4ed71c20f53fb7ed58e8a0f993125a70a19619f4 Fails as below:- 2024-07-30 02:40:39.534572 | controller | mv -f $depbase.Tpo $depbase.Plo 2024-07-30 02:40:39.600783 | controller | lib/actions.c: In function ‘encode_SAMPLE’: 2024-07-30 02:40:39.601802 | controller | lib/actions.c:4549:9: error: ‘struct ofpact_sample’ has no member named ‘obs_domain_imm’; did you mean ‘obs_domain_id’? 2024-07-30 02:40:39.601837 | controller | 4549 | os->obs_domain_imm = 2024-07-30 02:40:39.601860 | controller | | ^~ 2024-07-30 02:40:39.601881 | controller | | obs_domain_id 2024-07-30 02:40:39.603037 | controller | lib/actions.c:4553:13: error: ‘struct ofpact_sample’ has no member named ‘obs_point_imm’; did you mean ‘obs_point_id’? 2024-07-30 02:40:39.603071 | controller | 4553 | os->obs_point_imm = ep->lflow_uuid.parts[0]; 2024-07-30 02:40:39.603093 | controller | | ^ 2024-07-30 02:40:39.603113 | controller | | obs_point_id 2024-07-30 02:40:39.604224 | controller | lib/actions.c:4555:13: error: ‘struct ofpact_sample’ has no member named ‘obs_point_imm’; did you mean ‘obs_point_id’? 2024-07-30 02:40:39.604287 | controller | 4555 | os->obs_point_imm = sample->obs_point_id; 2024-07-30 02:40:39.604340 | controller | | ^ 2024-07-30 02:40:39.604368 | controller | | obs_point_id example failure:- https://c33b3f5b3ef18e057f86-4f1ab06e880eb7499aa5249ca5a98f04.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ipv6-only-ovs-master/eac2338/job-output.txt Builds:- https://zuul.openstack.org/builds?job_name=neutron-ovn- tempest-ipv6-only-ovs-master&project=openstack/neutron need to use OVS_BRANCH=branch3.4 to be compatible with OVN_BRANCH=main To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2075151/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2076430] [NEW] rally job broken with recent keystone change and fails with ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
Public bug reported: Test fails as:- 2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) 2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last): 2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref 2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess) 2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access 2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session) 2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref 2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs) 2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref 2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers, 2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post 2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs) 2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request 2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url) 2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) >From keyston log:- Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone Traceback (most recent call last): Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.wsgi_app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs) Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__ Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_respons
[Yahoo-eng-team] [Bug 2076430] Re: rally job broken with recent keystone change and fails with ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
<< Testing keystone revert in https://review.opendev.org/c/openstack/neutron/+/925968 The job passed with the revert. << Looks like the issue is with osprofiler being enabled for keystone in the neutron gate, but it's not being enabled in keystone gates, so that's why we didn't see the issue there. @xek yes exactly that's why it got missed. Wouldn't be good to have some coverage in keystone gate as well for this? << So the fix should be to extend oslo_serialization to support date type and not to revert keystone patch. Ok i can push a patch in oslo.serialization, but i also noticed the keystone patch is also being backported so oslo.serialization fixes would also require backports and releases and u-c update and that can take time so if this is the way would be good to hold keystone backports. << It is actually interesting that lots of other functional tests are not failing, that means the job is running some specific setup including osprofiler @gtema it's not doing anything extra than enabling osprofiler, for test i did enabled it in devstack jobs and all those too failed[1][2] with same issue. [1] https://review.opendev.org/c/openstack/devstack/+/926169/1 [2] https://zuul.opendev.org/t/openstack/buildset/b80b78109f344777afe00d02d62ef5fa ** Also affects: oslo.serialization Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2076430 Title: rally job broken with recent keystone change and fails with ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive Status in neutron: New Status in oslo.serialization: In Progress Bug description: Test fails as:- 2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) 2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last): 2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref 2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess) 2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access 2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session) 2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref 2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs) 2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref 2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers, 2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post 2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs) 2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request 2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url) 2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) From keyston log:- Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Canno
[Yahoo-eng-team] [Bug 2052915] Re: "neutron-ovs-grenade-multinode" and "neutron-ovn-grenade-multinode" failing in 2023.1 and Zed
Closing for neutron too as zed moved to unmaintained and we need not run grenade jobs there. ** Changed in: neutron Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2052915 Title: "neutron-ovs-grenade-multinode" and "neutron-ovn-grenade-multinode" failing in 2023.1 and Zed Status in neutron: Won't Fix Status in OpenStack Compute (nova): Won't Fix Bug description: The issue seems to be in the neutron-lib version installed: 2024-02-07 16:19:35.155231 | compute1 | ERROR: neutron 21.2.1.dev38 has requirement neutron-lib>=3.1.0, but you'll have neutron-lib 2.20.2 which is incompatible. That leads to an error when starting the Neutron API (an API definition is not found) [1]: Feb 07 16:13:54.385467 np0036680724 neutron-server[67288]: ERROR neutron ImportError: cannot import name 'port_mac_address_override' from 'neutron_lib.api.definitions' (/usr/local/lib/python3.8/dist-packages/neutron_lib/api/definitions/__init__.py) Setting priority to Critical because that affects to the CI. [1]https://9faad8159db8d6994977-b587eccfce0a645f527dfcbc49e54bb4.ssl.cf2.rackcdn.com/891397/4/check/neutron- ovs-grenade-multinode/ba47cef/controller/logs/screen-q-svc.txt To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2052915/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2011600] Re: functional test_get_datapath_id fails with neutron.common.utils.WaitTimeout: Timed out after 5 seconds
*** This bug is a duplicate of bug 2000163 *** https://bugs.launchpad.net/bugs/2000163 ** This bug has been marked a duplicate of bug 2000163 [FT] Error in "test_get_datapath_id" -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2011600 Title: functional test_get_datapath_id fails with neutron.common.utils.WaitTimeout: Timed out after 5 seconds Status in neutron: Confirmed Bug description: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ec1/876659/3/check/neutron- functional-with-uwsgi/ec1de9e/testr_results.html 2023-03-09 21:37:05.333441 | controller | {3} neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_get_datapath_id [5.136944s] ... FAILED ft1.7: neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_get_datapath_idtesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 745, in wait_until_true eventlet.sleep(sleep) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 36, in sleep hub.switch() File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch return self.greenlet.switch() eventlet.timeout.Timeout: 5 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/test_ovs_lib.py", line 228, in test_get_datapath_id utils.wait_until_true(_check_datapath_id, timeout=5) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 750, in wait_until_true raise WaitTimeout(_("Timed out after %d seconds") % timeout) neutron.common.utils.WaitTimeout: Timed out after 5 seconds To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2011600/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2024160] Re: [OVN][Trunk] subport doesn't reach status ACTIVE
** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2024160 Title: [OVN][Trunk] subport doesn't reach status ACTIVE Status in neutron: Fix Released Status in OpenStack Compute (nova): Invalid Bug description: Test test_live_migration_with_trunk has been failing for the last two days. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3f3/831018/30/check/nova-live-migration/3f3b065/testr_results.html It's a test about live-migration, but it is important to notice that it fails before any live migration happens. The test creates a VM with a port and a subport. The test waits until the VM status is ACTIVE -> this passes The test waits until the subport status is ACTIVE -> this started failing two days ago because the port status is DOWN There was only one neutron patch merged that day[1], but I checked the test failed during some jobs even before that patch was merged. I compared some logs. Neutron logs when the test passes: [2] Neutron logs when the test fails: [3] When it fails, I see this during the creation of the subport (and I don't see this event when it passes): Jun 14 18:13:43.052982 np0034303809 neutron-server[77531]: DEBUG ovsdbapp.backend.ovs_idl.event [None req-929dd199-4247-46f5-9466-622c7d538547 None None] Matched DELETE: PortBindingUpdateVirtualPortsEvent(events=('update', 'delete'), table='Port_Binding', conditions=None, old_conditions=None), priority=20 to row=Port_Binding(parent_port=[], mac=['fa:16:3e:93:9d:5a 19.80.0.42'], chassis=[], ha_chassis_group=[], options={'mcast_flood_reports': 'true', 'requested-chassis': ''}, type=, tag=[], requested_chassis=[], tunnel_key=2, up=[False], logical_port=f8c707ec-ecd8-4f1e-99ba-6f8303b598b2, gateway_chassis=[], encap=[], external_ids={'name': 'tempest-subport-2029248863', 'neutron:cidrs': '19.80.0.42/24', 'neutron:device_id': '', 'neutron:device_owner': '', 'neutron:network_name': 'neutron-5fd9faa7-ec1c-4f42-ab87-6ce19edda245', 'neutron:port_capabilities': '', 'neutron:port_name': 'tempest-subport-2029248863', 'neutron:project_id': '6f92a9f8e16144148026725b25711d3a', 'neutron:revision_n umber': '1', 'neutron:security_group_ids': '5eab41ef-c5c1-425c-a931-f5b6b4b330ad', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': '', 'neutron:vnic_type': 'normal'}, virtual_parent=[], nat_addresses=[], datapath=3c472399-d6ee-4b7c-aa97-6777f2bc2772) old= {{(pid=77531) matches /usr/local/lib/python3.10/dist-packages/ovsdbapp/backend/ovs_idl/event.py:43}} ... Jun 14 18:13:49.597911 np0034303809 neutron-server[77531]: DEBUG neutron.plugins.ml2.plugin [None req-3588521e-7878-408d-b1f8-15db562c69f8 None None] Port f8c707ec-ecd8-4f1e-99ba-6f8303b598b2 cannot update to ACTIVE because it is not bound. {{(pid=77531) _port_provisioned /opt/stack/neutron/neutron/plugins/ml2/plugin.py:361}} It seems the ovn version has changed between these jobs: Passes [4]: 2023-06-14 10:01:46.358875 | controller | Preparing to unpack .../ovn-common_22.03.0-0ubuntu1_amd64.deb ... Fails [5]: 2023-06-14 17:55:07.077377 | controller | Preparing to unpack .../ovn-common_22.03.2-0ubuntu0.22.04.1_amd64.deb ... [1] https://review.opendev.org/c/openstack/neutron/+/883687 [2] https://96b562ba0d2478fe5bc1-d58fbc463536b3122b4367e996d5e5b0.ssl.cf1.rackcdn.com/831018/30/check/nova-live-migration/312c2ab/controller/logs/screen-q-svc.txt [3] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3f3/831018/30/check/nova-live-migration/3f3b065/controller/logs/screen-q-svc.txt [4] https://96b562ba0d2478fe5bc1-d58fbc463536b3122b4367e996d5e5b0.ssl.cf1.rackcdn.com/831018/30/check/nova-live-migration/312c2ab/job-output.txt [5] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3f3/831018/30/check/nova-live-migration/3f3b065/job-output.txt To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2024160/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2007353] Re: Functional test neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalance fails intermittently
** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2007353 Title: Functional test neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalance fails intermittently Status in neutron: Fix Released Bug description: Failure example: https://32c2553a5b1b0d58bd31-9e074609c9fda723118db1a634ce1a12.ssl.cf5.rackcdn.com/873247/3/gate/neutron- functional-with-uwsgi/6195659/testr_results.html Stacktrace: ft1.9: neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalancetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 560, in test_gateway_chassis_rebalance self.assertTrue(self.cr_lrp_pb_event.wait(logical_port), File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue raise self.failureException(msg) AssertionError: False is not true : lrp cr-lrp-77e343ce-8e3c-472d-8eca-acde6196ea38 failed to bind To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2007353/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1789351] [NEW] Glance deployment with python3 + "keystone" paste_deploy flavor Fails
Public bug reported: This happens with oslo.config >= 6.3.0([1]) + python3 + "keystone" paste_deploy + current glance(before https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 it works) Testing in devstack: https://review.openstack.org/#/c/596380/ The glance api service fails to start with below Error, reproducing here: https://review.openstack.org/#/c/596380/:- ERROR: dictionary changed size during iteration , see logs below Failure logs from job:- http://logs.openstack.org/80/596380/2/check /tempest-full- py3/514fa29/controller/logs/screen-g-api.txt.gz#_Aug_27_07_26_10_698243 The Runtime Error is returned at keystonemiddleware:- https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L551 Adding code snippet here:- if self._conf.oslo_conf_obj != cfg.CONF: <-- Fails here oslo_cache.configure(self._conf.oslo_conf_obj) So with pdb found that an additional key(fatal_deprecations) was added to cfg.CONF at ^^, so Error is returned in python3. With python2 same key is added but no Error. There are multiple ways to avoid it, like use the paste_deploy configuration that works(ex: keystone+cachemanagement), use oslo.config <= 6.2.0, Use python2 or update glance(https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 as use_user_token is deprecated since long) with keystone+cachemanagement, all the config items were added before reaching the Failure point in keystonemiddleware and self._conf.oslo_conf_obj != cfg.CONF didn't raised an error and returned Boolean. Don't know why. But it seems a real issue to me as it may happen in python3 at different places. So it would be good if Teams from affected projects(oslo.config, keystonemiddleware, glance) can look at it and fix(not avoid) at the best place. To me it looks like keystonemiddleware is not handling(comparing the dict) it properly for python3, as the conf is dynamically updated(how ? and when ?). - so can oslo.config Team check if glance and keystonmiddleware are handling/using oslo.config properly. - i checked keystone+cachemanagement is default in devstack from last 6 years, is "keystone" flavor supported? if yes it should be fixed. Also it would be good to cleanup the deprecated options those are deprecated since Mitaka. - If it's wrongly used in keystonemiddleware/glance, it would be good to fix there. Initially detected while testing with Fedora[2], but later digged on why it's working in CI with Ubuntu and started [3]. [1] https://review.openstack.org/#/c/560094/ [2] https://review.rdoproject.org/r/#/c/14921/ [3] https://review.openstack.org/#/c/596380/ ** Affects: glance Importance: Undecided Status: New ** Affects: keystonemiddleware Importance: Undecided Status: New ** Affects: oslo.config Importance: Undecided Status: New ** Also affects: oslo.config Importance: Undecided Status: New ** Also affects: keystonemiddleware Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1789351 Title: Glance deployment with python3 + "keystone" paste_deploy flavor Fails Status in Glance: New Status in keystonemiddleware: New Status in oslo.config: New Bug description: This happens with oslo.config >= 6.3.0([1]) + python3 + "keystone" paste_deploy + current glance(before https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 it works) Testing in devstack: https://review.openstack.org/#/c/596380/ The glance api service fails to start with below Error, reproducing here: https://review.openstack.org/#/c/596380/:- ERROR: dictionary changed size during iteration , see logs below Failure logs from job:- http://logs.openstack.org/80/596380/2/check /tempest-full- py3/514fa29/controller/logs/screen-g-api.txt.gz#_Aug_27_07_26_10_698243 The Runtime Error is returned at keystonemiddleware:- https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L551 Adding code snippet here:- if self._conf.oslo_conf_obj != cfg.CONF: <-- Fails here oslo_cache.configure(self._conf.oslo_conf_obj) So with pdb found that an additional key(fatal_deprecations) was added to cfg.CONF at ^^, so Error is returned in python3. With python2 same key is added but no Error. There are multiple ways to avoid it, like use the paste_deploy configuration that works(ex: keystone+cachemanagement), use oslo.config <= 6.2.0, Use python2 or update glance(https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 as use_user_token is deprecated since long) with keystone+cachemanagement, all the config items were added before reaching the Failure point in keystonemiddleware and self._conf.oslo_conf_obj != cfg.CONF didn't raised an error and returned B
[Yahoo-eng-team] [Bug 1808951] [NEW] python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object
Public bug reported: Description:- So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:- 2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last): 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERR
[Yahoo-eng-team] [Bug 1808975] [NEW] python3 + Fedora + SSL + nova compute RecursionError: maximum recursion depth exceeded while calling a Python object
Public bug reported: Description:- While Testing python3 Fedora deployment for nova in [1] got below Recursion Error in nova-compute:- 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager [req-f908a9e0-e77a-4d35-9266-fc5e8d79dfde - - - - -] Error updating resources for node rdo-fedora-stable-rdo-cloud-358855.: RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager Traceback (most recent call last): 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7690, in _update_available_resource_for_node 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager rt.update_available_resource(context, nodename, startup=startup) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 738, in update_available_resource 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return f(*args, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 794, in _update_available_resource 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager instance_by_uuid) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1256, in _remove_deleted_instances_allocations 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager context, cn.uuid) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 2165, in get_allocations_for_resource_provider 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = self.get(url, global_request_id=context.global_id) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 297, in get 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self._client.get(url, microversion=version, headers=headers) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.session.request(url, method, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 684, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager auth_headers = self.get_auth_headers(auth) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return auth.get_headers(self, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager token = self.get_token(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.get_access(session).auth_token 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self.auth_ref = self.get_auth_ref(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self._plugin = self._do_create_plugin(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager authenticated=False) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager authenticated=authenticated) 2018-12-18 08:00
[Yahoo-eng-team] [Bug 1748399] [NEW] Config generation failing after recent patch
Public bug reported: tox -egenconfig failing after https://review.openstack.org/#/c/541740/ with Error:- Traceback (most recent call last): File "/root/glance/.tox/genconfig/bin/oslo-config-generator", line 11, in sys.exit(main()) File "/root/glance/.tox/genconfig/lib/python2.7/site-packages/oslo_config/generator.py", line 770, in main generate(conf) File "/root/glance/.tox/genconfig/lib/python2.7/site-packages/oslo_config/generator.py", line 731, in generate groups = _get_groups(_list_opts(conf.namespace)) File "/root/glance/.tox/genconfig/lib/python2.7/site-packages/oslo_config/generator.py", line 489, in _list_opts for group, group_opts in loader(): File "/root/glance/glance/opts.py", line 165, in list_image_import_opts return [(g, copy.deepcopy(o)) for g, o in opts] TypeError: 'NoneType' object is not iterable ERROR: InvocationError: '/root/glance/.tox/genconfig/bin/oslo-config-generator --config-file etc/oslo-config-generator/glance-image-import.conf' summary _ ERROR: genconfig: commands failed Review https://review.openstack.org/#/c/541740/ wrongly used "extend" method in image_import_opts. ** Affects: glance Importance: Critical Assignee: yatin (yatinkarel) Status: Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1748399 Title: Config generation failing after recent patch Status in Glance: Confirmed Bug description: tox -egenconfig failing after https://review.openstack.org/#/c/541740/ with Error:- Traceback (most recent call last): File "/root/glance/.tox/genconfig/bin/oslo-config-generator", line 11, in sys.exit(main()) File "/root/glance/.tox/genconfig/lib/python2.7/site-packages/oslo_config/generator.py", line 770, in main generate(conf) File "/root/glance/.tox/genconfig/lib/python2.7/site-packages/oslo_config/generator.py", line 731, in generate groups = _get_groups(_list_opts(conf.namespace)) File "/root/glance/.tox/genconfig/lib/python2.7/site-packages/oslo_config/generator.py", line 489, in _list_opts for group, group_opts in loader(): File "/root/glance/glance/opts.py", line 165, in list_image_import_opts return [(g, copy.deepcopy(o)) for g, o in opts] TypeError: 'NoneType' object is not iterable ERROR: InvocationError: '/root/glance/.tox/genconfig/bin/oslo-config-generator --config-file etc/oslo-config-generator/glance-image-import.conf' summary _ ERROR: genconfig: commands failed Review https://review.openstack.org/#/c/541740/ wrongly used "extend" method in image_import_opts. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1748399/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1765008] Re: Tempest API tests failing for stable/queens branch
The test is failing in tripleo as well:- master:- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/63a5b7a/undercloud/home/jenkins/tempest/tempest.html.gz Queens:- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/a1bc8c7/undercloud/home/jenkins/tempest/tempest.html.gz https://review.openstack.org/#/c/562324/ should fix the issue. ** Also affects: tripleo Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1765008 Title: Tempest API tests failing for stable/queens branch Status in neutron: In Progress Status in tripleo: New Bug description: neutron-tempest-plugin repo is branchless so same tests are running for master and stable/queens neutron code. Recently there was new test merged: https://review.openstack.org/#/c/558609/ and this cause problem like: http://logs.openstack.org/47/562147/1/check/neutron-tempest-plugin-api/5596592/logs/testr_results.html.gz for stable branches To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1765008/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1768498] [NEW] config generation(yaml/json) broken with python3
Public bug reported: config generation with format yaml/json is broken with python3 Reproduction steps:- git clone https://github.com/openstack/keystone cd keystone sed -i '/\[testenv\:genconfig\]/a basepython=python3' tox.ini tox -egenconfig .tox/genconfig/bin/oslo-config-generator --config-file=config-generator/keystone.conf --format yaml --output-file=keystone-schema.yaml The above command works fine when basepython=python2 with "ini" it works fine:- .tox/genconfig/bin/oslo-config-generator --config-file=config-generator/keystone.conf --format ini --output-file=keystone-schema.yaml Actual Output:- Traceback (most recent call last): File ".tox/genconfig/bin/oslo-config-generator", line 11, in sys.exit(main()) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 770, in main generate(conf) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 754, in generate conf=conf) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 690, in _output_machine_readable output_data = _generate_machine_readable_data(groups, conf) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 655, in _generate_machine_readable_data entry = _build_entry(opt, group_name, namespace[0], conf) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 601, in _build_entry entry['type'] = _format_type_name(entry['type']) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 157, in _format_type_name return _TYPE_NAMES[opt_type] TypeError: unhashable type: 'HostAddress' Expected Output:- keystone-schema.yaml should be generated This should be a bug for oslo.config, but found in keystone so reporting against it as well. ** Affects: keystone Importance: Undecided Status: New ** Affects: oslo.config Importance: Undecided Status: New ** Also affects: oslo.config Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1768498 Title: config generation(yaml/json) broken with python3 Status in OpenStack Identity (keystone): New Status in oslo.config: New Bug description: config generation with format yaml/json is broken with python3 Reproduction steps:- git clone https://github.com/openstack/keystone cd keystone sed -i '/\[testenv\:genconfig\]/a basepython=python3' tox.ini tox -egenconfig .tox/genconfig/bin/oslo-config-generator --config-file=config-generator/keystone.conf --format yaml --output-file=keystone-schema.yaml The above command works fine when basepython=python2 with "ini" it works fine:- .tox/genconfig/bin/oslo-config-generator --config-file=config-generator/keystone.conf --format ini --output-file=keystone-schema.yaml Actual Output:- Traceback (most recent call last): File ".tox/genconfig/bin/oslo-config-generator", line 11, in sys.exit(main()) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 770, in main generate(conf) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 754, in generate conf=conf) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 690, in _output_machine_readable output_data = _generate_machine_readable_data(groups, conf) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 655, in _generate_machine_readable_data entry = _build_entry(opt, group_name, namespace[0], conf) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 601, in _build_entry entry['type'] = _format_type_name(entry['type']) File "/home/ykarel/work/keystone/.tox/genconfig/lib/python3.6/site-packages/oslo_config/generator.py", line 157, in _format_type_name return _TYPE_NAMES[opt_type] TypeError: unhashable type: 'HostAddress' Expected Output:- keystone-schema.yaml should be generated This should be a bug for oslo.config, but found in keystone so reporting against it as well. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1768498/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1742648] [NEW] Doc build(both html and man) failing with sphinx 1.6.6
Public bug reported: There seems to be some issue when building docs with sphinx 1.6.6. Current u-c for sphinx is 1.6.5. Filing bug so that it can be fixed in keystone(if it's a keystone issue) or sphinx==1.6.6 can be blacklisted if issue is confirmed in sphinx. Reproduction steps:- git clone https://github.com/openstack/keystone cd keystone tox -edocs #SUCCESS .tox/docs/bin/pip install sphinx==1.6.6 tox -edocs#FAILS with below Error Expected result: Doc build should pass Actual result: Doc build fails ERROR log:- looking for now-outdated files... none found pickling environment... done checking consistency... Warning, treated as error: /home/ykarel/work/keystone/doc/source/api/keystone.assignment.backends.rst:document isn't included in any toctree ERROR: InvocationError: '/home/ykarel/work/keystone/.tox/docs/bin/python setup.py build_sphinx' ** Affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1742648 Title: Doc build(both html and man) failing with sphinx 1.6.6 Status in OpenStack Identity (keystone): New Bug description: There seems to be some issue when building docs with sphinx 1.6.6. Current u-c for sphinx is 1.6.5. Filing bug so that it can be fixed in keystone(if it's a keystone issue) or sphinx==1.6.6 can be blacklisted if issue is confirmed in sphinx. Reproduction steps:- git clone https://github.com/openstack/keystone cd keystone tox -edocs #SUCCESS .tox/docs/bin/pip install sphinx==1.6.6 tox -edocs#FAILS with below Error Expected result: Doc build should pass Actual result: Doc build fails ERROR log:- looking for now-outdated files... none found pickling environment... done checking consistency... Warning, treated as error: /home/ykarel/work/keystone/doc/source/api/keystone.assignment.backends.rst:document isn't included in any toctree ERROR: InvocationError: '/home/ykarel/work/keystone/.tox/docs/bin/python setup.py build_sphinx' To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1742648/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1906500] [NEW] [ovn] Tempest tests failing while creating security group driver with KeyError: 'remote_address_group_id'
Public bug reported: This is failing post https://review.opendev.org/c/openstack/neutron/+/751110, detected in a packstack job, it fails with below Traceback:- 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource Traceback (most recent call last): 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource result = method(request=request, **args) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 437, in create 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise() 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise() 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource self.force_reraise() 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource raise value 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 547, in _create 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource objs = do_create(body, bulk=True) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 543, in do_create 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource request.context, reservation.reservation_id) 2020-12-01 13:05:53.257 85629 ERROR neutron.api.v2.resource File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", li
[Yahoo-eng-team] [Bug 1939137] [NEW] ovn + log service plugin reports AttributeError: 'NoneType' object has no attribute 'tenant_id'
Public bug reported: Originally noticed in a TripleO job[1], and after enabling log service plugin in devstack seeing the similar error in neutron service log. Following Traceback is seen:- ERROR neutron_lib.callbacks.manager [None req-131d215c-1d03-48ce-a16e-28175c0f58ba tempest-DefaultSnatToExternal-1368745793 tempest-DefaultSnatToExternal-1368745793-project] Error during notification for neutron.services.logapi.common.sg_callback.SecurityGroupRuleCallBack.handle_event-423586 security_group_rule, after_create: AttributeError: 'NoneType' object has no attribute 'tenant_id' Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/services/logapi/common/sg_callback.py", line 32, in handle_event Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager log_resources = db_api.get_logs_bound_sg(context, sg_id) Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/services/logapi/common/db_api.py", line 186, in get_logs_bound_sg Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager project_id = context.tenant_id Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager AttributeError: 'NoneType' object has no attribute 'tenant_id' Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager Example log:- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c84/803712/1/check/neutron-tempest-plugin-scenario-ovn/c84b228/controller/logs/screen-q-svc.txt https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_65b/797823/16/check/tripleo-ci-centos-8-standalone/65b6831/logs/undercloud/var/log/containers/neutron/server.log The support was added as part of https://bugs.launchpad.net/neutron/+bug/1914757 Test patch:- https://review.opendev.org/c/openstack/neutron-tempest- plugin/+/803712 ** Affects: neutron Importance: Undecided Assignee: Kamil Sambor (ksambor) Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1939137 Title: ovn + log service plugin reports AttributeError: 'NoneType' object has no attribute 'tenant_id' Status in neutron: New Bug description: Originally noticed in a TripleO job[1], and after enabling log service plugin in devstack seeing the similar error in neutron service log. Following Traceback is seen:- ERROR neutron_lib.callbacks.manager [None req-131d215c-1d03-48ce-a16e-28175c0f58ba tempest-DefaultSnatToExternal-1368745793 tempest-DefaultSnatToExternal-1368745793-project] Error during notification for neutron.services.logapi.common.sg_callback.SecurityGroupRuleCallBack.handle_event-423586 security_group_rule, after_create: AttributeError: 'NoneType' object has no attribute 'tenant_id' Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/services/logapi/common/sg_callback.py", line 32, in handle_event Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager log_resources = db_api.get_logs_bound_sg(context, sg_id) Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/services/logapi/common/db_api.py", line 186,
[Yahoo-eng-team] [Bug 1939137] Re: ovn + log service plugin reports AttributeError: 'NoneType' object has no attribute 'tenant_id'
Seems it's fixed with https://github.com/openstack/neutron/commit/7f063223553a18345891bf42e88989edb67038e7, no longer seeing issue in TripleO job:- https://d0e3fd572414115f797b-39524de8c5a1fb89d206195b6f692473.ssl.cf5.rackcdn.com/808056/1/check/tripleo- ci- centos-8-standalone/6cb888b/logs/undercloud/var/log/containers/neutron/server.log ** Changed in: neutron Status: Triaged => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1939137 Title: ovn + log service plugin reports AttributeError: 'NoneType' object has no attribute 'tenant_id' Status in neutron: Fix Released Bug description: Originally noticed in a TripleO job[1], and after enabling log service plugin in devstack seeing the similar error in neutron service log. Following Traceback is seen:- ERROR neutron_lib.callbacks.manager [None req-131d215c-1d03-48ce-a16e-28175c0f58ba tempest-DefaultSnatToExternal-1368745793 tempest-DefaultSnatToExternal-1368745793-project] Error during notification for neutron.services.logapi.common.sg_callback.SecurityGroupRuleCallBack.handle_event-423586 security_group_rule, after_create: AttributeError: 'NoneType' object has no attribute 'tenant_id' Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/services/logapi/common/sg_callback.py", line 32, in handle_event Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager log_resources = db_api.get_logs_bound_sg(context, sg_id) Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/services/logapi/common/db_api.py", line 186, in get_logs_bound_sg Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager project_id = context.tenant_id Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager AttributeError: 'NoneType' object has no attribute 'tenant_id' Aug 06 09:28:02.985925 ubuntu-focal-airship-kna1-0025793663 neutron-server[107737]: ERROR neutron_lib.callbacks.manager Example log:- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c84/803712/1/check/neutron-tempest-plugin-scenario-ovn/c84b228/controller/logs/screen-q-svc.txt https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_65b/797823/16/check/tripleo-ci-centos-8-standalone/65b6831/logs/undercloud/var/log/containers/neutron/server.log The support was added as part of https://bugs.launchpad.net/neutron/+bug/1914757 Test patch:- https://review.opendev.org/c/openstack/neutron-tempest- plugin/+/803712 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1939137/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1952393] [NEW] [master] neutron-tempest-plugin-scenario-ovn broken with "ovn-northd did not start"
Public bug reported: Job failing consistently with below error since https://review.opendev.org/c/openstack/devstack/+/806858:- 2021-11-26 05:58:40.377912 | controller | + functions-common:test_with_retry:2339: timeout 60 sh -c 'while ! test -e /usr/local/var/run/openvswitch/ovn-northd.pid; do sleep 1; done' 2021-11-26 05:59:40.383253 | controller | + functions-common:test_with_retry:2340: die 2340 'ovn-northd did not start' 2021-11-26 05:59:40.386420 | controller | + functions-common:die:253 : local exitcode=0 Nov 26 05:58:40.329669 ubuntu-focal-inmotion-iad3-0027510462 bash[107881]: * Creating empty database /opt/stack/data/ovn/ovnsb_db.db Nov 26 05:58:40.330503 ubuntu-focal-inmotion-iad3-0027510462 bash[107922]: chown: changing ownership of '/opt/stack/data/ovn': Operation not permitted Nov 26 05:58:40.331416 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.pid': Operation not permitted Nov 26 05:58:40.331416 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.106684.ctl': Operation not permitted Nov 26 05:58:40.331647 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.mgmt': Operation not permitted Nov 26 05:58:40.331647 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.107192.ctl': Operation not permitted Example logs:- https://780a11778aeb29655ec5-5f07f27cfda9f7663453c94db6894b0a.ssl.cf5.rackcdn.com/818443/7/check/neutron-tempest-plugin-scenario-ovn/642b702/job-output.txt https://780a11778aeb29655ec5-5f07f27cfda9f7663453c94db6894b0a.ssl.cf5.rackcdn.com/818443/7/check/neutron-tempest-plugin-scenario-ovn/642b702/controller/logs/screen-ovn-northd.txt Job Builds:- https://zuul.openstack.org/builds?job_name=neutron-tempest- plugin-scenario-ovn Other ovn jobs which uses OVN_BUILD_FROM_SOURCE=True will also be impacted. So need to fix affected cases by https://review.opendev.org/c/openstack/devstack/+/806858. ** Affects: neutron Importance: Undecided Status: New ** Tags: gate-failure ovn ** Tags added: gate-failure ovn -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1952393 Title: [master] neutron-tempest-plugin-scenario-ovn broken with "ovn-northd did not start" Status in neutron: New Bug description: Job failing consistently with below error since https://review.opendev.org/c/openstack/devstack/+/806858:- 2021-11-26 05:58:40.377912 | controller | + functions-common:test_with_retry:2339: timeout 60 sh -c 'while ! test -e /usr/local/var/run/openvswitch/ovn-northd.pid; do sleep 1; done' 2021-11-26 05:59:40.383253 | controller | + functions-common:test_with_retry:2340: die 2340 'ovn-northd did not start' 2021-11-26 05:59:40.386420 | controller | + functions-common:die:253 : local exitcode=0 Nov 26 05:58:40.329669 ubuntu-focal-inmotion-iad3-0027510462 bash[107881]: * Creating empty database /opt/stack/data/ovn/ovnsb_db.db Nov 26 05:58:40.330503 ubuntu-focal-inmotion-iad3-0027510462 bash[107922]: chown: changing ownership of '/opt/stack/data/ovn': Operation not permitted Nov 26 05:58:40.331416 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.pid': Operation not permitted Nov 26 05:58:40.331416 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.106684.ctl': Operation not permitted Nov 26 05:58:40.331647 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.mgmt': Operation not permitted Nov 26 05:58:40.331647 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.107192.ctl': Operation not permitted Example logs:- https://780a11778aeb29655ec5-5f07f27cfda9f7663453c94db6894b0a.ssl.cf5.rackcdn.com/818443/7/check/neutron-tempest-plugin-scenario-ovn/642b702/job-output.txt https://780a11778aeb29655ec5-5f07f27cfda9f7663453c94db6894b0a.ssl.cf5.rackcdn.com/818443/7/check/neutron-tempest-plugin-scenario-ovn/642b702/controller/logs/screen-ovn-northd.txt Job Builds:- https://zuul.openstack.org/builds?job_name=neutron- tempest-plugin-scenario-ovn Other ovn jobs which uses OVN_BUILD_FROM_SOURCE=True will also be impacted. So need to fix affected cases by https://review.opendev.org/c/openstack/devstack/+/806858. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1952393/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-t
[Yahoo-eng-team] [Bug 1952393] Re: [OVN] neutron-tempest-plugin-scenario-ovn broken with "ovn-northd did not start"
** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1952393 Title: [OVN] neutron-tempest-plugin-scenario-ovn broken with "ovn-northd did not start" Status in devstack: Fix Released Status in neutron: Fix Released Bug description: Job failing consistently with below error since https://review.opendev.org/c/openstack/devstack/+/806858:- 2021-11-26 05:58:40.377912 | controller | + functions-common:test_with_retry:2339: timeout 60 sh -c 'while ! test -e /usr/local/var/run/openvswitch/ovn-northd.pid; do sleep 1; done' 2021-11-26 05:59:40.383253 | controller | + functions-common:test_with_retry:2340: die 2340 'ovn-northd did not start' 2021-11-26 05:59:40.386420 | controller | + functions-common:die:253 : local exitcode=0 Nov 26 05:58:40.329669 ubuntu-focal-inmotion-iad3-0027510462 bash[107881]: * Creating empty database /opt/stack/data/ovn/ovnsb_db.db Nov 26 05:58:40.330503 ubuntu-focal-inmotion-iad3-0027510462 bash[107922]: chown: changing ownership of '/opt/stack/data/ovn': Operation not permitted Nov 26 05:58:40.331416 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.pid': Operation not permitted Nov 26 05:58:40.331416 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovsdb-server.106684.ctl': Operation not permitted Nov 26 05:58:40.331647 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/br-ex.mgmt': Operation not permitted Nov 26 05:58:40.331647 ubuntu-focal-inmotion-iad3-0027510462 bash[107923]: chown: changing ownership of '/usr/local/var/run/openvswitch/ovs-vswitchd.107192.ctl': Operation not permitted Example logs:- https://780a11778aeb29655ec5-5f07f27cfda9f7663453c94db6894b0a.ssl.cf5.rackcdn.com/818443/7/check/neutron-tempest-plugin-scenario-ovn/642b702/job-output.txt https://780a11778aeb29655ec5-5f07f27cfda9f7663453c94db6894b0a.ssl.cf5.rackcdn.com/818443/7/check/neutron-tempest-plugin-scenario-ovn/642b702/controller/logs/screen-ovn-northd.txt Job Builds:- https://zuul.openstack.org/builds?job_name=neutron- tempest-plugin-scenario-ovn Other ovn jobs which uses OVN_BUILD_FROM_SOURCE=True will also be impacted. So need to fix affected cases by https://review.opendev.org/c/openstack/devstack/+/806858. To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1952393/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1957021] Re: openstack xena documenation has wrong neutron service name to start
Seems you following wrong documentation for CentOS Stream. You need to refer https://docs.openstack.org/neutron/xena/install/install-rdo.html instead of https://docs.openstack.org/neutron/xena/install/install- obs.html, The later one is for OpenSUSE. Marking the bug as invalid. https://docs.openstack.org/neutron/xena/install/install-rdo.html correctly describes the service names on CentOS Stream. https://docs.openstack.org/neutron/xena/install/ describes installation method for various distros. Marking the bug as invalid, feel free to re open if wrong. ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1957021 Title: openstack xena documenation has wrong neutron service name to start Status in neutron: Invalid Bug description: The manual install for neturon says to start - https://docs.openstack.org/neutron/xena/install/controller-install-obs.html# # systemctl enable openstack-neutron.service \ openstack-neutron-linuxbridge-agent.service \ openstack-neutron-dhcp-agent.service \ openstack-neutron-metadata-agent.service # systemctl start openstack-neutron.service \ openstack-neutron-linuxbridge-agent.service \ openstack-neutron-dhcp-agent.service \ openstack-neutron-metadata-agent.service But Xena neutron on centOS stream installs as [root@controller neutron]# cd /usr/lib/systemd/system [root@controller system]# ls -ltra neutron* -rw-r--r--. 1 root root 569 Oct 6 16:45 neutron-server.service -rw-r--r--. 1 root root 1024 Oct 6 16:45 neutron-ovs-cleanup.service -rw-r--r--. 1 root root 734 Oct 6 16:45 neutron-openvswitch-agent.service -rw-r--r--. 1 root root 987 Oct 6 16:45 neutron-netns-cleanup.service -rw-r--r--. 1 root root 536 Oct 6 16:45 neutron-metadata-agent.service -rw-r--r--. 1 root root 1039 Oct 6 16:45 neutron-linuxbridge-cleanup.service -rw-r--r--. 1 root root 645 Oct 6 16:45 neutron-linuxbridge-agent.service -rw-r--r--. 1 root root 512 Oct 6 16:45 neutron-l3-agent.service -rw-r--r--. 1 root root 516 Oct 6 16:45 neutron-dhcp-agent.service -rw-r--r--. 1 root root 579 Oct 6 16:50 neutron-destroy-patch-ports.service so the document has to be updated with the proper services to start. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1957021/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1958961] [NEW] [ovn-octavia-provider] lb create failing with with ValueError: invalid literal for int() with base 10: '24 2001:db8::131/64'
Public bug reported: When deployed with octavia-ovn-provider with below local.conf, loadbalancer create(openstack loadbalancer create --vip-network-id public --provider ovn) goes into ERROR state. >From o-api logs:- ERROR ovn_octavia_provider.helper Traceback (most recent call last): ERROR ovn_octavia_provider.helper File "/usr/local/lib/python3.8/dist-packages/netaddr/ip/__init__.py", line 811, in parse_ip_network ERROR ovn_octavia_provider.helper prefixlen = int(val2) ERROR ovn_octavia_provider.helper ValueError: invalid literal for int() with base 10: '24 2001:db8::131/64' Seems regression caused after https://review.opendev.org/c/openstack/ovn-octavia-provider/+/816868. # Logical switch ports output sudo ovn-nbctl find logical_switch_port type=router _uuid : 4865f50c-a2cd-4a5c-ae4a-bbc911985fb2 addresses : [router] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids: {"neutron:cidrs"="172.24.4.149/24 2001:db8::131/64", "neutron:device_id"="31a0e24f-6278-4714-b543-cba735a6c49d", "neutron:device_owner"="network:router_gateway", "neutron:network_name"=neutron-4708e992-cff8-4438-8142-1cc2ac7010db, "neutron:port_name"="", "neutron:project_id"="", "neutron:revision_number"="6", "neutron:security_group_ids"=""} ha_chassis_group: [] name: "c18869b9--49a8-bc8a-5d2c51db5b6e" options : {mcast_flood_reports="true", nat-addresses=router, requested-chassis=ykarel-devstack, router-port=lrp-c18869b9--49a8-bc8a-5d2c51db5b6e} parent_name : [] port_security : [] tag : [] tag_request : [] type: router up : true _uuid : f0ed6566-a942-4e2d-94f5-64ccd6bed568 addresses : [router] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids: {"neutron:cidrs"="fd25:38d5:1d9::1/64", "neutron:device_id"="31a0e24f-6278-4714-b543-cba735a6c49d", "neutron:device_owner"="network:router_interface", "neutron:network_name"=neutron-591d2b8c-3501-49b1-822c-731f2cc9b305, "neutron:port_name"="", "neutron:project_id"=f4c9948020024e13a1a091bd09d1fbba, "neutron:revision_number"="3", "neutron:security_group_ids"=""} ha_chassis_group: [] name: "e778ac75-a15b-441b-b334-6a7579f851fa" options : {router-port=lrp-e778ac75-a15b-441b-b334-6a7579f851fa} parent_name : [] port_security : [] tag : [] tag_request : [] type: router up : true _uuid : 9c2f3327-ac94-4881-a9c5-a6da87acf6a3 addresses : [router] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids: {"neutron:cidrs"="10.0.0.1/26", "neutron:device_id"="31a0e24f-6278-4714-b543-cba735a6c49d", "neutron:device_owner"="network:router_interface", "neutron:network_name"=neutron-591d2b8c-3501-49b1-822c-731f2cc9b305, "neutron:port_name"="", "neutron:project_id"=f4c9948020024e13a1a091bd09d1fbba, "neutron:revision_number"="3", "neutron:security_group_ids"=""} ha_chassis_group: [] name: "d728e2a3-f9fd-4fff-8a6f-0c55a26bc55c" options : {router-port=lrp-d728e2a3-f9fd-4fff-8a6f-0c55a26bc55c} parent_name : [] port_security : [] tag : [] tag_request : [] type: router up : true local.conf == [[local|localrc]] RECLONE=yes DATABASE_PASSWORD=password RABBIT_PASSWORD=password SERVICE_PASSWORD=password SERVICE_TOKEN=password ADMIN_PASSWORD=password Q_AGENT=ovn Q_ML2_PLUGIN_MECHANISM_DRIVERS=ovn,logger Q_ML2_PLUGIN_TYPE_DRIVERS=local,flat,vlan,geneve Q_ML2_TENANT_NETWORK_TYPE="geneve" OVN_BRANCH="v21.06.0" OVN_BUILD_FROM_SOURCE="True" OVS_BRANCH="branch-2.15" OVS_SYSCONFDIR="/usr/local/etc/openvswitch" OVN_L3_CREATE_PUBLIC_NETWORK=True OCTAVIA_NODE="api" DISABLE_AMP_IMAGE_BUILD=True enable_plugin barbican https://opendev.org/openstack/barbican enable_plugin octavia https://opendev.org/openstack/octavia enable_plugin octavia-dashboard https://opendev.org/openstack/octavia-dashboard LIBS_FROM_GIT+=python-octaviaclient enable_service octavia enable_service o-api enable_service o-hk enable_service o-da disable_service o-cw disable_service o-hm enable_plugin ovn-octavia-provider https://opendev.org/openstack/ovn-octavia-provider LOGFILE=$DEST/logs/stack.sh.log enable_service ovn-northd enable_service ovn-controller enable_service q-ovn-metadata-agent enable_service q-svc disable_service q-agt disable_service q-l3 disable_service q-dhcp disable_service q-meta enable_plugin neutron https://opendev.org/openstack/neutron enable_service q-trunk enable_service q-dns enable_service q-port-forwarding enable_service neutron-segments enable_service q-log enable_plugin neutron-tempest-plugin https://opendev.org/o
[Yahoo-eng-team] [Bug 1959176] [NEW] neutron fullstack/functional jobs TIMED_OUT randomly
Public bug reported: Neutron fullstack and functional jobs are timing out randomly across all branches, only few such failures though, 1 or 2 failures per week. Builds references:- - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional-with-uwsgi&result=TIMED_OUT&skip=0 - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack-with-uwsgi&result=TIMED_OUT&skip=0 - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional&result=TIMED_OUT&skip=0 - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack&result=TIMED_OUT&skip=0 ** Affects: neutron Importance: Undecided Status: New ** Tags: fullstack functional-tests ** Tags added: fullstack functional-tests -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1959176 Title: neutron fullstack/functional jobs TIMED_OUT randomly Status in neutron: New Bug description: Neutron fullstack and functional jobs are timing out randomly across all branches, only few such failures though, 1 or 2 failures per week. Builds references:- - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional-with-uwsgi&result=TIMED_OUT&skip=0 - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack-with-uwsgi&result=TIMED_OUT&skip=0 - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional&result=TIMED_OUT&skip=0 - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack&result=TIMED_OUT&skip=0 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1959176/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1959564] [NEW] Random Tempest test failures(SSH failure) in openvswitch jobs
Public bug reported: Seen multiple similar occurences in Stable/wallaby patches, where tempest tests fails with ssh to VM Timeouts, some examples:- - https://cfaa2d1e4f6a936642aa-ae5561c9d080274a217713c4553af257.ssl.cf5.rackcdn.com/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-wallaby/a7c128e/testr_results.html - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_803/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/803c276/testr_results.html - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e19/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/e19d9a7/testr_results.html - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6b0/826830/1/check/neutron-tempest-plugin-scenario-openvswitch-wallaby/6b05e5b/testr_results.html - https://f529caf5e8a0adc3d959-479aba3a0d5645603ea5f6db22bcd24f.ssl.cf5.rackcdn.com/826830/1/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/9ad9f43/testr_results.html In some failures seeing metadata request failed and thus ssh failed, and in some metadata requests passed, but ssh failed, so may be multiple issues are there Builds:- https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby&job_name=neutron-tempest-plugin-scenario-openvswitch-wallaby&project=openstack%2Fneutron&branch=stable%2Fwallaby&skip=0 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1959564 Title: Random Tempest test failures(SSH failure) in openvswitch jobs Status in neutron: New Bug description: Seen multiple similar occurences in Stable/wallaby patches, where tempest tests fails with ssh to VM Timeouts, some examples:- - https://cfaa2d1e4f6a936642aa-ae5561c9d080274a217713c4553af257.ssl.cf5.rackcdn.com/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-wallaby/a7c128e/testr_results.html - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_803/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/803c276/testr_results.html - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e19/824022/2/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/e19d9a7/testr_results.html - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6b0/826830/1/check/neutron-tempest-plugin-scenario-openvswitch-wallaby/6b05e5b/testr_results.html - https://f529caf5e8a0adc3d959-479aba3a0d5645603ea5f6db22bcd24f.ssl.cf5.rackcdn.com/826830/1/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby/9ad9f43/testr_results.html In some failures seeing metadata request failed and thus ssh failed, and in some metadata requests passed, but ssh failed, so may be multiple issues are there Builds:- https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-wallaby&job_name=neutron-tempest-plugin-scenario-openvswitch-wallaby&project=openstack%2Fneutron&branch=stable%2Fwallaby&skip=0 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1959564/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1959573] [NEW] test_local_ip_connectivity test failing in stable/xena jobs
Public bug reported: Example failures:- - https://zuul.opendev.org/t/openstack/build/dc2e6aa9be504a39a930b17645dee287 - https://zuul.opendev.org/t/openstack/build/4ad0d2fd426b40b2b249585e612dbf1f Failing since https://review.opendev.org/c/openstack/neutron-tempest- plugin/+/823007 is merged. It's happening as master version of openvswitch jobs is running in stable/xena branch and that feature is only available in master branch. This needs to be fixed by switch to xena-jobs in stable/xena branch. ** Affects: neutron Importance: Undecided Assignee: yatin (yatinkarel) Status: In Progress ** Tags: gate-failure ovs ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) ** Tags added: gate-failure ovs -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1959573 Title: test_local_ip_connectivity test failing in stable/xena jobs Status in neutron: In Progress Bug description: Example failures:- - https://zuul.opendev.org/t/openstack/build/dc2e6aa9be504a39a930b17645dee287 - https://zuul.opendev.org/t/openstack/build/4ad0d2fd426b40b2b249585e612dbf1f Failing since https://review.opendev.org/c/openstack/neutron-tempest- plugin/+/823007 is merged. It's happening as master version of openvswitch jobs is running in stable/xena branch and that feature is only available in master branch. This needs to be fixed by switch to xena-jobs in stable/xena branch. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1959573/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1930402] Re: SSH timeouts happens very often in the ovn based CI jobs
Closing it as neutron-ovn-tempest-slow is moved to periodic in master branch and is no longer seeing ssh failures, if it happen again this can be reopened or new issue can be created. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1930402 Title: SSH timeouts happens very often in the ovn based CI jobs Status in neutron: Fix Released Bug description: I saw those errors mostly in the neutron-ovn-tempest-slow but probably it happens also in other jobs. Example of failures: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0de/791365/7/check/neutron-ovn-tempest-slow/0de1c30/testr_results.html https://2f7fb53980d59c550f7f-e09de525732b656a1c483807eeb06fc8.ssl.cf2.rackcdn.com/793369/3/check/neutron-ovn-tempest-slow/8a116c7/testr_results.html https://f86d217b949ada82d82c-f669355b5e0e599ce4f84e6e473a124c.ssl.cf2.rackcdn.com/791365/6/check/neutron-ovn-tempest-slow/5dc8d92/testr_results.html In all those cases, common thing is that VMs seems to get IP address from dhcp properly, cloud-init seems to be working fine but SSH to the FIP is not possible. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1930402/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1962306] [NEW] [Prefix Delegation] router add segment fails with Subnet for router interface must have a gateway IP
Public bug reported: Steps to reproduce:- Update ipv6_pd_enabled = true in neutron.conf and restart neutron-api service. $ openstack network create ipv6-pd $ openstack subnet create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --use-default-subnet-pool --network ipv6-pd ipv6-pd-1 $ openstack router add subnet router1 ipv6-pd-1 Add subnet fails with below Error:- BadRequestException: 400: Client Error for url: http://192.168.0.5:9696/v2.0/routers/417bc2ad-019a-470c-b0ca-61a1d81c3d7d/add_router_interface, Bad router request: Subnet for router interface must have a gateway IP. The issue is caused with https://review.opendev.org/c/openstack/neutron/+/699465 in order to fix https://bugs.launchpad.net/neutron/+bug/1856675. Workaround:- Create segment with --gateway specified, this will cause the issue #1856675 but working prefix-delegation, so need to fix it without re introducing previous bug:- openstack subnet create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --use-default-subnet-pool --network ipv6-pd --gateway :: ipv6-pd-1 ** Affects: neutron Importance: Undecided Assignee: yatin (yatinkarel) Status: New ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1962306 Title: [Prefix Delegation] router add segment fails with Subnet for router interface must have a gateway IP Status in neutron: New Bug description: Steps to reproduce:- Update ipv6_pd_enabled = true in neutron.conf and restart neutron-api service. $ openstack network create ipv6-pd $ openstack subnet create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --use-default-subnet-pool --network ipv6-pd ipv6-pd-1 $ openstack router add subnet router1 ipv6-pd-1 Add subnet fails with below Error:- BadRequestException: 400: Client Error for url: http://192.168.0.5:9696/v2.0/routers/417bc2ad-019a-470c-b0ca-61a1d81c3d7d/add_router_interface, Bad router request: Subnet for router interface must have a gateway IP. The issue is caused with https://review.opendev.org/c/openstack/neutron/+/699465 in order to fix https://bugs.launchpad.net/neutron/+bug/1856675. Workaround:- Create segment with --gateway specified, this will cause the issue #1856675 but working prefix-delegation, so need to fix it without re introducing previous bug:- openstack subnet create --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --use-default-subnet-pool --network ipv6-pd --gateway :: ipv6-pd-1 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1962306/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1956344] Re: Functional test test_gateway_chassis_rebalance is failing intermittently
Closing it as the workaround is in place and recently we have not seen failures in this test and also the test code have a NOTE added for future cleanup. If we see again the failure bug can be re opened. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1956344 Title: Functional test test_gateway_chassis_rebalance is failing intermittently Status in neutron: Fix Released Bug description: It may looks like similar to https://bugs.launchpad.net/neutron/+bug/1912369 but I noticed recently that at least 2 times test neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalance failed due to DB integrity error: https://54389876f34a304893e1-7e513b902e81eb926eb2654dfc72839e.ssl.cf1.rackcdn.com/822299/4/check/neutron- functional-with-uwsgi/dd33a5d/testr_results.html https://fd3bdfefdb4656902679-237e20f2346dac25649a4e5e3b0485b4.ssl.cf1.rackcdn.com/815962/6/check/neutron- functional-with-uwsgi/ef35e56/testr_results.html Stacktrace: ft1.1: neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalancetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context self.dialect.do_execute( File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute cursor.execute(statement, parameters) sqlite3.IntegrityError: FOREIGN KEY constraint failed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 494, in test_gateway_chassis_rebalance router = self._create_router('router%d' % i, gw_info=gw_info) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 49, in _create_router return self.l3_plugin.create_router(self.context, router) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/services/ovn_l3/plugin.py", line 166, in create_router router = super(OVNL3RouterPlugin, self).create_router(context, router) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/l3_db.py", line 2088, in create_router router_dict = super(L3_NAT_db_mixin, self).create_router(context, File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 218, in wrapped return method(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 139, in wrapped setattr(e, '_RETRY_EXCEEDED', True) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 135, in wrapped return f(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 154, in wrapper ectxt.value = e.inner_exc File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 142, in wrapper return f(*args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 183, in wrapped LOG.debug("Retry wrapper got retriable exception: %s", e) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/
[Yahoo-eng-team] [Bug 1868110] Re: [OVN] neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log randomly fails
Closing at as not seeing this issue recently as timeout increased long back. ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1868110 Title: [OVN] neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log randomly fails Status in neutron: Fix Released Bug description: The functional test neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log Randomly fails on our CI. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1868110/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1864833] Re: [OVN] Functional tests start with OVSDB binary 2.9 instead 2.12
It's no longer an issue now as i see ovs 2.16 in test and same is compiled in job, so closing the bug. OVS_BRANCH=v2.16.0 2022-02-25T11:12:50.733Z|3|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.16.0 https://91ac997443c049736e18-7ff9e89a9bc5ab4bd512192457a69ff2.ssl.cf1.rackcdn.com/828687/4/check/neutron-functional-with-uwsgi/9014bfc/controller/logs/dsvm-functional-logs/neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_resources.TestNBDbResourcesOverSsl.test_dhcp_options/ovn_nb-22-02-25_11-12-57_log.txt ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1864833 Title: [OVN] Functional tests start with OVSDB binary 2.9 instead 2.12 Status in neutron: Fix Released Bug description: In OVN functional we start ovsdb per each test. We don't start ovsdb 2.12 but 2.9, even we compile OVS/OVN 2.12 on gates: 2020-02-26T02:39:25.824Z|3|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.9.5 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1864833/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1960022] Re: neutron-tempest-with-uwsgi job failing tempest tests with: Floating ip failed to associate with server in time
Fixed in tempest, closing it. ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1960022 Title: neutron-tempest-with-uwsgi job failing tempest tests with: Floating ip failed to associate with server in time Status in neutron: Fix Released Bug description: Tempest test failing in neutron-tempest-with-uwsgi with below TraceBack:- Last successful job is seen on 28th Jan, 2022. ft1.1: setUpClass (tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON)testtools.testresult.real._StringException: Traceback (most recent call last): File "/opt/stack/tempest/tempest/test.py", line 182, in setUpClass raise value.with_traceback(trace) File "/opt/stack/tempest/tempest/test.py", line 175, in setUpClass cls.resource_setup() File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 84, in resource_setup cls.server_id = cls.recreate_server(None, validatable=True) File "/opt/stack/tempest/tempest/api/compute/base.py", line 431, in recreate_server server = cls.create_test_server( File "/opt/stack/tempest/tempest/api/compute/base.py", line 270, in create_test_server body, servers = compute.create_test_server( File "/opt/stack/tempest/tempest/common/compute.py", line 268, in create_test_server LOG.exception('Server %s failed to delete in time', File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/opt/stack/tempest/tempest/common/compute.py", line 246, in create_test_server _setup_validation_fip() File "/opt/stack/tempest/tempest/common/compute.py", line 231, in _setup_validation_fip waiters.wait_for_server_floating_ip( File "/opt/stack/tempest/tempest/common/waiters.py", line 571, in wait_for_server_floating_ip raise lib_exc.TimeoutException(msg) tempest.lib.exceptions.TimeoutException: Request timed out Details: Floating ip {'id': '42ad634c-9fce-4fff-87e0-c16cfddaa3ca', 'tenant_id': '485663e37c554596aff00daf8b166106', 'floating_ip_address': '172.24.5.190', 'floating_network_id': 'ca44af5f-08e7-4463-a77b-9660d250a82b', 'router_id': None, 'port_id': None, 'fixed_ip_address': None, 'status': 'DOWN', 'project_id': '485663e37c554596aff00daf8b166106', 'description': '', 'port_details': None, 'tags': [], 'created_at': '2022-02-03T13:28:19Z', 'updated_at': '2022-02-03T13:28:19Z', 'revision_number': 0, 'ip': '172.24.5.190'} failed to associate with server 7329393c-ed1d-433f-8be0-7a15a72b60e7 in time. As per neutron log fip get's created/associated/disassociated:- Feb 03 13:28:19.390516 ubuntu-focal-ovh-bhs1-0028297773 devstack@neutron-api.service[63722]: DEBUG neutron.db.db_base_plugin_common [None req-bf1740b9-bf5c-4a21-a073-1af294df8f30 tempest-ServerActionsTestJSON-974914713 tempest-ServerActionsTestJSON-974914713-project] Allocated IP 172.24.5.190 (ca44af5f-08e7-4463-a77b-9660d250a82b/b8944f8b-e1ee-4e3c-9a3b-e8c2f93ea5eb/9ec9d1c3-5cce-4393-9fb2-ab9407711e99) {{(pid=63722) _store_ip_allocation /opt/stack/neutron/neutron/db/db_base_plugin_common.py:131}} Feb 03 13:28:29.818159 ubuntu-focal-ovh-bhs1-0028297773 devstack@neutron-api.service[63722]: INFO neutron.db.l3_db [None req-c75b805c-590a-4793-ac9d-51d93945926a tempest-ServerActionsTestJSON-974914713 tempest-ServerActionsTestJSON-974914713-project] Floating IP 42ad634c-9fce-4fff-87e0-c16cfddaa3ca associated. External IP: 172.24.5.190, port: a2859895-1f57-439f-8235-a9a7540e9a75. Feb 03 13:31:50.314556 ubuntu-focal-ovh-bhs1-0028297773 devstack@neutron-api.service[63722]: INFO neutron.db.l3_db [req-10cd545c-dfcf-48b5-9754-0fb68c3bb271 req-30c83f43-a0b2-4cdd-915a-4afd4f8e7ced tempest- ServerActionsTestJSON-974914713 tempest- ServerActionsTestJSON-974914713-project] Floating IP 42ad634c-9fce-4fff-87e0-c16cfddaa3ca disassociated. External IP: 172.24.5.190, port: a2859895-1f57-439f-8235-a9a7540e9a75. But the FIP is not found in server "addresses". Failure example:- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_879/827016/1/check/neutron-tempest-with-uwsgi/8794c4a/testr_results.html https://b09ab8a3bdb459203334-b023e6398111e1b55f35dc3d44640709.ssl.cf5.rackcdn.com/827016/1/check/neutron-tempest-with-uwsgi/6eb9664/testr_results.html Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron- tempest-with-uwsgi&project=openstack/neutron To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1960022/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-
[Yahoo-eng-team] [Bug 1960338] Re: [fullstack] Test "test_east_west_traffic" randomly failing
Job is much stable now[1], single failure since the patch[2] merge and that too different test, Closing for now, if we see the issue again, it can be reopened. [1] https://zuul.openstack.org/builds?job_name=neutron-fullstack&branch=stable%2Fwallaby&skip=0 [2] https://review.opendev.org/c/openstack/neutron/+/828231 ** Changed in: neutron Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1960338 Title: [fullstack] Test "test_east_west_traffic" randomly failing Status in neutron: Fix Released Bug description: Test "test_east_west_traffic" randomly failing, mostly in Wallaby CI jobs. Errors: - https://60c5a582ad68767b8b2b-7c6e098ee109e2cfc4ca8fa68868cc2d.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/wallaby/neutron-fullstack/229aac7/testr_results.html - https://cb91710b8557a2f6220d-98f49207ca8d4e649788d064c2e22814.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/wallaby/neutron-fullstack/7470cbf/testr_results.html Snippet: https://paste.opendev.org/show/812593/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1960338/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1886807] Re: neutron-ovn-tempest-full-multinode-ovs-master job is failing 100% times
*** This bug is a duplicate of bug 1904117 *** https://bugs.launchpad.net/bugs/1904117 ** This bug is no longer a duplicate of bug 1885898 test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job ** This bug has been marked a duplicate of bug 1904117 Nodes in the OVN scenario multinode jobs can't talk to each other -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1886807 Title: neutron-ovn-tempest-full-multinode-ovs-master job is failing 100% times Status in neutron: Confirmed Bug description: Job neutron-ovn-tempest-full-multinode-ovs-master since few days (around 7.07.2020) started failing all the time. In most cases it TIMEOUTS or many tests are failed due to problem with SSH to the instance. Examples: https://f14611c8909f3446de65-4bd8d5b2b9c85b41b7d53d514f475e69.ssl.cf2.rackcdn.com/720464/25/check/neutron-ovn-tempest-full-multinode-ovs-master/cf995a4/testr_results.html https://58c639024a13bcb07d67-8e6063eece8c96bdec38e25d6079d8b4.ssl.cf1.rackcdn.com/739139/4/check/neutron-ovn-tempest-full-multinode-ovs-master/bf22922/testr_results.html https://571a74dab618adaab10c-51ebcc6aa3b2e625a79952b9fff60e62.ssl.cf5.rackcdn.com/736386/3/check/neutron-ovn-tempest-full-multinode-ovs-master/149bce9/job-output.txt To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1886807/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1874447] Re: [OVN] Tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails randomly
** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1874447 Title: [OVN] Tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails randomly Status in neutron: Fix Released Bug description: We can see occasional test failures of tempest test with OVN: neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle Traceback (most recent call last): File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/common/utils.py", line 78, in wait_until_true eventlet.sleep(sleep) File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 36, in sleep hub.switch() File "/usr/local/lib/python3.6/dist-packages/eventlet/hubs/hub.py", line 298, in switch return self.greenlet.switch() eventlet.timeout.Timeout: 60 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 240, in test_trunk_subport_lifecycle self._wait_for_port(port) File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_trunk.py", line 141, in _wait_for_port "status {!r}.".format(port['id'], status))) File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/common/utils.py", line 82, in wait_until_true raise exception RuntimeError: Timed out waiting for port 'cffcacde-a34e-4e1a-90ca-8d48776b9851' to transition to get status 'ACTIVE'. Example failure: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d8c/717851/2/check/neutron-ovn-tempest-ovs-release/d8c0282/testr_results.html To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1874447/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1971569] [NEW] [neutron][api] test_log_deleted_with_corresponding_security_group failing randomly
Public bug reported: test_log_deleted_with_corresponding_security_group api tests randomly failing with below Trace:- Traceback (most recent call last): File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/api/admin/test_logging.py", line 99, in test_log_deleted_with_corresponding_security_group self.assertRaises(exceptions.NotFound, File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 467, in assertRaises self.assertThat(our_callable, matcher) File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 480, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: ._show at 0x7f784dd443a0> returned {'log': {'id': '4360d06d-7bd2-42e8-9468-2e760b1b246a', 'name': 'tempest-test-log-1596006254', 'resource_type': 'security_group', 'resource_id': '1df2df30-4901-4e41-b58f-bbef3b0c48ee', 'target_id': None, 'event': 'ALL', 'enabled': True, 'revision_number': 0, 'description': '', 'created_at': '2022-05-04T10:17:44Z', 'updated_at': '2022-05-04T10:17:44Z'}} Example logs:- https://c1d005e52a45f1c42de5-a40b9478ae1bd073dc32f331038fe6d7.ssl.cf1.rackcdn.com/839477/1/check/neutron-tempest-plugin-api/825f39d/testr_results.html https://9a2b40b9abe499183777-8788f9c43324a469e03ac2bb48dfb234.ssl.cf2.rackcdn.com/837301/1/check/neutron-tempest-plugin-api-yoga/f0c7ad9/testr_results.html https://5fb6cfbc0c5c1a5be079-29e820dbac3fa779e4aa716d6c5c5850.ssl.cf1.rackcdn.com/839477/2/gate/neutron-tempest-plugin-api/929ff45/testr_results.html >From neutron api: May 04 10:17:44.536866 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-69db86bc-5bbd-4b78-a6bc-00e84d0f5e88 None None] Running txn n=1 command(idx=14): DbSetCommand(table=ACL, record=8f71b6eb-d4cf-4e64-8bbb-709c99dc3ad8, col_values=(('log', False), ('meter', []), ('name', []), ('severity', []))) {{(pid=98372) do_commit /usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}} May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d1699144-a926-431c-8202-470b285baa8a tempest-LoggingTestJSON-1873065189 tempest-LoggingTestJSON-1873065189-project] Traceback (most recent call last): May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: txn.results.put(txn.do_commit()) May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: command.run_idl(txn) May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 139, in run_idl May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: record = self.api.lookup(self.table, self.record) May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 181, in lookup May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: return self._lookup(table, record) May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 200, in _lookup May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: raise idlutils.RowNotFound(table=table, col='uuid', May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find ACL with uuid=8f71b6eb-d4cf-4e64-8bbb-709c99dc3ad8 May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager [None req-d1699144-a926-431c-8202-470b285baa8a tempest-LoggingTestJSON-1873065189 tempest-LoggingTestJSON-1873065189-project] Extension driver 'ovn' failed in delete_log: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find ACL with uuid=8f71b6eb-d4cf-4e64-8bbb-709c99dc3ad8 May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager Traceback (most recent call last): May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager File "/opt/stack/neutron/neutron/services/logapi/drivers/manager.py", line 116, in call May 04 1
[Yahoo-eng-team] [Bug 1971050] Re: Nested KVM Networking Issue
@Modyn, Ok Thanks for confirming, will close the bug as INVALID for neutron as i don't see anything to fix for this on neutron side. ** Changed in: neutron Status: New => Incomplete ** Changed in: neutron Status: Incomplete => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1971050 Title: Nested KVM Networking Issue Status in neutron: Invalid Bug description: ## Host environment - Operating system: (ubuntu 20.04 server) - OS/kernel version: (5.13.0.40 Generic) - Architecture: (64 bit cpu architecture) - QEMU version: (latest using sudo apt install virt-manager) ## Emulated/Virtualized environment - Operating system: (ubuntu 20.04 server) - OS/kernel version: ( 5.13.0.40 Generic) - Architecture: (64 bit cpu architecture) ## Description of problem Hi, Inside openstack i have an instance of Ubuntu 20.04 and i have installed KVM ( using virt-manager ) to setup a Virtual Machine ... i have done that and i created a VM of ubuntu 20.04 inside the Openstack Instance but there are networking issue while i set the default parameter as setting up the VM ( i mean the networking is as default to NAT ) , So when the VM is up and running the PING to 8.8.8.8 is available and also ping to google.com is also valid which shows that the DNS is correctly working ... but there is not connectivity with packages while i do sudo apt update, it will not get any package update and also the wget to google.com is shows that its connected to it but it wont able to download!!! the same happen with curl to any other websites... I'm confirming that the openstack instance has full access to the internet including ping and wget , but the VM is not working correctly! P.S. I have set the ip forwarding, Iptables , ... also disabled firewals but notting changed!! Would you please fix this ? ## Steps to reproduce 1. creating an openstack instance from ubuntu 20.04 server image 2. updating and upgrading packages setting ip forwarding to 1 ( Enabled), firewall 3. and kernel to 5.13.0.40 and installing virt-manager then reboot 3. creating a VM with default KVM networking ( NAT ) using ubuntu 20.04 server image 4. trying ping, wget, curl , ... These are my commands after creating an instance with 8VCPU, 16VRAM, 100VDisk, ubuntu cloud 20.04 image: sudo apt update && sudo apt full-upgrade -y && sudo apt install linux-image-5.13.0-40-generic linux-headers-5.13.0-40-generic -y && sudo reboot sudo apt update && sudo uname -a Linux test 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux sudo apt install virt-manager -y && sudo reboot sudo systemctl status libvirtd Its runningIP range 192.168.122.2 sudo usermod -a -G libvirt ubuntu then download ubuntu server 20.04 image from https://releases.ubuntu.com/20.04/ubuntu-20.04.4-live-server-amd64.iso and create a new VM using KVM by virt-manager as shown bellow: https://gitlab.com/qemu-project/qemu/uploads/8bd4c7381a60832b3a5fcd9dbd3665de/image.png qemu-system-x86_64 --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.21) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers Here is my networking : ``` 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens3: mtu 1442 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:10:60:0e brd ff:ff:ff:ff:ff:ff altname enp0s3 inet 10.20.30.52/24 brd 10.20.30.255 scope global dynamic ens3 valid_lft 34758sec preferred_lft 34758sec inet6 fe80::f816:3eff:fe10:600e/64 scope link valid_lft forever preferred_lft forever 3: virbr0: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:98:07:1a brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 4: virbr0-nic: mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:98:07:1a brd ff:ff:ff:ff:ff:ff 5: vnet0: mtu 1500 qdisc fq_codel master virbr0 state UNKNOWN group default qlen 1000 link/ether fe:54:00:f9:5d:4d brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fef9:5d4d/64 scope link valid_lft forever preferred_lft forever ``` And this is my Iptable ``` iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination LIBVIRT_INP all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source
[Yahoo-eng-team] [Bug 1972764] [NEW] [Wallaby] OVNPortForwarding._handle_lb_on_ls fails with TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context'
Public bug reported: It's failing with below TraceBack:- 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager [req-2e5ef575-b18c-4091-8c15-b37f6bcf0fdd f1840520501c41b2a6a534525f0f90a4 bf49659cd4cb40edb393b914198ce3c9 - default default] Error during notification for neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding._handle_lb_on_ls-4305748 router_interface, after_create: TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager Was noticed in a TripleO job https://bugs.launchpad.net/tripleo/+bug/1972660. This method was added in https://review.opendev.org/q/I0c4d492887216cad7a8155dceb738389f2886376 and backported till wallaby. Xena+ are ok, only wallaby impacted because before xena old notification format is used where arguments are passed as kwargs. ** Affects: neutron Importance: Undecided Assignee: yatin (yatinkarel) Status: Confirmed ** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1972764 Title: [Wallaby] OVNPortForwarding._handle_lb_on_ls fails with TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' Status in neutron: Confirmed Bug description: It's failing with below TraceBack:- 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager [req-2e5ef575-b18c-4091-8c15-b37f6bcf0fdd f1840520501c41b2a6a534525f0f90a4 bf49659cd4cb40edb393b914198ce3c9 - default default] Error during notification for neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding._handle_lb_on_ls-4305748 router_interface, after_create: TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager Was noticed in a TripleO job https://bugs.launchpad.net/tripleo/+bug/1972660. This method was added in https://review.opendev.org/q/I0c4d492887216cad7a8155dceb738389f2886376 and backported till wallaby. Xena+ are ok, only wallaby impacted because before xena old notification format is used where arguments are passed as kwargs. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1972764/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1973221] [NEW] [master] pep8 jobs TIMING OUT randomly
Public bug reported: After https://review.opendev.org/c/openstack/neutron/+/806246 patch pep8 job is taking more than 30 minutes and TIMES_OUT. Specifically flake8 is taking much time now. Before the patch approx 2 minutes and now more than 12 minutes. builds:- https://zuul.opendev.org/t/openstack/builds?job_name=openstack- tox-pep8&project=openstack%2Fneutron&result=TIMED_OUT&skip=0 example log:- https://76c5414ef698e04295b6-922433d163de5a07cac84d974d42345f.ssl.cf2.rackcdn.com/841118/1/gate/openstack- tox-pep8/e9359b5/job-output.txt 2022-05-12 11:03:13.275284 | ubuntu-focal | [5198] /home/zuul/src/opendev.org/openstack/neutron$ /home/zuul/src/opendev.org/openstack/neutron/.tox/lint/bin/flake8 2022-05-12 11:25:07.414232 | ubuntu-focal | :582: DeprecationWarning: invalid escape sequence \u ** Affects: neutron Importance: Undecided Status: New ** Tags: gate-failure ** Tags added: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1973221 Title: [master] pep8 jobs TIMING OUT randomly Status in neutron: New Bug description: After https://review.opendev.org/c/openstack/neutron/+/806246 patch pep8 job is taking more than 30 minutes and TIMES_OUT. Specifically flake8 is taking much time now. Before the patch approx 2 minutes and now more than 12 minutes. builds:- https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox- pep8&project=openstack%2Fneutron&result=TIMED_OUT&skip=0 example log:- https://76c5414ef698e04295b6-922433d163de5a07cac84d974d42345f.ssl.cf2.rackcdn.com/841118/1/gate/openstack- tox-pep8/e9359b5/job-output.txt 2022-05-12 11:03:13.275284 | ubuntu-focal | [5198] /home/zuul/src/opendev.org/openstack/neutron$ /home/zuul/src/opendev.org/openstack/neutron/.tox/lint/bin/flake8 2022-05-12 11:25:07.414232 | ubuntu-focal | :582: DeprecationWarning: invalid escape sequence \u To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1973221/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1973783] [NEW] [devstack] Segment plugin reports Traceback as placement client not configured
ine 1141, in get May 17 12:08:25.071826 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: return self.request(url, 'GET', **kwargs) May 17 12:08:25.071826 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/session.py", line 811, in request May 17 12:08:25.071826 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: base_url = self.get_endpoint(auth, allow=allow, May 17 12:08:25.071826 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/session.py", line 1241, in get_endpoint May 17 12:08:25.071826 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: auth = self._auth_required(auth, 'determine endpoint URL') May 17 12:08:25.071826 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/session.py", line 1181, in _auth_required May 17 12:08:25.071826 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: raise exceptions.MissingAuthPlugin(msg_fmt % msg) May 17 12:08:25.071826 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL Ex. log:- https://8430494fc120d4e2add1-92777588630241a74fd2839fb5cc6a5d.ssl.cf5.rackcdn.com/841118/1/gate/neutron- tempest-plugin-scenario-openvswitch/92d6c29/controller/logs/screen-q- svc.txt This is happening as placement client is not configured with devstack deployments. ** Affects: neutron Importance: Undecided Assignee: yatin (yatinkarel) Status: New ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1973783 Title: [devstack] Segment plugin reports Traceback as placement client not configured Status in neutron: New Bug description: Following Traceback is reported although the job passes but creates a noise in logs, so should be cleared:- May 17 12:08:25.056617 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: DEBUG neutron_lib.callbacks.manager [None req-01d22b64-1fbc-4578-8cc2-c6565188c424 admin admin] Publish callbacks ['neutron.plugins.ml2.plugin.Ml2Plugin._handle_segment_change-1048155', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_segment_deleted-1983453', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_segment_deleted-495'] for segment (45896f0b-13b1-4cfc-ab32-297a8d8dae05), after_delete {{(pid=72995) _notify_loop /usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py:176}} May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: Traceback (most recent call last): May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: timer() May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/timer.py", line 59, in __call__ May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: cb(*args, **kw) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/common/utils.py", line 922, in wrapper May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: return func(*args, **kwargs) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/notifiers/batch_notifier.py", line 58, in synced_send May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: self._notify() May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/notifiers/batch_notifier.py", line 69, in _notify May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: self.callback(batched_events) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/services/segments/plugin.py", line 211, in _send_notifications May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: event.method(event) May 17 12:08:25.
[Yahoo-eng-team] [Bug 1976323] [NEW] [functional/fullstack][master] move fips periodic job to CentOS 9-Stream
Public bug reported: Currently these periodic jobs are running on CentOS 8-Stream(with python 3.6) and failing[1][2]. These are failing as master no longer supports py3.6. To unblock switching these jobs to run functional/fullstack tests with python3.8[3] and disabling dbcounter installation. Ideally these jobs can be switched to CentOS 9-Stream(have python3.9 as default). But found in testing[4] it has couple of issues:- 1. DNS issues are hit randomly during setup until unbound service is setupped after reboot. Can fix by waiting for unbound to be ready or workaround by not using unbound. 2. 4 functional tests are failing:- - test_delete_multiple_entries --> conntrack delete not deleting enteries (WARNING neutron.privileged.agent.linux.netlink_lib [-] Netlink query failed looks related) - test_delete_icmp_entry --> conntrack delete not deleting enteries (WARNING neutron.privileged.agent.linux.netlink_lib [-] Netlink query failed looks related) - test_rule_application_converges(IptablesFirewallDriver,with ipset) --> self.assertEqual([], self.firewall.iptables._apply()) fails as _apply not returning empty list as expected --> might be due to iptables behavior in C9-Stream - test_rule_application_converges(IptablesFirewallDriver,without ipset) --> self.assertEqual([], self.firewall.iptables._apply()) fails as _apply not returning empty list as expected --> might be due to iptables behavior in C9-Stream 3. 2 fullstack tests are failing:- - neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_north_south_traffic - neutron.tests.fullstack.test_local_ip.LocalIPTestCase.test_vm_is_accessible_by_local_ip(static_nat) Both failing at ping neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 2001:db8:1234:: neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 10.0.0.10 [1] https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi-fips&project=openstack%2Fneutron&branch=master&pipeline=periodic&skip=0 [2] https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi-fips&project=openstack%2Fneutron&branch=master&pipeline=periodic&skip=0 [3] https://review.opendev.org/c/openstack/neutron/+/843252 [4] https://review.opendev.org/c/openstack/neutron/+/843245 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1976323 Title: [functional/fullstack][master] move fips periodic job to CentOS 9-Stream Status in neutron: New Bug description: Currently these periodic jobs are running on CentOS 8-Stream(with python 3.6) and failing[1][2]. These are failing as master no longer supports py3.6. To unblock switching these jobs to run functional/fullstack tests with python3.8[3] and disabling dbcounter installation. Ideally these jobs can be switched to CentOS 9-Stream(have python3.9 as default). But found in testing[4] it has couple of issues:- 1. DNS issues are hit randomly during setup until unbound service is setupped after reboot. Can fix by waiting for unbound to be ready or workaround by not using unbound. 2. 4 functional tests are failing:- - test_delete_multiple_entries --> conntrack delete not deleting enteries (WARNING neutron.privileged.agent.linux.netlink_lib [-] Netlink query failed looks related) - test_delete_icmp_entry --> conntrack delete not deleting enteries (WARNING neutron.privileged.agent.linux.netlink_lib [-] Netlink query failed looks related) - test_rule_application_converges(IptablesFirewallDriver,with ipset) --> self.assertEqual([], self.firewall.iptables._apply()) fails as _apply not returning empty list as expected --> might be due to iptables behavior in C9-Stream - test_rule_application_converges(IptablesFirewallDriver,without ipset) --> self.assertEqual([], self.firewall.iptables._apply()) fails as _apply not returning empty list as expected --> might be due to iptables behavior in C9-Stream 3. 2 fullstack tests are failing:- - neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_north_south_traffic - neutron.tests.fullstack.test_local_ip.LocalIPTestCase.test_vm_is_accessible_by_local_ip(static_nat) Both failing at ping neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 2001:db8:1234:: neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 10.0.0.10 [1] https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi-fips&project=openstack%2Fneutron&branch=master&pipeline=periodic&skip=0 [2] https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi-fips&project=openstack%2Fneutron&branch=master&pipeline=periodic&skip=0 [3] https://review.opendev.org/c/openstack/neutron/+/84325
[Yahoo-eng-team] [Bug 1976360] Re: [sqlalchemy-20] Missing DB context in networking-sfc flow classifier
This should have opened against networking-sfc, it's fixed with https://review.opendev.org/c/openstack/networking-sfc/+/844251. Will update project. ** Project changed: neutron => networking-sfc ** Changed in: networking-sfc Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1976360 Title: [sqlalchemy-20] Missing DB context in networking-sfc flow classifier Status in networking-sfc: Fix Committed Bug description: Logs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ab7/842135/1/gate/neutron- tempest-plugin-sfc/ab703be/controller/logs/screen-q-svc.txt Snippet: https://paste.opendev.org/show/bYc7TKCf9X5f4yFhYu7O/ To manage notifications about this bug go to: https://bugs.launchpad.net/networking-sfc/+bug/1976360/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1978369] Re: [ovn] External Gateway Loop in NB SB DB
*** This bug is a duplicate of bug 1973347 *** https://bugs.launchpad.net/bugs/1973347 This looks duplicate of https://bugs.launchpad.net/neutron/+bug/1973347 and is fixed with https://review.opendev.org/c/openstack/neutron/+/842147. This should be backported to stable branches as well. @Ammad Can you try out the patch and confirm it fixes the issue for your? For now i will mark out it as duplicate of other lp i.e 1973347, please reopen if you still consider it different issue once you check other bug and the fix. ** This bug has been marked a duplicate of bug 1973347 OVN revision_number infinite update loop -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1978369 Title: [ovn] External Gateway Loop in NB SB DB Status in neutron: New Bug description: Hi, I have installed neutron 20.0 and OVN 22.03 on ubuntu 22.04. When I create a router and attach external network with it, it generate loop thousands of ovn NB and SB DB transection cause the DB size grows. In SB OVSDB JSON 300 0f200aa6397e53cd203c99e6674bda75bdd53151 {"_date":1654929577073,"Multicast_Group":{"9b50bf0f-f9fe-4b9a-9333-fe2d1744575c":{"ports":["uuid","efc3d1a7-56a6-4235-8a29-4d1defdb459c"]}},"_is_diff":true,"_comment":"ovn-northd","Port_Binding":{"efc3d1a7-56a6-4235-8a29-4d1defdb459c":{"external_ids":["map",[["neutron:revision_number","10678"]]]}}} OVSDB JSON 402 86de47a7521717bd9ab7182422a6ad9b424c93d0 {"_date":1654929577345,"Multicast_Group":{"9b50bf0f-f9fe-4b9a-9333-fe2d1744575c":{"ports":["uuid","efc3d1a7-56a6-4235-8a29-4d1defdb459c"]}},"_is_diff":true,"_comment":"ovn-northd","Port_Binding":{"d34d2dd5-260b-4253-8429-5a7a89f3a500":{"external_ids":["map",[["neutron:revision_number","10679"]]]},"2ce0135e-b9b5-441b-aaae-7ce580bcf600":{"external_ids":["map",[["neutron:revision_number","10679"]]]}}} and In NB OVSDB JSON 334 e0ee7ff61d595e6151abd694ce2179c11d9e2570 {"_date":1654929536919,"_is_diff":true,"Logical_Router_Port":{"a0c2e43e-f4cb-4331-b070-a726b3da7a17":{"external_ids":["map",[["neutron:revision_number","10567"]]]}},"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]]}}} OVSDB JSON 269 dd8f87d8b132415a423b0f020b23f07d2488acba {"_date":1654929536992,"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]],"external_ids":["map",[["neutron:revision_number","10567"]]]}},"_is_diff":true} OVSDB JSON 334 42d2a02531bd91d88b8783a45da47a33b5e3dc94 {"_date":1654929537262,"_is_diff":true,"Logical_Router_Port":{"a0c2e43e-f4cb-4331-b070-a726b3da7a17":{"external_ids":["map",[["neutron:revision_number","10568"]]]}},"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]]}}} OVSDB JSON 269 b8454f003de8cb14961aa37d5a557d2490d34049 {"_date":1654929537355,"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]],"external_ids":["map",[["neutron:revision_number","10568"]]]}},"_is_diff":true} OVSDB JSON 334 705b3007e83f0646642510903602965a6192fccf {"_date":1654929537648,"_is_diff":true,"Logical_Router_Port":{"a0c2e43e-f4cb-4331-b070-a726b3da7a17":{"external_ids":["map",[["neutron:revision_number","10569"]]]}},"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]]}}} OVSDB JSON 269 4506e6ee9336bf2b8bde3134badbea7d23e72d33 I also see below logs in ovn-northd.log 2022-06-11T06:46:55.927Z|00171|northd|WARN|Dropped 650 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate 2022-06-11T06:46:55.927Z|00172|northd|WARN|Unknown chassis '' set as options:requested-chassis on LSP '426cf7d5-4fd7-4aa9-806b-9dbe170c543e'. 2022-06-11T06:47:55.941Z|00173|northd|WARN|Dropped 644 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate 2022-06-11T06:47:55.941Z|00174|northd|WARN|Unknown chassis '' set as options:requested-chassis on LSP '426cf7d5-4fd7-4aa9-806b-9dbe170c543e'. I have tested it on ubuntu 20.04 via UCA AND 22.04. Below are the test scenerio. - Two gateway chassis - 5 compute nodes I have also tested this with one chassis as well, for which I am attaching neutron-server.log when I attached external interface to router and ovn nb and sb DBs as well. I would be happy to provide any further info that is needed. Ammad To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1978369/+subscriptions -- Mailing list: https://lau
[Yahoo-eng-team] [Bug 1940243] Re: Neutron-tempest-plugin scenario tests - oom-killer is killing mysql process
Reopening the bz as seeing few failures(5 failures in last 15 days as per https://opensearch.logs.openstack.org) in linuxbridge and openvswitch scenario jobs:- https://1b33868f301e2201a22c-a64bb815b8796eabf8a53948331bd878.ssl.cf5.rackcdn.com/845366/2/check/neutron-tempest-plugin-openvswitch/aa3106c/controller/logs/syslog.txt https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_82d/843649/2/gate/neutron-tempest-plugin-openvswitch-iptables_hybrid/82d4dbe/controller/logs/syslog.txt https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_052/842135/1/check/neutron-tempest-plugin-linuxbridge/0528022/controller/logs/syslog.txt https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f11/844680/1/check/neutron-tempest-plugin-linuxbridge/f111e3f/controller/logs/syslog.txt https://bd6c3da3b1723c5f3502-97cb3b32849366f5bed744685e46b266.ssl.cf1.rackcdn.com/844860/3/check/neutron-tempest-plugin-linuxbridge/35e5065/controller/logs/syslog.txt In all the above failures, SwapFree: 0 kB and many vms were active(using 256 mb/128 mb each) and mysql utilizing approx 10% of total memory. Also multiple(around 10) neutron-keepalived-state-change processes utilizing approx 12-14% of memory. Considering only seen in master jobs and with timings seems this triggered after https://review.opendev.org/c/openstack/neutron-tempest- plugin/+/836912 as more api tests running along with scenario tests now. The swap used in these jobs is 1 gb, so it can be increased to some extent like 3-4 gb to avoid these issues due to memory shortage. ** Changed in: neutron Status: Fix Released => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1940243 Title: Neutron-tempest-plugin scenario tests - oom-killer is killing mysql process Status in neutron: Triaged Bug description: It happens pretty often recently that during our scenario tests we are running out of memory and oom-killer is killing mysql process as it is number 1 in memory consumption. That is causing job's failures. It seems for me that it happens when there are running tests which uses vms with advanced image (ubuntu). Maybe we should extract those tests and run them as second stage with "--concurency 1"? Examples of failures: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_85a/803462/2/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/85afc13/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/803936/1/gate/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/9445e5f/testr_results.html https://09e003a6a650f320c43d-e30f275ad83ed88289c7399adb6c5ee6.ssl.cf1.rackcdn.com/804236/5/check/neutron- tempest-plugin-scenario-linuxbridge/770722a/testr_results.html https://bd90009aa1732b7b8d4a-e998c5625939f617052baaae6f827bb8.ssl.cf5.rackcdn.com/797221/1/check/neutron- tempest-plugin-scenario-openvswitch/2a7ab79/testr_results.html https://27020bbcd4882754b192-88656c065c39ed46f44b21a92a1cea67.ssl.cf5.rackcdn.com/800445/7/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/5e597ae/testr_results.html https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e7d/800446/9/check/neutron- tempest-plugin-scenario-openvswitch/e7d72c9/testr_results.html https://637b02491f0435a9a86b-ccec73fd7dde7a9826f6a9aeb49ab878.ssl.cf5.rackcdn.com/804397/1/gate/neutron- tempest-plugin-scenario-linuxbridge/64bae23/testr_results.html https://d1b1a7bc5606074c0db2-9f552c22a38891cd59267376a7a41496.ssl.cf5.rackcdn.com/802596/12/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/de03f1f/testr_results.html https://b395fe859a68f8d08e03-e48e76b6f53fcff59de7a7c1c3da6c62.ssl.cf1.rackcdn.com/804394/3/check/neutron- tempest-plugin-scenario-openvswitch- iptables_hybrid/98bfff3/testr_results.html https://6ee071d84f4801a650d3-2635c9269ad2bde2592553cd282ad960.ssl.cf2.rackcdn.com/804394/3/check/neutron- tempest-plugin-scenario-linuxbridge/a9282d0/testr_results.html Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20Unexpected%20API%20Error.%20Please%20report%20this%20at%20http%3A%2F%2Fbugs.launchpad.net%2Fnova%2F%20and%20attach%20the%20Nova%20API%20log%20if%20possible.%5C%22 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1940243/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1978938] [NEW] [9-stream][master][yoga] ovs/ovn fips jobs fails randomly with DNS issue
Public bug reported: These FIPS jobs running on centos-9-stream fails randomly at different points due to DNS issue. In these jobs after configuring fips, node get's rebooted. The unbound service may take some time to be ready after the reboot and until then any DNS resolution fails. Fails like below:- 2022-06-16 02:36:58.829027 | controller | + ./stack.sh:_install_rdo:314 : sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos9-master/delorean-deps.repo 2022-06-16 02:36:58.940595 | controller | % Total% Received % Xferd Average Speed TimeTime Time Current 2022-06-16 02:36:58.940689 | controller | Dload Upload Total SpentLeft Speed 2022-06-16 02:36:58.941657 | controller | 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: trunk.rdoproject.org 2022-06-16 02:36:58.946989 | controller | + ./stack.sh:_install_rdo:316 : sudo dnf -y update 2022-06-16 02:36:59.649717 | controller | CentOS Stream 9 - BaseOS 0.0 B/s | 0 B 00:00 2022-06-16 02:36:59.650078 | controller | Errors during downloading metadata for repository 'baseos': 2022-06-16 02:36:59.650117 | controller | - Curl error (6): Couldn't resolve host name for https://mirror-int.ord.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml [Could not resolve host: mirror-int.ord.rax.opendev.org] 2022-06-16 02:40:30.411317 | controller | + functions-common:sudo_with_proxies:2384 : sudo http_proxy= https_proxy= no_proxy= dnf install -y rabbitmq-server 2022-06-16 02:40:31.225502 | controller | Last metadata expiration check: 0:03:30 ago on Thu 16 Jun 2022 02:37:01 AM UTC. 2022-06-16 02:40:31.278762 | controller | No match for argument: rabbitmq-server 2022-06-16 02:40:31.286118 | controller | Error: Unable to find a match: rabbitmq-server Job builds:- - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-tempest-ovs-release-fips&project=openstack/neutron - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-tempest-fips&project=openstack/neutron ** Affects: neutron Importance: High Assignee: yatin (yatinkarel) Status: In Progress ** Changed in: neutron Status: New => Triaged ** Changed in: neutron Importance: Undecided => High ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1978938 Title: [9-stream][master][yoga] ovs/ovn fips jobs fails randomly with DNS issue Status in neutron: In Progress Bug description: These FIPS jobs running on centos-9-stream fails randomly at different points due to DNS issue. In these jobs after configuring fips, node get's rebooted. The unbound service may take some time to be ready after the reboot and until then any DNS resolution fails. Fails like below:- 2022-06-16 02:36:58.829027 | controller | + ./stack.sh:_install_rdo:314 : sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos9-master/delorean-deps.repo 2022-06-16 02:36:58.940595 | controller | % Total% Received % Xferd Average Speed TimeTime Time Current 2022-06-16 02:36:58.940689 | controller | Dload Upload Total SpentLeft Speed 2022-06-16 02:36:58.941657 | controller | 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: trunk.rdoproject.org 2022-06-16 02:36:58.946989 | controller | + ./stack.sh:_install_rdo:316 : sudo dnf -y update 2022-06-16 02:36:59.649717 | controller | CentOS Stream 9 - BaseOS 0.0 B/s | 0 B 00:00 2022-06-16 02:36:59.650078 | controller | Errors during downloading metadata for repository 'baseos': 2022-06-16 02:36:59.650117 | controller | - Curl error (6): Couldn't resolve host name for https://mirror-int.ord.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml [Could not resolve host: mirror-int.ord.rax.opendev.org] 2022-06-16 02:40:30.411317 | controller | + functions-common:sudo_with_proxies:2384 : sudo http_proxy= https_proxy= no_proxy= dnf install -y rabbitmq-server 2022-06-16 02:40:31.225502 | controller | Last metadata expiration check: 0:03:30 ago on Thu 16 Jun 2022 02:37:01 AM UTC. 2022-06-16 02:40:31.278762 | controller | No match for argument: rabbitmq-server 2022-06-16 02:40:31.286118 | controller | Error: Unable to find a match: rabbitmq-server Job builds:- - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-tempest-ovs-release-fips&project=openstack/neutron -
[Yahoo-eng-team] [Bug 1979031] [NEW] [master] fullstack functional jobs broken with pyrouet2-0.6.12
Public bug reported: After pyroute2-0.6.12 update[1], multiple tests in fullstack/functional jobs failing[2][3]. Noting some failures here:- functional:- AssertionError: CIDR 2001:db8::/64 not found in the list of routes AssertionError: Route not found: {'table': 'main', 'cidr': '192.168.0.0/24', 'source_prefix': None, 'scope': 'global', 'device': 'test_device', 'via': None, 'metric': 0, 'proto': 'static'} testtools.matchers._impl.MismatchError: 'via' not in None external_device.route.get_gateway().get('via')) AttributeError: 'NoneType' object has no attribute 'get' testtools.matchers._impl.MismatchError: !=: reference = [{'cidr': '0.0.0.0/0', 'device': 'rfp-403146e1-f', 'table': 16, 'via': '169.254.88.135'}, {'cidr': '::/0', 'device': 'rfp-403146e1-f', 'table': 'main', 'via': 'fe80::9005:f3ff:fe70:40b9'}] actual= [] fullstack neutron.tests.common.machine_fixtures.FakeMachineException: Address 10.0.0.87/24 or gateway 10.0.0.1 not configured properly on port port3f2663 Route related calls looks broken with new version. Example:- functional:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_736/797120/19/check/neutron-functional-with-uwsgi/7366408/testr_results.html fullstack:- https://983763dc8d641eb0d8f2-98d26e4bc9646b70fe4861772f4678c8.ssl.cf2.rackcdn.com/845363/4/gate/neutron-fullstack-with-uwsgi/1cc6a9d/testr_results.html [1] https://review.opendev.org/c/openstack/requirements/+/845871 [2] https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional-with-uwsgi&project=openstack%2Fneutron&branch=master&skip=0 [3] https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack-with-uwsgi&project=openstack%2Fneutron&branch=master&skip=0 ** Affects: neutron Importance: Critical Assignee: Rodolfo Alonso (rodolfo-alonso-hernandez) Status: New ** Tags: fullstack functional-tests gate-failure ** Changed in: neutron Importance: Undecided => Critical ** Tags added: fullstack functional-tests gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1979031 Title: [master] fullstack functional jobs broken with pyrouet2-0.6.12 Status in neutron: New Bug description: After pyroute2-0.6.12 update[1], multiple tests in fullstack/functional jobs failing[2][3]. Noting some failures here:- functional:- AssertionError: CIDR 2001:db8::/64 not found in the list of routes AssertionError: Route not found: {'table': 'main', 'cidr': '192.168.0.0/24', 'source_prefix': None, 'scope': 'global', 'device': 'test_device', 'via': None, 'metric': 0, 'proto': 'static'} testtools.matchers._impl.MismatchError: 'via' not in None external_device.route.get_gateway().get('via')) AttributeError: 'NoneType' object has no attribute 'get' testtools.matchers._impl.MismatchError: !=: reference = [{'cidr': '0.0.0.0/0', 'device': 'rfp-403146e1-f', 'table': 16, 'via': '169.254.88.135'}, {'cidr': '::/0', 'device': 'rfp-403146e1-f', 'table': 'main', 'via': 'fe80::9005:f3ff:fe70:40b9'}] actual= [] fullstack neutron.tests.common.machine_fixtures.FakeMachineException: Address 10.0.0.87/24 or gateway 10.0.0.1 not configured properly on port port3f2663 Route related calls looks broken with new version. Example:- functional:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_736/797120/19/check/neutron-functional-with-uwsgi/7366408/testr_results.html fullstack:- https://983763dc8d641eb0d8f2-98d26e4bc9646b70fe4861772f4678c8.ssl.cf2.rackcdn.com/845363/4/gate/neutron-fullstack-with-uwsgi/1cc6a9d/testr_results.html [1] https://review.opendev.org/c/openstack/requirements/+/845871 [2] https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional-with-uwsgi&project=openstack%2Fneutron&branch=master&skip=0 [3] https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack-with-uwsgi&project=openstack%2Fneutron&branch=master&skip=0 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1979031/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1981963] [NEW] Some jobs broken post pyroute2 update to 0.7.1
Public bug reported: pyroute2 updated to 0.7.1 with https://review.opendev.org/c/openstack/requirements/+/849790, since then couple of jobs are broken, like:- - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-grenade-dvr-multinode&branch=master - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-tempest-dvr-ha-multinode-full&branch=master - https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi&branch=master Example failures:- https://37a49967c371d50badcf-d0788a990c172de672e24591209c14b2.ssl.cf5.rackcdn.com/849122/8/check/neutron-functional-with-uwsgi/1971bae/testr_results.html Failing as:- ft7.4: neutron.tests.functional.privileged.agent.linux.test_ip_lib.RuleTestCase.test_add_rule_iptesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/privileged/agent/linux/test_ip_lib.py", line 323, in test_add_rule_ip priv_ip_lib.add_ip_rule(self.namespace, src=ip_address, File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap return self.channel.remote_call(name, args, kwargs, File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 215, in remote_call raise exc_type(*result[2]) pyroute2.netlink.exceptions.NetlinkError: (22, 'Invalid argument') testtools.matchers._impl.MismatchError: !=: reference = {'from': '0.0.0.0/0', 'iif': 'fpr-663cc9d3-b', 'priority': '2852018311', 'table': '2852018311', 'type': 'unicast'} actual= {'from': '0.0.0.0/0', 'iif': 'fpr-663cc9d3-b', 'priority': '2852018311', 'table': '2852018311', 'type': 'unspecified'} https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce1/849122/8/check/neutron-ovs-tempest-dvr-ha-multinode-full/ce10281/testr_results.html https://d0b0b53d30de16fbad20-5f381a9e8c14b627196c6ef3340b4d4e.ssl.cf5.rackcdn.com/849122/8/check/neutron-ovs-grenade-dvr-multinode/2a568da/testr_results.html ^ jobs failing at SSH timeout ** Affects: neutron Importance: Undecided Status: New ** Tags: functional-tests gate-failure ovs -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1981963 Title: Some jobs broken post pyroute2 update to 0.7.1 Status in neutron: New Bug description: pyroute2 updated to 0.7.1 with https://review.opendev.org/c/openstack/requirements/+/849790, since then couple of jobs are broken, like:- - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-grenade-dvr-multinode&branch=master - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-tempest-dvr-ha-multinode-full&branch=master - https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi&branch=master Example failures:- https://37a49967c371d50badcf-d0788a990c172de672e24591209c14b2.ssl.cf5.rackcdn.com/849122/8/check/neutron-functional-with-uwsgi/1971bae/testr_results.html Failing as:- ft7.4: neutron.tests.functional.privileged.agent.linux.test_ip_lib.RuleTestCase.test_add_rule_iptesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/privileged/agent/linux/test_ip_lib.py", line 323, in test_add_rule_ip priv_ip_lib.add_ip_rule(self.namespace, src=ip_address, File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap return self.channel.remote_call(name, args, kwargs, File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 215, in remote_call raise exc_type(*result[2]) pyroute2.netlink.exceptions.NetlinkError: (22, 'Invalid argument') testtools.matchers._impl.MismatchError: !=: reference = {'from': '0.0.0.0/0', 'iif': 'fpr-663cc9d3-b', 'priority': '2852018311', 'table': '2852018311', 'type': 'unicast'} actual= {'from': '0.0.0.0/0', 'iif': 'fpr-663cc9d3-b', 'priority': '2852018311', 'table': '2852018311', 'type': 'unspecified'} https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce1/849122/8/check/neutron-ovs-tempest-dvr-ha-multinode-full/ce10281/testr_results.html https://d0b0b53d30de16fbad20-5f381a9e8c14b627196c6ef3340b4d4e.ssl.cf5.rackcdn.com/849122/8/check/neutron-ovs-grenade-dv
[Yahoo-eng-team] [Bug 1989057] [NEW] [master] [functional] test__get_dvr_subnet_ids_on_host_query failing with oslo.db-12.1.0 release
Public bug reported: Fails as below:- ft1.12: neutron.tests.functional.services.l3_router.test_l3_dvr_ha_router_plugin.L3DvrHATestCase.test__get_dvr_subnet_ids_on_host_querytesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/l3_router/test_l3_dvr_router_plugin.py", line 1754, in test__get_dvr_subnet_ids_on_host_query self.core_plugin.update_port( File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 702, in inner raise RuntimeError(_("Method %s cannot be called within a " RuntimeError: Method cannot be called within a transaction. Logs:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fa7/828488/1/check/neutron-functional-with-uwsgi/fa7c800/testr_results.html https://ae61986a45a6121e3a31-a2f117574c92282cf0ccc3fc53b9f219.ssl.cf2.rackcdn.com/855851/2/gate/neutron-functional-with-uwsgi/f56ec26/testr_results.html Happening after oslo.db updated to 12.1.0 in u-c. ** Affects: neutron Importance: Critical Status: New ** Changed in: neutron Importance: Undecided => Critical -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1989057 Title: [master] [functional] test__get_dvr_subnet_ids_on_host_query failing with oslo.db-12.1.0 release Status in neutron: New Bug description: Fails as below:- ft1.12: neutron.tests.functional.services.l3_router.test_l3_dvr_ha_router_plugin.L3DvrHATestCase.test__get_dvr_subnet_ids_on_host_querytesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/l3_router/test_l3_dvr_router_plugin.py", line 1754, in test__get_dvr_subnet_ids_on_host_query self.core_plugin.update_port( File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 702, in inner raise RuntimeError(_("Method %s cannot be called within a " RuntimeError: Method cannot be called within a transaction. Logs:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fa7/828488/1/check/neutron-functional-with-uwsgi/fa7c800/testr_results.html https://ae61986a45a6121e3a31-a2f117574c92282cf0ccc3fc53b9f219.ssl.cf2.rackcdn.com/855851/2/gate/neutron-functional-with-uwsgi/f56ec26/testr_results.html Happening after oslo.db updated to 12.1.0 in u-c. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1989057/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1992352] [NEW] [OVN] POST requests stucks when rabbitmq is not available
Public bug reported: As some of the operations relies on Messaging Callbacks[1][2], these requests get's stuck when messaging driver like rabbitmq is not available, For OVN without any other agent running, there is no consumer for these messages so these operations should skip messaging callbacks. To reproduce:- - Setup Devstack with OVN using https://github.com/openstack/neutron/blob/master/devstack/ovn-local.conf.sample - Comment transport_url(or modify port 5672 -> 5673) in neutron.conf and restart neutron services with sudo systemctl restart devstack@q* - Try operations like, openstack network delete, openstack router add/remove subnet etc. # In neutron logs can see too many oslo.messaging Errors for Access Denied(transport_url commented) or Connection Refused(transport_url updated to non listening port). oslo_messaging connection requests are also not needed for such cases so that can also be fixed. Actual Result:- These operations get's stuck Expected Result:- - These operations should succeed as there are no consumers for those callbacks. [1] https://github.com/openstack/neutron/blob/master/neutron/api/rpc/agentnotifiers/l3_rpc_agent_api.py#L44-L126 [2] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/rpc.py#L478-L510 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1992352 Title: [OVN] POST requests stucks when rabbitmq is not available Status in neutron: New Bug description: As some of the operations relies on Messaging Callbacks[1][2], these requests get's stuck when messaging driver like rabbitmq is not available, For OVN without any other agent running, there is no consumer for these messages so these operations should skip messaging callbacks. To reproduce:- - Setup Devstack with OVN using https://github.com/openstack/neutron/blob/master/devstack/ovn-local.conf.sample - Comment transport_url(or modify port 5672 -> 5673) in neutron.conf and restart neutron services with sudo systemctl restart devstack@q* - Try operations like, openstack network delete, openstack router add/remove subnet etc. # In neutron logs can see too many oslo.messaging Errors for Access Denied(transport_url commented) or Connection Refused(transport_url updated to non listening port). oslo_messaging connection requests are also not needed for such cases so that can also be fixed. Actual Result:- These operations get's stuck Expected Result:- - These operations should succeed as there are no consumers for those callbacks. [1] https://github.com/openstack/neutron/blob/master/neutron/api/rpc/agentnotifiers/l3_rpc_agent_api.py#L44-L126 [2] https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/rpc.py#L478-L510 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1992352/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1970679] Re: neutron-tempest-plugin-designate-scenario cross project job is failing on OVN
Based on https://bugs.launchpad.net/neutron/+bug/1970679/comments/5 and https://review.opendev.org/c/openstack/devstack/+/848548 closing the issue. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1970679 Title: neutron-tempest-plugin-designate-scenario cross project job is failing on OVN Status in neutron: Fix Released Bug description: The cross-project neutron-tempest-plugin-designate-scenario job is failing during the Designate gate runs due to an OVN failure. + lib/neutron_plugins/ovn_agent:start_ovn:698 : wait_for_sock_file /var/run/openvswitch/ovnnb_db.sock + lib/neutron_plugins/ovn_agent:wait_for_sock_file:173 : local count=0 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:174 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:175 : sleep 1 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:176 : count=1 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:177 : '[' 1 -gt 5 ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:174 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:175 : sleep 1 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:176 : count=2 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:177 : '[' 2 -gt 5 ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:174 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:175 : sleep 1 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:176 : count=3 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:177 : '[' 3 -gt 5 ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:174 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:175 : sleep 1 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:176 : count=4 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:177 : '[' 4 -gt 5 ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:174 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:175 : sleep 1 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:176 : count=5 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:177 : '[' 5 -gt 5 ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:174 : '[' '!' -S /var/run/openvswitch/ovnnb_db.sock ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:175 : sleep 1 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:176 : count=6 + lib/neutron_plugins/ovn_agent:wait_for_sock_file:177 : '[' 6 -gt 5 ']' + lib/neutron_plugins/ovn_agent:wait_for_sock_file:178 : die 178 'Socket /var/run/openvswitch/ovnnb_db.sock not found' + functions-common:die:264 : local exitcode=0 [Call Trace] ./stack.sh:1284:start_ovn_services /opt/stack/devstack/lib/neutron-legacy:516:start_ovn /opt/stack/devstack/lib/neutron_plugins/ovn_agent:698:wait_for_sock_file /opt/stack/devstack/lib/neutron_plugins/ovn_agent:178:die [ERROR] /opt/stack/devstack/lib/neutron_plugins/ovn_agent:178 Socket /var/run/openvswitch/ovnnb_db.sock not found exit_trap: cleaning up child processes An example job run is here: https://zuul.opendev.org/t/openstack/build/b014e50e018d426b9367fd3219ed489e To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1970679/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1979047] Re: Interface attach fails with libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1979047 Title: Interface attach fails with libvirt.libvirtError: internal error: unable to execute QEMU command'netdev_add': File descriptor named '(null)' has not been found Status in neutron: Fix Released Status in OpenStack Compute (nova): Fix Released Bug description: The tempest-integrated-compute-centos-9-stream job is broken since 2022-06-16 02:26:37 [1]. Multiple interface attach tempest test fails with: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found Full exception stack trace: [None req-6f95599e-022a-42ab-a4de-07c7b8f73daf tempest- AttachInterfacesTestJSON-2035965943 tempest- AttachInterfacesTestJSON-2035965943-project] [instance: b41513e6-cb4e-4441-af14-c272393cdafb] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] Traceback (most recent call last): Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2850, in attach_interface Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] guest.attach_device(cfg, persistent=True, live=live) Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 321, in attach_device Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] self._domain.attachDeviceFlags(device_xml, flags=flags) Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] result = proxy_call(self._autowrap, f, *args, **kwargs) Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] rv = execute(f, *args, **kwargs) Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] six.reraise(c, e, tb) Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/six.py", line 719, in reraise Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] raise value Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] rv = meth(*args, **kwargs) Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova- compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/lib64/python3.9/site-packages/libvirt.py", line 706, in attachDeviceFlags Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1
[Yahoo-eng-team] [Bug 1973783] Re: [devstack] Segment plugin reports Traceback as placement client not configured
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1973783 Title: [devstack] Segment plugin reports Traceback as placement client not configured Status in neutron: Fix Released Bug description: Following Traceback is reported although the job passes but creates a noise in logs, so should be cleared:- May 17 12:08:25.056617 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: DEBUG neutron_lib.callbacks.manager [None req-01d22b64-1fbc-4578-8cc2-c6565188c424 admin admin] Publish callbacks ['neutron.plugins.ml2.plugin.Ml2Plugin._handle_segment_change-1048155', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_segment_deleted-1983453', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_segment_deleted-495'] for segment (45896f0b-13b1-4cfc-ab32-297a8d8dae05), after_delete {{(pid=72995) _notify_loop /usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py:176}} May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: Traceback (most recent call last): May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: timer() May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/timer.py", line 59, in __call__ May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: cb(*args, **kw) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/common/utils.py", line 922, in wrapper May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: return func(*args, **kwargs) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/notifiers/batch_notifier.py", line 58, in synced_send May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: self._notify() May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/notifiers/batch_notifier.py", line 69, in _notify May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: self.callback(batched_events) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/services/segments/plugin.py", line 211, in _send_notifications May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: event.method(event) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/services/segments/plugin.py", line 383, in _delete_nova_inventory May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: aggregate_id = self._get_aggregate_id(event.segment_id) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/opt/stack/neutron/neutron/services/segments/plugin.py", line 370, in _get_aggregate_id May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: aggregate_uuid = self.p_client.list_aggregates( May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/neutron_lib/placement/client.py", line 58, in wrapper May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: return f(self, *a, **k) May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/neutron_lib/placement/client.py", line 554, in list_aggregates May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: return self._get(url).json() May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/neutron_lib/placement/client.py", line 190, in _get May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: return self._client.get(url, endpoint_filter=self._ks_filter, May 17 12:08:25.069453 nested-virt-ubuntu-focal-ovh-bhs1-0029671721 neutron-server[72995]: File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/session.py", line 1141, in get May 17 12:08:25.069453 nested-virt-ubuntu
[Yahoo-eng-team] [Bug 1972764] Re: [Wallaby] OVNPortForwarding._handle_lb_on_ls fails with TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context'
** Changed in: neutron Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1972764 Title: [Wallaby] OVNPortForwarding._handle_lb_on_ls fails with TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' Status in neutron: Fix Released Bug description: It's failing with below TraceBack:- 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager [req-2e5ef575-b18c-4091-8c15-b37f6bcf0fdd f1840520501c41b2a6a534525f0f90a4 bf49659cd4cb40edb393b914198ce3c9 - default default] Error during notification for neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding._handle_lb_on_ls-4305748 router_interface, after_create: TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' 2022-05-10 00:38:36.539 ERROR /var/log/containers/neutron/server.log: 15 ERROR neutron_lib.callbacks.manager Was noticed in a TripleO job https://bugs.launchpad.net/tripleo/+bug/1972660. This method was added in https://review.opendev.org/q/I0c4d492887216cad7a8155dceb738389f2886376 and backported till wallaby. Xena+ are ok, only wallaby impacted because before xena old notification format is used where arguments are passed as kwargs. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1972764/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1999154] [NEW] ovs/ovn source deployment broken with ovs_branch=master
Public bug reported: Since [1] jobs running with OVS_BRANCH=master are broken, fails as below:- utilities/ovn-dbctl.c: In function ‘do_dbctl’: utilities/ovn-dbctl.c:724:9: error: too few arguments to function ‘ctl_context_init_command’ 724 | ctl_context_init_command(ctx, c); | ^~~~ In file included from utilities/ovn-dbctl.c:23: /opt/stack/ovs/lib/db-ctl-base.h:249:6: note: declared here 249 | void ctl_context_init_command(struct ctl_context *, struct ctl_command *, | ^~~~ make[1]: *** [Makefile:2352: utilities/ovn-dbctl.o] Error 1 make[1]: *** Waiting for unfinished jobs make[1]: Leaving directory '/opt/stack/ovn' make: *** [Makefile:1548: all] Error 2 + lib/neutron_plugins/ovn_agent:compile_ovn:1 : exit_trap Failure builds example:- - https://zuul.opendev.org/t/openstack/build/3a900a1cfe824746ac8ffc6a27fc8ec4 - https://zuul.opendev.org/t/openstack/build/7d862338d6194a4fb3a34e8c3c67f532 - https://zuul.opendev.org/t/openstack/build/ae092f4985af41908697240e3f64f522 Until OVN repo[2] get's updated to work with ovs master we have to pin ovs to working version to get these experimental jobs back to green. [1] https://github.com/openvswitch/ovs/commit/b8bf410a5c94173da02279b369d75875c4035959 [2] https://github.com/ovn-org/ovn ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1999154 Title: ovs/ovn source deployment broken with ovs_branch=master Status in neutron: New Bug description: Since [1] jobs running with OVS_BRANCH=master are broken, fails as below:- utilities/ovn-dbctl.c: In function ‘do_dbctl’: utilities/ovn-dbctl.c:724:9: error: too few arguments to function ‘ctl_context_init_command’ 724 | ctl_context_init_command(ctx, c); | ^~~~ In file included from utilities/ovn-dbctl.c:23: /opt/stack/ovs/lib/db-ctl-base.h:249:6: note: declared here 249 | void ctl_context_init_command(struct ctl_context *, struct ctl_command *, | ^~~~ make[1]: *** [Makefile:2352: utilities/ovn-dbctl.o] Error 1 make[1]: *** Waiting for unfinished jobs make[1]: Leaving directory '/opt/stack/ovn' make: *** [Makefile:1548: all] Error 2 + lib/neutron_plugins/ovn_agent:compile_ovn:1 : exit_trap Failure builds example:- - https://zuul.opendev.org/t/openstack/build/3a900a1cfe824746ac8ffc6a27fc8ec4 - https://zuul.opendev.org/t/openstack/build/7d862338d6194a4fb3a34e8c3c67f532 - https://zuul.opendev.org/t/openstack/build/ae092f4985af41908697240e3f64f522 Until OVN repo[2] get's updated to work with ovs master we have to pin ovs to working version to get these experimental jobs back to green. [1] https://github.com/openvswitch/ovs/commit/b8bf410a5c94173da02279b369d75875c4035959 [2] https://github.com/ovn-org/ovn To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1999154/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1999240] Re: [FT] Error in "test_sg_stateful_toggle_updates_ovn_acls"
Specific to patch https://review.opendev.org/c/openstack/neutron/+/867075 as older ovs/ovn versions are setup:- OVN_BRANCH: v20.06.1 OVS_BRANCH: 0047ca3a0290f1ef954f2c76b31477cf4b9755f5 ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1999240 Title: [FT] Error in "test_sg_stateful_toggle_updates_ovn_acls" Status in neutron: Invalid Bug description: Logs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce0/867075/3/check/neutron- functional-with-uwsgi/ce08f60/testr_results.html Snippet: https://paste.opendev.org/show/bKuQzDvF6UshmKItbs6r/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1999240/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1999239] Re: [FT] Error in "test_agent_list"
Specific to patch https://review.opendev.org/c/openstack/neutron/+/867075 as older ovs/ovn versions are setup:- OVN_BRANCH: v20.06.1 OVS_BRANCH: 0047ca3a0290f1ef954f2c76b31477cf4b9755f5 ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1999239 Title: [FT] Error in "test_agent_list" Status in neutron: Invalid Bug description: Log: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce0/867075/3/check/neutron- functional-with-uwsgi/ce08f60/testr_results.html Snippet: https://paste.opendev.org/show/bnHiQN58tZoFIHtjMhTi/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1999239/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1999238] Re: [FT] Error in "test_router_port_binding"
Specific to patch https://review.opendev.org/c/openstack/neutron/+/867075 as older ovs/ovn versions are setup:- OVN_BRANCH: v20.06.1 OVS_BRANCH: 0047ca3a0290f1ef954f2c76b31477cf4b9755f5 ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1999238 Title: [FT] Error in "test_router_port_binding" Status in neutron: Invalid Bug description: Log: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce0/867075/3/check/neutron- functional-with-uwsgi/ce08f60/testr_results.html Snippet: https://paste.opendev.org/show/bJ4hUYzj0tkqcjMU7o0f/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1999238/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2002629] Re: devstack build in the gate fails with: ovnnb_db.sock: database connection failed
27; -f /var/lib/ovn/ovnsb_db.db ']' 2023-01-12 10:45:50.215127 | controller | + lib/neutron_plugins/ovn_agent:start_ovn:716 : is_service_enabled tls-proxy 2023-01-12 10:45:50.232297 | controller | + functions-common:is_service_enabled:2089 : return 0 2023-01-12 10:45:50.234744 | controller | + lib/neutron_plugins/ovn_agent:start_ovn:717 : sudo ovn-nbctl --db=unix:/var/run/ovn/ovnnb_db.sock set-ssl /opt/stack/data/CA/int-ca/private/devstack-cert.key /opt/stack/data/CA/int-ca/devstack-cert.crt /opt/stack/data/CA/int-ca/ca-chain.pem 2023-01-12 10:45:50.244116 | controller | ovn-nbctl: unix:/var/run/ovn/ovnnb_db.sock: database connection failed (No such file or directory) ◠ovn-ovsdb-server-nb.service - Open vSwitch database server for OVN Northbound database Loaded: loaded (/lib/systemd/system/ovn-ovsdb-server-nb.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2023-01-12 10:45:49 UTC; 1min 4s ago Main PID: 77511 (ovsdb-server) Tasks: 1 (limit: 9247) Memory: 2.8M CGroup: /system.slice/ovn-ovsdb-server-nb.service └─77511 ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/ovn/ovsdb-server-nb.log --remote=punix:/var/run/ovn/ovnnb_db.sock --pidfile=/var/run/ovn/ovnnb_db.pid --unixctl=/var/run/ovn/ovnnb_db.ctl --remote=db:OVN_Northbound,NB_Global,connections --private-key=db:OVN_Northbound,SSL,private_key --certificate=db:OVN_Northbound,SSL,certificate --ca-cert=db:OVN_Northbound,SSL,ca_cert --ssl-protocols=db:OVN_Northbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Northbound,SSL,ssl_ciphers /var/lib/ovn/ovnnb_db.db Jan 12 10:45:49 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 systemd[1]: ovn-ovsdb-server-nb.service: Succeeded. Jan 12 10:45:49 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 systemd[1]: Stopped Open vSwitch database server for OVN Northbound database. Jan 12 10:45:49 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 systemd[1]: Started Open vSwitch database server for OVN Northbound database. Jan 12 10:45:50 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 ovn-ctl[77511]: * /var/lib/ovn/ovnnb_db.db does not exist Jan 12 10:46:05 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 ovn-ctl[77511]: * Creating empty database /var/lib/ovn/ovnnb_db.db Jan 12 10:46:05 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 ovsdb-server[77511]: ovs|1|vlog|INFO|opened log file /var/log/ovn/ovsdb-server-nb.log Jan 12 10:46:05 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 ovsdb-server[77511]: ovs|2|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.13.8 Jan 12 10:46:15 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 ovsdb-server[77511]: ovs|3|memory|INFO|7272 kB peak resident set size after 10.0 seconds Jan 12 10:46:15 nested-virt-ubuntu-focal-ovh-bhs1-0032709305 ovsdb-server[77511]: ovs|4|memory|INFO|cells:31 monitors:2 sessions:1 Ensuring sock files are absent just like dbs should be able to handle this issue. Will push a patch to devstack. ** Changed in: neutron Importance: Undecided => High ** Also affects: devstack Importance: Undecided Status: New ** Changed in: devstack Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2002629 Title: devstack build in the gate fails with: ovnnb_db.sock: database connection failed Status in devstack: New Status in neutron: New Bug description: Recently we seem to have many the same devstack build failure in many different gate jobs. The usual error message is: + lib/neutron_plugins/ovn_agent:start_ovn:714 : wait_for_db_file /var/lib/ovn/ovnsb_db.db + lib/neutron_plugins/ovn_agent:wait_for_db_file:175 : local count=0 + lib/neutron_plugins/ovn_agent:wait_for_db_file:176 : '[' '!' -f /var/lib/ovn/ovnsb_db.db ']' + lib/neutron_plugins/ovn_agent:start_ovn:716 : is_service_enabled tls-proxy + functions-common:is_service_enabled:2089 : return 0 + lib/neutron_plugins/ovn_agent:start_ovn:717 : sudo ovn-nbctl --db=unix:/var/run/ovn/ovnnb_db.sock set-ssl /opt/stack/data/CA/int-ca/private/devstack-cert.key /opt/stack/data/CA/int-ca/devstack-cert.crt /opt/stack/data/CA/int-ca/ca-chain.pem ovn-nbctl: unix:/var/run/ovn/ovnnb_db.sock: database connection failed (No such file or directory) + lib/neutron_plugins/ovn_agent:start_ovn:1 : exit_trap A few example logs: https://zuul.opendev.org/t/openstack/build/ec852d75c8094afcb4140871bc9ffa36 https://zuul.opendev.org/t/openstack/build/eae988aa8cd24c78894a3d3438392357 The search expression 'message:"ovnnb_db.sock: database connection failed"' gives me 1200+ hits in https://opensearch.logs.openstack.org for the last 2 weeks. To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/2002629/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2006683] [NEW] fwaas scenario job fails randomly
Public bug reported: fwass scenario job fails randomly on random tests Example failures:- test test_update_firewall_group fails:- - https://2c1464b5b351d9aa5e93-14387ee33f74a27c93fba699ca02403e.ssl.cf1.rackcdn.com/869152/3/gate/neutron-tempest-plugin-fwaas/d9df246/testr_results.html - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_26d/873060/1/check/neutron-tempest-plugin-fwaas/26d3325/testr_results.html traceback-5: {{{ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 330, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 720, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 841, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'FirewallRuleInUse', 'message': 'Firewall rule ddaf801f-b279-41f4-a793-bd02e37c886e is being used.', 'detail': ''} }}} Traceback (most recent call last): File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 341, in test_update_firewall_group self.firewall_groups_client.delete_firewall_group(fwg_id) File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 330, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 720, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 841, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 0845629f-cb8a-4796-8bb5-e8df06c6919f is still active.', 'detail': ''} Fails as:- Feb 07 22:44:15.544863 np0033003110 neutron-server[57277]: INFO neutron.api.v2.resource [None req-3d968b8a-c24c-43c1-9d82-ded509555450 tempest-FWaaSv2ExtensionTestJSON-1939451253 tempest-FWaaSv2ExtensionTestJSON-1939451253-project] create failed (client error): There was a conflict when trying to complete your request. Feb 07 22:44:15.545878 np0033003110 neutron-server[57277]: INFO neutron.wsgi [None req-3d968b8a-c24c-43c1-9d82-ded509555450 tempest-FWaaSv2ExtensionTestJSON-1939451253 tempest-FWaaSv2ExtensionTestJSON-1939451253-project] 149.202.171.72,149.202.171.72 "POST /networking/v2.0/fwaas/firewall_groups HTTP/1.1" status: 409 len: 378 time: 0.0383568 test_create_show_delete_firewall_group:- - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a1d/873060/1/check/neutron-tempest-plugin-fwaas/a1da5c4/testr_results.html - https://c4dea4e2ebaa3412b0a2-051d59c98d54ce1c08087cd83140c470.ssl.cf1.rackcdn.com/873060/1/check/neutron-tempest-plugin-fwaas-zed/7ade93d/testr_results.html Traceback as:- Feb 09 01:35:54.898784 np0033019570 neutron-server[57239]: DEBUG neutron_fwaas.db.firewall.v2.firewall_db_v2 [None req-f56695ab-2f6b-4035-8b61-74bcaa43898f tempest-FWaaSv2ExtensionTestJSON-845501349 tempest-FWaaSv2ExtensionTestJSON-845501349-project] Default FWG was concurrently created {{(pid=57239) _ensure_default_firewall_group /opt/stack/neutron-fwaas/neutron_fwaas/db/firewall/v2/firewall_db_v2.py:940}} Feb 09 01:35:54.920152 np0033019570 neutron-server[57239]: ERROR neutron.api.v2.resource [None req-f56695ab-2f6b-4035-8b61-74bcaa43898f tempest-FWaaSv2ExtensionTestJSON-845501349 tempest-FWaaSv2ExtensionTestJSON-845501349-project] create failed: No details.: sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'c119661f432d4bc5aaf2308e8a8bd47a' for key 'default_firewall_groups.PRIMARY'") Feb 09 01:35:54.920152
[Yahoo-eng-team] [Bug 2007986] Re: [neutron-lib][stable] Wallaby CI, tempest failing: unknown environment 'integrated-full'
Being fixed in Tempest https://review.opendev.org/c/openstack/tempest/+/874704 ** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2007986 Title: [neutron-lib][stable] Wallaby CI, tempest failing: unknown environment 'integrated-full' Status in neutron: Invalid Status in tempest: In Progress Bug description: Neutron-lib Wallaby CI, job "tempest-full-py3" is failing with error: controller | ERROR: unknown environment 'integrated-full' Logs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cf8/874412/1/check/tempest- full-py3/cf87e9c/job-output.txt To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2007986/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1463631] Re: 60_nova/resources.sh:106:ping_check_public fails intermittently
** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1463631 Title: 60_nova/resources.sh:106:ping_check_public fails intermittently Status in grenade: Confirmed Status in neutron: Invalid Status in OpenStack Compute (nova): Invalid Bug description: http://logs.openstack.org/12/186112/17/gate/gate-grenade- dsvm/4da364e/logs/grenade.sh.txt.gz#_2015-06-09_22_42_15_929 2015-06-09 22:42:13.960 | --- 172.24.5.1 ping statistics --- 2015-06-09 22:42:13.960 | 1 packets transmitted, 0 received, 100% packet loss, time 0ms 2015-06-09 22:42:13.960 | 2015-06-09 22:42:15.929 | + [[ True = \T\r\u\e ]] 2015-06-09 22:42:15.929 | + die 67 '[Fail] Couldn'\''t ping server' 2015-06-09 22:42:15.929 | + local exitcode=0 2015-06-09 22:42:15.929 | [Call Trace] 2015-06-09 22:42:15.929 | /opt/stack/new/grenade/projects/60_nova/resources.sh:134:verify 2015-06-09 22:42:15.929 | /opt/stack/new/grenade/projects/60_nova/resources.sh:101:verify_noapi 2015-06-09 22:42:15.929 | /opt/stack/new/grenade/projects/60_nova/resources.sh:106:ping_check_public 2015-06-09 22:42:15.929 | /opt/stack/new/grenade/functions:67:die 2015-06-09 22:42:15.931 | [ERROR] /opt/stack/new/grenade/functions:67 [Fail] Couldn't ping server 2015-06-09 22:42:16.933 | 1 die /opt/stack/old/devstack/functions-common 2015-06-09 22:42:16.933 | 67 ping_check_public /opt/stack/new/grenade/functions 2015-06-09 22:42:16.933 | 106 verify_noapi /opt/stack/new/grenade/projects/60_nova/resources.sh 2015-06-09 22:42:16.933 | 101 verify /opt/stack/new/grenade/projects/60_nova/resources.sh 2015-06-09 22:42:16.933 | 134 main /opt/stack/new/grenade/projects/60_nova/resources.sh 2015-06-09 22:42:16.933 | Exit code: 1 2015-06-09 22:42:16.961 | World dumping... see /opt/stack/old/worlddump-2015-06-09-224216.txt for details 2015-06-09 22:42:26.139 | [Call Trace] 2015-06-09 22:42:26.139 | ./grenade.sh:250:resources 2015-06-09 22:42:26.139 | /opt/stack/new/grenade/inc/plugin:82:die 2015-06-09 22:42:26.141 | [ERROR] /opt/stack/new/grenade/inc/plugin:82 Failed to run /opt/stack/new/grenade/projects/60_nova/resources.sh verify I wonder if there is a race in setting up security groups. http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiW0ZhaWxdIENvdWxkbid0IHBpbmcgc2VydmVyXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImN1c3RvbSIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNS0wNS0yN1QwMDozMDoxNiswMDowMCIsInRvIjoiMjAxNS0wNi0xMFQwMDozMDoxNiswMDowMCIsInVzZXJfaW50ZXJ2YWwiOiIwIn0sInN0YW1wIjoxNDMzODk2MjUwNTAyfQ== This hits in nova-network and neutron grenade jobs. To manage notifications about this bug go to: https://bugs.launchpad.net/grenade/+bug/1463631/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2015728] [NEW] ovs/ovn source(with master branch) deployments broken
Public bug reported: With [1] ovn/ovs jobs running with OVS_BRANCH=master,OVN_BRANCH=main are broken, fails as below:- utilities/ovn-dbctl.c: In function ‘server_loop’: utilities/ovn-dbctl.c:1105:5: error: too few arguments to function ‘daemonize_start’ 1105 | daemonize_start(false); | ^~~ In file included from utilities/ovn-dbctl.c:22: /opt/stack/ovs/lib/daemon.h:170:6: note: declared here 170 | void daemonize_start(bool access_datapath, bool access_hardware_ports); | ^~~ make[1]: *** [Makefile:2374: utilities/ovn-dbctl.o] Error 1 make[1]: *** Waiting for unfinished jobs Example failure:- https://zuul.openstack.org/build/b7b1700e2e5941f7a52b57ca411db722 Builds:- - https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ipv6-only-ovs-master - https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-full-multinode-ovs-master - https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master - https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master-centos-9-stream - https://zuul.openstack.org/builds?job_name=ovn-octavia-provider-functional-master - https://zuul.openstack.org/builds?job_name=ovn-octavia-provider-tempest-master Until ovn main branch is adapted to this change we need to pin ovs_branch to working commit or better stable branch(as done with [2]) Also i noticed some of these jobs running in neutron/ovn-octavia stable branches, that likely not needed to be run, so should be checked and cleaned up. [1] https://github.com/openvswitch/ovs/commit/07cf5810de [2] https://github.com/ovn-org/ovn/commit/b61e819bf9673 ** Affects: neutron Importance: High Assignee: yatin (yatinkarel) Status: Confirmed ** Tags: gat ** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Importance: Undecided => Critical ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) ** Tags added: gat ** Changed in: neutron Importance: Critical => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2015728 Title: ovs/ovn source(with master branch) deployments broken Status in neutron: Confirmed Bug description: With [1] ovn/ovs jobs running with OVS_BRANCH=master,OVN_BRANCH=main are broken, fails as below:- utilities/ovn-dbctl.c: In function ‘server_loop’: utilities/ovn-dbctl.c:1105:5: error: too few arguments to function ‘daemonize_start’ 1105 | daemonize_start(false); | ^~~ In file included from utilities/ovn-dbctl.c:22: /opt/stack/ovs/lib/daemon.h:170:6: note: declared here 170 | void daemonize_start(bool access_datapath, bool access_hardware_ports); | ^~~ make[1]: *** [Makefile:2374: utilities/ovn-dbctl.o] Error 1 make[1]: *** Waiting for unfinished jobs Example failure:- https://zuul.openstack.org/build/b7b1700e2e5941f7a52b57ca411db722 Builds:- - https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ipv6-only-ovs-master - https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-full-multinode-ovs-master - https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master - https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master-centos-9-stream - https://zuul.openstack.org/builds?job_name=ovn-octavia-provider-functional-master - https://zuul.openstack.org/builds?job_name=ovn-octavia-provider-tempest-master Until ovn main branch is adapted to this change we need to pin ovs_branch to working commit or better stable branch(as done with [2]) Also i noticed some of these jobs running in neutron/ovn-octavia stable branches, that likely not needed to be run, so should be checked and cleaned up. [1] https://github.com/openvswitch/ovs/commit/07cf5810de [2] https://github.com/ovn-org/ovn/commit/b61e819bf9673 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2015728/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2015364] Re: [skip-level] OVN tests constantly failing
** Also affects: devstack Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2015364 Title: [skip-level] OVN tests constantly failing Status in devstack: In Progress Status in neutron: Confirmed Bug description: In the new Zed-Bobcat skip-level jobs [1], the OVN job has 4 tests constantly failing (1 fail is actually a setup class method): *tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops *tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesUnderV243Test.test_add_remove_fixed_ip *setUpClass (tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON) *tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops Logs: *https://fd50651997fbb0337883-282d0b18354725863279cd3ebda4ab44.ssl.cf5.rackcdn.com/878632/6/experimental/neutron-ovn-grenade-multinode-skip-level/baf4ed5/controller/logs/grenade.sh_log.txt *https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_607/878632/6/experimental/neutron-ovn-grenade-multinode-skip-level/6072d85/controller/logs/grenade.sh_log.txt [1]https://review.opendev.org/c/openstack/neutron/+/878632 To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/2015364/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2017131] [NEW] neutron-loki job fails tests with ORM session: SQL execution without transaction in progress
Public bug reported: Job(neutron-ovn-tempest-with-uwsgi-loki) running neutron-loki service[1] fails tempest tests randomly with Tracebacks like below:- ORM session: SQL execution without transaction in progress, traceback: Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/greenthread.py", line 221, in main Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: result = function(*args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 837, in process_request Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: proto.__init__(conn_state, self) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 350, in __init__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: self.handle() Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 383, in handle Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: self.handle_one_request() Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 459, in handle_one_request Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: self.handle_one_response() Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 569, in handle_one_response Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: result = self.application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/paste/urlmap.py", line 216, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return app(environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: response = req.get_response(self.application) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: status, headers, app_iter = self.call_application( Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: app_iter = application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: response = req.get_response(self.application) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: status, headers, app_iter = self.call_application( Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: app_iter = application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 n
[Yahoo-eng-team] [Bug 2017478] [NEW] openstack-tox-py38 broken with tooz==4.0.0
Public bug reported: openstack-tox-py38 broken[1] since tooz update in upper-constratints [2], tooz-4.0.0 requires python>=3.9[3], fails as below:- 2023-04-23 22:45:20.084152 | ubuntu-focal | The conflict is caused by: 2023-04-23 22:45:20.084169 | ubuntu-focal | The user requested tooz>=1.58.0 2023-04-23 22:45:20.084184 | ubuntu-focal | The user requested (constraint) tooz===4.0.0 Looking at [4] we can drop py38 jobs as min supported python version is 3.9 in 2023.2. [1] https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py38&project=openstack%2Fneutron&branch=master&skip=0 [2] https://review.opendev.org/c/openstack/requirements/+/880738 [3] https://review.opendev.org/c/openstack/tooz/+/879930 [4] https://review.opendev.org/c/openstack/governance/+/872232 ** Affects: neutron Importance: Critical Assignee: yatin (yatinkarel) Status: Triaged ** Tags: gate-failure unittest ** Tags added: gate-failure unittest ** Changed in: neutron Importance: Undecided => Critical ** Changed in: neutron Status: New => Triaged ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2017478 Title: openstack-tox-py38 broken with tooz==4.0.0 Status in neutron: Triaged Bug description: openstack-tox-py38 broken[1] since tooz update in upper-constratints [2], tooz-4.0.0 requires python>=3.9[3], fails as below:- 2023-04-23 22:45:20.084152 | ubuntu-focal | The conflict is caused by: 2023-04-23 22:45:20.084169 | ubuntu-focal | The user requested tooz>=1.58.0 2023-04-23 22:45:20.084184 | ubuntu-focal | The user requested (constraint) tooz===4.0.0 Looking at [4] we can drop py38 jobs as min supported python version is 3.9 in 2023.2. [1] https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py38&project=openstack%2Fneutron&branch=master&skip=0 [2] https://review.opendev.org/c/openstack/requirements/+/880738 [3] https://review.opendev.org/c/openstack/tooz/+/879930 [4] https://review.opendev.org/c/openstack/governance/+/872232 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2017478/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2017992] [NEW] Jobs running on vexxhost provider failing with Mirror Issues
Public bug reported: Fails as below:- 2023-04-27 12:36:47.535483 | controller | ++ functions-common:apt_get_update:1155 : timeout 300 sh -c 'while ! sudo http_proxy= https_proxy= no_proxy= apt-get update; do sleep 30; done' 2023-04-27 12:36:50.877357 | controller | Err:1 https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu focal InRelease 2023-04-27 12:36:50.877408 | controller | Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (2604:e100:1:0:f816:3eff:fe0c:e2c0). - connect (113: No route to host) Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (199.204.45.149). - connect (113: No route to host) 2023-04-27 12:36:50.877419 | controller | Err:2 https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu focal-updates InRelease 2023-04-27 12:36:50.877428 | controller | Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.877437 | controller | Err:3 https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu focal-backports InRelease 2023-04-27 12:36:50.877446 | controller | Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.877454 | controller | Err:4 https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu focal-security InRelease 2023-04-27 12:36:50.877462 | controller | Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.892401 | controller | Reading package lists... 2023-04-27 12:36:50.895427 | controller | W: Failed to fetch https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu/dists/focal/InRelease Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (2604:e100:1:0:f816:3eff:fe0c:e2c0). - connect (113: No route to host) Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (199.204.45.149). - connect (113: No route to host) 2023-04-27 12:36:50.895462 | controller | W: Failed to fetch https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu/dists/focal-updates/InRelease Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.895539 | controller | W: Failed to fetch https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu/dists/focal-backports/InRelease Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.895604 | controller | W: Failed to fetch https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu/dists/focal-security/InRelease Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.895677 | controller | W: Some index files failed to download. They have been ignored, or old ones used instead. 2023-04-27 12:36:51.007845 | controller | 2023-04-27 12:36:51.008040 | controller | E: Unable to locate package apache2 2023-04-27 12:36:51.008098 | controller | E: Unable to locate package apache2-dev 2023-04-27 12:36:51.008157 | controller | E: Unable to locate package bc 2023-04-27 12:36:51.008218 | controller | E: Package 'bsdmainutils' has no installation candidate 2023-04-27 12:36:51.008321 | controller | E: Unable to locate package gawk 2023-04-27 12:36:51.008382 | controller | E: Unable to locate package gettext 2023-04-27 12:36:51.008438 | controller | E: Unable to locate package graphviz Example failures:- - https://db3cff29bdb713e861e1-7db0c1fa1bd98a0adf758f7c1d49f672.ssl.cf5.rackcdn.com/881735/4/check/neutron-tempest-plugin-openvswitch-enforce-scope-new-defaults/278dce1/job-output.txt - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/881742/4/check/neutron-tempest-plugin-designate-scenario/9444d54/job-output.txt It's happening randomly from last couple of weeks, seems some issue in general with IPv6 external connectivity. >From opensearch[1] it can be seen following host_ids are impacted:- c984fb897502bc826ccaf0e258b6071e76c29b305bc5b31b301de76a 1b1e841d5cdc8c40a480da993c1cbf0cd64900baff612378898e72ab 6cc97bc57f540569368fcc47255180c5d21ed00a22cad83eeb600cec 86c687840cd74cd63cbb095748afa5c9cd0f6fcea898d90aa030cc68 94cd367e7821f5d74cf44c5ebafd9af18d2b6dff64a9bee067337cf6 8926aa5796637312bf5e46a0671a88021c208235fafdfcf22931eb01 70670f45d0dc4eaae28e6553525eec409dfb6f80e8d6c8dcef7d7bf5 And seems like it started happening when nodes were upgraded as part of fixing nested-virt issue[2] [1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:filename,negate:!f,params:(query:job-output.txt),type:phrase),query:(match_phrase:(filename:job-output.txt,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Package%20bsdmainutils%20is%20not%20available,%20but%20is%20referred%20to%20by%20another%20package%22'),sort:!()) [2] https://bugs.launchpad.net/neutron/+bug/1999249/comments/3 ** Affects: neutron Importance: Critical
[Yahoo-eng-team] [Bug 2018130] [NEW] [master][functional] test_cascading_del_in_txn fails with ovsdbapp=2.3.0
Public bug reported: With ovsdbapp==2.3.0 [1] releaase functional test neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_cascading_del_in_txn fails consistently[2] as below:- ft1.7: neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_cascading_del_in_txntesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/test_ovs_lib.py", line 493, in test_cascading_del_in_txn self.assertRaises((RuntimeError, idlutils.RowNotFound), File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 468, in assertRaises self.assertThat(our_callable, matcher) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 481, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: .del_port_mod_iface at 0x7f35bbe8af80> returned None Seems to be caused with [3] as db_set will not fail if row is not found(if_exists=True). We need to add if_exists=false to keep the same behavior as test or update test so it succeed with current behavior [1] https://review.opendev.org/c/openstack/requirements/+/881736 [2] https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional-with-uwsgi&project=openstack%2Fneutron&branch=master&skip=0 [3] https://github.com/openstack/ovsdbapp/commit/6ab3f75388f583c99ac38d9c651f2529d68b617f ** Affects: neutron Importance: Critical Status: Confirmed ** Tags: functional-tests gate-failure ** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Importance: Undecided => Critical ** Tags added: functional-tests gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2018130 Title: [master][functional] test_cascading_del_in_txn fails with ovsdbapp=2.3.0 Status in neutron: Confirmed Bug description: With ovsdbapp==2.3.0 [1] releaase functional test neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_cascading_del_in_txn fails consistently[2] as below:- ft1.7: neutron.tests.functional.agent.test_ovs_lib.OVSBridgeTestCase.test_cascading_del_in_txntesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/test_ovs_lib.py", line 493, in test_cascading_del_in_txn self.assertRaises((RuntimeError, idlutils.RowNotFound), File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 468, in assertRaises self.assertThat(our_callable, matcher) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 481, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: .del_port_mod_iface at 0x7f35bbe8af80> returned None Seems to be caused with [3] as db_set will not fail if row is not found(if_exists=True). We need to add if_exists=false to keep the same behavior as test or update test so it succeed with current behavior [1] https://review.opendev.org/c/openstack/requirements/+/881736 [2] https://zuul.opendev.org/t/openstack/builds?job_name=neutron-functional-with-uwsgi&project=openstack%2Fneutron&branch=master&skip=0 [3] https://github.com/openstack/ovsdbapp/commit/6ab3f75388f583c99ac38d9c651f2529d68b617f To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2018130/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2018967] [NEW] [fwaas] test_update_firewall_group fails randomly
Public bug reported: Seen twice till now recently:- - https://a78793e982809689fe25-25fa16d377ec97c08c4e6ce3af683bd9.ssl.cf5.rackcdn.com/881232/1/check/neutron-tempest-plugin-fwaas/b0730f9/testr_results.html - https://53a7c53d508ecea7485c-f8ccc2b7c32dd8ba5caab7dc1c36a741.ssl.cf5.rackcdn.com/881232/1/gate/neutron-tempest-plugin-fwaas/5712826/testr_results.html Fails as below:- traceback-1: {{{ Traceback (most recent call last): File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 130, in _try_delete_firewall_group self.firewall_groups_client.delete_firewall_group(fwg_id) File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 57266ed6-c39c-4be2-80d8-649469adf7eb is still active.', 'detail': ''} }}} traceback-2: {{{ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 100, in delete_firewall_policy return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'FirewallPolicyInUse', 'message': 'Firewall policy 0e23c50e-28a9-41e5-829c-9a67d058bafd is being used.', 'detail': ''} }}} traceback-3: {{{ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 100, in delete_firewall_policy return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'FirewallPolicyInUse', 'message': 'Firewall policy a47d0031-65cb-40bf-86e1-ba3842c295aa is being used.', 'detail': ''} }}} traceback-4: {{{ Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc return func(*args, **kwargs) File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'Fi
[Yahoo-eng-team] [Bug 2017992] Re: Jobs running on vexxhost provider failing with Mirror Issues
We got an update[1] from vexxhost Team and now the vexxhost node provider is re enabled[2]. Since it reenabled we are not seeing the issue. Noticed till now the jobs ran on 3 impacted nodes and is passing so can consider the issue resolved. Closing the bug. [1] We have performed some optimizations on our compute pool. You can re-enable for now and see how it goes. [2] https://review.opendev.org/c/openstack/project-config/+/882787 ** Changed in: neutron Status: Triaged => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2017992 Title: Jobs running on vexxhost provider failing with Mirror Issues Status in neutron: Invalid Bug description: Fails as below:- 2023-04-27 12:36:47.535483 | controller | ++ functions-common:apt_get_update:1155 : timeout 300 sh -c 'while ! sudo http_proxy= https_proxy= no_proxy= apt-get update; do sleep 30; done' 2023-04-27 12:36:50.877357 | controller | Err:1 https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu focal InRelease 2023-04-27 12:36:50.877408 | controller | Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (2604:e100:1:0:f816:3eff:fe0c:e2c0). - connect (113: No route to host) Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (199.204.45.149). - connect (113: No route to host) 2023-04-27 12:36:50.877419 | controller | Err:2 https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu focal-updates InRelease 2023-04-27 12:36:50.877428 | controller | Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.877437 | controller | Err:3 https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu focal-backports InRelease 2023-04-27 12:36:50.877446 | controller | Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.877454 | controller | Err:4 https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu focal-security InRelease 2023-04-27 12:36:50.877462 | controller | Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.892401 | controller | Reading package lists... 2023-04-27 12:36:50.895427 | controller | W: Failed to fetch https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu/dists/focal/InRelease Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (2604:e100:1:0:f816:3eff:fe0c:e2c0). - connect (113: No route to host) Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (199.204.45.149). - connect (113: No route to host) 2023-04-27 12:36:50.895462 | controller | W: Failed to fetch https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu/dists/focal-updates/InRelease Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.895539 | controller | W: Failed to fetch https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu/dists/focal-backports/InRelease Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.895604 | controller | W: Failed to fetch https://mirror.ca-ymq-1.vexxhost.opendev.org/ubuntu/dists/focal-security/InRelease Unable to connect to mirror.ca-ymq-1.vexxhost.opendev.org:https: 2023-04-27 12:36:50.895677 | controller | W: Some index files failed to download. They have been ignored, or old ones used instead. 2023-04-27 12:36:51.007845 | controller | 2023-04-27 12:36:51.008040 | controller | E: Unable to locate package apache2 2023-04-27 12:36:51.008098 | controller | E: Unable to locate package apache2-dev 2023-04-27 12:36:51.008157 | controller | E: Unable to locate package bc 2023-04-27 12:36:51.008218 | controller | E: Package 'bsdmainutils' has no installation candidate 2023-04-27 12:36:51.008321 | controller | E: Unable to locate package gawk 2023-04-27 12:36:51.008382 | controller | E: Unable to locate package gettext 2023-04-27 12:36:51.008438 | controller | E: Unable to locate package graphviz Example failures:- - https://db3cff29bdb713e861e1-7db0c1fa1bd98a0adf758f7c1d49f672.ssl.cf5.rackcdn.com/881735/4/check/neutron-tempest-plugin-openvswitch-enforce-scope-new-defaults/278dce1/job-output.txt - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_944/881742/4/check/neutron-tempest-plugin-designate-scenario/9444d54/job-output.txt It's happening randomly from last couple of weeks, seems some issue in general with IPv6 external connectivity. From opensearch[1] it can be seen following host_ids are impacted:- c984fb897502bc826ccaf0e258b6071e76c29b305bc5b31b301de76a 1b1e841d5cdc8c40a480da993c1cbf0cd64900baff612378898e72ab 6cc97bc57f540569368fcc47255180c5d21ed00a22cad83eeb600cec 86c687840cd74cd63cbb095748afa5c9cd0f6fcea898d90aa030cc68 94cd367e7821f5d74cf44c5ebafd9af18d2b6dff64a9bee067337cf6 8926aa5796637312bf5e46a0671a88021c208235fafdfcf22931eb01 70670f45d0dc4eaae28e6553525eec409dfb6f80e8d6c8dcef7d7bf5 And see
[Yahoo-eng-team] [Bug 2019802] [NEW] [master] Not all fullstack tests running in CI
Public bug reported: >From last couple of days Only few tests(just 6 tests) are running in job neutron-fullstack-with-uwsgi job. Example:- https://d16311159baa9c9fc692-58e8a805a242f8a07eac2fd1c3f6b11b.ssl.cf1.rackcdn.com/880867/6/gate/neutron- fullstack-with-uwsgi/6b4c3ba/testr_results.html Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron- fullstack-with-uwsgi&project=openstack%2Fneutron&branch=master&skip=0 The job logs have below Traceback:- 2023-05-16 06:18:13.539943 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron Traceback (most recent call last): 2023-05-16 06:18:13.539958 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context 2023-05-16 06:18:13.539983 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron self.dialect.do_execute( 2023-05-16 06:18:13.539997 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute 2023-05-16 06:18:13.540008 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron cursor.execute(statement, parameters) 2023-05-16 06:18:13.540021 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/cursors.py", line 158, in execute 2023-05-16 06:18:13.540032 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron result = self._query(query) 2023-05-16 06:18:13.540044 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/cursors.py", line 325, in _query 2023-05-16 06:18:13.540056 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron conn.query(q) 2023-05-16 06:18:13.540067 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 549, in query 2023-05-16 06:18:13.540078 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2023-05-16 06:18:13.540090 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 779, in _read_query_result 2023-05-16 06:18:13.540118 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron result.read() 2023-05-16 06:18:13.540131 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 1157, in read 2023-05-16 06:18:13.540143 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron first_packet = self.connection._read_packet() 2023-05-16 06:18:13.540154 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 729, in _read_packet 2023-05-16 06:18:13.540165 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron packet.raise_for_error() 2023-05-16 06:18:13.540177 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error 2023-05-16 06:18:13.540188 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron err.raise_mysql_exception(self._data) 2023-05-16 06:18:13.540199 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception 2023-05-16 06:18:13.540210 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron raise errorclass(errno, errval) 2023-05-16 06:18:13.540221 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron pymysql.err.OperationalError: (1049, "Unknown database 'ybptypesko'") 2023-05-16 06:18:13.540233 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron 2023-05-16 06:18:13.540244 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron The above exception was the direct cause of the following exception: 2023-05-16 06:18:13.540255 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron 2023-05-16 06:18:13.540266 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron Traceback (most recent call last): 2023-05-16 06:18:13.540277 | controller | 2023-05-16 06:18:13.529 15888 ERROR neutron File "/usr/
[Yahoo-eng-team] [Bug 2024674] [NEW] Unit tests fails with oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: ml2_geneve_allocations when run with low concurrency or on lo
Public bug reported: The issue is noticed in RDO openstack-neutron package build[1], the package builds fails as unit tests fails randomly with below Traceback:- DEBUG: neutron.tests.unit.services.trunk.test_utils.UtilsTestCase.test_is_driver_compatible_multiple_drivers DEBUG: - DEBUG: Captured traceback: DEBUG: ~~~ DEBUG: Traceback (most recent call last): DEBUG: File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context DEBUG: self.dialect.do_execute( DEBUG: File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute DEBUG: cursor.execute(statement, parameters) DEBUG: sqlite3.OperationalError: no such table: ml2_geneve_allocations DEBUG: DEBUG: The above exception was the direct cause of the following exception: DEBUG: Traceback (most recent call last): DEBUG: File "/usr/lib/python3.9/site-packages/fixtures/fixture.py", line 196, in setUp DEBUG: self._setUp() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/tests/unit/plugins/ml2/test_plugin.py", line 110, in _setUp DEBUG: self.parent_setup() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 166, in setUp DEBUG: self.api = router.APIRouter() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/api/v2/router.py", line 21, in APIRouter DEBUG: return pecan_app.v2_factory(None, **local_config) DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/pecan_wsgi/app.py", line 47, in v2_factory DEBUG: startup.initialize_all() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/pecan_wsgi/startup.py", line 39, in initialize_all DEBUG: manager.init() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/manager.py", line 301, in init DEBUG: NeutronManager.get_instance() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/manager.py", line 252, in get_instance DEBUG: cls._create_instance() DEBUG: File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 414, in inner DEBUG: return f(*args, **kwargs) DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/manager.py", line 238, in _create_instance DEBUG: cls._instance = cls() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/manager.py", line 126, in __init__ DEBUG: plugin = self._get_plugin_instance(CORE_PLUGINS_NAMESPACE, DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/manager.py", line 162, in _get_plugin_instance DEBUG: plugin_inst = plugin_class() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/quota/resource_registry.py", line 124, in wrapper DEBUG: return f(*args, **kwargs) DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/plugins/ml2/plugin.py", line 282, in __init__ DEBUG: self.type_manager.initialize() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/plugins/ml2/managers.py", line 205, in initialize DEBUG: driver.obj.initialize() DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/plugins/ml2/drivers/type_geneve.py", line 47, in initialize DEBUG: self._initialize(cfg.CONF.ml2_type_geneve.vni_ranges) DEBUG: File "/builddir/build/BUILD/neutron-23.0.0.0b3.dev123/neutron/plugins/ml2/drivers/type_tunnel.py", line 131, in _initialize DEBUG: self.sync_allocations() DEBUG: File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped DEBUG: setattr(e, '_RETRY_EXCEEDED', True) DEBUG: File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ DEBUG: self.force_reraise() DEBUG: File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise DEBUG: raise self.value DEBUG: File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped DEBUG: return f(*args, **kwargs) DEBUG: File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper DEBUG: ectxt.value = e.inner_exc DEBUG: File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ DEBUG: self.force_reraise() DEBUG: File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise DEBUG: raise self.value DEBUG: File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper DEBUG: return f(*args, **kwargs) DEBUG: File "/usr/lib/python3.9/site-packages/neutron_lib/db/api.py", line 187, in wrapped DEBUG: context_reference.session.rollback() DEBUG: File "/u
[Yahoo-eng-team] [Bug 1932043] Re: nova-ceph-multistore test_resize_server_revert fails with rbd.ReadOnlyImage: [errno 30] RBD read-only image
Seen it today in https://c83b20527acf2b0f8494-4a0455790e56cb733d68b35ced7c28e7.ssl.cf5.rackcdn.com/886250/2/check/nova- ceph-multistore/2231918/testr_results.html ** Changed in: nova Status: Expired => New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1932043 Title: nova-ceph-multistore test_resize_server_revert fails with rbd.ReadOnlyImage: [errno 30] RBD read-only image Status in OpenStack Compute (nova): New Bug description: Traceback (most recent call last): File "/opt/stack/tempest/tempest/api/compute/base.py", line 222, in server_check_teardown waiters.wait_for_server_status(cls.servers_client, File "/opt/stack/tempest/tempest/common/waiters.py", line 75, in wait_for_server_status raise exceptions.BuildErrorException(body['fault'], tempest.exceptions.BuildErrorException: Server a1fd599b-909a-4e77-b69d-253b0995bd3d failed to build and is in ERROR status Details: {'code': 500, 'created': '2021-06-15T14:47:21Z', 'message': 'ReadOnlyImage'} https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_037/796269/3/check/nova- ceph-multistore/0373da6/controller/logs/screen-n-cpu.txt Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [None req-ac59ec60-ec07-465d-a82b-ae72be0f2454 tempest-ServerActionsTestJSON-45354220 tempest-ServerActionsTestJSON-45354220-project] [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] Setting instance vm_state to ERROR: rbd.ReadOnlyImage: [errno 30] RBD read-only image (error creating snapshot b'nova-resize' from b'a1fd599b-909a-4e77-b69d-253b0995bd3d_disk') Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] Traceback (most recent call last): Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] File "/opt/stack/nova/nova/compute/manager.py", line 10171, in _error_out_instance_on_exception Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] yield Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] File "/opt/stack/nova/nova/compute/manager.py", line 5862, in _finish_resize_helper Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] network_info = self._finish_resize(context, instance, migration, Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] File "/opt/stack/nova/nova/compute/manager.py", line 5800, in _finish_resize Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] self._set_instance_info(instance, old_flavor) Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] self.force_reraise() Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] raise self.value Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] File "/opt/stack/nova/nova/compute/manager.py", line 5783, in _finish_resize Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] self.driver.finish_migration(context, migration, instance, Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.compute.manager [instance: a1fd599b-909a-4e77-b69d-253b0995bd3d] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10939, in finish_migration Jun 15 14:47:20.281985 ubuntu-focal-ovh-bhs1-0025123352 nova-compute[114395]: ERROR nova.comput
[Yahoo-eng-team] [Bug 2027817] [NEW] neutron slow jobs before xena broken with ERROR: unknown environment 'slow'
Public bug reported: Broken since https://review.opendev.org/c/openstack/tempest/+/887237 merged as neutron slow jobs inherit from tempest-slow-py3. These fails with ERROR: unknown environment 'slow' In releases before xena tempest is pinned and don't have the patch[1] which added slow toxenv. The job needs to be fixed such that these use available toxenv. Build failures:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-tempest- slow-py3&project=openstack/neutron Example failure:- https://1aa32956600eed195a6f-1e93f2912422e0b9805cacc7d195dee2.ssl.cf1.rackcdn.com/887279/1/gate/neutron- tempest-slow-py3/c9ff905/job-output.txt [1] https://github.com/openstack/tempest/commit/6bb98c2aa478f7ad32838fec4b59c4acb73ccf21 ** Affects: neutron Importance: Critical Status: Triaged ** Affects: tempest Importance: Critical Assignee: Ghanshyam Mann (ghanshyammann) Status: New ** Changed in: neutron Importance: Undecided => Critical ** Changed in: neutron Status: New => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2027817 Title: neutron slow jobs before xena broken with ERROR: unknown environment 'slow' Status in neutron: Triaged Status in tempest: New Bug description: Broken since https://review.opendev.org/c/openstack/tempest/+/887237 merged as neutron slow jobs inherit from tempest-slow-py3. These fails with ERROR: unknown environment 'slow' In releases before xena tempest is pinned and don't have the patch[1] which added slow toxenv. The job needs to be fixed such that these use available toxenv. Build failures:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-tempest- slow-py3&project=openstack/neutron Example failure:- https://1aa32956600eed195a6f-1e93f2912422e0b9805cacc7d195dee2.ssl.cf1.rackcdn.com/887279/1/gate/neutron- tempest-slow-py3/c9ff905/job-output.txt [1] https://github.com/openstack/tempest/commit/6bb98c2aa478f7ad32838fec4b59c4acb73ccf21 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2027817/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1969592] Re: [OVN] Frequent DB leader changes causes 'VIF creation failed' on nova side
There were many fixes related to the reported issue in neutron and ovs since this bug report, some of these that i quickly catched are:- - https://patchwork.ozlabs.org/project/openvswitch/patch/20220819230810.2626573-1-i.maxim...@ovn.org/ - https://review.opendev.org/c/openstack/ovsdbapp/+/856200 - https://review.opendev.org/c/openstack/ovsdbapp/+/862524 - https://review.opendev.org/c/openstack/neutron/+/857775 - https://review.opendev.org/c/openstack/neutron/+/871825 One of the issue that i know still open but that happens very rarely https://bugzilla.redhat.com/show_bug.cgi?id=2214289 Closing it based on above and Comment #2 and #3. If the issues are still seen with python-ovs>=2.17 and above fixes included please feel free to open the issue along with neutron and nova debug logs. ** Bug watch added: Red Hat Bugzilla #2214289 https://bugzilla.redhat.com/show_bug.cgi?id=2214289 ** Changed in: neutron Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1969592 Title: [OVN] Frequent DB leader changes causes 'VIF creation failed' on nova side Status in neutron: Fix Released Status in OpenStack Compute (nova): Incomplete Bug description: Hello guys. As I found in 2021 there was a commit to OVS [since OVS v2.16 or 2.15 with that backport] that changed behavior during OVN DBs snapshoting. Now before the leader creates a snapshot it will transfer leadership to another node. We've run tests with rally and tempest and looks like there is a problem now when there is interaction between nova and neutron. For example, simple rally test like 'create {network,router,subnet} -> add interface to router' looks okay even with 256 concurrent same tests/threads. But something like 'neutron.create_subnet -> nova.boot_server -> nova.attach_interface' will fail in time when transfer leadership happens. Since it happens to often [ each 10m + rand(10) ] we will get a lot of errors. This problem can be observed on all versions where OVS 2.16 [or backport] or higher invited :) Some tracing from logs [neutron, nova, ovn-sb-db]: CONTROL-NODES: ctl01-ovn-sb-db.log:2022-04-19T12:30:03.089Z|01002|raft|INFO|Transferring leadership to write a snapshot. ctl01-ovn-sb-db.log:2022-04-19T12:30:03.099Z|01003|raft|INFO|server 1c5f is leader for term 42 ctl03-ovn-sb-db.log:2022-04-19T12:30:03.090Z|00938|raft|INFO|received leadership transfer from 1f46 in term 41 ctl03-ovn-sb-db.log:2022-04-19T12:30:03.092Z|00940|raft|INFO|term 42: elected leader by 2+ of 3 servers ctl03-ovn-sb-db.log:2022-04-19T12:30:10.941Z|00941|jsonrpc|WARN|tcp:xx.yy.zz.26:41882: send error: Connection reset by peer ctl03-ovn-sb-db.log:2022-04-19T12:30:27.324Z|00943|jsonrpc|WARN|tcp:xx.yy.zz.26:41896: send error: Connection reset by peer ctl03-ovn-sb-db.log:2022-04-19T12:30:27.325Z|00945|jsonrpc|WARN|tcp:xx.yy.zz.26:41880: send error: Connection reset by peer ctl03-ovn-sb-db.log:2022-04-19T12:30:27.325Z|00947|jsonrpc|WARN|tcp:xx.yy.zz.26:41892: send error: Connection reset by peer ctl03-ovn-sb-db.log:2022-04-19T12:30:27.327Z|00949|jsonrpc|WARN|tcp:xx.yy.zz.26:41884: send error: Connection reset by peer ctl03-ovn-sb-db.log:2022-04-19T12:31:49.244Z|00951|jsonrpc|WARN|tcp:xx.yy.zz.25:40260: send error: Connection timed out ctl03-ovn-sb-db.log:2022-04-19T12:31:49.244Z|00953|jsonrpc|WARN|tcp:xx.yy.zz.25:40264: send error: Connection timed out ctl03-ovn-sb-db.log:2022-04-19T12:31:49.244Z|00955|jsonrpc|WARN|tcp:xx.yy.zz.24:37440: send error: Connection timed out ctl03-ovn-sb-db.log:2022-04-19T12:31:49.244Z|00957|jsonrpc|WARN|tcp:xx.yy.zz.24:37442: send error: Connection timed out ctl03-ovn-sb-db.log:2022-04-19T12:31:49.245Z|00959|jsonrpc|WARN|tcp:xx.yy.zz.24:37446: send error: Connection timed out ctl03-ovn-sb-db.log:2022-04-19T12:32:01.533Z|01001|jsonrpc|WARN|tcp:xx.yy.zz.67:57586: send error: Connection timed out 2022-04-19 12:30:08.898 27 INFO neutron.db.ovn_revision_numbers_db [req-7fcfdd74-482d-46b2-9f76-07190669d76d ff1516be452b4b939314bf3864a63f35 9d3ae9a7b121488285203b0fdeabc3a3 - default default] Successfully bumped revision number for resource be178a9a-26d7-4bf0-a4e8-d206a6965205 (type: ports) to 1 2022-04-19 12:30:09.644 27 INFO neutron.db.ovn_revision_numbers_db [req-a8278418-3ad9-450c-89bb-e7a5c1c0a06d a9864cd890224c079051b3f56021be64 72db34087b9b401d842b66643b647e16 - default default] Successfully bumped revision number for resource be178a9a-26d7-4bf0-a4e8-d206a6965205 (type: ports) to 2 2022-04-19 12:30:10.235 27 INFO neutron.wsgi [req-571b53cc-ca04-46f7-89f9-fdf8e5931f4c a9864cd890224c079051b3f56021be64 72db34087b9b401d842b66643b647e16 - default default] xx.yy.zz.68,xx.yy.zz.26 "GET /v2.0/ports?tenant_id=9d3ae9a7b121488285203b0fdeabc3a3&device_id=7560fbb7-3ec7-41ef-b7a5-5e955ca4ff
[Yahoo-eng-team] [Bug 1996594] Re: OVN metadata randomly stops working
There were many fixes related to the reported issue in neutron and ovs since this bug report, some of these that i quickly catched are:- - https://patchwork.ozlabs.org/project/openvswitch/patch/20220819230810.2626573-1-i.maxim...@ovn.org/ - https://review.opendev.org/c/openstack/ovsdbapp/+/856200 - https://review.opendev.org/c/openstack/ovsdbapp/+/862524 - https://review.opendev.org/c/openstack/neutron/+/857775 - https://review.opendev.org/c/openstack/neutron/+/871825 Closing it based on above and Comment #5. If the issues are still seen with python-ovs>=2.17 and above fixes included please feel free to open the issue along with ovsdb-server sb logs and neutron server and metadata agent debug logs. ** Bug watch added: Red Hat Bugzilla #2214289 https://bugzilla.redhat.com/show_bug.cgi?id=2214289 ** Changed in: neutron Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1996594 Title: OVN metadata randomly stops working Status in neutron: Fix Released Bug description: We found that OVN metadata will not work randomly when OVN is writing a snapshot. 1, At 12:30:35, OVN started to transfer leadership to write a snapshot $ find sosreport-juju-2752e1-*/var/log/ovn/* |xargs zgrep -i -E 'Transferring leadership' sosreport-juju-2752e1-6-lxd-24-xxx-2022-08-18-entowko/var/log/ovn/ovsdb-server-sb.log:2022-08-18T12:30:35.322Z|80962|raft|INFO|Transferring leadership to write a snapshot. sosreport-juju-2752e1-6-lxd-24-xxx-2022-08-18-entowko/var/log/ovn/ovsdb-server-sb.log:2022-08-18T17:52:53.024Z|82382|raft|INFO|Transferring leadership to write a snapshot. sosreport-juju-2752e1-7-lxd-27-xxx-2022-08-18-hhxxqci/var/log/ovn/ovsdb-server-sb.log:2022-08-18T12:30:35.330Z|92698|raft|INFO|Transferring leadership to write a snapshot. 2, At 12:30:36, neutron-ovn-metadata-agent reported OVSDB Error $ find sosreport-srv1*/var/log/neutron/* |xargs zgrep -i -E 'OVSDB Error' sosreport-srv1xxx2d-xxx-2022-08-18-cuvkufw/var/log/neutron/neutron-ovn-metadata-agent.log:2022-08-18 12:30:36.103 75556 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: no error details available sosreport-srv1xxx6d-xxx-2022-08-18-bgnovqu/var/log/neutron/neutron-ovn-metadata-agent.log:2022-08-18 12:30:36.104 2171 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: no error details available 3, At 12:57:53, we saw the error 'No port found in network', then we will hit the problem that OVN metadata does not work randomly 2022-08-18 12:57:53.800 3730 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 63e2c276-60dd-40e3-baa1-c16342eacce2 with IP address 100.94.98.135 After the problem occurs, restarting neutron-ovn-metadata-agent or restarting haproxy instance as follows can be used as a workaround. /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec ovnmeta-63e2c276-60dd-40e3-baa1-c16342eacce2 haproxy -f /var/lib/neutron/ovn-metadata- proxy/63e2c276-60dd-40e3-baa1-c16342eacce2.conf One lp bug #1990978 [1] is trying to reducing the frequency of transfers, it should be beneficial to this problem. But it only reduces the occurrence of problems, not completely avoiding them. I wonder if we need to add some retry logic on the neutron side NOTE: The openstack version we are using is focal-xena, and openvswitch's version is 2.16.0-0ubuntu2.1~cloud0 [1] https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1990978 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1996594/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2028123] Re: Jobs setting up more than 1 image fails with ValueError: invalid literal for int() with base 10: ''
Fix proposed https://review.opendev.org/c/openstack/devstack/+/888906 ** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2028123 Title: Jobs setting up more than 1 image fails with ValueError: invalid literal for int() with base 10: '' Status in devstack: In Progress Status in neutron: New Bug description: Happening since the devstack patch https://review.opendev.org/c/openstack/devstack/+/886795 merged. Fails as below:- 2023-07-19 01:12:44.183383 | controller | +++ lib/tempest:configure_tempest:301: image_size_in_gib 2023-07-19 01:12:44.186369 | controller | +++ lib/tempest:image_size_in_gib:119: local size 2023-07-19 01:12:44.190924 | controller | lib/tempest:image_size_in_gib:120: oscwrap --os-cloud devstack-admin image show -c size -f value 2023-07-19 01:12:45.055357 | controller | usage: openstack image show [-h] [-f {json,shell,table,value,yaml}] 2023-07-19 01:12:45.055438 | controller | [-c COLUMN] [--noindent] [--prefix PREFIX] 2023-07-19 01:12:45.055450 | controller | [--max-width ] [--fit-width] 2023-07-19 01:12:45.055460 | controller | [--print-empty] [--human-readable] 2023-07-19 01:12:45.055469 | controller | 2023-07-19 01:12:45.055479 | controller | openstack image show: error: the following arguments are required: 2023-07-19 01:12:45.170317 | controller | functions-common:oscwrap:2427 : return 2 2023-07-19 01:12:45.174387 | controller | +++ lib/tempest:image_size_in_gib:120: size= 2023-07-19 01:12:45.178301 | controller | +++ lib/tempest:image_size_in_gib:121: echo 2023-07-19 01:12:45.178524 | controller | +++ lib/tempest:image_size_in_gib:121: python3 -c 'import math; print(int(math.ceil(float(int(input()) / 1024.0 ** 3' 2023-07-19 01:12:45.197214 | controller | Traceback (most recent call last): 2023-07-19 01:12:45.197249 | controller | File "", line 1, in 2023-07-19 01:12:45.197343 | controller | ValueError: invalid literal for int() with base 10: '' 2023-07-19 01:12:45.205412 | controller | ++ lib/tempest:configure_tempest:301: disk= 2023-07-19 01:12:45.209824 | controller | + lib/tempest:configure_tempest:1 : exit_trap 2023-07-19 01:12:45.213259 | controller | + ./stack.sh:exit_trap:547 : local r=1 2023-07-19 01:12:45.218464 | controller | ++ ./stack.sh:exit_trap:548 : jobs -p 2023-07-19 01:12:45.224425 | controller | + ./stack.sh:exit_trap:548 : jobs= 2023-07-19 01:12:45.227487 | controller | + ./stack.sh:exit_trap:551 : [[ -n '' ]] 2023-07-19 01:12:45.232326 | controller | + ./stack.sh:exit_trap:557 : '[' -f /tmp/tmp.phuFS9io2I ']' 2023-07-19 01:12:45.236057 | controller | + ./stack.sh:exit_trap:558 : rm /tmp/tmp.phuFS9io2I 2023-07-19 01:12:45.241367 | controller | + ./stack.sh:exit_trap:562 : kill_spinner 2023-07-19 01:12:45.245280 | controller | + ./stack.sh:kill_spinner:457 : '[' '!' -z '' ']' 2023-07-19 01:12:45.249184 | controller | + ./stack.sh:exit_trap:564 : [[ 1 -ne 0 ]] 2023-07-19 01:12:45.254924 | controller | + ./stack.sh:exit_trap:565 : echo 'Error on exit' 2023-07-19 01:12:45.254945 | controller | Error on exit 2023-07-19 01:12:45.258724 | controller | + ./stack.sh:exit_trap:567 : type -p generate-subunit 2023-07-19 01:12:45.261436 | controller | + ./stack.sh:exit_trap:568 : generate-subunit 1689727847 1318 fail 2023-07-19 01:12:45.402879 | controller | + ./stack.sh:exit_trap:570 : [[ -z /opt/stack/logs ]] 2023-07-19 01:12:45.406897 | controller | + ./stack.sh:exit_trap:573 : /usr/bin/python3.10 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs 2023-07-19 01:12:45.440788 | controller | Traceback (most recent call last): 2023-07-19 01:12:45.440856 | controller | File "/opt/stack/devstack/tools/worlddump.py", line 271, in 2023-07-19 01:12:45.440891 | controller | sys.exit(main()) 2023-07-19 01:12:45.440911 | controller | File "/opt/stack/devstack/tools/worlddump.py", line 248, in main 2023-07-19 01:12:45.440933 | controller | with io.open(fname, 'w') as f: 2023-07-19 01:12:45.440958 | controller | PermissionError: [Errno 13] Permission denied: '/opt/stack/logs/worlddump-2023-07-19-011245.txt' 2023-07-19 01:12:45.447843 | controller | *** FINISHED *** 2023-07-19 01:12:46.151871 | controller | ERROR Example failure:- https://e6631573c9d13b55a73e-3868013975e8773906368e2a70184
[Yahoo-eng-team] [Bug 2028285] [NEW] [unit test][xena+] test_port_deletion_prevention fails when runs in isolation
Public bug reported: Can be reproduced by Just running:- tox -epy3 -- test_port_deletion_prevention or run any of the below tests individually:- neutron.tests.unit.extensions.test_l3.L3NatDBSepTestCase.test_port_deletion_prevention_handles_missing_port neutron.tests.unit.extensions.test_extraroute.ExtraRouteDBSepTestCase.test_port_deletion_prevention_handles_missing_port Fails as below:- neutron.tests.unit.extensions.test_extraroute.ExtraRouteDBSepTestCase.test_port_deletion_prevention_handles_missing_port Captured traceback: ~~~ Traceback (most recent call last): File "/home/ykarel/work/openstack/neutron/neutron/tests/base.py", line 178, in func return f(self, *args, **kwargs) File "/home/ykarel/work/openstack/neutron/neutron/tests/base.py", line 178, in func return f(self, *args, **kwargs) File "/home/ykarel/work/openstack/neutron/neutron/tests/unit/extensions/test_l3.py", line 4491, in test_port_deletion_prevention_handles_missing_port pl.prevent_l3_port_deletion(context.get_admin_context(), 'fakeid') File "/home/ykarel/work/openstack/neutron/neutron/db/l3_db.py", line 1742, in prevent_l3_port_deletion port = port or self._core_plugin.get_port(context, port_id) File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped return f_with_retry(*args, **kwargs, File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped with excutils.save_and_reraise_exception(): File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped return f(*args, **kwargs) File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper with excutils.save_and_reraise_exception() as ectxt: File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper return f(*args, **kwargs) File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped with excutils.save_and_reraise_exception(): File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped return f(*dup_args, **dup_kwargs) File "/home/ykarel/work/openstack/neutron/.tox/py3/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1022, in wrapper return fn(*args, **kwargs) File "/home/ykarel/work/openstack/neutron/neutron/db/db_base_plugin_v2.py", line 1628, in get_port lazy_fields = [models_v2.Port.port_forwardings, AttributeError: type object 'Port' has no attribute 'port_forwardings' It's reproducible Since Xena+ since the inclusion of patch https://review.opendev.org/c/openstack/neutron/+/790691 It do not reproduce if there are other test runs(from the test class) before this test which involve other requests(like network get/create etc) apart from the ones modified in above patch. Considering above point if this test is modified to run other requests like below then it succeeds:- self.plugin.get_ports_count(context.get_admin_context()) or self.plugin.get_networks_count(context.get_admin_context()) or with self.port/network(): self.assertIsNone( pl.prevent_l3_port_deletion(context.get_admin_context(), 'fakeid') ) The issue was originally noticed in a downstream job where unit tests were executed as part of package build. As the tests suite was executed with 100+ concurrency, by chance this test got executed in a worker first of any other test of the class. Me not sure if there are other test cases which will fail like this when run in isolation. ** Affec
[Yahoo-eng-team] [Bug 2033752] [NEW] test_reboot_server_hard fails with AssertionError: time.struct_time() not greater than time.struct_time()
Public bug reported: Seen many occurrences recently, fails as below:- Traceback (most recent call last): File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 259, in test_reboot_server_hard self._test_reboot_server('HARD') File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 127, in _test_reboot_server self.assertGreater(new_boot_time, boot_time, File "/usr/lib/python3.10/unittest/case.py", line 1244, in assertGreater self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.10/unittest/case.py", line 675, in fail raise self.failureException(msg) AssertionError: time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) not greater than time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) : time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) > time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) Example logs:- https://1e11be38b60141dbb290-777f110ca49a5cd01022e1e8aeff1ed5.ssl.cf1.rackcdn.com/893401/5/check/neutron-ovn-tempest-ovs-release/f379752/testr_results.html https://1b9f88b068db0ff45f98-b11b73e0c31560154dece88f25c72a10.ssl.cf2.rackcdn.com/893401/5/check/neutron-linuxbridge-tempest/0bf1039/testr_results.html https://30b3c23edbff5d871c4c-595cfa47540877e41ce912cd21563e42.ssl.cf1.rackcdn.com/886988/10/check/neutron-ovs-tempest-multinode-full/e57a62a/testr_results.html https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e5/886988/10/check/neutron-ovn-tempest-ipv6-only-ovs-release/0e538d1/testr_results.html Opensearch:- https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22not%20greater%20than%20time.struct_time%22'),sort:!()) As per opensearch it's started to be seen just few hours back. ** Affects: neutron Importance: Undecided Status: New ** Affects: tempest Importance: Undecided Status: New ** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033752 Title: test_reboot_server_hard fails with AssertionError: time.struct_time() not greater than time.struct_time() Status in neutron: New Status in tempest: New Bug description: Seen many occurrences recently, fails as below:- Traceback (most recent call last): File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 259, in test_reboot_server_hard self._test_reboot_server('HARD') File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 127, in _test_reboot_server self.assertGreater(new_boot_time, boot_time, File "/usr/lib/python3.10/unittest/case.py", line 1244, in assertGreater self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.10/unittest/case.py", line 675, in fail raise self.failureException(msg) AssertionError: time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) not greater than time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) : time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) > time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) Example logs:- https://1e11be38b60141dbb290-777f110ca49a5cd01022e1e8aeff1ed5.ssl.cf1.rackcdn.com/893401/5/check/neutron-ovn-tempest-ovs-release/f379752/testr_results.html https://1b9f88b068db0ff45f98-b11b73e0c31560154dece88f25c72a10.ssl.cf2.rackcdn.com/893401/5/check/neutron-linuxbridge-tempest/0bf1039/testr_results.html https://30b3c23edbff5d871c4c-595cfa47540877e41ce912cd21563e42.ssl.cf1.rackcdn.com/886988/10/check/neutron-ovs-tempest-multinode-full/e57a62a/testr_results.html https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e5/886988/10/check/neutron-ovn-tempest-ipv6-only-ovs-release/0e538d1/testr_results.html Opensearch:- https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(la
[Yahoo-eng-team] [Bug 2033752] Re: test_reboot_server_hard fails with AssertionError: time.struct_time() not greater than time.struct_time()
Closed for tempest and neutron, as regression is fixed in nova with https://review.opendev.org/c/openstack/nova/+/893502, jobs are back to green. ** Changed in: neutron Status: Confirmed => Invalid ** Changed in: tempest Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033752 Title: test_reboot_server_hard fails with AssertionError: time.struct_time() not greater than time.struct_time() Status in neutron: Invalid Status in OpenStack Compute (nova): Fix Released Status in tempest: Invalid Bug description: Seen many occurrences recently, fails as below:- Traceback (most recent call last): File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 259, in test_reboot_server_hard self._test_reboot_server('HARD') File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 127, in _test_reboot_server self.assertGreater(new_boot_time, boot_time, File "/usr/lib/python3.10/unittest/case.py", line 1244, in assertGreater self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.10/unittest/case.py", line 675, in fail raise self.failureException(msg) AssertionError: time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) not greater than time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) : time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) > time.struct_time(tm_year=2023, tm_mon=9, tm_mday=1, tm_hour=7, tm_min=26, tm_sec=33, tm_wday=4, tm_yday=244, tm_isdst=0) Example logs:- https://1e11be38b60141dbb290-777f110ca49a5cd01022e1e8aeff1ed5.ssl.cf1.rackcdn.com/893401/5/check/neutron-ovn-tempest-ovs-release/f379752/testr_results.html https://1b9f88b068db0ff45f98-b11b73e0c31560154dece88f25c72a10.ssl.cf2.rackcdn.com/893401/5/check/neutron-linuxbridge-tempest/0bf1039/testr_results.html https://30b3c23edbff5d871c4c-595cfa47540877e41ce912cd21563e42.ssl.cf1.rackcdn.com/886988/10/check/neutron-ovs-tempest-multinode-full/e57a62a/testr_results.html https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e5/886988/10/check/neutron-ovn-tempest-ipv6-only-ovs-release/0e538d1/testr_results.html Opensearch:- https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22not%20greater%20than%20time.struct_time%22'),sort:!()) As per opensearch it's started to be seen just few hours back. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2033752/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2034016] [NEW] openstack-tox-py310-with-sqlalchemy-master broken with latest alembic commit
Public bug reported: The job runs with latest alembic/sqlalchemy commits and is broken with recent alembic commit[1]. Test neutron.tests.unit.db.test_migration.TestCli.test_autogen_process_directives fails as below:- ft1.27: neutron.tests.unit.db.test_migration.TestCli.test_autogen_process_directivestesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func return f(self, *args, **kwargs) File "/usr/lib/python3.10/unittest/mock.py", line 1379, in patched return func(*newargs, **newkeywargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/db/test_migration.py", line 708, in test_autogen_process_directives self.assertThat( File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/testtools/testcase.py", line 481, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: "# ### commands auto generated by Alembic - please adjust! ###\nop.drop_constraint('user', 'uq_user_org')\n op.drop_column('user', 'organization_name')\n# ### end Alembic commands ###" does not match /(# )?### commands\ auto\ generated\ by\ Alembic\ \-\ please\ adjust!\ \#\#\#\\n\ \ \ \ op\.drop_constraint\('user',\ 'uq_user_org',\ type_=None\)\\n\ \ \ \ op\.drop_column\('user',\ 'organization_name'\)\\n\ \ \ \ (# )?### end\ Alembic\ commands\ \#\#\#/ Example failure:- https://76069f8f7ed36a9de08a-3a08f5dd30775d36f5503265ac08cfa4.ssl.cf2.rackcdn.com/886988/10/check/openstack-tox-py310-with-sqlalchemy-master/70a282c/testr_results.html [1] https://github.com/sqlalchemy/alembic/commit/733197e0e794ed6aec3e21542257d6cec5bb7d1f ** Affects: neutron Importance: Critical Status: Triaged ** Tags: gate-failure unittest ** Changed in: neutron Importance: Undecided => Critical ** Changed in: neutron Status: New => Triaged ** Tags added: gate-failure unittest -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2034016 Title: openstack-tox-py310-with-sqlalchemy-master broken with latest alembic commit Status in neutron: Triaged Bug description: The job runs with latest alembic/sqlalchemy commits and is broken with recent alembic commit[1]. Test neutron.tests.unit.db.test_migration.TestCli.test_autogen_process_directives fails as below:- ft1.27: neutron.tests.unit.db.test_migration.TestCli.test_autogen_process_directivestesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func return f(self, *args, **kwargs) File "/usr/lib/python3.10/unittest/mock.py", line 1379, in patched return func(*newargs, **newkeywargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/db/test_migration.py", line 708, in test_autogen_process_directives self.assertThat( File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/testtools/testcase.py", line 481, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: "# ### commands auto generated by Alembic - please adjust! ###\nop.drop_constraint('user', 'uq_user_org')\n op.drop_column('user', 'organization_name')\n# ### end Alembic commands ###" does not match /(# )?### commands\ auto\ generated\ by\ Alembic\ \-\ please\ adjust!\ \#\#\#\\n\ \ \ \ op\.drop_constraint\('user',\ 'uq_user_org',\ type_=None\)\\n\ \ \ \ op\.drop_column\('user',\ 'organization_name'\)\\n\ \ \ \ (# )?### end\ Alembic\ commands\ \#\#\#/ Example failure:- https://76069f8f7ed36a9de08a-3a08f5dd30775d36f5503265ac08cfa4.ssl.cf2.rackcdn.com/886988/10/check/openstack-tox-py310-with-sqlalchemy-master/70a282c/testr_results.html [1] https://github.com/sqlalchemy/alembic/commit/733197e0e794ed6aec3e21542257d6cec5bb7d1f To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2034016/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2034096] [NEW] OVN source(main branch) deploy jobs fails with make: *** No rule to make target '/opt/stack/ovs/python/ovs_build_helpers/soutil.py', needed by 'manpages.mk'. Stop
Public bug reported: Fails as below:- 2023-09-05 02:43:36.168367 | controller | checking that generated files are newer than configure... done 2023-09-05 02:43:36.169129 | controller | configure: creating ./config.status 2023-09-05 02:43:36.994278 | controller | config.status: creating lib/libovn.sym 2023-09-05 02:43:37.014160 | controller | config.status: creating Makefile 2023-09-05 02:43:37.035733 | controller | config.status: creating tests/atlocal 2023-09-05 02:43:37.056014 | controller | config.status: creating include/ovn/version.h 2023-09-05 02:43:37.076031 | controller | config.status: creating config.h 2023-09-05 02:43:37.093144 | controller | config.status: executing tests/atconfig commands 2023-09-05 02:43:37.096437 | controller | config.status: executing depfiles commands 2023-09-05 02:43:37.502518 | controller | config.status: executing libtool commands 2023-09-05 02:43:37.522280 | controller | config.status: executing include/openflow/openflow.h.stamp commands 2023-09-05 02:43:37.529722 | controller | config.status: executing utilities/bugtool/dummy commands 2023-09-05 02:43:37.537607 | controller | config.status: executing utilities/dummy commands 2023-09-05 02:43:37.546272 | controller | config.status: executing ipsec/dummy commands 2023-09-05 02:43:37.623678 | controller | ++ lib/neutron_plugins/ovn_agent:compile_ovn:340 : nproc 2023-09-05 02:43:37.628062 | controller | + lib/neutron_plugins/ovn_agent:compile_ovn:340 : make -j9 2023-09-05 02:43:37.651547 | controller | make: *** No rule to make target '/opt/stack/ovs/python/ovs_build_helpers/soutil.py', needed by 'manpages.mk'. Stop. 2023-09-05 02:43:37.654863 | controller | + lib/neutron_plugins/ovn_agent:compile_ovn:1 : exit_trap 2023-09-05 02:43:37.657653 | controller | + ./stack.sh:exit_trap:551 : local r=2 2023-09-05 02:43:37.660729 | controller | ++ ./stack.sh:exit_trap:552 : jobs -p 2023-09-05 02:43:37.664619 | controller | + ./stack.sh:exit_trap:552 : jobs= 2023-09-05 02:43:37.667207 | controller | + ./stack.sh:exit_trap:555 : [[ -n '' ]] 2023-09-05 02:43:37.669954 | controller | + ./stack.sh:exit_trap:561 : '[' -f '' ']' 2023-09-05 02:43:37.672302 | controller | + ./stack.sh:exit_trap:566 : kill_spinner Example failure:- https://f73d480b17c2b34dc38c-939d58bab8c2106dbf157cafeea8359a.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron- ovn-tempest-ipv6-only-ovs-master/7c297ec/job-output.txt Builds:- https://zuul.openstack.org/builds?job_name=neutron-ovn- tempest-ipv6-only-ovs-master&job_name=neutron-ovn-tempest-ovs-master- centos-9-stream&project=openstack%2Fneutron&skip=0 Broken with https://github.com/ovn- org/ovn/commit/558da0cd21ad0172405f7d93c5d0e7533edbf653 Need to update OVS_BRANCH in jobs to fix it. ** Affects: neutron Importance: High Assignee: yatin (yatinkarel) Status: Triaged ** Tags: ovn ** Changed in: neutron Status: New => Triaged ** Changed in: neutron Importance: Undecided => High ** Tags added: ovn ** Changed in: neutron Assignee: (unassigned) => yatin (yatinkarel) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2034096 Title: OVN source(main branch) deploy jobs fails with make: *** No rule to make target '/opt/stack/ovs/python/ovs_build_helpers/soutil.py', needed by 'manpages.mk'. Stop. Status in neutron: Triaged Bug description: Fails as below:- 2023-09-05 02:43:36.168367 | controller | checking that generated files are newer than configure... done 2023-09-05 02:43:36.169129 | controller | configure: creating ./config.status 2023-09-05 02:43:36.994278 | controller | config.status: creating lib/libovn.sym 2023-09-05 02:43:37.014160 | controller | config.status: creating Makefile 2023-09-05 02:43:37.035733 | controller | config.status: creating tests/atlocal 2023-09-05 02:43:37.056014 | controller | config.status: creating include/ovn/version.h 2023-09-05 02:43:37.076031 | controller | config.status: creating config.h 2023-09-05 02:43:37.093144 | controller | config.status: executing tests/atconfig commands 2023-09-05 02:43:37.096437 | controller | config.status: executing depfiles commands 2023-09-05 02:43:37.502518 | controller | config.status: executing libtool commands 2023-09-05 02:43:37.522280 | controller | config.status: executing include/openflow/openflow.h.stamp commands 2023-09-05 02:43:37.529722 | controller | config.status: executing utilities/bugtool/dummy commands 2023-09-05 02:43:37.537607 | controller | config.status: executing utilities/dummy commands 2023-09-05 02:43:37.546272 | controller | config.status: executing ipsec/dummy c
[Yahoo-eng-team] [Bug 2033683] Re: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n']
Hi Alex, << Can someone take a look why the above patch https://review.opendev.org/c/openstack/kolla/+/761182 mentioned here has been excluded from the neutron image? It would have been just missed, since train release Tripleo builds container images natively and not use kolla, You can propose a patch in tripleo-common to fix it. As said i was more interested to know why the issue seen now as /usr/sbin/update-alternatives used to be the path from long back. But considering you are using CentOS8-stream containers on CentOS9-stream host i think you are hitting a recent iptables issue in CentOS8-stream[1], you can check version in your running container, if it matches iptables-1.8.5-8 you can downgrade it to resolve the issue temporary, as the fix for it is not yet merged. If there is no real reason to use CentOS8 images can move to use CentOS 9-Stream based images[2] [1] https://bugzilla.redhat.com/show_bug.cgi?id=2236501 [2] https://quay.io/repository/tripleowallabycentos9/openstack-neutron-server?tab=tags Again marking it as invalid for neutron, feel free to reopen but share what's expected fix is required in neutron project. ** Bug watch added: Red Hat Bugzilla #2236501 https://bugzilla.redhat.com/show_bug.cgi?id=2236501 ** Changed in: neutron Status: New => Invalid ** Changed in: tripleo Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033683 Title: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n'] Status in neutron: Invalid Status in tripleo: Confirmed Bug description: Description === Wallaby deployment via undercloud/overcloud started to fail recently on overcloud node provision Neutron constantly reports inability to update iptables that in turn makes baremetal to fail to boot from PXE From the review it seems that /usr/bin/update-alternatives set to legacy fails since neutron user doesn't have sudo to run it In the info I can see that neutron user has the following subset of commands it's able to run: ... (root) NOPASSWD: /usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --auto iptables (root) NOPASSWD: /usr/bin/update-alternatives --auto ip6tables But the issue is the fact that command isn't found as it was moved to /usr/sbin/update-alternatives Steps to reproduce == 1. Deploy undercloud 2. Deploy networks and VIP 3. Add and introspect a node 4. Execute overcloud node provision ... that will timeout Expected result === Successful overcloud node baremetal provisioning Logs & Configs == 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-18d52177-9c93-401c-b97d-0334e488a257 - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent COMMIT 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent # Completed by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ; Stdout: ; Stderr: iptables-restore: line 23 failed Environment === Centos 9 Stream and undercloud deployment tool To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2033683/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2012731] Re: [CI] "neutron-ovs-grenade-multinode-skip-level" and "neutron-ovn-grenade-multinode-skip-level" failing always
Required patch merged long back, jobs are green, Closing it:- * https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-grenade-multinode-skip-level&skip=0 * https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-grenade-multinode-skip-level&skip=0 ** Changed in: neutron Status: New => Fix Released ** Changed in: neutron Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2012731 Title: [CI] "neutron-ovs-grenade-multinode-skip-level" and "neutron-ovn- grenade-multinode-skip-level" failing always Status in neutron: Fix Released Bug description: Logs: * https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-grenade-multinode-skip-level&skip=0 * https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-grenade-multinode-skip-level&skip=0 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2012731/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1949606] Re: QEMU >= 5.0.0 with -accel tcg uses a tb-size of 1GB causing OOM issues in CI
Fixed with https://review.opendev.org/c/openstack/nova/+/868419 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1949606 Title: QEMU >= 5.0.0 with -accel tcg uses a tb-size of 1GB causing OOM issues in CI Status in OpenStack Compute (nova): Fix Released Bug description: This is a Nova tracker for a set of issues being seen in OpenStack CI jobs using QEMU >= 5.0.0 caused by the following change in defaults witin QEMU: https://github.com/qemu/qemu/commit/600e17b26 https://gitlab.com/qemu-project/qemu/-/issues/693 At present most of the impacted jobs are being given an increased amount of swap with lower Tempest concurrency settings to avoid the issue, for example for CentOS 8 stream: https://review.opendev.org/c/openstack/devstack/+/803706 https://review.opendev.org/c/openstack/tempest/+/797614 Longer term a libvirt RFE has been raised to allow Nova to control the size of the cache: https://gitlab.com/libvirt/libvirt/-/issues/229 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1949606/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2035578] [NEW] [stable branches] devstack-tobiko-neutron job Fails with InvocationError('could not find executable python', None)
Public bug reported: It started failing[1] since the job switched to ubuntu-jammy[2]. Fails as below:- 2023-09-13 16:46:18.124882 | TASK [tobiko-tox : run sanity test cases before creating resources] 2023-09-13 16:46:19.463567 | controller | neutron_sanity create: /home/zuul/src/opendev.org/x/tobiko/.tox/py3 2023-09-13 16:46:20.518574 | controller | neutron_sanity installdeps: -c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt 2023-09-13 16:46:20.519390 | controller | ERROR: could not install deps [-c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt]; v = InvocationError('could not find executable python', None) 2023-09-13 16:46:20.520263 | controller | ___ summary 2023-09-13 16:46:20.555843 | controller | ERROR: neutron_sanity: could not install deps [-c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt]; v = InvocationError('could not find executable python', None) 2023-09-13 16:46:21.141713 | controller | ERROR 2023-09-13 16:46:21.142024 | controller | { 2023-09-13 16:46:21.142117 | controller | "delta": "0:00:01.484351", 2023-09-13 16:46:21.142197 | controller | "end": "2023-09-13 16:46:20.556249", 2023-09-13 16:46:21.142276 | controller | "failed_when_result": true, 2023-09-13 16:46:21.142353 | controller | "msg": "non-zero return code", 2023-09-13 16:46:21.142688 | controller | "rc": 1, 2023-09-13 16:46:21.142770 | controller | "start": "2023-09-13 16:46:19.071898" 2023-09-13 16:46:21.142879 | controller | } 2023-09-13 16:46:21.142972 | controller | ERROR: Ignoring Errors Example failures zed/stable2023.1:- - https://zuul.opendev.org/t/openstack/build/591dae67122444daa35195f7458ffafe - https://zuul.opendev.org/t/openstack/build/5838bf0704b247dc8f1eb12367b1d33e - https://zuul.opendev.org/t/openstack/build/8d2e22ff171944b0b549c12e1aaac476 Wallaby/Xena/Yoga builds started failing with:- ++ functions:write_devstack_version:852 : git log '--format=%H %s %ci' -1 + ./stack.sh:main:230 : SUPPORTED_DISTROS='bullseye|focal|f35|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel9|openEuler-20.03' + ./stack.sh:main:232 : [[ ! jammy =~ bullseye|focal|f35|opensuse-15.2|opensuse-tumbleweed|rhel8|rhel9|openEuler-20.03 ]] + ./stack.sh:main:233 : echo 'WARNING: this script has not been tested on jammy' Example:- - https://zuul.opendev.org/t/openstack/build/0bd0421e30804b7aa9b6ea032d271be7 - https://zuul.opendev.org/t/openstack/build/8e06dfc0ccd940f3ab71edc0ec93466c - https://zuul.opendev.org/t/openstack/build/899634e90ee94e0294985747075fb26c Even before these jobs were broken but there tests used to fail not the test setup, that can be handled once the current issues are cleared. [1] https://zuul.opendev.org/t/openstack/builds?job_name=devstack-tobiko-neutron&branch=stable%2F2023.1 [2] https://review.opendev.org/c/x/devstack-plugin-tobiko/+/893662?usp=search ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2035578 Title: [stable branches] devstack-tobiko-neutron job Fails with InvocationError('could not find executable python', None) Status in neutron: New Bug description: It started failing[1] since the job switched to ubuntu-jammy[2]. Fails as below:- 2023-09-13 16:46:18.124882 | TASK [tobiko-tox : run sanity test cases before creating resources] 2023-09-13 16:46:19.463567 | controller | neutron_sanity create: /home/zuul/src/opendev.org/x/tobiko/.tox/py3 2023-09-13 16:46:20.518574 | controller | neutron_sanity installdeps: -c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt 2023-09-13 16:46:20.519390 | controller | ERROR: could not install deps [-c/home/zuul/src/opendev.org/x/tobiko/upper-constraints.txt, -r/home/zuul/src/opendev.org/x/tobiko/requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/test-requirements.txt, -r/home/zuul/src/opendev.org/x/tobiko/extra-requirements.txt]; v = InvocationError('could not find executable python', None) 2023-09-13 16:46:20.520263 | controller | ___
[Yahoo-eng-team] [Bug 2038541] [NEW] LinuxBridgeARPSpoofTestCase functional tests fails with latest jammy kernel 5.15.0-86.96
Public bug reported: Tests fails while running ebtables(['-D', chain] + rule.split()) with:- 2023-10-05 12:09:19.307 41358 ERROR neutron.agent.linux.utils [None req-defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test-veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc 2023-10-05 12:09:29.576 41358 ERROR neutron.agent.linux.utils [None req- defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test- veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc 2023-10-05 12:09:50.099 41358 ERROR neutron.agent.linux.utils [None req- defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test- veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc The new kernel includes below changes which have triggered this, described in https://launchpad.net/ubuntu/+source/linux/5.15.0-86.96:- - netfilter: nf_tables: disallow element updates of bound anonymous sets - netfilter: nf_tables: reject unbound anonymous set before commit phase - netfilter: nf_tables: reject unbound chain set before commit phase - netfilter: nf_tables: disallow updates of anonymous sets Following two test fails:- - test_arp_protection_update - test_arp_fails_incorrect_mac_protection ** Affects: neutron Importance: Critical Status: Triaged ** Tags: functional-tests gate-failure ** Tags added: functional-tests gate-failure ** Changed in: neutron Status: New => Triaged ** Changed in: neutron Importance: Undecided => Critical ** Description changed: Tests fails while running ebtables(['-D', chain] + rule.split()) with:- 2023-10-05 12:09:19.307 41358 ERROR neutron.agent.linux.utils [None req-defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test-veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc 2023-10-05 12:09:29.576 41358 ERROR neutron.agent.linux.utils [None req- defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test- veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc 2023-10-05 12:09:50.099 41358 ERROR neutron.agent.linux.utils [None req- defd197a-c4e2-4761-a4cc-cc960a3ff71a - - - - - -] Exit code: 4; Cmd: ['ip', 'netns', 'exec', 'test-b58b5cf9-5018-4801-aacb-8b00fae3fe37', 'ebtables', '-t', 'nat', '--concurrent', '-D', 'neutronMAC-test- veth09e6dc', '-i', 'test-veth09e6dc', '--among-src', 'fa:16:3e:ac:fd:b6', '-j', 'RETURN']; Stdin: ; Stdout: ; Stderr: ebtables v1.8.7 (nf_tables): RULE_DELETE failed (Invalid argument): rule in chain neutronMAC-test-veth09e6dc - The new kernel includes below changes which have triggered this:- - - netfilter: nf_tables: disallow element updates of bound anonymous sets - - netfilter: nf_tables: reject unbound anonymous set before commit phase - - netfilter: nf_tables: reject unbound chain set before commit phase - - netfilter: nf_tables: disallow updates of anonymous sets + The new kernel includes below changes which have triggered this, described in https://launchpad.net/ubuntu/+source/linux/5.15.0-86.96:- + - netfilter: nf_tables: disallow element updates of bound anonymous sets + - netfilter: nf_tables: reject unbound anonymous set before commit phase + - netfilter: nf_tables: reject unbound chain set before commit phase + - netfilter: nf_tables: disallow updates of anonymous sets Following two test fails:- - test_arp_protection_update - test_arp_fails_incorrect_mac_protection -- You received
[Yahoo-eng-team] [Bug 2039012] [NEW] [fwaas] Scenario job fails randomly
Public bug reported: Seen couple of occurrences across different releases:- Failures:- - https://zuul.opendev.org/t/openstack/build/30c625cd86aa40e6b6252689a7e88910 neutron-tempest-plugin-fwaas-2023-1 Traceback (most recent call last): File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 269, in test_create_show_delete_firewall_group body = self.firewall_groups_client.create_firewall_group( File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 25, in create_firewall_group return self.create_resource(uri, post_data) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 62, in create_resource resp, body = self.post(req_uri, req_post_data) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 300, in post return self.request('POST', url, extra_headers, headers, body, chunked) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 922, in _error_checker raise exceptions.ServerFault(resp_body, resp=resp, tempest.lib.exceptions.ServerFault: Got server fault Details: Request Failed: internal server error while processing your request. - https://zuul.opendev.org/t/openstack/build/0d0fbfc009cb4142920ddb96e9695ec0 neutron-tempest-plugin-fwaas-2023-2 Traceback (most recent call last): File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/scenario/test_fwaas_v2.py", line 220, in test_icmp_reachability_scenarios self.check_vm_connectivity( File "/opt/stack/tempest/tempest/scenario/manager.py", line 983, in check_vm_connectivity self.assertTrue(self.ping_ip_address(ip_address, File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue raise self.failureException(msg) AssertionError: False is not true : Timed out waiting for 172.24.5.235 to become reachable - https://zuul.opendev.org/t/openstack/build/5ed8731220654e9fb67ba910a3f08c25 neutron-tempest-plugin-fwaas-zed Traceback (most recent call last): File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 344, in test_update_firewall_group self.firewall_groups_client.delete_firewall_group(fwg_id) File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group return self.delete_resource(uri) File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource resp, body = self.delete(req_uri) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete return self.request('DELETE', url, extra_headers, headers, body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request self._error_checker(resp, resp_body) File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker raise exceptions.Conflict(resp_body, resp=resp) tempest.lib.exceptions.Conflict: Conflict with state of target resource Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group a20ead7e-40ca-488b-a751-a36e5fb4119a is still active.', 'detail': ''} - https://zuul.opendev.org/t/openstack/build/fbe68294562a43b492dd2ba66dec9d43 neutron-tempest-plugin-fwaas-2023-2 Traceback (most recent call last): File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/scenario/test_fwaas_v2.py", line 220, in test_icmp_reachability_scenarios self.check_vm_connectivity( File "/opt/stack/tempest/tempest/scenario/manager.py", line 983, in check_vm_connectivity self.assertTrue(self.ping_ip_address(ip_address, File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue raise self.failureException(msg) AssertionError: False is not true : Timed out waiting for 172.24.5.126 to become reachable ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2039012 Title: [fwaas] Scenario job fails randomly Status in neutron: New Bug description: Seen couple of occurrences across different releases:- Failures:- - https://zuul.opendev.org/t/openstack/build/30c625cd86aa40e6b6252689a7e88910 neutron-tempest-plugin-fwaas-2023-1 Traceback (most recent call last): File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 269, in test_create_show_delete_firewall_group body = self.firewall_groups_client.create_firewall_group( File "/opt/stack/tempest/.tox/tempest/lib/pyt
[Yahoo-eng-team] [Bug 2039066] [NEW] openstacksdk-functional-devstack-networking periodic job broken in xena/yoga/zed
Public bug reported: These periodic jobs are failing since https://review.opendev.org/c/openstack/openstacksdk/+/874695 merged as it added some new tests for designate zone share api. Tests fails as:- ft2.2: openstack.tests.functional.dns.v2.test_zone_share.TestZoneShare.test_find_zone_sharetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/tests/functional/dns/v2/test_zone_share.py", line 89, in test_find_zone_share orig_zone_share = self.operator_cloud.dns.create_zone_share( File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/dns/v2/_proxy.py", line 631, in create_zone_share return self._create( File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/proxy.py", line 605, in _create return res.create(self, base_path=base_path) File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/resource.py", line 1535, in create self._translate_response(response, **response_kwargs) File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/resource.py", line 1287, in _translate_response exceptions.raise_from_response(response, error_message=error_message) File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/exceptions.py", line 250, in raise_from_response raise cls( openstack.exceptions.HttpException: HttpException: 405: Client Error for url: https://158.69.78.71/dns/v2/zones/5f647760-ebd2-4e3b-84c1-e2f4e7bd4dea/shares, METHOD NOT ALLOWED zone sharing is supported in designate only since 2023.1 https://review.opendev.org/c/openstack/designate/+/726334 We do run this job in wallaby as well but is not impacted because openstacksdk stable/wallaby is used in those jobs as done in https://review.opendev.org/c/openstack/openstacksdk/+/885898 Builds:- https://zuul.openstack.org/builds?job_name=openstacksdk- functional-devstack- networking&project=openstack%2Fneutron&branch=stable%2Fxena&branch=stable%2Fyoga&branch=stable%2Fzed ** Affects: neutron Importance: Medium Assignee: yatin (yatinkarel) Status: Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2039066 Title: openstacksdk-functional-devstack-networking periodic job broken in xena/yoga/zed Status in neutron: Triaged Bug description: These periodic jobs are failing since https://review.opendev.org/c/openstack/openstacksdk/+/874695 merged as it added some new tests for designate zone share api. Tests fails as:- ft2.2: openstack.tests.functional.dns.v2.test_zone_share.TestZoneShare.test_find_zone_sharetesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/tests/functional/dns/v2/test_zone_share.py", line 89, in test_find_zone_share orig_zone_share = self.operator_cloud.dns.create_zone_share( File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/dns/v2/_proxy.py", line 631, in create_zone_share return self._create( File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/proxy.py", line 605, in _create return res.create(self, base_path=base_path) File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/resource.py", line 1535, in create self._translate_response(response, **response_kwargs) File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/resource.py", line 1287, in _translate_response exceptions.raise_from_response(response, error_message=error_message) File "/home/zuul/src/opendev.org/openstack/openstacksdk/openstack/exceptions.py", line 250, in raise_from_response raise cls( openstack.exceptions.HttpException: HttpException: 405: Client Error for url: https://158.69.78.71/dns/v2/zones/5f647760-ebd2-4e3b-84c1-e2f4e7bd4dea/shares, METHOD NOT ALLOWED zone sharing is supported in designate only since 2023.1 https://review.opendev.org/c/openstack/designate/+/726334 We do run this job in wallaby as well but is not impacted because openstacksdk stable/wallaby is used in those jobs as done in https://review.opendev.org/c/openstack/openstacksdk/+/885898 Builds:- https://zuul.openstack.org/builds?job_name=openstacksdk- functional-devstack- networking&project=openstack%2Fneutron&branch=stable%2Fxena&branch=stable%2Fyoga&branch=stable%2Fzed To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2039066/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2039417] [NEW] [master][stable/2023.2][functional] test_maintenance.TestMaintenance tests fails randomly
Public bug reported: The functional test fails randomly as:- ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_maintenance.TestMaintenance.test_porttesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 306, in test_port neutron_net = self._create_network('network1') File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 68, in _create_network return self.deserialize(self.fmt, res)['network'] KeyError: 'network' neutron server log when it fails:- 2023-10-06 08:16:33.966 37713 DEBUG neutron.db.ovn_revision_numbers_db [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] create_initial_revision uuid=6ab7bd53-277b-4133-ac32-52b1d0c90f78, type=security_groups, rev=-1 create_initial_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:108 2023-10-06 08:16:33.973 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.974 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run txn.results.put(txn.do_commit()) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit raise RuntimeError(msg) RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._create_security_group-11049860 security_group, after_create: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager Traceback (most recent call last): 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, payload=payload) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 409, in _create_security_group 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager self._ovn_client.create_security_group(context, 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2328, in create_security_group 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with self._nb_idl.transaction(check_error=True) as txn: 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager next(self.gen) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t: 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager next(self.gen) 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction 2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with self.cre
[Yahoo-eng-team] [Bug 2039940] [NEW] test_resize_volume_backed_server_confirm which fails randomly with Kernel panic
Public bug reported: tempest.api.compute.servers.test_server_actions.ServerActionsTestOtherA.test_resize_volume_backed_server_confirm fails randomly with:- Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 136, in _get_ssh_connection ssh.connect(self.host, port=self.port, username=self.username, File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) TimeoutError: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/stack/tempest/tempest/lib/decorators.py", line 106, in wrapper raise exc File "/opt/stack/tempest/tempest/lib/decorators.py", line 98, in wrapper return f(*func_args, **func_kwargs) File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper return f(*func_args, **func_kwargs) File "/opt/stack/tempest/tempest/api/compute/servers/test_server_actions.py", line 510, in test_resize_volume_backed_server_confirm linux_client.validate_authentication() File "/opt/stack/tempest/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper return function(self, *args, **kwargs) File "/opt/stack/tempest/tempest/lib/common/utils/linux/remote_client.py", line 123, in validate_authentication self.ssh_client.test_connection_auth() File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 245, in test_connection_auth connection = self._get_ssh_connection() File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 155, in _get_ssh_connection raise exceptions.SSHTimeout(host=self.host, tempest.lib.exceptions.SSHTimeout: Connection to the 172.24.5.65 via SSH timed out. User: cirros, Password: None Guest console log says it's kernel panic:- info: initramfs: up at 7.36 [8.403290] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [8.440219] GPT:Primary header thinks Alt. header is not at the end of the disk. [8.440726] GPT:229375 != 2097151 [8.440967] GPT:Alternate GPT header not at the end of the disk. [8.441252] GPT:229375 != 2097151 [8.441503] GPT: Use GNU Parted to correct GPT errors. [8.974064] virtio_gpu virtio0: [drm] drm_plane_enable_fb_damage_clips() not called [9.068224] random: crng init done currently loaded modules: 8021q 8139cp 8390 9pnet 9pnet_virtio ahci cec drm drm_kms_helper e1000 e1000e failover fb_sys_fops garp hid hid_generic ip6_udp_tunnel ip_tables isofs libahci libcrc32c llc mii mrp ne2k_pci net_failover nls_ascii nls_iso8859_1 nls_utf8 pcnet32 qemu_fw_cfg rc_core sctp stp syscopyarea sysfillrect sysimgblt udp_tunnel usbhid virtio_blk virtio_dma_buf virtio_gpu virtio_input virtio_net virtio_rng virtio_scsi x_tables info: initramfs loading root from /dev/vda1 /sbin/init: can't load library 'libtirpc.so.3' [ 11.288963] Kernel panic - not syncing: Attempted to kill init! exitcode=0x1000 [ 11.290203] CPU: 0 PID: 1 Comm: init Not tainted 5.15.0-71-generic #78-Ubuntu [ 11.290952] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.15.0-1 04/01/2014 [ 11.291870] Call Trace: [ 11.292973] [ 11.293458] show_stack+0x52/0x5c [ 11.294280] dump_stack_lvl+0x4a/0x63 [ 11.294720] dump_stack+0x10/0x16 [ 11.295179] panic+0x15c/0x334 [ 11.295587] ? exit_to_user_mode_prepare+0x37/0xb0 [ 11.296118] do_exit.cold+0x15/0xa0 [ 11.296460] __x64_sys_exit+0x1b/0x20 [ 11.296880] do_syscall_64+0x5c/0xc0 [ 11.297283] ? ksys_write+0x67/0xf0 [ 11.297672] ? exit_to_user_mode_prepare+0x37/0xb0 [ 11.298172] ? syscall_exit_to_user_mode+0x27/0x50 [ 11.298683] ? __x64_sys_write+0x19/0x20 [ 11.299151] ? do_syscall_64+0x69/0xc0 [ 11.299644] entry_SYSCALL_64_after_hwframe+0x61/0xcb [ 11.300611] RIP: 0033:0x7f147a37555e [ 11.301938] Code: 05 d7 2a 00 00 4c 89 f9 bf 02 00 00 00 48 8d 35 fb 0d 00 00 48 8b 10 31 c0 e8 50 d2 ff ff bf 10 00 00 00 b8 3c 00 00 00 0f 05 <48> 8d 15 f3 2a 00 00 f7 d8 89 02 48 83 ec 20 49 8b 8c 24 b8 00 00 [ 11.312012] RSP: 002b:7fff85488500 EFLAGS: 0207 ORIG_RAX: 003c [ 11.318360] RAX: ffda RBX: 7fff854897b0 RCX: 7f147a37555e [ 11.324215] RDX: 0002 RSI: 1000 RDI: 0010 [ 11.331344] RBP: 7fff85489790 R08: 7f147a36e000 R09: 7f147a36e01a [ 11.338406] R10: 0001 R11: 0207 R12: 7f147a36f040 [ 11.347090] R13: 004bae50 R14: R15: 00403d66 [ 11.354220] [ 11.362227] Kernel Offset: 0x3640 from 0x8100 (relocation range: 0x8000-0xbfff) [ 11.369248] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x1000 ]--- As per opensearch[1] there are 16 hits in last 12 days across multiple jobs in master/stable2023.2 branch. Jobs:- tempest-integrated-networking 31.3% cinder-tempest
[Yahoo-eng-team] [Bug 2040517] [NEW] host not removed from table ml2_vxlan_endpoints with the agent delete
Public bug reported: After deleting an agent, there is stale entry for the host in table 'ml2_vxlan_endpoints'. An use case is during node scale down, a agent is deleted, but the host entry is not removed from ml2_vxlan_endpoints; I have not checked other topologies but same should apply to other similar tables 'ml2_gre_endpoints' and 'ml2_geneve_endpoints' # Ensure agent is stopped or node is removed. $ openstack network agent show 338d13fc-3483-414f-bc55-5b2cbb0db189 --fit-width +---++ | Field | Value | +---++ | admin_state_up| UP | | agent_type| Open vSwitch agent | | alive | XXX | | availability_zone | None | | binary| neutron-openvswitch-agent | | configuration | {'arp_responder_enabled': True, 'baremetal_smartnic': False, 'bridge_mappings': {'public': 'br-ex'}, 'datapath_type': 'system', | | | 'devices': 0, 'enable_distributed_routing': True, 'extensions': [], 'in_distributed_mode': True, 'integration_bridge': 'br-int', | | | 'l2_population': True, 'log_agent_heartbeats': False, 'ovs_capabilities': {'datapath_types': ['netdev', 'system'], 'iface_types': | | | ['bareudp', 'erspan', 'geneve', 'gre', 'gtpu', 'internal', 'ip6erspan', 'ip6gre', 'lisp', 'patch', 'stt', 'system', 'tap', 'vxlan']}, | | | 'ovs_hybrid_plug': False, 'resource_provider_bandwidths': {'br-ex': {'egress': 100, 'ingress': 100}}, | | | 'resource_provider_hypervisors': {'br-ex': 'ykarel-temp3', 'rp_tunnelled': 'ykarel-temp3'}, 'resource_provider_inventory_defaults':| | | {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}, 'resource_provider_packet_processing_inventory_defaults': | | | {'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}, 'resource_provider_packet_processing_with_direction': {}, | | | 'resource_provider_packet_processing_without_direction': {}, 'tunnel_types': ['vxlan'], 'tunneling_ip': '10.0.109.173',| | | 'vhostuser_socket_dir': '/var/run/openvswitch'} | | created_at| 2023-10-25 14:30:17 | | description | None | | ha_state | None | | host | ykarel-temp3 | | id| 338d13fc-3483-414f-bc55-5b2cbb0db189 | | last_heartbeat_at | 2023-10-25 14:30:17 | | resources_synced | None | | started_at| 2023-10-25 14:30:17 | | topic | N/A | +---++ $ openstack network agent
[Yahoo-eng-team] [Bug 2042939] [NEW] sqlalchemy-master jobs broken with ValueError: invalid literal for int() with base 10: '0b1'
Public bug reported: Broken since https://github.com/sqlalchemy/sqlalchemy/commit/e93a5e89. Fails like:- 2023-11-07 10:51:53.328284 | ubuntu-jammy | Failed to import test module: neutron.tests.unit.agent.common.test_ovs_lib 2023-11-07 10:51:53.328310 | ubuntu-jammy | Traceback (most recent call last): 2023-11-07 10:51:53.328335 | ubuntu-jammy | File "/usr/lib/python3.10/unittest/loader.py", line 436, in _find_test_path 2023-11-07 10:51:53.328360 | ubuntu-jammy | module = self._get_module_from_name(name) 2023-11-07 10:51:53.328385 | ubuntu-jammy | File "/usr/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name 2023-11-07 10:51:53.328411 | ubuntu-jammy | __import__(name) 2023-11-07 10:51:53.328436 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/common/test_ovs_lib.py", line 18, in 2023-11-07 10:51:53.328461 | ubuntu-jammy | from neutron_lib import exceptions 2023-11-07 10:51:53.328486 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/neutron_lib/__init__.py", line 17, in 2023-11-07 10:51:53.328511 | ubuntu-jammy | from neutron_lib.db import api # noqa 2023-11-07 10:51:53.328537 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/neutron_lib/db/api.py", line 22, in 2023-11-07 10:51:53.328569 | ubuntu-jammy | from oslo_db.sqlalchemy import enginefacade 2023-11-07 10:51:53.328916 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 28, in 2023-11-07 10:51:53.328952 | ubuntu-jammy | from oslo_db.sqlalchemy import engines 2023-11-07 10:51:53.328979 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 36, in 2023-11-07 10:51:53.329004 | ubuntu-jammy | from oslo_db.sqlalchemy import compat 2023-11-07 10:51:53.329030 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/oslo_db/sqlalchemy/compat/__init__.py", line 18, in 2023-11-07 10:51:53.329055 | ubuntu-jammy | _vers = versionutils.convert_version_to_tuple(__version__) 2023-11-07 10:51:53.329080 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/oslo_utils/versionutils.py", line 90, in convert_version_to_tuple 2023-11-07 10:51:53.329105 | ubuntu-jammy | return tuple(int(part) for part in version_str.split('.')) 2023-11-07 10:51:53.329130 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/oslo_utils/versionutils.py", line 90, in 2023-11-07 10:51:53.329162 | ubuntu-jammy | return tuple(int(part) for part in version_str.split('.')) 2023-11-07 10:51:53.329469 | ubuntu-jammy | ValueError: invalid literal for int() with base 10: '0b1' Example failure:- https://46050a1a8fa787c1d655-839a87b8a49ad5b15d7c39aaacbfa49e.ssl.cf5.rackcdn.com/900087/2/check/openstack- tox-py310-with-sqlalchemy-master/429b0c2/job-output.txt Builds:- https://zuul.openstack.org/builds?job_name=openstack-tox- py310-with-sqlalchemy-master&job_name=neutron-functional-with- sqlalchemy-master&skip=0 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2042939 Title: sqlalchemy-master jobs broken with ValueError: invalid literal for int() with base 10: '0b1' Status in neutron: New Bug description: Broken since https://github.com/sqlalchemy/sqlalchemy/commit/e93a5e89. Fails like:- 2023-11-07 10:51:53.328284 | ubuntu-jammy | Failed to import test module: neutron.tests.unit.agent.common.test_ovs_lib 2023-11-07 10:51:53.328310 | ubuntu-jammy | Traceback (most recent call last): 2023-11-07 10:51:53.328335 | ubuntu-jammy | File "/usr/lib/python3.10/unittest/loader.py", line 436, in _find_test_path 2023-11-07 10:51:53.328360 | ubuntu-jammy | module = self._get_module_from_name(name) 2023-11-07 10:51:53.328385 | ubuntu-jammy | File "/usr/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name 2023-11-07 10:51:53.328411 | ubuntu-jammy | __import__(name) 2023-11-07 10:51:53.328436 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/agent/common/test_ovs_lib.py", line 18, in 2023-11-07 10:51:53.328461 | ubuntu-jammy | from neutron_lib import exceptions 2023-11-07 10:51:53.328486 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py310/lib/python3.10/site-packages/neutron_lib/__init__.py", line 17, in 2023-11-07 10:51:53.328511 | ubuntu-jammy | from neutron_lib.db import api # noqa