[Yahoo-eng-team] [Bug 2036877] [NEW] radvd seems to crash when ipv4 addresses are supplied as nameservers to ipv6 subnets
Public bug reported: I'll copy from this report, please notice that I'm NOT the original reporter!: https://bugs.launchpad.net/kolla-ansible/+bug/2033980/comments/8 Before cleaning the PID file, I did take a look at the config of radvd: ``` $ cat /var/lib/neutron/ra/aee91f41-1945-40b4-b72f-8be2eb369b44.radvd.conf interface qr-caa16d7e-26 { AdvSendAdvert on; MinRtrAdvInterval 30; MaxRtrAdvInterval 100; AdvLinkMTU 1450; RDNSS 2a02:74a0:x:0::53 10.40.3.53 2a02:74a0:x:0::54 {}; prefix 2a02:74a0:x:y::/64 { AdvOnLink on; AdvAutonomous on; }; route fe80::a9fe:a9fe/128 { }; }; ``` We've been configuring the router with terraform, assigning the ipv4 resolvers to the IPv4 subnet and the IPv6 resolvers to the IPv6 subnet. After deleting the router, adjusting the subnets (no resolvers on v4, only ipv6 resolvers on ipv6), and recreating the router, radvd is now active and everything's fine. It seems that due to misconfiguration (and incomplete template parsing), IPv4 nameservers ended up in the config of radvd, which failed to start. Neutron was then unable to clean up the pidfile, thus failing to start radvd again. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2036877 Title: radvd seems to crash when ipv4 addresses are supplied as nameservers to ipv6 subnets Status in neutron: New Bug description: I'll copy from this report, please notice that I'm NOT the original reporter!: https://bugs.launchpad.net/kolla-ansible/+bug/2033980/comments/8 Before cleaning the PID file, I did take a look at the config of radvd: ``` $ cat /var/lib/neutron/ra/aee91f41-1945-40b4-b72f-8be2eb369b44.radvd.conf interface qr-caa16d7e-26 { AdvSendAdvert on; MinRtrAdvInterval 30; MaxRtrAdvInterval 100; AdvLinkMTU 1450; RDNSS 2a02:74a0:x:0::53 10.40.3.53 2a02:74a0:x:0::54 {}; prefix 2a02:74a0:x:y::/64 { AdvOnLink on; AdvAutonomous on; }; route fe80::a9fe:a9fe/128 { }; }; ``` We've been configuring the router with terraform, assigning the ipv4 resolvers to the IPv4 subnet and the IPv6 resolvers to the IPv6 subnet. After deleting the router, adjusting the subnets (no resolvers on v4, only ipv6 resolvers on ipv6), and recreating the router, radvd is now active and everything's fine. It seems that due to misconfiguration (and incomplete template parsing), IPv4 nameservers ended up in the config of radvd, which failed to start. Neutron was then unable to clean up the pidfile, thus failing to start radvd again. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2036877/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2033980] Re: Neutron fails to respawn radvd due to corrupt pid file
This is not a bug in kolla-ansible ** Changed in: kolla-ansible Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033980 Title: Neutron fails to respawn radvd due to corrupt pid file Status in kolla-ansible: Invalid Status in neutron: In Progress Bug description: **Bug Report** What happened: I have had issues periodically where radvd seems to die and neutron is not able to respawn it. I'm not sure why it dies. In my neutron-l3-agent.log, the following error occurs once per minute: ``` 2023-09-03 14:37:07.514 16 ERROR neutron.agent.linux.utils [-] Unable to convert value in /var/lib/neutron/external/pids/ea759c71-0f4d-4be9-a761-83843ce04d9a.pid.radvd 2023-09-03 14:37:07.514 16 ERROR neutron.agent.linux.external_process [-] radvd for router with uuid ea759c71-0f4d-4be9-a761-83843ce04d9a not found. The process should not have died 2023-09-03 14:37:07.514 16 WARNING neutron.agent.linux.external_process [-] Respawning radvd for uuid ea759c71-0f4d-4be9-a761-83843ce04d9a 2023-09-03 14:37:07.514 16 ERROR neutron.agent.linux.utils [-] Unable to convert value in /var/lib/neutron/external/pids/ea759c71-0f4d-4be9-a761-83843ce04d9a.pid.radvd 2023-09-03 14:37:07.762 16 ERROR neutron.agent.linux.utils [-] Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-ea759c71-0f4d-4be9-a761-83843ce04d9a', 'env', 'PROCESS_TAG=radvd-ea759c71-0f4d-4be9-a761-83843ce04d9a', 'radvd', '-C', '/var/lib/neutron/ra/ea759c71-0f4d-4be9-a761-83843ce04d9a.radvd.conf', '-p', '/var/lib/neutron/external/pids/ea759c71-0f4d-4be9-a761-83843ce04d9a.pid.radvd', '-m', 'syslog', '-u', 'neutron']; Stdin: ; Stdout: ; Stderr: ``` Inspecting the pid file, it appears to have 2 pids, one on each line: ``` $ docker exec -it neutron_l3_agent cat /var/lib/neutron/external/pids/ea759c71-0f4d-4be9-a761-83843ce04d9a.pid.radvd 853 1161 ``` Deleting the file then properly respawns radvd: ``` 2023-09-03 14:38:07.515 16 ERROR neutron.agent.linux.external_process [-] radvd for router with uuid ea759c71-0f4d-4be9-a761-83843ce04d9a not found. The process should not have died 2023-09-03 14:38:07.516 16 WARNING neutron.agent.linux.external_process [-] Respawning radvd for uuid ea759c71-0f4d-4be9-a761-83843ce04d9a ``` What you expected to happen: Radvd is respawned without needing manual intervention. Likely what is meant to happen is neutron should write the pid to the file, whereas instead it appends it. I'm not sure if this is a kolla issue or a neutron issue. How to reproduce it (minimal and precise): Unsure, I'm not sure how radvd ends up dying in the first place. You could likely reproduce this by deploying kolla-ansible and then manually killing radvd. **Environment**: * OS (e.g. from /etc/os-release): NAME="Rocky Linux" VERSION="9.2 (Blue Onyx)" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="9.2" PLATFORM_ID="platform:el9" PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)" ANSI_COLOR="0;32" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:rocky:rocky:9::baseos" HOME_URL="https://rockylinux.org/; BUG_REPORT_URL="https://bugs.rockylinux.org/; SUPPORT_END="2032-05-31" ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9" ROCKY_SUPPORT_PRODUCT_VERSION="9.2" REDHAT_SUPPORT_PRODUCT="Rocky Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.2" * Kernel (e.g. `uname -a`): Linux lon1 5.14.0-284.25.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 2 14:53:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux * Docker version if applicable (e.g. `docker version`): Client: Docker Engine - Community Version: 24.0.5 API version: 1.43 Go version:go1.20.6 Git commit:ced0996 Built: Fri Jul 21 20:36:54 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.5 API version: 1.43 (minimum version 1.12) Go version: go1.20.6 Git commit: a61e2b4 Built:Fri Jul 21 20:35:17 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.22 GitCommit:8165feabfdfe38c65b599c4993d227328c231fca runc: Version: 1.1.8 GitCommit:v1.1.8-0-g82f18fe docker-init: Version: 0.19.0 GitCommit:de40ad0 * Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): 16.1.0 (stable/2023.1) * Docker image Install type (source/binary): Default installed by kolla-ansible * Docker image distribution: rocky * Are you using official images from Docker Hub or self built? official * If self built - Kolla version and environment used to build: not applicable * Share your inventory file, globals.yml and other configuration files if
[Yahoo-eng-team] [Bug 1452641] Re: Static Ceph mon IP addresses in connection_info can prevent VM startup
billy-olsen: from the launchpad bug status description: "invalid" means: "the report describes the software's normal behaviour, or is unsuitable for any other reason." I'd argue that it is not normal behaviour but a missing feature or a bug, depending on how you look at it. imho it's clearly a nova shortcoming. if nova doesn't want this to be fixed please close it as "won't fix" instead. thanks. ** Changed in: nova Status: Invalid => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1452641 Title: Static Ceph mon IP addresses in connection_info can prevent VM startup Status in OpenStack Compute (nova): Confirmed Status in nova package in Ubuntu: Triaged Bug description: The Cinder rbd driver extracts the IP addresses of the Ceph mon servers from the Ceph mon map when the instance/volume connection is established. This info is then stored in nova's block-device-mapping table and is never re-validated down the line. Changing the Ceph mon servers' IP adresses will prevent the instance from booting as the stale connection info will enter the instance's XML. One idea to fix this would be to use the information from ceph.conf, which should be an alias or a loadblancer, directly. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452641/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp