Public bug reported: Recent failure: https://zuul.opendev.org/t/openstack/build/beb7c6f697bf4c1eb9752ee4e6fed1f2
https://zuul.opendev.org/t/openstack/build/beb7c6f697bf4c1eb9752ee4e6fed1f2/log/compute1/logs/screen- n-cpu.txt#1328 Jul 04 11:53:21.361972 np386776bf5ad74 nova-compute[41485]: ERROR oslo.messaging._drivers.impl_rabbit [None req-4d1d00d9-92c7-4ced-86e0-b1d0cc44c8bc None None] Connection failed: timed out (retrying in 31.0 seconds): TimeoutError: timed out It is failing since 2025-07-01T11:12:19 and it is always compute1 that fails to connect the compute service on the controller node can connect to rabbit. ❯ logsearch log --job-group nova-devstack --result FAILURE --file controller/logs/screen-n-cpu.txt --file compute1/logs/screen-n-cpu.txt --days 7 'Connection failed: timed out .retrying in 31.0 seconds.' Builds with matching logs 36/211: +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+ | uuid | finished | project | pipeline | review | branch | job | +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+ | beb7c6f697bf4c1eb9752ee4e6fed1f2 | 2025-07-04T11:59:13 | openstack/nova | check | https://review.opendev.org/952282 | master | nova-next | | 01f1c2c124d341d59670da90f5705acd | 2025-07-03T18:02:55 | openstack/nova | check | https://review.opendev.org/948076 | master | nova-grenade-multinode | | 5a9064b84094476fa5bcdbd658e85d0f | 2025-07-03T19:37:35 | openstack/nova | gate | https://review.opendev.org/940508 | master | nova-next | | 77e40958cb3041b6ace884ebb3dc9270 | 2025-07-03T19:17:16 | openstack/nova | gate | https://review.opendev.org/940508 | master | nova-multi-cell | | 8cb281a99de14fb98cd7043c8103b997 | 2025-07-03T17:33:48 | openstack/nova | gate | https://review.opendev.org/952283 | master | nova-grenade-multinode | | 66557ea92bae4e809c09526eba619c07 | 2025-07-03T16:22:57 | openstack/nova | check | https://review.opendev.org/948079 | master | nova-next | | c3601c4a41b04f0eac25f312fb808b13 | 2025-07-03T16:20:29 | openstack/nova | check | https://review.opendev.org/951957 | master | nova-next | | 860170b59a244a569b1645587f5c499a | 2025-07-02T21:32:11 | openstack/nova | check | https://review.opendev.org/940770 | master | nova-multi-cell | | 8603d4f4ca464477b760412907a6e2b8 | 2025-07-02T21:52:44 | openstack/nova | check | https://review.opendev.org/950643 | master | nova-multi-cell | | 5c6afb98f32842d0b7ff662322b53365 | 2025-07-02T19:00:55 | openstack/nova | check | https://review.opendev.org/948340 | master | nova-multi-cell | | 30c9b97130554a97bb1f0193b9c0c817 | 2025-07-02T17:55:57 | openstack/nova | check | https://review.opendev.org/948186 | master | nova-grenade-multinode | | 3e306bc27af0466fa1e2dd7956275862 | 2025-07-02T18:01:50 | openstack/nova | check | https://review.opendev.org/948187 | master | nova-grenade-multinode | | 17102220c090463bb4094119f1f0b7ae | 2025-07-02T19:09:52 | openstack/nova | check | https://review.opendev.org/948311 | master | nova-multi-cell | | 44f4b5d5575149c2967e7ebf728f91c5 | 2025-07-02T17:40:44 | openstack/nova | check | https://review.opendev.org/948079 | master | nova-grenade-multinode | | 947f3f05eb214692b7bf9951cf04875d | 2025-07-02T19:51:06 | openstack/nova | check | https://review.opendev.org/948079 | master | nova-next | | 406d88ca4b074fa3b0d6dc825d39ae83 | 2025-07-02T18:36:30 | openstack/nova | check | https://review.opendev.org/948079 | master | nova-multi-cell | | 99a1ba90c43a46eb80d24aa6fa42359f | 2025-07-02T19:11:22 | openstack/nova | check | https://review.opendev.org/949754 | master | nova-next | | 0174eb89649346c5be4374c3ab37f76c | 2025-07-02T19:41:38 | openstack/nova | check | https://review.opendev.org/948311 | master | nova-next | | 1e7c62a5b2f54ca884803ea202fa54ac | 2025-07-02T17:35:03 | openstack/nova | check | https://review.opendev.org/953815 | master | nova-grenade-multinode | | 8bf4e36445d94abdb7cc3f4d984823d5 | 2025-07-02T17:31:48 | openstack/nova | check | https://review.opendev.org/948086 | master | nova-grenade-multinode | | 336b33d030ed4835b7c0c6d0ac3980cb | 2025-07-02T17:29:25 | openstack/nova | check | https://review.opendev.org/953338 | master | nova-grenade-multinode | | eacaf25f04a6455eac504957fe6497b8 | 2025-07-02T17:08:47 | openstack/nova | check | https://review.opendev.org/948087 | master | nova-grenade-multinode | | e5c2fac9eefa4cc786c4087b45ee3f1c | 2025-07-02T19:25:00 | openstack/nova | check | https://review.opendev.org/948186 | master | nova-next | | 0fc3b1ccc5ac4d168abe9bc657195272 | 2025-07-02T16:27:41 | openstack/nova | check | https://review.opendev.org/948072 | master | nova-next | | 55cba550524f4a5c8b9b724d90f24b30 | 2025-07-02T13:24:46 | openstack/nova | check | https://review.opendev.org/948187 | master | nova-grenade-multinode | | 89393b4964944033a191e636ead84783 | 2025-07-01T21:26:32 | openstack/nova | check | https://review.opendev.org/940508 | master | nova-multi-cell | | b5517cb864b446f2912cb12aa59b0cf3 | 2025-07-01T19:19:03 | openstack/nova | check | https://review.opendev.org/940508 | master | nova-next | | 2d7e48fa335a4d8f93a4df394f634e33 | 2025-07-01T17:45:38 | openstack/nova | check | https://review.opendev.org/940508 | master | nova-multi-cell | | 64b4d81acef6427794197dfc5e9d71dc | 2025-07-01T13:35:41 | openstack/nova | check | https://review.opendev.org/948311 | master | nova-next | | 24a9a891c7854df190b0bb8eed67312a | 2025-07-01T12:54:41 | openstack/nova | check | https://review.opendev.org/950994 | master | nova-multi-cell | | 080485a9eb8d45b8ac91f2ec1ac59663 | 2025-07-01T13:22:21 | openstack/nova | check | https://review.opendev.org/948086 | master | nova-next | | 0361c38ab1964ac385727bf311594ac1 | 2025-07-01T12:18:16 | openstack/nova | check | https://review.opendev.org/948311 | master | nova-multi-cell | | c7169554320b4eb2970e2d93774cf75f | 2025-07-01T12:21:18 | openstack/nova | check | https://review.opendev.org/952666 | master | nova-next | | b2f0dc55e7cb4bb6822303a45618873f | 2025-07-01T12:29:05 | openstack/nova | check | https://review.opendev.org/948087 | master | nova-next | | d2ca59dba76549f58768b688d074e7c4 | 2025-07-01T11:56:50 | openstack/nova | check | https://review.opendev.org/948186 | master | nova-multi-cell | | 6890c3bdc3ef400a8c561c03e659047f | 2025-07-01T11:12:19 | openstack/nova | check | https://review.opendev.org/948076 | master | nova-multi-cell | +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+ ** Affects: nova Importance: Undecided Status: New ** Tags: gate-failure ** Tags added: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2115980 Title: Nova multi-node jobs are failing as second compute cannot connect to rabbit Status in OpenStack Compute (nova): New Bug description: Recent failure: https://zuul.opendev.org/t/openstack/build/beb7c6f697bf4c1eb9752ee4e6fed1f2 https://zuul.opendev.org/t/openstack/build/beb7c6f697bf4c1eb9752ee4e6fed1f2/log/compute1/logs/screen- n-cpu.txt#1328 Jul 04 11:53:21.361972 np386776bf5ad74 nova-compute[41485]: ERROR oslo.messaging._drivers.impl_rabbit [None req-4d1d00d9-92c7-4ced-86e0-b1d0cc44c8bc None None] Connection failed: timed out (retrying in 31.0 seconds): TimeoutError: timed out It is failing since 2025-07-01T11:12:19 and it is always compute1 that fails to connect the compute service on the controller node can connect to rabbit. ❯ logsearch log --job-group nova-devstack --result FAILURE --file controller/logs/screen-n-cpu.txt --file compute1/logs/screen-n-cpu.txt --days 7 'Connection failed: timed out .retrying in 31.0 seconds.' Builds with matching logs 36/211: +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+ | uuid | finished | project | pipeline | review | branch | job | +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+ | beb7c6f697bf4c1eb9752ee4e6fed1f2 | 2025-07-04T11:59:13 | openstack/nova | check | https://review.opendev.org/952282 | master | nova-next | | 01f1c2c124d341d59670da90f5705acd | 2025-07-03T18:02:55 | openstack/nova | check | https://review.opendev.org/948076 | master | nova-grenade-multinode | | 5a9064b84094476fa5bcdbd658e85d0f | 2025-07-03T19:37:35 | openstack/nova | gate | https://review.opendev.org/940508 | master | nova-next | | 77e40958cb3041b6ace884ebb3dc9270 | 2025-07-03T19:17:16 | openstack/nova | gate | https://review.opendev.org/940508 | master | nova-multi-cell | | 8cb281a99de14fb98cd7043c8103b997 | 2025-07-03T17:33:48 | openstack/nova | gate | https://review.opendev.org/952283 | master | nova-grenade-multinode | | 66557ea92bae4e809c09526eba619c07 | 2025-07-03T16:22:57 | openstack/nova | check | https://review.opendev.org/948079 | master | nova-next | | c3601c4a41b04f0eac25f312fb808b13 | 2025-07-03T16:20:29 | openstack/nova | check | https://review.opendev.org/951957 | master | nova-next | | 860170b59a244a569b1645587f5c499a | 2025-07-02T21:32:11 | openstack/nova | check | https://review.opendev.org/940770 | master | nova-multi-cell | | 8603d4f4ca464477b760412907a6e2b8 | 2025-07-02T21:52:44 | openstack/nova | check | https://review.opendev.org/950643 | master | nova-multi-cell | | 5c6afb98f32842d0b7ff662322b53365 | 2025-07-02T19:00:55 | openstack/nova | check | https://review.opendev.org/948340 | master | nova-multi-cell | | 30c9b97130554a97bb1f0193b9c0c817 | 2025-07-02T17:55:57 | openstack/nova | check | https://review.opendev.org/948186 | master | nova-grenade-multinode | | 3e306bc27af0466fa1e2dd7956275862 | 2025-07-02T18:01:50 | openstack/nova | check | https://review.opendev.org/948187 | master | nova-grenade-multinode | | 17102220c090463bb4094119f1f0b7ae | 2025-07-02T19:09:52 | openstack/nova | check | https://review.opendev.org/948311 | master | nova-multi-cell | | 44f4b5d5575149c2967e7ebf728f91c5 | 2025-07-02T17:40:44 | openstack/nova | check | https://review.opendev.org/948079 | master | nova-grenade-multinode | | 947f3f05eb214692b7bf9951cf04875d | 2025-07-02T19:51:06 | openstack/nova | check | https://review.opendev.org/948079 | master | nova-next | | 406d88ca4b074fa3b0d6dc825d39ae83 | 2025-07-02T18:36:30 | openstack/nova | check | https://review.opendev.org/948079 | master | nova-multi-cell | | 99a1ba90c43a46eb80d24aa6fa42359f | 2025-07-02T19:11:22 | openstack/nova | check | https://review.opendev.org/949754 | master | nova-next | | 0174eb89649346c5be4374c3ab37f76c | 2025-07-02T19:41:38 | openstack/nova | check | https://review.opendev.org/948311 | master | nova-next | | 1e7c62a5b2f54ca884803ea202fa54ac | 2025-07-02T17:35:03 | openstack/nova | check | https://review.opendev.org/953815 | master | nova-grenade-multinode | | 8bf4e36445d94abdb7cc3f4d984823d5 | 2025-07-02T17:31:48 | openstack/nova | check | https://review.opendev.org/948086 | master | nova-grenade-multinode | | 336b33d030ed4835b7c0c6d0ac3980cb | 2025-07-02T17:29:25 | openstack/nova | check | https://review.opendev.org/953338 | master | nova-grenade-multinode | | eacaf25f04a6455eac504957fe6497b8 | 2025-07-02T17:08:47 | openstack/nova | check | https://review.opendev.org/948087 | master | nova-grenade-multinode | | e5c2fac9eefa4cc786c4087b45ee3f1c | 2025-07-02T19:25:00 | openstack/nova | check | https://review.opendev.org/948186 | master | nova-next | | 0fc3b1ccc5ac4d168abe9bc657195272 | 2025-07-02T16:27:41 | openstack/nova | check | https://review.opendev.org/948072 | master | nova-next | | 55cba550524f4a5c8b9b724d90f24b30 | 2025-07-02T13:24:46 | openstack/nova | check | https://review.opendev.org/948187 | master | nova-grenade-multinode | | 89393b4964944033a191e636ead84783 | 2025-07-01T21:26:32 | openstack/nova | check | https://review.opendev.org/940508 | master | nova-multi-cell | | b5517cb864b446f2912cb12aa59b0cf3 | 2025-07-01T19:19:03 | openstack/nova | check | https://review.opendev.org/940508 | master | nova-next | | 2d7e48fa335a4d8f93a4df394f634e33 | 2025-07-01T17:45:38 | openstack/nova | check | https://review.opendev.org/940508 | master | nova-multi-cell | | 64b4d81acef6427794197dfc5e9d71dc | 2025-07-01T13:35:41 | openstack/nova | check | https://review.opendev.org/948311 | master | nova-next | | 24a9a891c7854df190b0bb8eed67312a | 2025-07-01T12:54:41 | openstack/nova | check | https://review.opendev.org/950994 | master | nova-multi-cell | | 080485a9eb8d45b8ac91f2ec1ac59663 | 2025-07-01T13:22:21 | openstack/nova | check | https://review.opendev.org/948086 | master | nova-next | | 0361c38ab1964ac385727bf311594ac1 | 2025-07-01T12:18:16 | openstack/nova | check | https://review.opendev.org/948311 | master | nova-multi-cell | | c7169554320b4eb2970e2d93774cf75f | 2025-07-01T12:21:18 | openstack/nova | check | https://review.opendev.org/952666 | master | nova-next | | b2f0dc55e7cb4bb6822303a45618873f | 2025-07-01T12:29:05 | openstack/nova | check | https://review.opendev.org/948087 | master | nova-next | | d2ca59dba76549f58768b688d074e7c4 | 2025-07-01T11:56:50 | openstack/nova | check | https://review.opendev.org/948186 | master | nova-multi-cell | | 6890c3bdc3ef400a8c561c03e659047f | 2025-07-01T11:12:19 | openstack/nova | check | https://review.opendev.org/948076 | master | nova-multi-cell | +----------------------------------+---------------------+----------------+----------+-----------------------------------+--------+------------------------+ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2115980/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
