Public bug reported:
In multinode devstack setup nova scheduler fails to start after reboot
Steps to reproduce
==================
1 - deploy multinode devstack
https://docs.openstack.org/devstack/latest/guides/multinode-lab.html
2 - Verify all compute nodes are listed and setup is working as expected
$ openstack compute service list
create vm, assign floating IP and access VM
3 - Restart compute nodes, and controller node
$ sudo init 6
4 - Once controller and all other nodes are rebooted, check whether all nova
services are running
$ openstack compute service list
$ sudo systemctl status devstack@n-*
Expected result
===============
$ sudo systemctl status devstack@n-*
All services should be running
$ openstack compute service list
openstack cmds should run without a issue,
Actual result
=============
nova-schduler fails to start with error:
============================================
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
self._init_plugins(extensions)
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova File
"/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in
_init_plugins
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
raise NoMatches('No %r driver found, looking for %r' %
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking
for 'filter_scheduler'
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: INFO
oslo_service.periodic_task [-] Skipping periodic task _discover_hosts_in_cells
because its interval is negative
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: WARNING
stevedore.named [-] Could not load filter_scheduler
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: CRITICAL nova
[-] Unhandled error: stevedore.exception.NoMatches: No 'nova.scheduler.driver'
driver found, looking for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
Traceback (most recent call last):
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/bin/nova-scheduler", line 10, in <module>
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
sys.exit(main())
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/opt/stack/nova/nova/cmd/scheduler.py", line 47, in main
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
server = service.Service.create(binary='nova-scheduler',
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/opt/stack/nova/nova/service.py", line 252, in create
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
service_obj = cls(host, binary, topic, manager,
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/opt/stack/nova/nova/service.py", line 116, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
self.manager = manager_class(host=self.host, *args, **kwargs)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/opt/stack/nova/nova/scheduler/manager.py", line 60, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
self.driver = driver.DriverManager(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in
__init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
super(DriverManager, self).__init__(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in
__init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
self._init_plugins(extensions)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in
_init_plugins
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
raise NoMatches('No %r driver found, looking for %r' %
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking
for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
============================================
$ openstack compute service list
HttpException: 500: Server Error for url:
http://22.0.2.5/compute/v2.1/os-services, Internal Server Error
$ sudo systemctl status devstack@n-sch
● devstack@n-sch.service - Devstack devstack@n-sch.service
Loaded: loaded (/etc/systemd/system/devstack@n-sch.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2022-09-26 05:09:16 UTC; 3min
34s ago
Process: 11226 ExecStart=/usr/local/bin/nova-scheduler --config-file
/etc/nova/nova.conf (code=exited, status=1/FAILURE)
Main PID: 11226 (code=exited, status=1/FAILURE)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in
__init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
super(DriverManager, self).__init__(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in
__init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
self._init_plugins(extensions)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in
_init_plugins
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
raise NoMatches('No %r driver found, looking for %r' %
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking
for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service:
Main process exited, code=exited, status=1/FAILURE
Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service:
Failed with result 'exit-code'.
Environment
===========
Devstack multinode setup
1 controller
2 computes
** Affects: nova
Importance: Undecided
Status: New
** Attachment added: "nova-sch logs"
https://bugs.launchpad.net/bugs/1990809/+attachment/5618985/+files/sch.log
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1990809
Title:
multinode setup, devstack scheduler fails to start after controller
restart
Status in OpenStack Compute (nova):
New
Bug description:
In multinode devstack setup nova scheduler fails to start after reboot
Steps to reproduce
==================
1 - deploy multinode devstack
https://docs.openstack.org/devstack/latest/guides/multinode-lab.html
2 - Verify all compute nodes are listed and setup is working as expected
$ openstack compute service list
create vm, assign floating IP and access VM
3 - Restart compute nodes, and controller node
$ sudo init 6
4 - Once controller and all other nodes are rebooted, check whether all nova
services are running
$ openstack compute service list
$ sudo systemctl status devstack@n-*
Expected result
===============
$ sudo systemctl status devstack@n-*
All services should be running
$ openstack compute service list
openstack cmds should run without a issue,
Actual result
=============
nova-schduler fails to start with error:
============================================
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
self._init_plugins(extensions)
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in
_init_plugins
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
raise NoMatches('No %r driver found, looking for %r' %
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking
for 'filter_scheduler'
Sep 26 04:59:14 multinodesetupcontroller nova-scheduler[926]: ERROR nova
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: INFO
oslo_service.periodic_task [-] Skipping periodic task _discover_hosts_in_cells
because its interval is negative
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: WARNING
stevedore.named [-] Could not load filter_scheduler
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: CRITICAL nova
[-] Unhandled error: stevedore.exception.NoMatches: No 'nova.scheduler.driver'
driver found, looking for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
Traceback (most recent call last):
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/bin/nova-scheduler", line 10, in <module>
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
sys.exit(main())
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/opt/stack/nova/nova/cmd/scheduler.py", line 47, in main
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
server = service.Service.create(binary='nova-scheduler',
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/opt/stack/nova/nova/service.py", line 252, in create
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
service_obj = cls(host, binary, topic, manager,
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/opt/stack/nova/nova/service.py", line 116, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
self.manager = manager_class(host=self.host, *args, **kwargs)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/opt/stack/nova/nova/scheduler/manager.py", line 60, in __init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
self.driver = driver.DriverManager(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in
__init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
super(DriverManager, self).__init__(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in
__init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
self._init_plugins(extensions)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in
_init_plugins
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
raise NoMatches('No %r driver found, looking for %r' %
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking
for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
============================================
$ openstack compute service list
HttpException: 500: Server Error for url:
http://22.0.2.5/compute/v2.1/os-services, Internal Server Error
$ sudo systemctl status devstack@n-sch
● devstack@n-sch.service - Devstack devstack@n-sch.service
Loaded: loaded (/etc/systemd/system/devstack@n-sch.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2022-09-26 05:09:16 UTC;
3min 34s ago
Process: 11226 ExecStart=/usr/local/bin/nova-scheduler --config-file
/etc/nova/nova.conf (code=exited, status=1/FAILURE)
Main PID: 11226 (code=exited, status=1/FAILURE)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in
__init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
super(DriverManager, self).__init__(
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 89, in
__init__
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
self._init_plugins(extensions)
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
File "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 113, in
_init_plugins
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
raise NoMatches('No %r driver found, looking for %r' %
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
stevedore.exception.NoMatches: No 'nova.scheduler.driver' driver found, looking
for 'filter_scheduler'
Sep 26 05:09:16 multinodesetupcontroller nova-scheduler[11226]: ERROR nova
Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service:
Main process exited, code=exited, status=1/FAILURE
Sep 26 05:09:16 multinodesetupcontroller systemd[1]: devstack@n-sch.service:
Failed with result 'exit-code'.
Environment
===========
Devstack multinode setup
1 controller
2 computes
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1990809/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
