Opening this back up against the package and adding upstream as well. I may be missing something, but I think this is still an issue upstream.
** Also affects: nova Importance: Undecided Status: New ** Changed in: nova (Ubuntu) Status: Won't Fix => Triaged ** Changed in: nova (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1818239 Title: scheduler: build failure high negative weighting Status in OpenStack nova-cloud-controller charm: Fix Committed Status in OpenStack Compute (nova): New Status in nova package in Ubuntu: Triaged Bug description: Whilst debugging a Queens cloud which seems to be landing all new instances on 3 out of 9 hypervisors (which resulted in three very heavily overloaded servers) I noticed that the weighting of the build failure weighter is -1000000.0 * number of failures: https://github.com/openstack/nova/blob/master/nova/conf/scheduler.py#L495 This means that a server which has any sort of build failure instantly drops to the bottom of the weighed list of hypervisors for scheduling of instances. Why might a instance fail to build? Could be a timeout due to load, might also be due to a bad image (one that won't actually boot under qemu). This second cause could be triggered by an end user of the cloud inadvertently causing all instances to be pushed to a small subset of hypervisors (which is what I think happened in our case). This feels like quite a dangerous default to have given the potential to DOS hypervisors intentionally or otherwise. ProblemType: Bug DistroRelease: Ubuntu 18.04 Package: nova-scheduler 2:17.0.7-0ubuntu1 ProcVersionSignature: Ubuntu 4.15.0-43.46-generic 4.15.18 Uname: Linux 4.15.0-43-generic x86_64 ApportVersion: 2.20.9-0ubuntu7.5 Architecture: amd64 Date: Fri Mar 1 13:57:39 2019 NovaConf: Error: [Errno 13] Permission denied: '/etc/nova/nova.conf' PackageArchitecture: all ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=C.UTF-8 SHELL=/bin/bash SourcePackage: nova UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1818239/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp