[Yahoo-eng-team] [Bug 1840862] Re: Missing description of hacking rule N363 in HACKING.rst
Reviewed: https://review.opendev.org/677589 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ccdbc7189640d02cc17e2156c635e8f7eafb02e Submitter: Zuul Branch:master commit 5ccdbc7189640d02cc17e2156c635e8f7eafb02e Author: Takashi NATSUME Date: Wed Aug 21 10:34:51 2019 +0900 Fix missing rule description in HACKING.rst The description of N363 hacking rule is missing in HACKING.rst. Add the description. Change-Id: I036a48612fcd256df4ccbd2ebba814bf3ed7a1c2 Closes-Bug: #1840862 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1840862 Title: Missing description of hacking rule N363 in HACKING.rst Status in OpenStack Compute (nova): Fix Released Bug description: The description of hacking rule N363 is missing in HACKING.rst. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1840862/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1826695] Re: [L3][QoS] cache does not removed when router is down or deleted
Reviewed: https://review.opendev.org/656105 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=78629e0d3780af4a8a8cc1b5b0762e0bc8a48f1f Submitter: Zuul Branch:master commit 78629e0d3780af4a8a8cc1b5b0762e0bc8a48f1f Author: LIU Yulong Date: Wed Mar 27 08:40:31 2019 +0800 Remove L3 IP QoS cache when router is down When router admin-state is down or removed, fip-qos and gateway-ip-qos extension should delete the router IPs' QoS rate limit cache. Then if the router is up again the router floating IPs QoS can be reconfigured. This patch achives these: 1. make sure floating IP or gateway IP QoS cache removed. 2. floating IP QoS can be re-configured to snat device when router doing admin_state down/up. Closes-Bug: #1826695 Change-Id: I24fcecd9686ad17fa50093bb8bccab0d6c711298 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1826695 Title: [L3][QoS] cache does not removed when router is down or deleted Status in neutron: Fix Released Bug description: [L3][QoS] cache does not removed when router is down or deleted ENV: stable/queens (master has the same issue) When router is DOWN or deleted, the L3 IP QoS extensions (floating IP and gateway IP) should remove the QoS cache. Otherwise, if the router is UP again, the floating IP QoS rules may not set to the devices back because the bandwidth value does not change. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1826695/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840930] Re: Networking service in neutron - install guide says to configure nova with [neutron]/url which is deprecated
Reviewed: https://review.opendev.org/677810 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e8653f28dee5044b943c4ca1ee0329e0bd7b34d2 Submitter: Zuul Branch:master commit e8653f28dee5044b943c4ca1ee0329e0bd7b34d2 Author: Matt Riedemann Date: Wed Aug 21 16:09:09 2019 -0400 doc: remove deprecated [neutron]/url from compute install guide The nova [neutron]/url config option was deprecated in Queens [1] and is being removed in Train [2]. The neutron install guide sections about configuring compute to work with neutron were still using the url option so this change removes them. There are a few things to note here: 1. The url option functionality is replaced with the endpoint_override option from keystoneauth1 but we don't really want users using that unless there is a real need. One of the main reasons for moving the nova configuration to use keystoneauth1 was so that the network service endpoint can be looked up via KSA dynamically based on the configurable interfaces (public, internal, private) and service types authority so the endpoint URL will just be pulled from the service catalog. That means not having to hard-code the endpoint URL in nova config which makes deployment and config management simpler. As such, the url option removed in the install guide here is not replaced with the endpoint_override option. 2. Following on #1, the install guide portion about the nova/neutron config is updated with a link back to the nova config guide for the full set of options in case an operator needs to tweak the config, e.g. to set valid_interfaces or endpoint_override because the KSA defaults don't work for their deployment. 3. With the old url option, if region_name was not specified, nova would default to 'RegionOne'. That is not the case if not using the url option so we leave the region_one config in the install guide example, otherwise region_name would default to None. [1] I41724a612a5f3eabd504f3eaa9d2f9d141ca3f69 [2] I6c068a84c4c0bd88f088f9328d7897bfc1f843f1 Change-Id: I30445edeb8509330571db28c7d61dd63886e9e61 Closes-Bug: #1840930 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1840930 Title: Networking service in neutron - install guide says to configure nova with [neutron]/url which is deprecated Status in neutron: Fix Released Bug description: - [x] This doc is inaccurate in this way: The [neutron]/url option https://docs.openstack.org/nova/latest/configuration/config.html#neutron.url in nova has been deprecated since the Queens release and is being removed in Train. The neutron/compute config guide in the neutron install guides still says to use the url option though. Since Queens when nova started using KSA adapters for working with neutron config: https://review.opendev.org/#/c/509892/ I think we want to avoid configuring the [neutron] section in nova.conf with url or endpoint_override but install rely on KSA to use the service types authority to find the endpoint to use based on service name/type and interface, in other words things should just work without needing to explicitly define an endpoint url to use for nova talking to neutron - nova can go through KSA and the service catalog to get the endpoint it needs. --- Release: 14.1.0.dev665 on 2017-06-30 05:58:47 SHA: 490471ebd3ac56d0cee164b9c1c1211687e49437 Source: https://opendev.org/openstack/neutron/src/doc/source/install/index.rst URL: https://docs.openstack.org/neutron/latest/install/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1840930/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840978] [NEW] nova-manage commands with unexpected errors returning 1 conflict with expected cases of 1 for flow control
Public bug reported: The archive_deleted_rows command returns 1 meaning some records were archived and the code documents that if automating and not using --until-complete, you should keep going while you get rc=1 until you get rc=0: https://github.com/openstack/nova/blob/0bf81cfe73340ba5cfd9cf44a38905014ba780f0/nova/cmd/manage.py#L505 The problem is if some unexpected error happens, let's say there is a TypeError in the code or something, the command will also return 1: https://github.com/openstack/nova/blob/0bf81cfe73340ba5cfd9cf44a38905014ba780f0/nova/cmd/manage.py#L2625 That unexpected error should probably be a 255 which generally means a command failed in some unexpected way. There might be other nova-manage commands that return 1 for flow control as well. Note that changing the "unexpected error" code from 1 to 255 is an upgrade impacting change worth a release note. ** Affects: nova Importance: Low Status: Triaged ** Tags: nova-manage ** Tags added: nova-manage ** Changed in: nova Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1840978 Title: nova-manage commands with unexpected errors returning 1 conflict with expected cases of 1 for flow control Status in OpenStack Compute (nova): Triaged Bug description: The archive_deleted_rows command returns 1 meaning some records were archived and the code documents that if automating and not using --until-complete, you should keep going while you get rc=1 until you get rc=0: https://github.com/openstack/nova/blob/0bf81cfe73340ba5cfd9cf44a38905014ba780f0/nova/cmd/manage.py#L505 The problem is if some unexpected error happens, let's say there is a TypeError in the code or something, the command will also return 1: https://github.com/openstack/nova/blob/0bf81cfe73340ba5cfd9cf44a38905014ba780f0/nova/cmd/manage.py#L2625 That unexpected error should probably be a 255 which generally means a command failed in some unexpected way. There might be other nova- manage commands that return 1 for flow control as well. Note that changing the "unexpected error" code from 1 to 255 is an upgrade impacting change worth a release note. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1840978/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840979] [NEW] [L2] [opinion] update the port DB status directly in agent-side
Public bug reported: When ovs-agent done processing the port, it will call neutron-server to make some DB update. Especially when restart the ovs-agent, all ports in one agent will do such RPC and DB update again to make port status consistent. When a large number of concurrent agent restart happen, neutron-server may not work fine. So how about making the following DB updating locally in neutron agent side directly? It may have some mechanism driver notification, IMO, this can also be done in agent-side. def update_device_down(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_down', device=device, agent_id=agent_id, host=host) def update_device_up(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_up', device=device, agent_id=agent_id, host=host) def update_device_list(self, context, devices_up, devices_down, ret = cctxt.call(context, 'update_device_list', ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1840979 Title: [L2] [opinion] update the port DB status directly in agent-side Status in neutron: New Bug description: When ovs-agent done processing the port, it will call neutron-server to make some DB update. Especially when restart the ovs-agent, all ports in one agent will do such RPC and DB update again to make port status consistent. When a large number of concurrent agent restart happen, neutron-server may not work fine. So how about making the following DB updating locally in neutron agent side directly? It may have some mechanism driver notification, IMO, this can also be done in agent-side. def update_device_down(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_down', device=device, agent_id=agent_id, host=host) def update_device_up(self, context, device, agent_id, host=None): cctxt = self.client.prepare() return cctxt.call(context, 'update_device_up', device=device, agent_id=agent_id, host=host) def update_device_list(self, context, devices_up, devices_down, ret = cctxt.call(context, 'update_device_list', To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1840979/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840967] [NEW] nova-next job does not fail when 'nova-manage db purge' fails
Public bug reported: Happened upon this while working on another patch to add more testing to our post_test_hook.sh script, excerpt from the log [1]: + /usr/local/bin/nova-manage db purge --all --verbose --all-cells + RET=3 + [[ 3 -eq 0 ]] + echo Purge failed with result 3 Purge failed with result 3 + return 3 + set -e + set +x WARNING: setting legacy OS_TENANT_NAME to support cli tools. + /opt/stack/nova/gate/post_test_hook.sh:main:54 : echo 'Verifying that instances were archived from all cells' Verifying that instances were archived from all cells ++ /opt/stack/nova/gate/post_test_hook.sh:main:55 : openstack server list --deleted --all-projects -c ID -f value + /opt/stack/nova/gate/post_test_hook.sh:main:55 : deleted_servers='e4727a33-796e-4173-b369-24d7ee45d7fd b213a354-0830-4cc3-abf7-e9dd068cefa9 33569d93-d7b6-4a92-825e-f36e972722db 521e4a84-c313-433e-8cc7-6d66c821d78c Because of a bug in my WIP patch, the purge command failed, but the job continued to run and didn't fail at that point because the 'nova-manage db purge' command comes before the 'set -e' command [that makes the script exit with any non-zero return value]. So, we need to move the purge command after 'set -e'. Note that we should *not* move the archive command though, because during its intermediate runs, it is expected to return 1, and we don't want to fail the job when that happens. The archive_deleted_rows function does its own explicit exiting in the case of actual failures. [1] https://object-storage-ca- ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_40/672840/8/check /nova-next/9d13cfb/ara-report/result/d13f888f-d187-4c3b-b5ab- 9326f611e534/ ** Affects: nova Importance: Undecided Assignee: melanie witt (melwitt) Status: New ** Tags: testing -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1840967 Title: nova-next job does not fail when 'nova-manage db purge' fails Status in OpenStack Compute (nova): New Bug description: Happened upon this while working on another patch to add more testing to our post_test_hook.sh script, excerpt from the log [1]: + /usr/local/bin/nova-manage db purge --all --verbose --all-cells + RET=3 + [[ 3 -eq 0 ]] + echo Purge failed with result 3 Purge failed with result 3 + return 3 + set -e + set +x WARNING: setting legacy OS_TENANT_NAME to support cli tools. + /opt/stack/nova/gate/post_test_hook.sh:main:54 : echo 'Verifying that instances were archived from all cells' Verifying that instances were archived from all cells ++ /opt/stack/nova/gate/post_test_hook.sh:main:55 : openstack server list --deleted --all-projects -c ID -f value + /opt/stack/nova/gate/post_test_hook.sh:main:55 : deleted_servers='e4727a33-796e-4173-b369-24d7ee45d7fd b213a354-0830-4cc3-abf7-e9dd068cefa9 33569d93-d7b6-4a92-825e-f36e972722db 521e4a84-c313-433e-8cc7-6d66c821d78c Because of a bug in my WIP patch, the purge command failed, but the job continued to run and didn't fail at that point because the 'nova- manage db purge' command comes before the 'set -e' command [that makes the script exit with any non-zero return value]. So, we need to move the purge command after 'set -e'. Note that we should *not* move the archive command though, because during its intermediate runs, it is expected to return 1, and we don't want to fail the job when that happens. The archive_deleted_rows function does its own explicit exiting in the case of actual failures. [1] https://object-storage-ca- ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_40/672840/8/check /nova-next/9d13cfb/ara-report/result/d13f888f-d187-4c3b-b5ab- 9326f611e534/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1840967/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1818734] Re: The endpoint group API doesn't use default roles
Reviewed: https://review.opendev.org/676115 Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=cf572f9e843745fd3b0fb2040e8a7a9dec94432e Submitter: Zuul Branch:master commit cf572f9e843745fd3b0fb2040e8a7a9dec94432e Author: Vishakha Agarwal Date: Tue Aug 13 12:07:19 2019 +0530 Add tests for project users interacting with endpoint_groups This commit introduces some tests that show how project users are expected to behave with the endpoint_groups API. Change-Id: I0f32de4ea615c89a7500a8098c44ef543fe45a02 Closes-bug: #1818734 ** Changed in: keystone Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1818734 Title: The endpoint group API doesn't use default roles Status in OpenStack Identity (keystone): Fix Released Bug description: In Rocky, keystone implemented support to ensure at least three default roles were available [0]. An endpoint group is a collection of endpoints that can be populated in a users service catalog through association to projects. Ultimately, endpoint groups are system-specific resources and shouldn't be accessible directly by domain or project users. The report is to track the work for implementing system `member` and system `reader` role support for endpoint groups. [0] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/define-default-roles.html [1] http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/policies/endpoint_group.py?id=6e3f1f6e46787ed4542609c935c13cb85e91d7fc API Reference: https://developer.openstack.org/api-ref/identity/v3-ext/index.html#os-ep-filter-api To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1818734/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840952] [NEW] no tenant network is available for allocation
Public bug reported: hi i don t know if it is a bug but i spent 3 days with this error. i read and redo the neutron option 2 network as the service . i use stein version . i installed a br0 bridge my questions are : . in linuxbridgeagent.ini must i set provider:eno1 or provider:br0 ? . how to create provider tenant with the new openstack client cli ? this message rised with : openstack network create edge --enable --internal (all previous commands are all ok (horizon, controller, compute node, domain, project, user, endpoint, etc). at startup and verification all documentation commands are ok i don t know where to investigate. thx Bruno ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1840952 Title: no tenant network is available for allocation Status in neutron: New Bug description: hi i don t know if it is a bug but i spent 3 days with this error. i read and redo the neutron option 2 network as the service . i use stein version . i installed a br0 bridge my questions are : . in linuxbridgeagent.ini must i set provider:eno1 or provider:br0 ? . how to create provider tenant with the new openstack client cli ? this message rised with : openstack network create edge --enable --internal (all previous commands are all ok (horizon, controller, compute node, domain, project, user, endpoint, etc). at startup and verification all documentation commands are ok i don t know where to investigate. thx Bruno To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1840952/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1704179] Re: Too many period db actions in large scale clusters increase the load of database
*** This bug is a duplicate of bug 1729621 *** https://bugs.launchpad.net/bugs/1729621 ** This bug has been marked a duplicate of bug 1729621 Inconsistent value for vcpu_used -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1704179 Title: Too many period db actions in large scale clusters increase the load of database Status in OpenStack Compute (nova): In Progress Bug description: Too many period db actions in large scale clusters increase the load of database, especially un-necessary db update or query. For example， over 1000 nodes, it will be 2 * 1000=2000 db update for compute_node table in 60s in _update_available_resource, but this two db update is not necessary if resource usage not changed. Delete the first and second _update() in _init_compute_node can reduce two db update for one node in 60s, if resource usage not changed for this compute_node. Then the funtion self._resource_change(compute_node) in _update() make sense To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1704179/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1729621] Re: Inconsistent value for vcpu_used
** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Changed in: nova/queens Status: New => Fix Released ** Changed in: nova/rocky Status: New => Fix Released ** Changed in: nova/pike Assignee: Tony Breeds (o-tony) => Radoslav Gerganov (rgerganov) ** Changed in: nova/pike Status: In Progress => Won't Fix ** Changed in: nova/queens Assignee: (unassigned) => Radoslav Gerganov (rgerganov) ** Changed in: nova/rocky Assignee: (unassigned) => Radoslav Gerganov (rgerganov) ** No longer affects: nova/ocata ** Changed in: nova/queens Importance: Undecided => High ** Changed in: nova/rocky Importance: Undecided => High ** Changed in: nova/pike Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1729621 Title: Inconsistent value for vcpu_used Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: Won't Fix Status in OpenStack Compute (nova) queens series: Fix Released Status in OpenStack Compute (nova) rocky series: Fix Released Bug description: Description === Nova updates hypervisor resources using function called ./nova/compute/resource_tracker.py:update_available_resource(). In case of *shutdowned* instances it could impact inconsistent values for resources like vcpu_used. Resources are taken from function self.driver.get_available_resource(): https://github.com/openstack/nova/blob/f974e3c3566f379211d7fdc790d07b5680925584/nova/compute/resource_tracker.py#L617 https://github.com/openstack/nova/blob/f974e3c3566f379211d7fdc790d07b5680925584/nova/virt/libvirt/driver.py#L5766 This function calculates allocated vcpu's based on function _get_vcpu_total(). https://github.com/openstack/nova/blob/f974e3c3566f379211d7fdc790d07b5680925584/nova/virt/libvirt/driver.py#L5352 As we see in _get_vcpu_total() function calls *self._host.list_guests()* without "only_running=False" parameter. So it doesn't respect shutdowned instances. At the end of resource update process function _update_available_resource() is beign called: > /opt/stack/nova/nova/compute/resource_tracker.py(733) 677 @utils.synchronized(COMPUTE_RESOURCE_SEMAPHORE) 678 def _update_available_resource(self, context, resources): 679 681 # initialize the compute node object, creating it 682 # if it does not already exist. 683 self._init_compute_node(context, resources) It initialize compute node object with resources that are calculated without shutdowned instances. If compute node object already exists it *UPDATES* its fields - *for a while nova-api has other resources values than it its in real.* 731 # update the compute_node 732 self._update(context, cn) The inconsistency is automatically fixed during other code execution: https://github.com/openstack/nova/blob/f974e3c3566f379211d7fdc790d07b5680925584/nova/compute/resource_tracker.py#L709 But for heavy-loaded hypervisors (like 100 active instances and 30 shutdowned instances) it creates wrong informations in nova database for about 4-5 seconds (in my usecase) - it could impact other issues like spawning on already full hypervisor (because scheduler has wrong informations about hypervisor usage). Steps to reproduce == 1) Start devstack 2) Create 120 instances 3) Stop some instances 4) Watch blinking values in nova hypervisor-show nova hypervisor-show e6dfc16b-7914-48fb-a235-6fe3a41bb6db Expected result === Returned values should be the same during test. Actual result = while true; do echo -n "$(date) "; echo "select hypervisor_hostname, vcpus_used from compute_nodes where hypervisor_hostname='example.compute.node.com';" | mysql nova_cell1; sleep 0.3; done Thu Nov 2 14:50:09 UTC 2017 example.compute.node.com 120 Thu Nov 2 14:50:10 UTC 2017 example.compute.node.com 120 Thu Nov 2 14:50:10 UTC 2017 example.compute.node.com 120 Thu Nov 2 14:50:10 UTC 2017 example.compute.node.com 120 Thu Nov 2 14:50:11 UTC 2017 example.compute.node.com 120 Thu Nov 2 14:50:11 UTC 2017 example.compute.node.com 120 Thu Nov 2 14:50:11 UTC 2017 example.compute.node.com 120 Thu Nov 2 14:50:11 UTC 2017 example.compute.node.com 120 Thu Nov 2 14:50:12 UTC 2017 example.compute.node.com 117 Thu Nov 2 14:50:12 UTC 2017 example.compute.node.com 117 Thu Nov 2 14:50:12 UTC 2017 example.compute.node.com 117 Thu Nov 2 14:50:13 UTC 2017 example.compute.node.com 117 Thu Nov 2 14:50:13 UTC 2017 example.compute.node.com 117 Thu Nov 2 14:50:13 UTC 2017 example.compute.node.com 117 Thu Nov 2 14:50:14 UTC 20
[Yahoo-eng-team] [Bug 1789991] Re: nova-compute error after enrolling ironic baremetal nodes
*** This bug is a duplicate of bug 1839674 ***
https://bugs.launchpad.net/bugs/1839674
** This bug has been marked a duplicate of bug 1839674
ResourceTracker.compute_nodes won't try to create a ComputeNode a second
time if the first create() fails
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1789991
Title:
nova-compute error after enrolling ironic baremetal nodes
Status in OpenStack Compute (nova):
New
Bug description:
Description
===
After enrolling some ironic baremetal nodes, I noticed the following in
nova-compute.log (longer trace below):
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager [req-73ba9d4b-
b51d-4ab7-88c8-5fc3f27fd89e - - - - -] Error updating resources for
node 0e5705cc-e872-49aa-aff4-1a91278b5cb3.: NotImplementedError:
Cannot load 'id' in the base class
Steps to reproduce
==
* Enroll ironic baremetal nodes (openstack baremetal node provide)
* Wait
* Error repeatedly appears in nova-compute.log
Expected result
===
No errors in log
Actual result
=
Errors in log
Environment
===
openstack-nova-compute-18.0.0-0.20180829095234.45fc232.el7.noarch
puppet-nova-13.3.1-0.20180825165256.5d1889b.el7.noarch
python-nova-18.0.0-0.20180829095234.45fc232.el7.noarch
python-novajoin-1.0.19-0.20180828183900.3d58511.el7.noarch
openstack-nova-common-18.0.0-0.20180829095234.45fc232.el7.noarch
python2-novaclient-11.0.0-0.20180807085257.f1005ce.el7.noarch
Logs & Configs
=
2018-08-30 17:00:51.142 7 DEBUG oslo_concurrency.lockutils
[req-73ba9d4b-b51d-4ab7-88c8-5fc3f27fd89e - - - - -] Lock "compute_resources"
release\
d by "nova.compute.resource_tracker._update_available_resource" :: held
0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils\
.py:285
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager
[req-73ba9d4b-b51d-4ab7-88c8-5fc3f27fd89e - - - - -] Error updating resources
for node 0e57\
05cc-e872-49aa-aff4-1a91278b5cb3.: NotImplementedError: Cannot load 'id' in
the base class
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager Traceback (most recent
call last):
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7729, in
_update_av\
ailable_resource_for_node
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager
rt.update_available_resource(context, nodename)
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 700,
in up\
date_available_resource
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager
self._update_available_resource(context, resources)
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in
inner
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager return f(*args,
**kwargs)
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 723,
in _u\
pdate_available_resource
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager
self._init_compute_node(context, resources)
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 563,
in _i\
nit_compute_node
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager
self._setup_pci_tracker(context, cn, resources)
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 594,
in _s\
etup_pci_tracker
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager n_id =
compute_node.id
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in
getter
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager
self.obj_load_attr(name)
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 603, in
obj_l\
oad_attr
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager _("Cannot load '%s'
in the base class") % attrname)
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager NotImplementedError:
Cannot load 'id' in the base class
2018-08-30 17:00:51.142 7 ERROR nova.compute.manager
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1789991/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1838003] Re: Horizon hides password requirements
Reviewed: https://review.opendev.org/672927 Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=8f5a3cd7dad96a0a764e0b7bd295fca8878a5166 Submitter: Zuul Branch:master commit 8f5a3cd7dad96a0a764e0b7bd295fca8878a5166 Author: Radomir Dopieralski Date: Fri Jul 26 11:22:41 2019 +0200 Display password_regex_description when password change fails When keystone has password_regex set and the new password doesn't match it, the attempt to change the password will fail, and keystone will helpfully include password_regex_description in its error message. Unfortunately Horizon discards that and displays "Unable to update the user password.", which is considerably less helpful. This change extracts the description from the error message in this case, and displays it to the user together with a translated error message. Unfortunately we have no way of also translating the description. Closes-bug: #1838003 Change-Id: I535c020deed76188c0f7150072fbdced9c5454fc ** Changed in: horizon Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1838003 Title: Horizon hides password requirements Status in OpenStack Dashboard (Horizon): Fix Released Bug description: When keystone has password_regex configured, and a new password doesn't match it, it will include password_regex_description in the error message, so that the users have a chance at coming up with a password that matches the requirements. Unfortunately Horizon hides that message and displays a completely unhelpful "Unable to update the user password." instead. Ideally we would like to display password_regex_description right in the password change form, in its description, but there is currently no API in keystone that would let us retrieve it. So instead it would be nice to at least display the description from the error message. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1838003/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840930] [NEW] Networking service in neutron - install guide says to configure nova with [neutron]/url which is deprecated
Public bug reported: - [x] This doc is inaccurate in this way: The [neutron]/url option https://docs.openstack.org/nova/latest/configuration/config.html#neutron.url in nova has been deprecated since the Queens release and is being removed in Train. The neutron/compute config guide in the neutron install guides still says to use the url option though. Since Queens when nova started using KSA adapters for working with neutron config: https://review.opendev.org/#/c/509892/ I think we want to avoid configuring the [neutron] section in nova.conf with url or endpoint_override but install rely on KSA to use the service types authority to find the endpoint to use based on service name/type and interface, in other words things should just work without needing to explicitly define an endpoint url to use for nova talking to neutron - nova can go through KSA and the service catalog to get the endpoint it needs. --- Release: 14.1.0.dev665 on 2017-06-30 05:58:47 SHA: 490471ebd3ac56d0cee164b9c1c1211687e49437 Source: https://opendev.org/openstack/neutron/src/doc/source/install/index.rst URL: https://docs.openstack.org/neutron/latest/install/ ** Affects: neutron Importance: Undecided Status: New ** Tags: doc -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1840930 Title: Networking service in neutron - install guide says to configure nova with [neutron]/url which is deprecated Status in neutron: New Bug description: - [x] This doc is inaccurate in this way: The [neutron]/url option https://docs.openstack.org/nova/latest/configuration/config.html#neutron.url in nova has been deprecated since the Queens release and is being removed in Train. The neutron/compute config guide in the neutron install guides still says to use the url option though. Since Queens when nova started using KSA adapters for working with neutron config: https://review.opendev.org/#/c/509892/ I think we want to avoid configuring the [neutron] section in nova.conf with url or endpoint_override but install rely on KSA to use the service types authority to find the endpoint to use based on service name/type and interface, in other words things should just work without needing to explicitly define an endpoint url to use for nova talking to neutron - nova can go through KSA and the service catalog to get the endpoint it needs. --- Release: 14.1.0.dev665 on 2017-06-30 05:58:47 SHA: 490471ebd3ac56d0cee164b9c1c1211687e49437 Source: https://opendev.org/openstack/neutron/src/doc/source/install/index.rst URL: https://docs.openstack.org/neutron/latest/install/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1840930/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840927] [NEW] Rewrite doc on L2 population
Public bug reported: In the Networking Guide back in Kilo https://docs.openstack.org/neutron/latest/admin/config-ml2.html there was a section on L2 population. This section no longer exists, and nothing similar is in existance. As a replacement, this blog post was used in the documentation: https://networkop.co.uk/blog/2016/05/06/neutron-l2pop/ as per this review: https://review.opendev.org/#/c/676920/ It would be best if this knowledge could be transferred. Thanks, Alex ** Affects: neutron Importance: Undecided Status: New ** Tags: docs ** Tags added: docs -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1840927 Title: Rewrite doc on L2 population Status in neutron: New Bug description: In the Networking Guide back in Kilo https://docs.openstack.org/neutron/latest/admin/config-ml2.html there was a section on L2 population. This section no longer exists, and nothing similar is in existance. As a replacement, this blog post was used in the documentation: https://networkop.co.uk/blog/2016/05/06/neutron-l2pop/ as per this review: https://review.opendev.org/#/c/676920/ It would be best if this knowledge could be transferred. Thanks, Alex To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1840927/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840912] [NEW] libvirt calls aren't reliably using tpool.Proxy
Public bug reported: A customer is hitting an issue with symptoms identical to bug 1045152 (from 2012). Specifically, we are frequently seeing the compute host being marked down. From log correlation, we can see that when this occurs the relevant compute is always in the middle of executing LibvirtDriver._get_disk_over_committed_size_total(). The reason for this appears to be a long-running libvirt call which is not using tpool.Proxy, and therefore blocks all other greenthreads during execution. We do not yet know why the libvirt call is slow, but we have identified the reason it is not using tpool.Proxy. Because eventlet, we proxy libvirt calls at the point we create the libvirt connection in libvirt.Host._connect: return tpool.proxy_call( (libvirt.virDomain, libvirt.virConnect), libvirt.openAuth, uri, auth, flags) This means: run libvirt.openAuth(uri, auth, flags) in a native thread. If the returned object is a libvirt.virDomain or libvirt.virConnect, wrap the returned object in a tpool.Proxy with the same autowrap rules. There are 2 problems with this. Firstly, the autowrap list is incomplete. At the very least we need to add libvirt.virNodeDevice, libvirt.virSecret, and libvirt.NWFilter to this list as we use all of these objects in Nova. Currently none of our interactions with these objects are using the tpool proxy. Secondly, and the specific root cause of this bug, it doesn't understand lists: https://github.com/eventlet/eventlet/blob/ca8dd0748a1985a409e9a9a517690f46e05cae99/eventlet/tpool.py#L149 In LibvirtDriver._get_disk_over_committed_size_total() we get a list of running libvirt domains with libvirt.Host.list_instance_domains, which calls virConnect.listAllDomains(). listAllDomains() returns a *list* of virDomain, which the above code in tpool doesn't match. Consequently, none of the subsequent virDomain calls use the tpool proxy, which starves all other greenthreads. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1840912 Title: libvirt calls aren't reliably using tpool.Proxy Status in OpenStack Compute (nova): New Bug description: A customer is hitting an issue with symptoms identical to bug 1045152 (from 2012). Specifically, we are frequently seeing the compute host being marked down. From log correlation, we can see that when this occurs the relevant compute is always in the middle of executing LibvirtDriver._get_disk_over_committed_size_total(). The reason for this appears to be a long-running libvirt call which is not using tpool.Proxy, and therefore blocks all other greenthreads during execution. We do not yet know why the libvirt call is slow, but we have identified the reason it is not using tpool.Proxy. Because eventlet, we proxy libvirt calls at the point we create the libvirt connection in libvirt.Host._connect: return tpool.proxy_call( (libvirt.virDomain, libvirt.virConnect), libvirt.openAuth, uri, auth, flags) This means: run libvirt.openAuth(uri, auth, flags) in a native thread. If the returned object is a libvirt.virDomain or libvirt.virConnect, wrap the returned object in a tpool.Proxy with the same autowrap rules. There are 2 problems with this. Firstly, the autowrap list is incomplete. At the very least we need to add libvirt.virNodeDevice, libvirt.virSecret, and libvirt.NWFilter to this list as we use all of these objects in Nova. Currently none of our interactions with these objects are using the tpool proxy. Secondly, and the specific root cause of this bug, it doesn't understand lists: https://github.com/eventlet/eventlet/blob/ca8dd0748a1985a409e9a9a517690f46e05cae99/eventlet/tpool.py#L149 In LibvirtDriver._get_disk_over_committed_size_total() we get a list of running libvirt domains with libvirt.Host.list_instance_domains, which calls virConnect.listAllDomains(). listAllDomains() returns a *list* of virDomain, which the above code in tpool doesn't match. Consequently, none of the subsequent virDomain calls use the tpool proxy, which starves all other greenthreads. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1840912/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840898] [NEW] INSTR can not supported for postgresql
Public bug reported: Recently added migration script doesn't support INSTR function. INSTR function does support sqlite3 and mysql. [1] https://review.opendev.org/#/c/665606/5 ** Affects: glance Importance: High Assignee: Abhishek Kekane (abhishek-kekane) Status: New ** Changed in: glance Assignee: (unassigned) => Abhishek Kekane (abhishek-kekane) ** Changed in: glance Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1840898 Title: INSTR can not supported for postgresql Status in Glance: New Bug description: Recently added migration script doesn't support INSTR function. INSTR function does support sqlite3 and mysql. [1] https://review.opendev.org/#/c/665606/5 To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1840898/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840895] [NEW] segment parameter check failed when creating network
Public bug reported: neutron net-create test --provider:network_type vlan --provider:segmentation_id 0 Execute commands like this, all vlan in ml2_vlan_allocations table is set to allocated, no vlan network can be created. validate_provider_segment function should check whether provider:segmentation_id is 0. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1840895 Title: segment parameter check failed when creating network Status in neutron: New Bug description: neutron net-create test --provider:network_type vlan --provider:segmentation_id 0 Execute commands like this, all vlan in ml2_vlan_allocations table is set to allocated, no vlan network can be created. validate_provider_segment function should check whether provider:segmentation_id is 0. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1840895/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
