[Yahoo-eng-team] [Bug 1718788] Re: DVR: Migrate centralized unbound floatingip to the respective host when the port is bound
Reviewed: https://review.openstack.org/505324 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=27fcf86bcbf4a66f52385f0efea956f13c38d7b2 Submitter: Jenkins Branch:master commit 27fcf86bcbf4a66f52385f0efea956f13c38d7b2 Author: Swaminathan VasudevanDate: Tue Sep 19 06:54:14 2017 -0700 DVR: Fix unbound fip port migration to bound port With the current change in allowing the unbound fip to be associated with the snat node, we are seeing that all floating IPs that are associated with an unbound port are created at the snat node. This is also applicable for floating IPs that are created just before associating the port to a VM. We have seen such scenarios in the test cases. This is the right behavior as per design. But when the port is bound to a host, the floating IP should be migrated to the respective host. This patch fixes the issue by sending notification to the respective node, when the port is bound and also clear the fip from the snat node. Closes-Bug: #1718788 Change-Id: I6b1f3ffc3c3336035632f6a82d3a87b3be57b403 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1718788 Title: DVR: Migrate centralized unbound floatingip to the respective host when the port is bound Status in neutron: Fix Released Bug description: When unbound ports are associated with floatingIP in DVR, it implements the floatingIP in the dvr_snat node under the snat_namespace. When the private ports are bound to a specific host, the floatingIPs are not moved or migrated to their respective hosts. This can be reproduced by 1. Create a network 2. Create a subnet 3. Create a router and associate the subnet to the router 4. Assign a gateway to the router. 5. Then create a port on the given network with a specific IP. 6. Now create a FloatingIP on the external network. 7. Associate the FloatingIP to the created port. 8. At this point the port is not bound and so the floatingIP gets implemented in the Snat_namespace in the dvr_snat node. 9. Then within a few seconds, we create a VM with the given port-id instead of network-id. 10. Now when the VM is built then the port gets bound. 11. Now the floatingIP is not seen on the host where the VM resides. Theoretically the FloatingIP should be migrated to the host where it is currently bound. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1718788/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1722293] Re: Keystone not removing mapping between deleted LDAP user and Openstack
LDAP is a read-only backend for keystone. We do provide a way to purge users from the id_mapping backend [0][1][2]. Keystone does not receive push notifications from LDAP when users or group changes are made there. Not that this must be done using the `keystone-manage` command. If you need more information on how this works, please swing by #openstack- keystone. [0] https://github.com/openstack/keystone/blob/47dbd256258d747d95cb5320bd02ae207ecf60d6/keystone/cmd/cli.py#L824 [1] https://github.com/openstack/keystone/blob/47dbd256258d747d95cb5320bd02ae207ecf60d6/keystone/identity/core.py#L1470 [2] https://docs.openstack.org/keystone/latest/cli/index.html#keystone-manage ** Changed in: keystone Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1722293 Title: Keystone not removing mapping between deleted LDAP user and Openstack Status in OpenStack Keystone LDAP integration: Invalid Status in OpenStack Identity (keystone): Invalid Bug description: Keystone not removing mapping between deleted LDAP user and Openstack The client is using LDAP for authentication and has used uid as a key for user_id_attribute. The client created a LDAP user say ABC with UID=100, this user is associated with an OpenStack user ABC. The relationship is recorded in id_mapping table within keystone database. Now when the client delete the ldap user ABC, the entry is not deleted from the id_mapping table. Thus when the client create a new ldap user XYZ which get the same UID=100, the incorrect record in id_mapping restrict the new user XYZ from authenticating and successfully log on to OpenStack. Note: there is not record for XYZ within the id_mapping table. Details of domain config: # User supplied configuration flags user_filter = (memberof=cn=xxx,ou=Group,dc=xxx,dc=xxx) user_id_attribute = uidNumber user_name_attribute = uid user_objectclass = posixAccount user_tree_dn = ou=x,dc=xxx,dc=xx [identity] driver = ldap Table Description mysql> desc id_mapping; +-+--+--+-+-+---+ | Field | Type | Null | Key | Default | Extra | +-+--+--+-+-+---+ | public_id | varchar(64) | NO | PRI | NULL| | | domain_id | varchar(64) | NO | MUL | NULL| | | local_id| varchar(64) | NO | | NULL| | | entity_type | enum('user','group') | NO | | NULL| | +-+--+--+-+-+---+ To manage notifications about this bug go to: https://bugs.launchpad.net/charm-keystone-ldap/+bug/1722293/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1718747] Re: Unable to delete domain with users in it
** Also affects: keystone/newton Importance: Undecided Status: New ** Also affects: keystone/ocata Importance: Undecided Status: New ** Also affects: keystone/pike Importance: Undecided Status: New ** Changed in: keystone/newton Importance: Undecided => High ** Changed in: keystone/ocata Importance: Undecided => Medium ** Changed in: keystone/pike Importance: Undecided => Medium ** Changed in: keystone/ocata Importance: Medium => High ** Changed in: keystone/pike Importance: Medium => High ** Changed in: keystone/newton Status: New => Confirmed ** Changed in: keystone/pike Status: New => Confirmed ** Changed in: keystone/ocata Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1718747 Title: Unable to delete domain with users in it Status in OpenStack Identity (keystone): In Progress Status in OpenStack Identity (keystone) newton series: Confirmed Status in OpenStack Identity (keystone) ocata series: Confirmed Status in OpenStack Identity (keystone) pike series: Confirmed Bug description: Attempting to delete a domain which contains users and projects may yield an UnexpectedError similiar to this Sep 21 19:37:17 vagrant-openSUSE-Leap devstack@keystone.service[23894]: DEBUG keystone.common.sql.core [None req-707ec264-b10c-4079-94bb-2af01db58aab None None] Conflict project: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`keystone`.`user`, CONSTRAINT `user_ibfk_1` FOREIGN KEY (`domain_id`) REFERENCES `project` (`id`))') [SQL: u'DELETE FROM project WHERE project.id = %(id)s'] [parameters: {'id': u'63d2d5446e364f00b3181bf49c62c5b8'}] {{(pid=23897) wrapper /opt/stack/keystone/keystone/common/sql/core.py:550}} Sep 21 19:37:17 vagrant-openSUSE-Leap devstack@keystone.service[23894]: WARNING keystone.common.wsgi [None req-707ec264-b10c-4079-94bb-2af01db58aab None None] An unexpected error prevented the server from fulfilling your request.: UnexpectedError: An unexpected error prevented the server from fulfilling your request. Steps to reproduce: 1. Install devstack 2. create a domain 'foo' openstack domain create foo 3. create a user in domain 'foo' openstack user create --password equifax --domain foo foo_user 4. create a project in domain 'foo' openstack project create --domain foo foo_project 5. enable domain user 'foo_user' access to project 'foo_project' openstack role add --user foo_user --project foo_project admin 6. now disable domain 'foo' openstack domain set --disable foo 7. attempt to delete domain 'foo' will yield an expected error mentioned above openstack domain delete foo This was introduced in: https://github.com/openstack/keystone/commit/2bd88d30e1d2873470af7f40db45a99e07e12ce6 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1718747/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1722444] Re: WARNING message related to keystone is present in all services API logs
This looks specific to how you're configuration is set up. Can you share how you have keystone configured at each of the services (omitting sensitive information)? ** Also affects: keystonemiddleware Importance: Undecided Status: New ** No longer affects: keystone ** Changed in: keystonemiddleware Status: New => Incomplete -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1722444 Title: WARNING message related to keystone is present in all services API logs Status in keystonemiddleware: Incomplete Bug description: 2017-08-31 02:57:29.236 23010 WARNING keystonemiddleware._common.config [-] The option "__file__" in conf is not known to auth_token 2017-08-31 02:57:29.244 23010 WARNING keystonemiddleware._common.config [-] The option "configkey" in conf is not known to auth_token 2017-08-31 02:57:29.245 23010 WARNING keystonemiddleware._common.config [-] The option "here" in conf is not known to auth_token The above warning messages are cluttering all the service api log files because of this change https://github.com/openstack/keystonemiddleware/commit/ba78db0b4b559dcc3b22c37834e430d8b27c45ae To manage notifications about this bug go to: https://bugs.launchpad.net/keystonemiddleware/+bug/1722444/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1488111] Re: Boot from volumes that fail in initialize_connection are not rescheduled
I wouldn't say that we won't ever fix this, since I've wondered why we don't reschedule on volume failures like we do with networking failures, but it's not a high priority. ** Tags removed: liberty-backport-potential ** No longer affects: nova/liberty ** Changed in: nova Status: Won't Fix => Opinion ** Changed in: nova Status: Opinion => Confirmed ** Changed in: nova Importance: Low => Wishlist -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1488111 Title: Boot from volumes that fail in initialize_connection are not rescheduled Status in OpenStack Compute (nova): Confirmed Bug description: Version: OpenStack Liberty Boot from volumes that fail in volume initialize_connection are not rescheduled. Initialize connection failures can be very host-specific and in many cases the boot would succeed if the instance build was rescheduled to another host. The instance is not rescheduled because the initialize_connection is being called down this stack: nova.compute.manager _build_resources nova.compute.manager _prep_block_device nova.virt.block_device attach_block_devices nova.virt.block_device.DriverVolumeBlockDevice.attach When this fails an exception is thrown which lands in this block: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1740 and throws an InvalidBDM exception which is caught by this block: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2110 this in turn throws a BuildAbortException which causes the instance to not be rescheduled by landing the flow in this block: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2004 To fix this we likely need a different exception thrown from nova.virt.block_device.DriverVolumeBlockDevice.attach when the failure is in initialize_connection and then work back up the stack to ensure that when this different exception is thrown a BuildAbortException is not thrown so the reschedule can happen. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1488111/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1722577] Re: test_list_get_volume_attachments failing with 400 error on teardown when detaching an already detached volume
Reviewed: https://review.openstack.org/510951 Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=0d4551b367c9d3f37117a5426a9ca500d791ac25 Submitter: Jenkins Branch:master commit 0d4551b367c9d3f37117a5426a9ca500d791ac25 Author: Matt RiedemannDate: Tue Oct 10 13:00:48 2017 -0400 Only attempt to detach an in-use volume during cleanup test_list_get_volume_attachments intermittently fails during cleanup because it tries to detach an already detached volume, which results in a 400 response. Tempest, as the client, should be checking the volume status before making the detach request. The only reason this ever worked before Pike was because of some (incorrect) ordering of operations in the compute service which affected how the API behaved during detach, and the compute API would return a 404 rather than a 400. That changed with I2581ff9f9c0e7cfc14a25acf45eb1860df69eacf in Pike, which exposed the race on the Tempest side by deleting the BDM in nova *after* marking the volume as 'available' in Cinder, and the os-volume_attachments API checks for the existence of the BDM and if it exists, attempts the detach (which then fails with the 400 from Cinder). Change-Id: Id2d22cbb86d8d5fa7f71202b274260c1367e8a0f Closes-Bug: #1722577 ** Changed in: tempest Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1722577 Title: test_list_get_volume_attachments failing with 400 error on teardown when detaching an already detached volume Status in OpenStack Compute (nova): Won't Fix Status in tempest: Fix Released Bug description: Noticed this today: http://logs.openstack.org/04/510704/1/check/gate-tempest-dsvm-py35 -ubuntu-xenial/084e7c7/console.html#_2017-10-10_07_50_05_759627 2017-10-10 07:50:05.759627 | tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments[id-7fa563fe-f0f7-43eb-9e22-a1ece036b513] 2017-10-10 07:50:05.759770 | - 2017-10-10 07:50:05.759801 | 2017-10-10 07:50:05.759859 | Captured traceback: 2017-10-10 07:50:05.759942 | ~~~ 2017-10-10 07:50:05.75 | b'Traceback (most recent call last):' 2017-10-10 07:50:05.760082 | b' File "/opt/stack/new/tempest/tempest/lib/common/utils/test_utils.py", line 84, in call_and_ignore_notfound_exc' 2017-10-10 07:50:05.760144 | b'return func(*args, **kwargs)' 2017-10-10 07:50:05.760226 | b' File "/opt/stack/new/tempest/tempest/lib/services/compute/servers_client.py", line 441, in detach_volume' 2017-10-10 07:50:05.760266 | b'(server_id, volume_id))' 2017-10-10 07:50:05.760335 | b' File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 301, in delete' 2017-10-10 07:50:05.760411 | b"return self.request('DELETE', url, extra_headers, headers, body)" 2017-10-10 07:50:05.760545 | b' File "/opt/stack/new/tempest/tempest/lib/services/compute/base_compute_client.py", line 48, in request' 2017-10-10 07:50:05.760602 | b'method, url, extra_headers, headers, body, chunked)' 2017-10-10 07:50:05.760670 | b' File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 659, in request' 2017-10-10 07:50:05.760716 | b'self._error_checker(resp, resp_body)' 2017-10-10 07:50:05.760787 | b' File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 770, in _error_checker' 2017-10-10 07:50:05.760852 | b'raise exceptions.BadRequest(resp_body, resp=resp)' 2017-10-10 07:50:05.760903 | b'tempest.lib.exceptions.BadRequest: Bad request' 2017-10-10 07:50:05.761081 | b'Details: {\'message\': "Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be \'in-use\' and attach_status must be \'attached\' to detach. (HTTP 400) (Request-ID: req-e4faf8b8-a431-4021-8480-486c94a91543)", \'code\': 400}' 2017-10-10 07:50:05.761119 | b'' This test does the following: 1. create a server 2. create a volume vol1 3. attach vol1 to the server and wait for the volume status to be in-use 4. repeat steps 3 and 4 for vol2 5. for each volume attachment, detach it and wait for it's status to be 'available' That all works. The failure is during the cleanup routines on teardown, they try to detach the volume but the volumes are already detached, which results in a 400 response from the compute API (because the compute API gets a 400 response from the volumev3 API): http://logs.openstack.org/04/510704/1/check/gate-tempest-dsvm-py35
[Yahoo-eng-team] [Bug 1723511] [NEW] 05_logging.conf file broken in /etc/cloud/cloud.log.d broke cloud-init-output logs
Public bug reported: - Cloud Provider: AWS - dpkg-query = ? (CentOS, but I suppose you're asking for the package version): cloud-init x86_64 0.7.9-9.el7.centos.2base - Any appropriate cloud-init config: https://github.com/number5/cloud- init/blob/master/config/cloud.cfg.d/05_logging.cfg This file contains the appropriate config. -- Hello there, There is a new 05_logging.cfg file in the latest cloud-init package which brokes the cloud-init-output.log file and all the logs are dumped into messages log instead. This file matches with the new rpm package from September 5th. Before this update, there been a 05_logging.conf with the proper configuration to dump the output log, and this file is from April 1st. ** Affects: cloud-init Importance: Undecided Status: New ** Attachment added: "cloud-init.tar" https://bugs.launchpad.net/bugs/1723511/+attachment/4970327/+files/cloud-init.tar -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1723511 Title: 05_logging.conf file broken in /etc/cloud/cloud.log.d broke cloud- init-output logs Status in cloud-init: New Bug description: - Cloud Provider: AWS - dpkg-query = ? (CentOS, but I suppose you're asking for the package version): cloud-init x86_64 0.7.9-9.el7.centos.2base - Any appropriate cloud-init config: https://github.com/number5/cloud- init/blob/master/config/cloud.cfg.d/05_logging.cfg This file contains the appropriate config. -- Hello there, There is a new 05_logging.cfg file in the latest cloud-init package which brokes the cloud-init-output.log file and all the logs are dumped into messages log instead. This file matches with the new rpm package from September 5th. Before this update, there been a 05_logging.conf with the proper configuration to dump the output log, and this file is from April 1st. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1723511/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1723512] [NEW] 05_logging.conf file broken in /etc/cloud/cloud.log.d broke cloud-init-output logs
Public bug reported: - Cloud Provider: AWS - dpkg-query = ? (CentOS, but I suppose you're asking for the package version): cloud-init x86_64 0.7.9-9.el7.centos.2base - Any appropriate cloud-init config: https://github.com/number5/cloud- init/blob/master/config/cloud.cfg.d/05_logging.cfg This file contains the appropriate config. -- Hello there, There is a new 05_logging.cfg file in the latest cloud-init package which brokes the cloud-init-output.log file and all the logs are dumped into messages log instead. This file matches with the new rpm package from September 5th. Before this update, there been a 05_logging.conf with the proper configuration to dump the output log, and this file is from April 1st. ** Affects: cloud-init Importance: Undecided Status: New ** Attachment added: "cloud-init.tar" https://bugs.launchpad.net/bugs/1723512/+attachment/4970328/+files/cloud-init.tar -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1723512 Title: 05_logging.conf file broken in /etc/cloud/cloud.log.d broke cloud- init-output logs Status in cloud-init: New Bug description: - Cloud Provider: AWS - dpkg-query = ? (CentOS, but I suppose you're asking for the package version): cloud-init x86_64 0.7.9-9.el7.centos.2base - Any appropriate cloud-init config: https://github.com/number5/cloud- init/blob/master/config/cloud.cfg.d/05_logging.cfg This file contains the appropriate config. -- Hello there, There is a new 05_logging.cfg file in the latest cloud-init package which brokes the cloud-init-output.log file and all the logs are dumped into messages log instead. This file matches with the new rpm package from September 5th. Before this update, there been a 05_logging.conf with the proper configuration to dump the output log, and this file is from April 1st. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1723512/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1723476] [NEW] py27 job failing on testtools.matchers._impl.MismatchError
Public bug reported: Related patches: https://review.openstack.org/#/c/511835/ https://review.openstack.org/#/c/511767/ when I remove some useless methods in nova/objects/compute_node.py, I find the error: Failed 2 tests - output below: 2017-10-13 13:07:54.110687 | == 2017-10-13 13:07:54.110704 | 2017-10-13 13:07:54.110758 | nova.tests.unit.notifications.objects.test_notification.TestNotificationObjectVersions.test_versions 2017-10-13 13:07:54.110820 | 2017-10-13 13:07:54.110838 | 2017-10-13 13:07:54.110850 | Captured traceback: 2017-10-13 13:07:54.110861 | ~~~ 2017-10-13 13:07:54.110879 | Traceback (most recent call last): 2017-10-13 13:07:54.110911 | File "nova/tests/unit/notifications/objects/test_notification.py", line 420, in test_versions 2017-10-13 13:07:54.110933 | 'Some notification objects have changed; please make ' 2017-10-13 13:07:54.110992 | File "/home/jenkins/workspace/gate-nova-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual 2017-10-13 13:07:54.111018 | self.assertThat(observed, matcher, message) 2017-10-13 13:07:54.111062 | File "/home/jenkins/workspace/gate-nova-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat 2017-10-13 13:07:54.111075 | raise mismatch_error 2017-10-13 13:07:54.111093 | testtools.matchers._impl.MismatchError: !=: 2017-10-13 13:07:54.17 | reference = {'ComputeNodeList': '1.17-52f3b0962b1c86b98590144463ebb192'} 2017-10-13 13:07:54.41 | actual= {'ComputeNodeList': '1.17-badecf5a910a5e0df8fffb270f30c7da'} 2017-10-13 13:07:54.78 | : Some notification objects have changed; please make sure the versions have been bumped, and then update their hashes here. py27 error: http://logs.openstack.org/35/511835/1/check/gate-nova-python27-ubuntu-xenial/477e1b9/console.html ** Affects: nova Importance: Undecided Assignee: zhangyangyang (zhangyangyang) Status: In Progress ** Changed in: nova Assignee: (unassigned) => zhangyangyang (zhangyangyang) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1723476 Title: py27 job failing on testtools.matchers._impl.MismatchError Status in OpenStack Compute (nova): In Progress Bug description: Related patches: https://review.openstack.org/#/c/511835/ https://review.openstack.org/#/c/511767/ when I remove some useless methods in nova/objects/compute_node.py, I find the error: Failed 2 tests - output below: 2017-10-13 13:07:54.110687 | == 2017-10-13 13:07:54.110704 | 2017-10-13 13:07:54.110758 | nova.tests.unit.notifications.objects.test_notification.TestNotificationObjectVersions.test_versions 2017-10-13 13:07:54.110820 | 2017-10-13 13:07:54.110838 | 2017-10-13 13:07:54.110850 | Captured traceback: 2017-10-13 13:07:54.110861 | ~~~ 2017-10-13 13:07:54.110879 | Traceback (most recent call last): 2017-10-13 13:07:54.110911 | File "nova/tests/unit/notifications/objects/test_notification.py", line 420, in test_versions 2017-10-13 13:07:54.110933 | 'Some notification objects have changed; please make ' 2017-10-13 13:07:54.110992 | File "/home/jenkins/workspace/gate-nova-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual 2017-10-13 13:07:54.111018 | self.assertThat(observed, matcher, message) 2017-10-13 13:07:54.111062 | File "/home/jenkins/workspace/gate-nova-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat 2017-10-13 13:07:54.111075 | raise mismatch_error 2017-10-13 13:07:54.111093 | testtools.matchers._impl.MismatchError: !=: 2017-10-13 13:07:54.17 | reference = {'ComputeNodeList': '1.17-52f3b0962b1c86b98590144463ebb192'} 2017-10-13 13:07:54.41 | actual= {'ComputeNodeList': '1.17-badecf5a910a5e0df8fffb270f30c7da'} 2017-10-13 13:07:54.78 | : Some notification objects have changed; please make sure the versions have been bumped, and then update their hashes here. py27 error: http://logs.openstack.org/35/511835/1/check/gate-nova-python27-ubuntu-xenial/477e1b9/console.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1723476/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe :
[Yahoo-eng-team] [Bug 1723472] [NEW] [DVR] Lowering the MTU breaks FIP traffic
Public bug reported: In a DVR environment, when lowering the MTU of a network, traffic going to an instance through a floating IP is broken. Description: * Ping traffic to a VM through its FIP works. * Change the MTU of its network through "neutron net-update --mtu 1440". * Ping to the same FIP doesn't work. After a long debugging session with Anil Venkata, we've found that packets reach br-ex and then they hit this OF rule with normal action: cookie=0x1f847e4bf0de0aea, duration=70306.532s, table=3, n_packets=1579251, n_bytes=796614220, idle_age=0, hard_age=65534, priority=1 actions=NORMAL We would expect this rule to switch the packet to br-int so that it can be forwarded to the fip namespace (ie. with dst MAC address set to the floating ip gw (owner=network:floatingip_agent_gateway): $ sudo ovs-vsctl list interface _uuid : 1f2b6e86-d303-42f4-9467-5dab78fc7199 admin_state : down bfd : {} bfd_status : {} cfm_fault : [] cfm_fault_status: [] cfm_flap_count : [] cfm_health : [] cfm_mpid: [] cfm_remote_mpids: [] cfm_remote_opstate : [] duplex : [] error : [] external_ids: {attached-mac="fa:16:3e:9d:0c:4f", iface-id="8ec34826-b1a6-48ce-9c39-2fd3e8167eb4", iface-status=active} name: "fg-8ec34826-b1" [heat-admin@overcloud-novacompute-0 ~]$ sudo ovs-appctl fdb/show br-ex port VLAN MACAge [...] 710 fa:16:3e:9d:0c:4f0 $ sudo ovs-ofctl show br-ex | grep "7(" 7(phy-br-ex): addr:36:63:93:fc:af:e2 And from there, to the fip namespace which would route the packet to the qrouter namespace, etc. However, when we change the MTU through the following command: "neutron net-update --mtu 1440" We see that, after a few seconds, the MAC address of the FIP changes so when traffic arrives br-ex and NORMAL action is performed, it will not be output to br-int through the patch-port but instead, through eth1 and traffic won't work anymore. [heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113" 10.0.0.113 ether fa:16:3e:9d:0c:4f C vlan10 neutron port-set x --mtu 1440 $ arp -n | grep ".113" 10.0.0.113 ether fa:16:3e:20:f9:85 C vlan10 When setting the MAC address manually, ping starts working again: $ arp -s 10.0.0.113 fa:16:3e:9d:0c:4f $ ping 10.0.0.113 PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data. 64 bytes from 10.0.0.113: icmp_seq=1 ttl=62 time=1.17 ms 64 bytes from 10.0.0.113: icmp_seq=2 ttl=62 time=0.561 ms Additional notes: When we set the MAC address manually and traffic gets working back again, lowering the MTU doesn't change the MAC address (we can't see any gARP's coming through). When we delete the ARP entry for the FIP and try to ping the FIP, the wrong MAC address is set. [heat-admin@overcloud-novacompute-0 ~]$ sudo arp -d 10.0.0.113 [heat-admin@overcloud-novacompute-0 ~]$ ping 10.0.0.113 -c 2 PING 10.0.0.113 (10.0.0.113) 56(84) bytes of data. --- 10.0.0.113 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 999ms [heat-admin@overcloud-novacompute-0 ~]$ arp -n | grep ".113" 10.0.0.113 ether fa:16:3e:20:f9:85 C vlan10 ** Affects: neutron Importance: Undecided Assignee: Daniel Alvarez (dalvarezs) Status: New ** Tags: l3-dvr-backlog ** Changed in: neutron Assignee: (unassigned) => Daniel Alvarez (dalvarezs) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1723472 Title: [DVR] Lowering the MTU breaks FIP traffic Status in neutron: New Bug description: In a DVR environment, when lowering the MTU of a network, traffic going to an instance through a floating IP is broken. Description: * Ping traffic to a VM through its FIP works. * Change the MTU of its network through "neutron net-update --mtu 1440". * Ping to the same FIP doesn't work. After a long debugging session with Anil Venkata, we've found that packets reach br-ex and then they hit this OF rule with normal action: cookie=0x1f847e4bf0de0aea, duration=70306.532s, table=3, n_packets=1579251, n_bytes=796614220, idle_age=0, hard_age=65534, priority=1 actions=NORMAL We would expect this rule to switch the packet to br-int so that it can be forwarded to the fip namespace (ie. with dst MAC address set to the floating ip gw (owner=network:floatingip_agent_gateway): $ sudo ovs-vsctl list interface _uuid : 1f2b6e86-d303-42f4-9467-5dab78fc7199 admin_state : down bfd : {} bfd_status : {} cfm_fault : [] cfm_fault_status: [] cfm_flap_count
[Yahoo-eng-team] [Bug 1723452] [NEW] Move placement API to neutron-lib
Public bug reported: Once the current development is done (bug 1578989), the migration of this API [1] to neutron-lib could be very useful for other Neutron related projects. https://github.com/openstack/neutron/blob/stable/pike/neutron/services/segments/placement_client.py ** Affects: neutron Importance: Undecided Assignee: Rodolfo Alonso (rodolfo-alonso-hernandez) Status: New ** Tags: placement ** Changed in: neutron Assignee: (unassigned) => Rodolfo Alonso (rodolfo-alonso-hernandez) ** Tags added: placement -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1723452 Title: Move placement API to neutron-lib Status in neutron: New Bug description: Once the current development is done (bug 1578989), the migration of this API [1] to neutron-lib could be very useful for other Neutron related projects. https://github.com/openstack/neutron/blob/stable/pike/neutron/services/segments/placement_client.py To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1723452/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1723431] [NEW] Hyper-V: device metadata not updated
Public bug reported: The Hyper-V driver does not update the instance device metadata when adding/detaching volumes or network interfaces to already existing instances. ** Affects: compute-hyperv Importance: Undecided Status: New ** Affects: nova Importance: Undecided Status: New ** Tags: hyper-v ** Also affects: nova Importance: Undecided Status: New ** Tags removed: drivers ** Description changed: The Hyper-V driver does not update the instance device metadata when - adding/detaching volumes or network interfaces. + adding/detaching volumes or network interfaces to already existing + instances. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1723431 Title: Hyper-V: device metadata not updated Status in compute-hyperv: New Status in OpenStack Compute (nova): New Bug description: The Hyper-V driver does not update the instance device metadata when adding/detaching volumes or network interfaces to already existing instances. To manage notifications about this bug go to: https://bugs.launchpad.net/compute-hyperv/+bug/1723431/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1723429] [NEW] Mitaka Series Release Notes in Neutron Release Notes missing notes for versions 8.3 & 8.4
Public bug reported: This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [x] This doc is inaccurate in this way: changes for versions 8.0.0, 8.1.0, 8.2.0 & 8.2.0-42 are shown but changes for versions 8.3.0 & 8.4.0 are not, while they were a couple of days ago! - [ ] This is a doc addition request. - [ ] I have a fix to the document that I can paste below including example: input and output. If you have a troubleshooting or support issue, use the following resources: - Ask OpenStack: http://ask.openstack.org - The mailing list: http://lists.openstack.org - IRC: 'openstack' channel on Freenode --- Release: 11.0.0.0rc2.dev354 on 2017-10-13 00:36 SHA: 9a7c5a1ff667a0649c81b41ef56cc1fd8d1e947b Source: https://git.openstack.org/cgit/openstack/neutron/tree/doc/source/mitaka.rst URL: https://docs.openstack.org/releasenotes/neutron/mitaka.html ** Affects: neutron Importance: Undecided Status: New ** Tags: doc -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1723429 Title: Mitaka Series Release Notes in Neutron Release Notes missing notes for versions 8.3 & 8.4 Status in neutron: New Bug description: This bug tracker is for errors with the documentation, use the following as a template and remove or add fields as you see fit. Convert [ ] into [x] to check boxes: - [x] This doc is inaccurate in this way: changes for versions 8.0.0, 8.1.0, 8.2.0 & 8.2.0-42 are shown but changes for versions 8.3.0 & 8.4.0 are not, while they were a couple of days ago! - [ ] This is a doc addition request. - [ ] I have a fix to the document that I can paste below including example: input and output. If you have a troubleshooting or support issue, use the following resources: - Ask OpenStack: http://ask.openstack.org - The mailing list: http://lists.openstack.org - IRC: 'openstack' channel on Freenode --- Release: 11.0.0.0rc2.dev354 on 2017-10-13 00:36 SHA: 9a7c5a1ff667a0649c81b41ef56cc1fd8d1e947b Source: https://git.openstack.org/cgit/openstack/neutron/tree/doc/source/mitaka.rst URL: https://docs.openstack.org/releasenotes/neutron/mitaka.html To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1723429/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1685333] Re: Fatal Python error: Cannot recover from stack overflow. - in py35 unit test job
** Also affects: nova/pike Importance: Undecided Status: New ** Changed in: nova/pike Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1685333 Title: Fatal Python error: Cannot recover from stack overflow. - in py35 unit test job Status in OpenStack Compute (nova): Confirmed Status in OpenStack Compute (nova) pike series: Confirmed Bug description: Seeing this in the py35 job, looks like it's related to an infinite recursion in oslo.config: Fatal Python error: Cannot recover from stack overflow. http://logs.openstack.org/34/458834/2/check/gate-nova- python35/c55b003/console.html#_2017-04-21_16_36_11_981505 I'm not entirely sure which test it is, but I suspect this one which is still in progress when the job dies: {0} nova.tests.unit.test_rpc.TestRPC.test_add_extra_exmods [] ... inprogress To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1685333/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1680616] Re: instance_get_all_by_host joins tables even if told not to
** Also affects: nova/ocata Importance: Undecided Status: New ** Changed in: nova/ocata Status: New => In Progress ** Changed in: nova/ocata Importance: Undecided => Medium ** Changed in: nova/ocata Assignee: (unassigned) => Matt Riedemann (mriedem) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1680616 Title: instance_get_all_by_host joins tables even if told not to Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: In Progress Bug description: The instance_get_all_by_host DB API will join the info_cache and security_groups tables even if told not to by passing in columns_to_join=[], which the _sync_instance_scheduler_info periodic task from the compute manager does. That is because instance_get_all_by_host doesn't pass columns_to_join through to _instance_get_all_query which will default to join on info_cache and security_groups: https://github.com/openstack/nova/blob/6103ec7c113121866344cdca2fbbbf7b80dfa975/nova/db/sqlalchemy/api.py#L2530 https://github.com/openstack/nova/blob/6103ec7c113121866344cdca2fbbbf7b80dfa975/nova/db/sqlalchemy/api.py#L2513 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1680616/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1723423] [NEW] Ironic node cannot be used if it does not report VCPU
Public bug reported: Even though reporting VCPU is optional, starting with Pike, due to code at https://github.com/openstack/nova/blob/10661dc5e2e3f2e2f37b6a86ea22f71b4b833fc9/nova/virt/ironic/driver.py#L762 nodes without VCPU are not recognized. This has to be changes to properly call _node_resources_unavailable instead. ** Affects: nova Importance: Undecided Assignee: Dmitry Tantsur (divius) Status: New ** Tags: ironic ** Changed in: nova Assignee: (unassigned) => Dmitry Tantsur (divius) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1723423 Title: Ironic node cannot be used if it does not report VCPU Status in OpenStack Compute (nova): New Bug description: Even though reporting VCPU is optional, starting with Pike, due to code at https://github.com/openstack/nova/blob/10661dc5e2e3f2e2f37b6a86ea22f71b4b833fc9/nova/virt/ironic/driver.py#L762 nodes without VCPU are not recognized. This has to be changes to properly call _node_resources_unavailable instead. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1723423/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1715066] Re: Cannot launch instance with Django Launch Instance form with multiple networks for a tenant
Reviewed: https://review.openstack.org/507560 Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=c13d6da80e7a68c28aa0adfe63459be8a6dbedfd Submitter: Jenkins Branch:master commit c13d6da80e7a68c28aa0adfe63459be8a6dbedfd Author: Ivan KolodyazhnyDate: Tue Sep 26 17:26:45 2017 +0300 Add render method to ThemableCheckboxSelectMultiple Django 1.11+ doesn't have RenderMixin class for widgets, so we have to implement render method for ThemableCheckboxSelectMultiple to get it rendered in a correct way with all supported Django versions until we find a better solution for our widgets implementation. Change-Id: I656afb162e130f2b77853368945b74330bedf747 Closes-Bug: #1715066 ** Changed in: horizon Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1715066 Title: Cannot launch instance with Django Launch Instance form with multiple networks for a tenant Status in OpenStack Dashboard (Horizon): Fix Released Bug description: When LAUNCH_INSTANCE_LEGACY_ENABLED is set to True and the Django launch instance form is used, a user cannot launch an instance when there is multiple networks. I selected network(s) and submitted the form, but the launch instance workflow was shown again. The selected networks are reset and all networks are shown in the Available Networks. As a result, users cannot launch instances with the Django Launch Instance form. It seems something has changed in Django 1.11. Previously network ID is shown with a small gray characters but now "(undefined)" is shown, so it looks like generate_html() JS function does not work expectedly. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1715066/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1416000] Re: VMware: write error lost while transferring volume
This has been fixed in Nova as part of the image transfer refactoring long time ago: https://github.com/openstack/nova/commit/2df83abaa0a5c828421fc38602cc1e5145b46ff4 ** Changed in: nova Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1416000 Title: VMware: write error lost while transferring volume Status in OpenStack Compute (nova): Fix Released Status in oslo.vmware: Confirmed Bug description: I'm running the following command: cinder create --image-id a24f216f-9746-418e-97f9-aebd7fa0e25f 1 The write side of the data transfer (a VMwareHTTPWriteFile object) returns an error in write() which I haven't debugged, yet. However, this error is never reported to the user, although it does show up in the logs. The effect is that the transfer sits in the 'downloading' state until the 7200 second timeout, when it reports the timeout. The reason is that the code which waits on transfer completion (in start_transfer) does: try: # Wait on the read and write events to signal their end read_event.wait() write_event.wait() except (timeout.Timeout, Exception) as exc: ... That is, it waits for the read thread to signal completion via read_event before checking write_event. However, because write_thread has died, read_thread is blocking and will never signal completion. You can demonstrate this by swapping the order. If you want for write first it will die immediately, which is what you want. However, that's not right either because now you're missing read errors. Ideally this code needs to be able to notice an error at either end and stop immediately. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1416000/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1723354] [NEW] No validation for DHCP options may lead to potential issues with different dhcp backends
Public bug reported: DHCP option names are not standardized and option names that works with dnsmasq doesn't work with Contrail. By trying to switch to dhcp option values we found that it breaks dnsmasq case. As dnsmasq always set `siaddr` field in dhcp reply to dnsmasq host ip unless `server-ip-address` is not configured. This field is treated by pxe client as tftp server more information about this may be found here https://tools.ietf.org/html/rfc5859. We left this dnsmasq internal option `server-ip-address`. All dhcp providers we tried contrail/dnsmasq/isc just silently ignore unknown options. But still we concern that it may blow up with others. This is not an actual bug, it is only our concern. And we wondered that it maybe standardized on Neutron side somehow or at least basic validation is provided. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1723354 Title: No validation for DHCP options may lead to potential issues with different dhcp backends Status in neutron: New Bug description: DHCP option names are not standardized and option names that works with dnsmasq doesn't work with Contrail. By trying to switch to dhcp option values we found that it breaks dnsmasq case. As dnsmasq always set `siaddr` field in dhcp reply to dnsmasq host ip unless `server-ip-address` is not configured. This field is treated by pxe client as tftp server more information about this may be found here https://tools.ietf.org/html/rfc5859. We left this dnsmasq internal option `server-ip-address`. All dhcp providers we tried contrail/dnsmasq/isc just silently ignore unknown options. But still we concern that it may blow up with others. This is not an actual bug, it is only our concern. And we wondered that it maybe standardized on Neutron side somehow or at least basic validation is provided. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1723354/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1516936] Re: [VMware]: migrate/resize failed in same host with vmware driver
I am not able to reproduce this problem running latest Nova and Neutron with DVS plugin. I think this was a problem when using nova-network which is no longer supported, so I will resolve the bug as "Invalid". Feel free to reopen if you manage to reproduce. ** Changed in: nova Status: Confirmed => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1516936 Title: [VMware]: migrate/resize failed in same host with vmware driver Status in OpenStack Compute (nova): Invalid Bug description: Same to bug 1161226, this issue exist for vmware driver , when resizing a vmware instance with option "allow_resize_to_same_host = True" in nova.conf, the resize operation failed due to report using duplicated mac address . Cold migrate also failed .This only occurred resize/migrate in same host, not between two hosts . To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1516936/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp