[Yahoo-eng-team] [Bug 1836692] [NEW] Useless unit test for pre_live_migration method in compute manager

[Takashi NATSUME]

Public bug reported:

In nova/tests/unit/compute/test_compute.py, the following method does
not check anything.

def test_pre_live_migration_instance_has_no_fixed_ip(self):
# Confirm that no exception is raised if there is no fixed ip on
# pre_live_migration
self.compute.driver.pre_live_migration(
test.MatchType(nova.context.RequestContext),
test.MatchType(objects.Instance),
{'block_device_mapping': []},
mock.ANY, mock.ANY, mock.ANY)

The method was mentioned in https://review.opendev.org/#/c/330297/ .
It was necessary to follow it up.
The follow-up patch was https://review.opendev.org/#/c/340713/, but it has not 
been merged.

** Affects: nova
 Importance: Undecided
 Assignee: Takashi NATSUME (natsume-takashi)
 Status: New


** Tags: testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1836692

Title:
  Useless unit test for pre_live_migration method in compute manager

Status in OpenStack Compute (nova):
  New

Bug description:
  In nova/tests/unit/compute/test_compute.py, the following method does
  not check anything.

  def test_pre_live_migration_instance_has_no_fixed_ip(self):
  # Confirm that no exception is raised if there is no fixed ip on
  # pre_live_migration
  self.compute.driver.pre_live_migration(
  test.MatchType(nova.context.RequestContext),
  test.MatchType(objects.Instance),
  {'block_device_mapping': []},
  mock.ANY, mock.ANY, mock.ANY)

  The method was mentioned in https://review.opendev.org/#/c/330297/ .
  It was necessary to follow it up.
  The follow-up patch was https://review.opendev.org/#/c/340713/, but it has 
not been merged.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1836692/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1829296] Re: keystone-manage fails silently

[Launchpad Bug Tracker]

[Expired for OpenStack Identity (keystone) because there has been no
activity for 60 days.]

** Changed in: keystone
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1829296

Title:
  keystone-manage fails silently

Status in OpenStack Identity (keystone):
  Expired

Bug description:
  When using keystone-manage interactively to install keystone [1], it
  does not provide feedback if something goes awry.

  Steps to recreate
  -

  Enter a bad database connection string. eg. mysql+pymsql://... (note
  the wrong pymsql driver)

  Run `keystone-manage db_sync` to sync the database.

  Expected Result
  ---

  keystone-manage prints an error message to stderr

  Actual Result
  -

  cli returns with no output, so you have to run something like `echo $?` to 
check the exit code,
  and then read the keystone.log file to find out why it failed.

  [1] https://docs.openstack.org/keystone/stein/install/keystone-
  install-rdo.html#install-and-configure-components

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1829296/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1836680] [NEW] attach volume succeeded but device not found on guest machine

[norman shen]

Public bug reported:

sorry post bug at wrong place.

** Affects: neutron
 Importance: Undecided
 Status: Invalid

** Changed in: neutron
   Status: New => Invalid

** Description changed:

- we are using OpenStack Queens: 
- nova-common/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed]
- nova-compute/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed,automatic]
- nova-compute-kvm/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed]
- 
- guest vm uses windows 2012 datacenter edition
- 
- after successfully executing openstack server add volume ${instance_id}
- ${volume_id}, we observe volume status has changed to in-used and
- attachments info are correctly stored in both nova and neutron. But
- device does not show up in guest machine.
- 
- we execute `virsh dumpxml ${instance_id}` but device is not there. We
- then try to edit directly by executing `virsh edit ${instance_id}` and
- we see the device with proper attachments info...
- 
- At last we have to shutdown the vm and boot again to solve the problem.
- 
- 
- command line outputs are put below,
- 
- /var/lib/libvirt/qemu# virsh dumpxml 55 --inactive
- 
- 
-   
-   
-   
- 
- .
- 
- # virsh domblklist 55
- Target Source
- 
- vdavms/xxx
- vdbvms/
- 
- manually attach vdc reports `vdc` in-used
+ sorry post bug at wrong place.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1836680

Title:
  attach volume succeeded but device not found on guest machine

Status in neutron:
  Invalid

Bug description:
  sorry post bug at wrong place.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1836680/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1836681] [NEW] attach volume succeeded but device not found on guest machine

[norman shen]

Public bug reported:

we are using OpenStack Queens: 
nova-common/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed]
nova-compute/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed,automatic]
nova-compute-kvm/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed]

guest vm uses windows 2012 datacenter edition

after successfully executing openstack server add volume ${instance_id}
${volume_id}, we observe volume status has changed to in-used and
attachments info are correctly stored in both nova and neutron. But
device does not show up in guest machine.

we execute `virsh dumpxml ${instance_id}` but device is not there. We
then try to edit directly by executing `virsh edit ${instance_id}` and
we see the device with proper attachments info...

At last we have to shutdown the vm and boot again to solve the problem.


command line outputs are put below,

/var/lib/libvirt/qemu# virsh dumpxml 55 --inactive


  
  
  

.

# virsh domblklist 55
Target Source

vdavms/xxx
vdbvms/

manually attach vdc reports `vdc` in-used

** Affects: nova
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1836681

Title:
  attach volume succeeded but device not found on guest machine

Status in OpenStack Compute (nova):
  New

Bug description:
  we are using OpenStack Queens: 
  nova-common/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed]
  nova-compute/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed,automatic]
  nova-compute-kvm/xenial,now 2:17.0.9-6~u16.01+mcp189 all [installed]

  guest vm uses windows 2012 datacenter edition

  after successfully executing openstack server add volume
  ${instance_id} ${volume_id}, we observe volume status has changed to
  in-used and attachments info are correctly stored in both nova and
  neutron. But device does not show up in guest machine.

  we execute `virsh dumpxml ${instance_id}` but device is not there. We
  then try to edit directly by executing `virsh edit ${instance_id}` and
  we see the device with proper attachments info...

  At last we have to shutdown the vm and boot again to solve the
  problem.

  
  command line outputs are put below,

  /var/lib/libvirt/qemu# virsh dumpxml 55 --inactive
  
  



  
  .

  # virsh domblklist 55
  Target Source
  
  vdavms/xxx
  vdbvms/

  manually attach vdc reports `vdc` in-used

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1836681/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1754104] Re: install guide: keystone_authtoken/auth_url shows incorrect port

[zengjia]

** Also affects: heat
   Importance: Undecided
   Status: New

** Changed in: heat
 Assignee: (unassigned) => zengjia (zengjia87)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1754104

Title:
  install guide: keystone_authtoken/auth_url shows incorrect port

Status in Cinder:
  In Progress
Status in Glance:
  Fix Released
Status in Glance queens series:
  Fix Released
Status in OpenStack Heat:
  New
Status in Manila:
  Fix Released
Status in OpenStack Object Storage (swift):
  Fix Released

Bug description:

  This bug tracker is for errors with the documentation, use the
  following as a template and remove or add fields as you see fit.
  Convert [ ] into [x] to check boxes:

  - [X] This doc is inaccurate in this way: The code shown for the 
keystone_authtoken needs an update regarding the ports for queens. Following 
the guides, keystone only listens on port 5000 instead of 5000 & 35357
  - [ ] This is a doc addition request.
  - [x] I have a fix to the document that I can paste below including example: 
input and output. 
  Input:
  [keystone_authtoken]
  # ...
  auth_uri = http://controller:5000
  auth_url = http://controller:35357
  memcached_servers = controller:11211
  auth_type = password
  project_domain_name = default
  user_domain_name = default
  project_name = service
  username = glance
  password = GLANCE_PASS

  output:
  [keystone_authtoken]
  # ...
  auth_uri = http://controller:5000
  auth_url = http://controller:5000
  memcached_servers = controller:11211
  auth_type = password
  project_domain_name = default
  user_domain_name = default
  project_name = service
  username = glance
  password = GLANCE_PASS

  If you have a troubleshooting or support issue, use the following
  resources:

   - Ask OpenStack: http://ask.openstack.org
   - The mailing list: http://lists.openstack.org
   - IRC: 'openstack' channel on Freenode

  ---
  Release: 16.0.1.dev1 on 'Thu Mar 1 07:26:57 2018, commit 968f4ae'
  SHA: 968f4ae9ce244d9372cb3e8f45acea9d557f317d
  Source: 
https://git.openstack.org/cgit/openstack/glance/tree/doc/source/install/install-ubuntu.rst
  URL: https://docs.openstack.org/glance/queens/install/install-ubuntu.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1754104/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1836671] [NEW] string_concat is removed from django as of 2.1

[Corey Bryant]

Public bug reported:

I just tripped over this in Ubuntu development of Eoan (Train) where we
are moving from python-django 1.11.22 to python-django 2.2.3.

The traceback ends with:

File 
"/usr/lib/python3/dist-packages/openstack_dashboard/dashboards/project/instances/tables.py",
 line 27, in 
  from django.utils.translation import string_concat
ImportError: cannot import name 'string_concat' from 'django.utils.translation' 
(/usr/lib/python3/dist-packages/django/utils/translation/__init__.py)

https://github.com/django/django/blob/master/docs/releases/2.1.txt
states that "django.utils.translation.string_concat() is removed."

** Affects: horizon
 Importance: Undecided
 Status: New

** Affects: horizon (Ubuntu)
 Importance: High
 Status: Triaged

** Description changed:

  I just tripped over this in Ubuntu development of Eoan (Train) where we
  are moving from python-django 1.11.22 to python-django 2.2.3.
  
  The traceback ends with:
  
  File 
"/usr/lib/python3/dist-packages/openstack_dashboard/dashboards/project/instances/tables.py",
 line 27, in 
-   from django.utils.translation import string_concat
+   from django.utils.translation import string_concat
  ImportError: cannot import name 'string_concat' from 
'django.utils.translation' 
(/usr/lib/python3/dist-packages/django/utils/translation/__init__.py)
+ 
+ https://github.com/django/django/blob/master/docs/releases/2.1.txt
+ states that "django.utils.translation.string_concat() is removed."

** Also affects: horizon (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: horizon (Ubuntu)
   Status: New => Triaged

** Changed in: horizon (Ubuntu)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1836671

Title:
  string_concat is removed from django as of 2.1

Status in OpenStack Dashboard (Horizon):
  New
Status in horizon package in Ubuntu:
  Triaged

Bug description:
  I just tripped over this in Ubuntu development of Eoan (Train) where
  we are moving from python-django 1.11.22 to python-django 2.2.3.

  The traceback ends with:

  File 
"/usr/lib/python3/dist-packages/openstack_dashboard/dashboards/project/instances/tables.py",
 line 27, in 
    from django.utils.translation import string_concat
  ImportError: cannot import name 'string_concat' from 
'django.utils.translation' 
(/usr/lib/python3/dist-packages/django/utils/translation/__init__.py)

  https://github.com/django/django/blob/master/docs/releases/2.1.txt
  states that "django.utils.translation.string_concat() is removed."

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1836671/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1836650] Re: Bug when configuring Keystone events format

[Rafael Weingartner]

** Also affects: keystone
   Importance: Undecided
   Status: New

** Changed in: kolla-ansible
 Assignee: (unassigned) => Rafael Weingartner (rafaelweingartner)

** Changed in: keystone
 Assignee: (unassigned) => Rafael Weingartner (rafaelweingartner)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1836650

Title:
  Bug when configuring Keystone events format

Status in OpenStack Identity (keystone):
  New
Status in kolla-ansible:
  New

Bug description:
  By default Kolla-ansible defines 'enable_cadf_notifications' as 'no'.
  This variable is used to enable/disable CADF format in Keystone. The
  default of Keystone is 'CADF' already, but it (Keystone) does not set
  any messaging drivers; as a consequence, the default behavior of
  Keystone is not to send event messages to the queueing system. We were
  led to believe that using 'enable_cadf_notifications' with value 'no'
  would lead Kolla-ansible to configure the 'basic' message format.
  However, that is not what happens.

  Kolla-ansible will configure Keystone without setting the oslo.messaging 
driver as messagingv2 when 'enable_cadf_notifications: no'. This will create a 
configuration that does not publish events in RabbitMQ. A PR was pull request 
(PR) was proposed to fix this misunderstanding in 
https://review.opendev.org/#/c/670626. That PR is introducing a few things: 
  * Moving the definition of 'enable_cadf_notifications' to Keystone role as it 
is only used there
  * Changing the default value defined to 'yes' because that is the default 
behavior in Keystone. Keystone uses CADF format by Default. 
  * Add an else condition in Keystone.conf template. When CADF is not enabled, 
we need to explicitly set the format as 'basic'. Moreover, enabling the  
message driver to allow us to get messages in the queueing system.

  After opening the PR, the fellow Radosław Piliszek questioned the proposed 
changes. More details can be found there 
(https://review.opendev.org/#/c/670626/2), at the PR's comments. In summary, it 
was questioned the use of a parameter in Kolla-ansible to enable/disable a 
feature in Keystone. It is argued that this is not the goal of Kolla-ansible. 
Right now, we have a few options with respect to this issue:
  option 1 -- we can use the PR as is;
  option 2 -- we can remove the "feature" (enable_cadf_notifications) in 
kolla-ansible to configure CADF notification format;
  option 3 -- do nothing (abandon this PR), and leave things as they are.

  The community now has to decide on which path we will follow to
  address this situation. Afterwards, we can move on and propose a PR to
  apply/address the selected option into Kolla-ansible.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1836650/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1832267] Re: add raw format link to keystone config sample.

[Colleen Murphy]

** Changed in: keystone
   Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1832267

Title:
  add raw format link to keystone config sample.

Status in OpenStack Identity (keystone):
  Invalid

Bug description:
  https://review.opendev.org/663063
  Dear bug triager. This bug was created since a commit was marked with 
DOCIMPACT.
  Your project "openstack/keystone" is set up so that we directly report the 
documentation bugs against it. If this needs changing, the docimpact-group 
option needs to be added for the project. You can ask the OpenStack infra team 
(#openstack-infra on freenode) for help if you need to.

  commit 21387e0a668fcdffd05d719f5f0bbff196f947e5
  Author: Dominic Schlegel 
  Date:   Tue Jun 4 17:17:43 2019 +0200

  add raw format link to keystone config sample.
  
  This change adjusts the documentation to include a link to the
  original raw file for better copy purpose.
  
  DocImpact
  
  Change-Id: I85841a7c1c2242fca9849793908f6eb6cd6c89b1

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1832267/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1836642] [NEW] Metadata responses are very slow sometimes

[Slawek Kaplonski]

Public bug reported:

It happens from time to time in CI that VM spawned in test don't get
public-keys from metadata service. Due to that SSH to instance using
ssh-key is not possible thus test fails.

Example of such failures:

http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/testr_results.html.gz
http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/testr_results.html.gz
http://logs.openstack.org/45/645645/8/gate/tempest-full/4d7874e/testr_results.html.gz

In each of those cases it looks that neutron metadata agent was waiting
more than 10 seconds for response from n-api-meta service:

http://logs.openstack.org/09/666409/7/check/tempest-
full/08f4c53/controller/logs/screen-n-api-
meta.txt.gz#_Jul_11_23_43_16_704979 ~ 16 seconds,

http://logs.openstack.org/35/521035/8/check/tempest-
full/031b0b9/controller/logs/screen-n-api-
meta.txt.gz#_Jul_09_17_23_47_871054 ~ 13 seconds,

http://logs.openstack.org/45/645645/8/gate/tempest-
full/4d7874e/controller/logs/screen-n-api-
meta.txt.gz#_Jul_09_01_43_56_549916 ~ 17 seconds.

I have no idea why nova is responding so slow but it would be worth if
someone from Nova team would take a look into that.

Logstash query which can help to find another examples:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%20to%20get%20http%3A%2F%2F169.254.169.254%2F2009-04-04
%2Fmeta-data%2Fpublic-keys%5C%22

It is possible that in some cases of failed tests, reason of failure may
be different but problem described above is quite common in those failed
tests IMO.

** Affects: neutron
 Importance: Undecided
 Status: New

** Affects: nova
 Importance: Undecided
 Status: New

** Also affects: neutron
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1836642

Title:
  Metadata responses are very slow sometimes

Status in neutron:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  It happens from time to time in CI that VM spawned in test don't get
  public-keys from metadata service. Due to that SSH to instance using
  ssh-key is not possible thus test fails.

  Example of such failures:

  
http://logs.openstack.org/09/666409/7/check/tempest-full/08f4c53/testr_results.html.gz
  
http://logs.openstack.org/35/521035/8/check/tempest-full/031b0b9/testr_results.html.gz
  
http://logs.openstack.org/45/645645/8/gate/tempest-full/4d7874e/testr_results.html.gz

  In each of those cases it looks that neutron metadata agent was
  waiting more than 10 seconds for response from n-api-meta service:

  http://logs.openstack.org/09/666409/7/check/tempest-
  full/08f4c53/controller/logs/screen-n-api-
  meta.txt.gz#_Jul_11_23_43_16_704979 ~ 16 seconds,

  http://logs.openstack.org/35/521035/8/check/tempest-
  full/031b0b9/controller/logs/screen-n-api-
  meta.txt.gz#_Jul_09_17_23_47_871054 ~ 13 seconds,

  http://logs.openstack.org/45/645645/8/gate/tempest-
  full/4d7874e/controller/logs/screen-n-api-
  meta.txt.gz#_Jul_09_01_43_56_549916 ~ 17 seconds.

  I have no idea why nova is responding so slow but it would be worth if
  someone from Nova team would take a look into that.

  Logstash query which can help to find another examples:
  
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%20to%20get%20http%3A%2F%2F169.254.169.254%2F2009-04-04
  %2Fmeta-data%2Fpublic-keys%5C%22

  It is possible that in some cases of failed tests, reason of failure
  may be different but problem described above is quite common in those
  failed tests IMO.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1836642/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1834632] Re: apt-key add fails with raw key input

[Dan Watkins]

Hi shine,

Thanks for the additional information.  It looks to me like these keys
aren't valid; both of them are missing a leading "-" on their first
line, which causes gpg to not be able to operate on them.  For example,
comparing the Rabbit key to the one provided upstream:

# diff their.gpg our.gpg 
1c1
< -BEGIN PGP PUBLIC KEY BLOCK-
---
> BEGIN PGP PUBLIC KEY BLOCK-


If changing that doesn't fix the problem, please feel free to let us know and 
set this back to New.


Thanks!

Dan

** Changed in: cloud-init
   Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1834632

Title:
  apt-key add fails with raw key input

Status in cloud-init:
  Invalid

Bug description:
  Upon adding a raw key within cloud-config in apt > sources > 
  > key, the apt-key add operation hits a
  cloudinit.util.ProcessExecutionError.

  Environment Information :
  Operating System : Ubuntu 18.04.2 LTS
  Cloud Init : 19.1-1-gbaa47854-0ubuntu1~18.04.1

  Relevant logs :

  ==> /var/log/cloud-init-output.log <==
  Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:config' at 
Fri, 28 Jun 2019 11:31:03 +. Up 24.14 seconds.
  2019-06-28 11:31:04,390 - cc_apt_configure.py[ERROR]: failed to add apt GPG 
Key to apt keyring
  Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", 
line 551, in add_apt_key_raw
  util.subp(['apt-key', 'add', '-'], data=key.encode(), target=target)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2069, in subp
  cmd=args)
  cloudinit.util.ProcessExecutionError: Unexpected error while running command.
  Command: ['apt-key', 'add', '-']
  Exit code: 2
  Reason: -
  Stdout: 
  Stderr: Warning: apt-key output should not be parsed (stdout is not a 
terminal)
  gpg: no valid OpenPGP data found.

  ==> /var/log/cloud-init.log <==
  2019-06-28 11:31:04,390 - cc_apt_configure.py[ERROR]: failed to add apt GPG 
Key to apt keyring
  Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", 
line 551, in add_apt_key_raw
  util.subp(['apt-key', 'add', '-'], data=key.encode(), target=target)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2069, in subp
  cmd=args)
  cloudinit.util.ProcessExecutionError: Unexpected error while running command.
  Command: ['apt-key', 'add', '-']
  Exit code: 2
  Reason: -
  Stdout: 
  Stderr: Warning: apt-key output should not be parsed (stdout is not a 
terminal)
  gpg: no valid OpenPGP data found.
  2019-06-28 11:31:04,401 - handlers.py[DEBUG]: finish: 
modules-config/config-apt-configure: FAIL: running config-apt-configure with 
frequency once-per-instance

  ==> /var/log/cloud-init-output.log <==
  2019-06-28 11:31:04,401 - util.py[WARNING]: Running module apt-configure 
() failed

  ==> /var/log/cloud-init.log <==
  2019-06-28 11:31:04,401 - util.py[WARNING]: Running module apt-configure 
() failed
  2019-06-28 11:31:04,402 - util.py[DEBUG]: Running module apt-configure 
() failed
  Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 812, in 
_run_modules
  freq=freq)
File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
  return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
  results = functor(*args)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", 
line 283, in handle
  apply_apt(cfg, cloud, target)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", 
line 331, in apply_apt
  template_params=params, aa_repo_match=matcher)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", 
line 600, in add_apt_sources
  add_apt_key(ent, target)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", 
line 571, in add_apt_key
  add_apt_key_raw(ent['key'], target)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", 
line 551, in add_apt_key_raw
  util.subp(['apt-key', 'add', '-'], data=key.encode(), target=target)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2069, in subp
  cmd=args)
  cloudinit.util.ProcessExecutionError: Unexpected error while running command.
  Command: ['apt-key', 'add', '-']
  Exit code: 2
  Reason: -
  Stdout: 
  Stderr: Warning: apt-key output should not be parsed (stdout is not a 
terminal)
  gpg: no valid OpenPGP data found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1834632/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : 

[Yahoo-eng-team] [Bug 1836095] Re: Improve "OVSFirewallDriver.process_trusted_ports"

[OpenStack Infra]

Reviewed:  https://review.opendev.org/670162
Committed: 
https://git.openstack.org/cgit/openstack/neutron/commit/?id=ae1d36fa9d8e2115a5241b5da2e941cdefa2c463
Submitter: Zuul
Branch:master

commit ae1d36fa9d8e2115a5241b5da2e941cdefa2c463
Author: Rodolfo Alonso Hernandez 
Date:   Wed Jul 10 18:57:02 2019 +

Improve "OVSFirewallDriver.process_trusted_ports"

FirewallDriver.process_trusted_ports" is called with many ports,
"_initialize_egress_no_port_security" retrieves the VIF ports
("Interface" registers in OVS DB), one per iteration, based in the
port_id. Instead of this procedure, if the DB is called only once to
retrieve all the VIF ports, the performance increase is noticeable.
E.g.: bridge with 1000 ports and interfaces.

Retrieving 100 ports:
- Bulk operation: 0.08 secs
- Loop operation: 5.6 secs

Retrieving 1000 ports:
- Bulk operation: 0.08 secs
- Loop operation: 59 secs

Closes-Bug: #1836095
Related-Bug: #1836023

Change-Id: I5b259717c0fdb8991f1df86b1ef4fb8ad0f18e70


** Changed in: neutron
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1836095

Title:
  Improve "OVSFirewallDriver.process_trusted_ports"

Status in neutron:
  Fix Released

Bug description:
  When "OVSFirewallDriver.process_trusted_ports" is called with many
  ports, "_initialize_egress_no_port_security" retrieves the VIF ports
  ("Interface" registers in OVS DB), one per iteration, based in the
  port_id. Instead of this procedure, if the DB is called only once to
  retrieve all the VIF ports, the performance increase is noticeable.
  E.g.: bridge with 1000 ports and interfaces.

  port_ids = ['id%s' % i for i in range(1, 1000)]
  ts1 = timeutils.utcnow_ts(microsecond=True)
  vifs = ovs.get_vifs_by_ids(port_ids)
  ts2 = timeutils.utcnow_ts(microsecond=True)
  print("Time lapsed: %s" % str(ts2 - ts1))

  ts1 = timeutils.utcnow_ts(microsecond=True)
  for i in range(1, 1000):
  id = "id%s" % i
  vif = ovs.get_vif_port_by_id(id)
  ts2 = timeutils.utcnow_ts(microsecond=True)
  print("Time lapsed: %s" % str(ts2 - ts1))

  Retrieving 100 ports:
  - Bulk operation: 0.08 secs
  - Loop operation: 5.6 secs

  Retrieving 300 ports:
  - Bulk operation: 0.08 secs
  - Loop operation: 16.44 secs

  Retrieving 1000 ports:
  - Bulk operation: 0.08 secs
  - Loop operation: 59 secs

  
[1]https://github.com/openstack/neutron/blob/06754907e241af76570f19301093c2abab97e627/neutron/agent/linux/openvswitch_firewall/firewall.py#L667
  
[2]https://github.com/openstack/neutron/blob/06754907e241af76570f19301093c2abab97e627/neutron/agent/linux/openvswitch_firewall/firewall.py#L747

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1836095/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1836598] [NEW] cc_ntp generates wrong config for ntpd in debian (buster)

[Dmitry Shkuratov]

Public bug reported:

when i use the following configuration:

ntp:
  enabled: true
  ntp_client: ntp
  pools: [0.int.pool.ntp.org, 1.int.pool.ntp.org, ntp.myorg.org]
  servers:
- ntp.server.local
- ntp.ubuntu.com
- 192.168.23.2

wrong configuration pools received,
cat /etc/ntpd.conf

...skip...
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: 
# poolspool 0.int.pool.ntp.org iburst
pool 1.int.pool.ntp.org iburst
pool ntp.myorg.org iburst
# servers
server ntp.server.local iburst
server ntp.ubuntu.com iburst
server 192.168.23.2 iburst

# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
...skip...

wrong line >> # poolspool 0.int.pool.ntp.org iburst


system>
Debian GNU/Linux 10 (buster)
cloud-init 18.3-6

** Affects: cloud-init
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1836598

Title:
  cc_ntp generates wrong config for ntpd in debian (buster)

Status in cloud-init:
  New

Bug description:
  when i use the following configuration:

  ntp:
enabled: true
ntp_client: ntp
pools: [0.int.pool.ntp.org, 1.int.pool.ntp.org, ntp.myorg.org]
servers:
  - ntp.server.local
  - ntp.ubuntu.com
  - 192.168.23.2

  wrong configuration pools received,
  cat /etc/ntpd.conf

  ...skip...
  # pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
  # pick a different set every time it starts up.  Please consider joining the
  # pool: 
  # poolspool 0.int.pool.ntp.org iburst
  pool 1.int.pool.ntp.org iburst
  pool ntp.myorg.org iburst
  # servers
  server ntp.server.local iburst
  server ntp.ubuntu.com iburst
  server 192.168.23.2 iburst

  # Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html 
for
  ...skip...

  wrong line >> # poolspool 0.int.pool.ntp.org iburst

  
  system>
  Debian GNU/Linux 10 (buster)
  cloud-init 18.3-6

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1836598/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1722584] Re: [SRU] Return traffic from metadata service may get dropped by hypervisor due to wrong checksum

[Corey Bryant]

This bug was fixed in the package neutron - 2:13.0.3-0ubuntu2~cloud0
---

 neutron (2:13.0.3-0ubuntu2~cloud0) bionic-rocky; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 neutron (2:13.0.3-0ubuntu2) cosmic; urgency=medium
 .
   * d/p/revert-iptables-tcp-checksum-fill-code.patch: Cherry-picked
 from upstream to revert invalid use of iptables -j CHECKSUM
 (LP: #1722584).
 .
 neutron (2:13.0.3-0ubuntu1) cosmic; urgency=medium
 .
   * New stable point release for OpenStack Rocky (LP: #1830695).
   * d/p/Spawn-metadata-proxy-on-dvr-ha-standby-routers.patch:
 Dropped. Fixed upstream in 13.0.3.
   * d/p/bug1823038.patch: Dropped. Fixed upstream in 13.0.3.
   * d/p/fix-KeyError-in-OVS-firewall.patch: Dropped. Fixed upstream in 13.0.3.
   * d/p/set-initial-ha-router-state-in-neutron-keepalived-st.patch:
 Dropped. Fixed upstream in 13.0.3.
   * d/p/bug1826419.patch: Rebased to the last version published upstream.


** Changed in: cloud-archive/rocky
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1722584

Title:
  [SRU] Return traffic from metadata service may get dropped by
  hypervisor due to wrong checksum

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Fix Released
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Released
Status in neutron source package in Cosmic:
  Fix Released
Status in neutron source package in Disco:
  Fix Released
Status in neutron source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  Prior addition of code to add checksum rules was found to cause problems with 
newer kernels. Patch subsequently reverted so this request is to backport those 
patches to the ubuntu archives.

  [Test Case]
  * deploy openstack (>= queens)
  * create router/network/instance (dvr=false,l3ha=false)
  * go to router ns on neutron-gateway and check that the following returns 
nothing
  sudo ip netns exec qrouter- iptables -t mangle -S| grep '\--sport 9697 -j 
CHECKSUM --checksum-fill'

  [Regression Potential]
  Backporting the revert patch will mean that routers created with this patch 
will no longer have a checksum rule added for metadata tcp packets. The 
original patch added a rule that turned out not to be the fix for the root 
issue and was subsequently found to cause problems with kernels < 4.19 since it 
was never intended for gso tcp packets to have their checksum verified using 
this type of rule. So, removal of this rule (by addition of the revert patch) 
is not intended to change behaviour at all. The only potential side-effect is 
that rules that were already created will not be cleaned up (until node reboot 
or router recreate) and in an L3HA config you could end up with some router 
instances having the rule and some not depending on whether they were created 
before or after the patch was included.

  [Other Info]
  This revert patch does not remove rules added by the original patch so manual 
cleanup of those old rules is required.

  -
  We have a problem with the metadata service not being responsive, when the 
proxied in the router namespace on some of our networking nodes after upgrading 
to Ocata (Running on CentOS 7.4, with the RDO packages).

  Instance routes traffic to 169.254.169.254 to it's default gateway.
  Default gateway is an OpenStack router in a namespace on a networking node.

  - Traffic gets sent from the guest,
  - to the router,
  - iptables routes it to the metadata proxy service,
  - response packet gets routed back, leaving the namespace
  - Hypervisor gets the packet in
  - Checksum of packet is wrong, and the packet gets dropped before putting it 
on the bridge

  Based on the following bug https://bugs.launchpad.net/openstack-
  ansible/+bug/1483603, we found that adding the following iptable rule
  in the router namespace made this work again: 'iptables -t mangle -I
  POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill'

  (NOTE: The rule from the 1st comment to the bug did solve access to
  the metadata service, but the lack of precision introduced other
  problems with the network)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1722584/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1722584] Re: [SRU] Return traffic from metadata service may get dropped by hypervisor due to wrong checksum

[Corey Bryant]

This bug was fixed in the package neutron - 2:12.0.6-0ubuntu2~cloud0
---

 neutron (2:12.0.6-0ubuntu2~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:12.0.6-0ubuntu2) bionic; urgency=medium
 .
   * d/p/revert-iptables-tcp-checksum-fill-code.patch: Cherry-picked
 from upstream to revert invalid use of iptables -j CHECKSUM
 (LP: #1722584).


** Changed in: cloud-archive/queens
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1722584

Title:
  [SRU] Return traffic from metadata service may get dropped by
  hypervisor due to wrong checksum

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Fix Released
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Released
Status in neutron source package in Cosmic:
  Fix Released
Status in neutron source package in Disco:
  Fix Released
Status in neutron source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  Prior addition of code to add checksum rules was found to cause problems with 
newer kernels. Patch subsequently reverted so this request is to backport those 
patches to the ubuntu archives.

  [Test Case]
  * deploy openstack (>= queens)
  * create router/network/instance (dvr=false,l3ha=false)
  * go to router ns on neutron-gateway and check that the following returns 
nothing
  sudo ip netns exec qrouter- iptables -t mangle -S| grep '\--sport 9697 -j 
CHECKSUM --checksum-fill'

  [Regression Potential]
  Backporting the revert patch will mean that routers created with this patch 
will no longer have a checksum rule added for metadata tcp packets. The 
original patch added a rule that turned out not to be the fix for the root 
issue and was subsequently found to cause problems with kernels < 4.19 since it 
was never intended for gso tcp packets to have their checksum verified using 
this type of rule. So, removal of this rule (by addition of the revert patch) 
is not intended to change behaviour at all. The only potential side-effect is 
that rules that were already created will not be cleaned up (until node reboot 
or router recreate) and in an L3HA config you could end up with some router 
instances having the rule and some not depending on whether they were created 
before or after the patch was included.

  [Other Info]
  This revert patch does not remove rules added by the original patch so manual 
cleanup of those old rules is required.

  -
  We have a problem with the metadata service not being responsive, when the 
proxied in the router namespace on some of our networking nodes after upgrading 
to Ocata (Running on CentOS 7.4, with the RDO packages).

  Instance routes traffic to 169.254.169.254 to it's default gateway.
  Default gateway is an OpenStack router in a namespace on a networking node.

  - Traffic gets sent from the guest,
  - to the router,
  - iptables routes it to the metadata proxy service,
  - response packet gets routed back, leaving the namespace
  - Hypervisor gets the packet in
  - Checksum of packet is wrong, and the packet gets dropped before putting it 
on the bridge

  Based on the following bug https://bugs.launchpad.net/openstack-
  ansible/+bug/1483603, we found that adding the following iptable rule
  in the router namespace made this work again: 'iptables -t mangle -I
  POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill'

  (NOTE: The rule from the 1st comment to the bug did solve access to
  the metadata service, but the lack of precision introduced other
  problems with the network)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1722584/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1836595] [NEW] test_server_connectivity_cold_migration_revert failing

[Artom Lifshitz]

Public bug reported:

test_server_connectivity_cold_migration_revert has started failing now
that we've un-skipped it [1]. It appears as though Nova is doing
everything right in terms of external events and VIF plugging (see
analysis on PS6 of [2]), so the thinking is that it's something with the
Neutron L3 agent. Sean Mooney's noticed that the guest does get an IP
from DHCP, but then we fail to ping the VM from tempest.

Logstash query:
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert%5C%22%20AND%20message:%5C%22FAILED%5C%22%20AND%20tags:%5C
%22job-output.txt%5C%22

[1] https://review.opendev.org/#/c/663405/
[2] https://review.opendev.org/#/c/668631/6

** Affects: neutron
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1836595

Title:
  test_server_connectivity_cold_migration_revert failing

Status in neutron:
  New

Bug description:
  test_server_connectivity_cold_migration_revert has started failing now
  that we've un-skipped it [1]. It appears as though Nova is doing
  everything right in terms of external events and VIF plugging (see
  analysis on PS6 of [2]), so the thinking is that it's something with
  the Neutron L3 agent. Sean Mooney's noticed that the guest does get an
  IP from DHCP, but then we fail to ping the VM from tempest.

  Logstash query:
  
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert%5C%22%20AND%20message:%5C%22FAILED%5C%22%20AND%20tags:%5C
  %22job-output.txt%5C%22

  [1] https://review.opendev.org/#/c/663405/
  [2] https://review.opendev.org/#/c/668631/6

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1836595/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1821594] Re: [SRU] Error in confirm_migration leaves stale allocations and 'confirming' migration state

[Edward Hope-Morley]

** Changed in: nova/stein
   Status: Fix Committed => Fix Released

** Changed in: cloud-archive/stein
   Status: Fix Committed => Fix Released

** Changed in: nova (Ubuntu Disco)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1821594

Title:
  [SRU] Error in confirm_migration leaves stale allocations and
  'confirming' migration state

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Fix Committed
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Committed
Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) pike series:
  Triaged
Status in OpenStack Compute (nova) queens series:
  Fix Committed
Status in OpenStack Compute (nova) rocky series:
  Fix Committed
Status in OpenStack Compute (nova) stein series:
  Fix Released
Status in nova package in Ubuntu:
  Fix Committed
Status in nova source package in Bionic:
  Fix Committed
Status in nova source package in Cosmic:
  Fix Released
Status in nova source package in Disco:
  Fix Released
Status in nova source package in Eoan:
  Fix Committed

Bug description:
  Description:

  When performing a cold migration, if an exception is raised by the
  driver during confirm_migration (this runs in the source node), the
  migration record is stuck in "confirming" state and the allocations
  against the source node are not removed.

  The instance is fine at the destination in this stage, but the source
  host has allocations that is not possible to clean without going to
  the database or invoking the Placement API via curl. After several
  migration attempts that fail in the same spot, the source node is
  filled with these allocations that prevent new instances from being
  created or instances migrated to this node.

  When confirm_migration fails in this stage, the migrating instance can
  be saved through a hard reboot or a reset state to active.

  Steps to reproduce:

  Unfortunately, I don't have logs of the real root cause of the problem
  inside driver.confirm_migration running libvirt driver. However, the
  stale allocations and migration status problem can be easily
  reproduced by raising an exception in libvirt driver's
  confirm_migration method, and it would affect any driver.

  Expected results:

  Discussed this issue with efried and mriedem over #openstack-nova on
  March 25th, 2019. They confirmed that allocations not being cleared up
  is a bug.

  Actual results:

  Instance is fine at the destination after a reset-state. Source node
  has stale allocations that prevent new instances from being
  created/migrated to the source node. Migration record is stuck in
  "confirming" state.

  Environment:

  I verified this bug on on pike, queens and stein branches. Running
  libvirt KVM driver.

  ===

  [Impact]

  If users attempting to perform cold migrations face any issues when
  the virt driver is running the "Confirm Migration" step, the failure leaves 
stale allocation records in the database, in migration records in "confirming" 
state. The stale allocations are not cleaned up by nova, consuming the user's 
quota indefinitely.

  This bug was confirmed from pike to stein release, and a fix was
  implemented for queens, rocky and stein. It should be backported to
  those releases to prevent the issue from reoccurring.

  This fix prevents new stale allocations being left over, by cleaning
  them up immediately when the failures occur. At the moment, the users
  affected by this bug have to clean their previous stale allocations
  manually.

  [Test Case]

  1. Reproducing the bug

  1a. Inject failure

  The root cause for this problem may vary for each driver and
  environment, so to reproduce the bug, it is necessary first to inject
  a failure in the driver's confirm_migration method to cause an
  exception to be raised.

  An example when using libvirt is to add a line:

  raise Exception("TEST")

  in
  
https://github.com/openstack/nova/blob/a57b990c6bffa4c7447081b86573972866c696d2/nova/virt/libvirt/driver.py#L9012

  1b. Restart nova-compute service: systemctl restart nova-compute

  1c. Create a VM

  1d. Then, invoke a cold migration: "openstack server migrate {id}"

  1e. Wait for instance status: VERIFY_RESIZE

  1f. Invoke "openstack server resize {id} --confirm"

  1g. Wait for instance status: ERROR

  1h. Check migration stuck in "confirming" status: nova migration-list

  1i. Check allocations, you should see 2 allocations, one with the VM
  ID, the other with the migration uuid

  export ENDPOINT=
  export TOKEN=`openstack token issue| grep ' id '| awk '{print $4}'`
  for id in $(curl 

[Yahoo-eng-team] [Bug 1836568] [NEW] Logis filled with uneccesery policy derecation warning

[Attila Fazekas]

Public bug reported:

My today master version of keystone log is full with:

2019-07-15 10:47:25.316828 As of the Stein release, the domain API now 
understands how to handle
2019-07-15 10:47:25.316831 system-scoped tokens in addition to project-scoped 
tokens, making the API more
2019-07-15 10:47:25.316834 accessible to users without compromising security or 
manageability for
2019-07-15 10:47:25.316837 administrators. The new default policies for this 
API account for these changes
2019-07-15 10:47:25.316840 automatically
2019-07-15 10:47:25.316843 . Either ensure your deployment is ready for the new 
default or copy/paste the deprecated policy into your policy file and maintain 
it manually.
2019-07-15 10:47:25.316846   warnings.warn(deprecated_msg)
2019-07-15 10:47:25.316849 \x1b[00m

2019-07-15 10:47:25.132244 2019-07-15 10:47:25.131 22582 WARNING py.warnings 
[req-0162c9d3-9953-4b2d-9587-6046651033c3 7b0f3387e0f942f3bae75cea0a5766a3 
98500c83d03e4ba38aa27a78675d2b1b - default default] /usr/lo
cal/lib/python3.7/site-packages/oslo_policy/policy.py:695: UserWarning: Policy 
"identity:delete_credential":"rule:admin_required" was deprecated in S in favor 
of "identity:delete_credential":"(role:admin and sys
tem_scope:all) or user_id:%(target.credential.user_id)s". Reason: As of the 
Stein release, the credential API now understands how to handle system-scoped 
tokens in addition to project-scoped tokens, making the A
PI more accessible to users without compromising security or manageability for 
administrators. The new default policies for this API account for these changes 
automatically.. Either ensure your deployment is rea
dy for the new default or copy/paste the deprecated policy into your policy 
file and maintain it manually.
2019-07-15 10:47:25.132262   warnings.warn(deprecated_msg)
2019-07-15 10:47:25.132266 \x1b[00m
2019-07-15 10:47:25.132979 2019-07-15 10:47:25.132 22582 WARNING


This is fresh setup from `master` without any policy configuration, therefore 
keystone defaults itself triggers the warning.

grep -R  'As of the Stein release' keystone-error.log |wc -l
820


Current master is for `T` , there is no point to have 820 warning (first ~ 10 
minute) for using the keystone default.


Please make these warnings less noise .

** Affects: keystone
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1836568

Title:
  Logis  filled with uneccesery policy derecation warning

Status in OpenStack Identity (keystone):
  New

Bug description:
  My today master version of keystone log is full with:

  2019-07-15 10:47:25.316828 As of the Stein release, the domain API now 
understands how to handle
  2019-07-15 10:47:25.316831 system-scoped tokens in addition to project-scoped 
tokens, making the API more
  2019-07-15 10:47:25.316834 accessible to users without compromising security 
or manageability for
  2019-07-15 10:47:25.316837 administrators. The new default policies for this 
API account for these changes
  2019-07-15 10:47:25.316840 automatically
  2019-07-15 10:47:25.316843 . Either ensure your deployment is ready for the 
new default or copy/paste the deprecated policy into your policy file and 
maintain it manually.
  2019-07-15 10:47:25.316846   warnings.warn(deprecated_msg)
  2019-07-15 10:47:25.316849 \x1b[00m

  2019-07-15 10:47:25.132244 2019-07-15 10:47:25.131 22582 WARNING py.warnings 
[req-0162c9d3-9953-4b2d-9587-6046651033c3 7b0f3387e0f942f3bae75cea0a5766a3 
98500c83d03e4ba38aa27a78675d2b1b - default default] /usr/lo
  cal/lib/python3.7/site-packages/oslo_policy/policy.py:695: UserWarning: 
Policy "identity:delete_credential":"rule:admin_required" was deprecated in S 
in favor of "identity:delete_credential":"(role:admin and sys
  tem_scope:all) or user_id:%(target.credential.user_id)s". Reason: As of the 
Stein release, the credential API now understands how to handle system-scoped 
tokens in addition to project-scoped tokens, making the A
  PI more accessible to users without compromising security or manageability 
for administrators. The new default policies for this API account for these 
changes automatically.. Either ensure your deployment is rea
  dy for the new default or copy/paste the deprecated policy into your policy 
file and maintain it manually.
  2019-07-15 10:47:25.132262   warnings.warn(deprecated_msg)
  2019-07-15 10:47:25.132266 \x1b[00m
  2019-07-15 10:47:25.132979 2019-07-15 10:47:25.132 22582 WARNING

  
  This is fresh setup from `master` without any policy configuration, therefore 
keystone defaults itself triggers the warning.

  grep -R  'As of the Stein release' keystone-error.log |wc -l
  820

  
  Current master is for `T` , there is no point to have 820 warning (first ~ 10 
minute) for using the keystone default.

  
  Please make these warnings less noise .

To manage 

[Yahoo-eng-team] [Bug 1836570] [NEW] VNC connections from horizon not working due to certs not set

[Paul Henien]

Public bug reported:

VNC connections from horizon were not working, console-ssl-cert and
console-ssl-key are unset in n-c-c, yet "openstack console url show" is
returning.

Once SSL certificates set up for VNC access the console worked.

Perhaps related to any sort of patching that may have affected nova
providing https urls instead of http urls when that value was not set.
Going forward, the next charms would fix it, but should be aware that
it's a hazard.

** Affects: horizon
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1836570

Title:
  VNC connections from horizon not working due to certs not set

Status in OpenStack Dashboard (Horizon):
  New

Bug description:
  VNC connections from horizon were not working, console-ssl-cert and
  console-ssl-key are unset in n-c-c, yet "openstack console url show"
  is returning.

  Once SSL certificates set up for VNC access the console worked.

  Perhaps related to any sort of patching that may have affected nova
  providing https urls instead of http urls when that value was not set.
  Going forward, the next charms would fix it, but should be aware that
  it's a hazard.

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1836570/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1836565] [NEW] Functional test test_keepalived_state_change_notification may fail do to race condition

[Slawek Kaplonski]

Public bug reported:

Functional test
neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_keepalived_state_change_notification
may fail due to setting initial status of keepalived_state during
starting of IP monitor.

Some time ago patch https://review.opendev.org/#/c/642295/ introduced checking 
initial status of VIP in router's namespace during IPMonitor initialisation 
process.
It works fine but in this specific test, if keepalived will first set router to 
master and than IPMonitor will be spawned, enqueue_state_change() function will 
be called 4 instead of 3 times and test will fail.

Example of failure http://logs.openstack.org/69/668569/3/check/neutron-
functional-python27/9d345e8/testr_results.html.gz

Logstash query:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%2048%2C%20in%20test_keepalived_state_change_notification%5C%22

** Affects: neutron
 Importance: Medium
 Assignee: Slawek Kaplonski (slaweq)
 Status: Confirmed


** Tags: functional-tests gate-failure l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1836565

Title:
  Functional test test_keepalived_state_change_notification may fail do
  to race condition

Status in neutron:
  Confirmed

Bug description:
  Functional test
  
neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_keepalived_state_change_notification
  may fail due to setting initial status of keepalived_state during
  starting of IP monitor.

  Some time ago patch https://review.opendev.org/#/c/642295/ introduced 
checking initial status of VIP in router's namespace during IPMonitor 
initialisation process.
  It works fine but in this specific test, if keepalived will first set router 
to master and than IPMonitor will be spawned, enqueue_state_change() function 
will be called 4 instead of 3 times and test will fail.

  Example of failure http://logs.openstack.org/69/668569/3/check
  /neutron-functional-python27/9d345e8/testr_results.html.gz

  Logstash query:
  
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%2048%2C%20in%20test_keepalived_state_change_notification%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1836565/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1722584] Re: [SRU] Return traffic from metadata service may get dropped by hypervisor due to wrong checksum

[Launchpad Bug Tracker]

This bug was fixed in the package neutron - 2:12.0.6-0ubuntu2

---
neutron (2:12.0.6-0ubuntu2) bionic; urgency=medium

  * d/p/revert-iptables-tcp-checksum-fill-code.patch: Cherry-picked
from upstream to revert invalid use of iptables -j CHECKSUM
(LP: #1722584).

 -- Corey Bryant   Mon, 17 Jun 2019 13:30:49
-0400

** Changed in: neutron (Ubuntu Bionic)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1722584

Title:
  [SRU] Return traffic from metadata service may get dropped by
  hypervisor due to wrong checksum

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Fix Committed
Status in Ubuntu Cloud Archive rocky series:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Released
Status in neutron source package in Cosmic:
  Fix Released
Status in neutron source package in Disco:
  Fix Released
Status in neutron source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  Prior addition of code to add checksum rules was found to cause problems with 
newer kernels. Patch subsequently reverted so this request is to backport those 
patches to the ubuntu archives.

  [Test Case]
  * deploy openstack (>= queens)
  * create router/network/instance (dvr=false,l3ha=false)
  * go to router ns on neutron-gateway and check that the following returns 
nothing
  sudo ip netns exec qrouter- iptables -t mangle -S| grep '\--sport 9697 -j 
CHECKSUM --checksum-fill'

  [Regression Potential]
  Backporting the revert patch will mean that routers created with this patch 
will no longer have a checksum rule added for metadata tcp packets. The 
original patch added a rule that turned out not to be the fix for the root 
issue and was subsequently found to cause problems with kernels < 4.19 since it 
was never intended for gso tcp packets to have their checksum verified using 
this type of rule. So, removal of this rule (by addition of the revert patch) 
is not intended to change behaviour at all. The only potential side-effect is 
that rules that were already created will not be cleaned up (until node reboot 
or router recreate) and in an L3HA config you could end up with some router 
instances having the rule and some not depending on whether they were created 
before or after the patch was included.

  [Other Info]
  This revert patch does not remove rules added by the original patch so manual 
cleanup of those old rules is required.

  -
  We have a problem with the metadata service not being responsive, when the 
proxied in the router namespace on some of our networking nodes after upgrading 
to Ocata (Running on CentOS 7.4, with the RDO packages).

  Instance routes traffic to 169.254.169.254 to it's default gateway.
  Default gateway is an OpenStack router in a namespace on a networking node.

  - Traffic gets sent from the guest,
  - to the router,
  - iptables routes it to the metadata proxy service,
  - response packet gets routed back, leaving the namespace
  - Hypervisor gets the packet in
  - Checksum of packet is wrong, and the packet gets dropped before putting it 
on the bridge

  Based on the following bug https://bugs.launchpad.net/openstack-
  ansible/+bug/1483603, we found that adding the following iptable rule
  in the router namespace made this work again: 'iptables -t mangle -I
  POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill'

  (NOTE: The rule from the 1st comment to the bug did solve access to
  the metadata service, but the lack of precision introduced other
  problems with the network)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1722584/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1722584] Re: [SRU] Return traffic from metadata service may get dropped by hypervisor due to wrong checksum

[Launchpad Bug Tracker]

This bug was fixed in the package neutron - 2:13.0.3-0ubuntu2

---
neutron (2:13.0.3-0ubuntu2) cosmic; urgency=medium

  * d/p/revert-iptables-tcp-checksum-fill-code.patch: Cherry-picked
from upstream to revert invalid use of iptables -j CHECKSUM
(LP: #1722584).

neutron (2:13.0.3-0ubuntu1) cosmic; urgency=medium

  * New stable point release for OpenStack Rocky (LP: #1830695).
  * d/p/Spawn-metadata-proxy-on-dvr-ha-standby-routers.patch:
Dropped. Fixed upstream in 13.0.3.
  * d/p/bug1823038.patch: Dropped. Fixed upstream in 13.0.3.
  * d/p/fix-KeyError-in-OVS-firewall.patch: Dropped. Fixed upstream in 13.0.3.
  * d/p/set-initial-ha-router-state-in-neutron-keepalived-st.patch:
Dropped. Fixed upstream in 13.0.3.
  * d/p/bug1826419.patch: Rebased to the last version published upstream.

 -- Corey Bryant   Mon, 17 Jun 2019 13:23:45
-0400

** Changed in: neutron (Ubuntu Cosmic)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1722584

Title:
  [SRU] Return traffic from metadata service may get dropped by
  hypervisor due to wrong checksum

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive queens series:
  Fix Committed
Status in Ubuntu Cloud Archive rocky series:
  Fix Committed
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Released
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in neutron source package in Bionic:
  Fix Committed
Status in neutron source package in Cosmic:
  Fix Released
Status in neutron source package in Disco:
  Fix Released
Status in neutron source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  Prior addition of code to add checksum rules was found to cause problems with 
newer kernels. Patch subsequently reverted so this request is to backport those 
patches to the ubuntu archives.

  [Test Case]
  * deploy openstack (>= queens)
  * create router/network/instance (dvr=false,l3ha=false)
  * go to router ns on neutron-gateway and check that the following returns 
nothing
  sudo ip netns exec qrouter- iptables -t mangle -S| grep '\--sport 9697 -j 
CHECKSUM --checksum-fill'

  [Regression Potential]
  Backporting the revert patch will mean that routers created with this patch 
will no longer have a checksum rule added for metadata tcp packets. The 
original patch added a rule that turned out not to be the fix for the root 
issue and was subsequently found to cause problems with kernels < 4.19 since it 
was never intended for gso tcp packets to have their checksum verified using 
this type of rule. So, removal of this rule (by addition of the revert patch) 
is not intended to change behaviour at all. The only potential side-effect is 
that rules that were already created will not be cleaned up (until node reboot 
or router recreate) and in an L3HA config you could end up with some router 
instances having the rule and some not depending on whether they were created 
before or after the patch was included.

  [Other Info]
  This revert patch does not remove rules added by the original patch so manual 
cleanup of those old rules is required.

  -
  We have a problem with the metadata service not being responsive, when the 
proxied in the router namespace on some of our networking nodes after upgrading 
to Ocata (Running on CentOS 7.4, with the RDO packages).

  Instance routes traffic to 169.254.169.254 to it's default gateway.
  Default gateway is an OpenStack router in a namespace on a networking node.

  - Traffic gets sent from the guest,
  - to the router,
  - iptables routes it to the metadata proxy service,
  - response packet gets routed back, leaving the namespace
  - Hypervisor gets the packet in
  - Checksum of packet is wrong, and the packet gets dropped before putting it 
on the bridge

  Based on the following bug https://bugs.launchpad.net/openstack-
  ansible/+bug/1483603, we found that adding the following iptable rule
  in the router namespace made this work again: 'iptables -t mangle -I
  POSTROUTING -p tcp --sport 9697 -j CHECKSUM --checksum-fill'

  (NOTE: The rule from the 1st comment to the bug did solve access to
  the metadata service, but the lack of precision introduced other
  problems with the network)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1722584/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp