[Yahoo-eng-team] [Bug 1784155] Re: nova_placement service start not coordinated with api db sync on multiple controllers
** Changed in: nova/rocky
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1784155
Title:
nova_placement service start not coordinated with api db sync on
multiple controllers
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) rocky series:
Fix Released
Status in tripleo:
Fix Released
Bug description:
On a loaded HA / galera environment using VMs I can fairly
consistently reproduce a race condition where the nova_placement
service is started on controllers where the database is not yet
available. The nova_placement service itself does not seem to be
able to tolerate this condition upon startup and it then fails to
recover. Mitigation here can either involve synchronizing these
conditions or getting nova-placement to be more resilient.
The symptoms of overcloud deploy failure look like two out of three
controllers having the nova_placement container in an unhealthy state:
TASK [Debug output for task which failed: Check for unhealthy containers
after step 3] ***
Saturday 28 July 2018 10:19:29 + (0:00:00.663) 0:30:26.152
*
fatal: [stack2-overcloud-controller-2]: FAILED! => {
"failed_when_result": true,
"outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [
"3597b92e9714
192.168.25.1:8787/tripleomaster/centos-binary-nova-placement-api:959e1d7f755ee681b6f23b498d262a9e4dd6326f_4cbb1814
\"kolla_start\" 2 minutes ago Up 2 minutes (unhealthy)
nova_placement"
]
}
fatal: [stack2-overcloud-controller-1]: FAILED! => {
"failed_when_result": true,
"outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [
"322c5ea53895
192.168.25.1:8787/tripleomaster/centos-binary-nova-placement-api:959e1d7f755ee681b6f23b498d262a9e4dd6326f_4cbb1814
\"kolla_start\" 2 minutes ago Up 2 minutes (unhealthy)
nova_placement"
]
}
ok: [stack2-overcloud-controller-0] => {
"failed_when_result": false,
"outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": []
}
ok: [stack2-overcloud-compute-0] => {
"failed_when_result": false,
"outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": []
}
NO MORE HOSTS LEFT
*
inspecting placement_wsgi_error.log shows the first stack trace that the
nova_placement database is missing the "traits" table:
[Sat Jul 28 10:17:06.525018 2018] [:error] [pid 14] [remote 10.1.20.15:0]
mod_wsgi (pid=14): Target WSGI script
'/var/www/cgi-bin/nova/nova-placement-api' cannot be loaded as Python module.
[Sat Jul 28 10:17:06.525067 2018] [:error] [pid 14] [remote 10.1.20.15:0]
mod_wsgi (pid=14): Exception occurred processing WSGI script
'/var/www/cgi-bin/nova/nova-placement-api'.
[Sat Jul 28 10:17:06.525101 2018] [:error] [pid 14] [remote 10.1.20.15:0]
Traceback (most recent call last):
[Sat Jul 28 10:17:06.525124 2018] [:error] [pid 14] [remote 10.1.20.15:0]
File "/var/www/cgi-bin/nova/nova-placement-api", line 54, in
[Sat Jul 28 10:17:06.525165 2018] [:error] [pid 14] [remote 10.1.20.15:0]
application = init_application()
[Sat Jul 28 10:17:06.525174 2018] [:error] [pid 14] [remote 10.1.20.15:0]
File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi.py",
line 88, in init_application
[Sat Jul 28 10:17:06.525198 2018] [:error] [pid 14] [remote 10.1.20.15:0]
return deploy.loadapp(conf.CONF)
[Sat Jul 28 10:17:06.525205 2018] [:error] [pid 14] [remote 10.1.20.15:0]
File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/deploy.py",
line 111, in loadapp
[Sat Jul 28 10:17:06.525300 2018] [:error] [pid 14] [remote 10.1.20.15:0]
update_database()
[Sat Jul 28 10:17:06.525310 2018] [:error] [pid 14] [remote 10.1.20.15:0]
File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/deploy.py",
line 92, in update_database
[Sat Jul 28 10:17:06.525329 2018] [:error] [pid 14] [remote 10.1.20.15:0]
resource_provider.ensure_trait_sync(ctx)
[Sat Jul 28 10:17:06.525337 2018] [:error] [pid 14] [remote 10.1.20.15:0]
File
"/usr/lib/python2.7/site-packages/nova/api/openstack/placement/objects/resource_provider.py",
line 146, in ensure_trait_sync
[Sat Jul 28 10:17:06.526277 2018] [:error] [pid 14] [remote 10.1.20.15:0]
_trait_sync(ctx)
...
[Sat Jul 28 10:17:06.531950 2018] [:error] [pid 14] [remote 10.1.20.15:0]
raise errorclass(errno, errval)
[Sat Jul 28 10:17:06.532049 2018] [:error] [pid 14] [remote 10.1.20.15:0]
ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table
'nova_placement.traits' doesn't exist") [SQL:
[Yahoo-eng-team] [Bug 1811644] [NEW] There are multiple spaces in the middle of the name. cannot delete the data.
Public bug reported: On the Volume Type -> Management Specification page, create a key name of ab, then delete it and find that the data cannot be deleted. ** Affects: horizon Importance: Undecided Assignee: pengyuesheng (pengyuesheng) Status: New ** Changed in: horizon Assignee: (unassigned) => pengyuesheng (pengyuesheng) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1811644 Title: There are multiple spaces in the middle of the name. cannot delete the data. Status in OpenStack Dashboard (Horizon): New Bug description: On the Volume Type -> Management Specification page, create a key name of ab, then delete it and find that the data cannot be deleted. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1811644/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1811613] [NEW] OVF Vmware IMC does not render gateway to netplan
Public bug reported: With cloud-init 18.5-1-g5b065316-0ubuntu1 and the following VMware cust.cfg: [NETWORK] NETWORKING = yes BOOTPROTO = dhcp HOSTNAME = test DOMAINNAME = zeit.de [NIC-CONFIG] NICS = NIC1 [NIC1] MACADDR = 00:50:56:9e:ce:5f ONBOOT = yes IPv4_MODE = BACKWARDS_COMPATIBLE BOOTPROTO = static IPADDR = 10.100.4.238 NETMASK = 255.255.0.0 GATEWAY = 10.100.0.3 [DNS] DNSFROMDHCP=no SUFFIX|1 = zeit.de NAMESERVER|1 = 172.20.10.7 NAMESERVER|2 = 172.20.10.2 [DATETIME] TIMEZONE = Europe/Berlin UTC = yes The following output is productd by the netplan renderer: network: version: 2 ethernets: ens192: addresses: - 10.100.4.238/16 match: macaddress: 00:50:56:9e:ce:5f nameservers: addresses: - 172.30.20.105 - 172.20.10.3 search: [] routes: - metric: 1 to: 10.100.0.0/16 via: 10.100.0.3 set-name: ens192 This leads to a missing default route. What I would expect is a 'gateay4' parameter for the interface. It is probably related to https://github.com/cloud-init/cloud- init/commit/cb44ad6f42ac015d7d8eaf2ab0bb5ab125ed04b6 which seems to address a similar problem. ** Affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1811613 Title: OVF Vmware IMC does not render gateway to netplan Status in cloud-init: New Bug description: With cloud-init 18.5-1-g5b065316-0ubuntu1 and the following VMware cust.cfg: [NETWORK] NETWORKING = yes BOOTPROTO = dhcp HOSTNAME = test DOMAINNAME = zeit.de [NIC-CONFIG] NICS = NIC1 [NIC1] MACADDR = 00:50:56:9e:ce:5f ONBOOT = yes IPv4_MODE = BACKWARDS_COMPATIBLE BOOTPROTO = static IPADDR = 10.100.4.238 NETMASK = 255.255.0.0 GATEWAY = 10.100.0.3 [DNS] DNSFROMDHCP=no SUFFIX|1 = zeit.de NAMESERVER|1 = 172.20.10.7 NAMESERVER|2 = 172.20.10.2 [DATETIME] TIMEZONE = Europe/Berlin UTC = yes The following output is productd by the netplan renderer: network: version: 2 ethernets: ens192: addresses: - 10.100.4.238/16 match: macaddress: 00:50:56:9e:ce:5f nameservers: addresses: - 172.30.20.105 - 172.20.10.3 search: [] routes: - metric: 1 to: 10.100.0.0/16 via: 10.100.0.3 set-name: ens192 This leads to a missing default route. What I would expect is a 'gateay4' parameter for the interface. It is probably related to https://github.com/cloud-init/cloud- init/commit/cb44ad6f42ac015d7d8eaf2ab0bb5ab125ed04b6 which seems to address a similar problem. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1811613/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1811605] [NEW] Tokenless authentication is broken
Public bug reported:
When trying to use tokenless authentication, authentication fails with
the following traceback:
http://paste.openstack.org/show/742271/
git bisect shows this is the commit that introduced the bug:
0dc5c4edabd5cb0455ffe1c4f8cf8369f64b2197
Steps to reproduce:
(Can start out with configuring devstack with the tls-proxy service to
have devstack generate a root CA but then you need to remove the default
proxy configuration in /etc/apache2/sites-available/http-services-tls-
proxy.conf it generates)
Configure keystone behind Apache with mod_ssl and the following mod_ssl
options:
SSLEngine On
SSLCertificateFile /opt/stack/data/devstack-cert.pem
SSLCACertificateFile /opt/stack/data/CA/root-ca/cacert.pem
SSLOptions +StdEnvVars
SSLVerifyClient optional
SSLUserName SSL_CLIENT_S_DN_CN
SetEnv REMOTE_DOMAIN openstack
In keystone.conf set up external authentication and tokenless auth:
[tokenless_auth]
trusted_issuer = CN=Root CA,OU=DevStack Certificate Authority,O=OpenStack
[auth]
methods = password,token,external
external = Domain
Create a client certificate with the example user values from the
tokenless auth docs, signed by the root CA:
$ openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key
$ openssl x509 -req -in CSR.csr -CA /opt/stack/data/CA/root-ca/cacert.pem
-CAkey /opt/stack/data/CA/root-ca/private/cacert.key -days 365 -out john.pem
-CAcreateserial
Create the IdP, mapping and protocol:
$ openstack identity provider create
ad9b5af1ba36ffc36e1fbf7af5e83e25aebf66bbfefc12eed0313a875e6f9663
$ openstack mapping create x509map --rules rules.json
$ openstack federation protocol create x509 --mapping x509map
--identity-provider
ad9b5af1ba36ffc36e1fbf7af5e83e25aebf66bbfefc12eed0313a875e6f9663
Create a local user with role assignments:
$ openstack domain create openstack
$ openstack user create john --domain openstack
$ openstack role add --user john --user-domain openstack --project demo member
Get a token for the user:
$ curl -v -k -s -X POST --cert john.pem --key privateKey.key -H "x
-project-name: demo" -H "x-project-domain-id: default"
https://192.168.122.248/identity/v3/auth/tokens -d '{"auth":
{"identity": { "methods": [ "external" ], "external": { "user": {
"name": "john", "domain": { "name": "openstack" } } } } } }' -H
'content-type: application/json'
Try to validate the token with tokenless auth (as in the documented
example):
$ curl -v -k -s -X GET --cert /home/devuser/john.pem --key
/home/devuser/privateKey.key -H "x-project-name: demo" -H "x-project-
domain-id: default" https://192.168.122.248/identity/v3/auth/tokens -H
"x-subject-token: "
** Affects: keystone
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1811605
Title:
Tokenless authentication is broken
Status in OpenStack Identity (keystone):
New
Bug description:
When trying to use tokenless authentication, authentication fails with
the following traceback:
http://paste.openstack.org/show/742271/
git bisect shows this is the commit that introduced the bug:
0dc5c4edabd5cb0455ffe1c4f8cf8369f64b2197
Steps to reproduce:
(Can start out with configuring devstack with the tls-proxy service to
have devstack generate a root CA but then you need to remove the
default proxy configuration in /etc/apache2/sites-available/http-
services-tls-proxy.conf it generates)
Configure keystone behind Apache with mod_ssl and the following
mod_ssl options:
SSLEngine On
SSLCertificateFile /opt/stack/data/devstack-cert.pem
SSLCACertificateFile /opt/stack/data/CA/root-ca/cacert.pem
SSLOptions +StdEnvVars
SSLVerifyClient optional
SSLUserName SSL_CLIENT_S_DN_CN
SetEnv REMOTE_DOMAIN openstack
In keystone.conf set up external authentication and tokenless auth:
[tokenless_auth]
trusted_issuer = CN=Root CA,OU=DevStack Certificate Authority,O=OpenStack
[auth]
methods = password,token,external
external = Domain
Create a client certificate with the example user values from the
tokenless auth docs, signed by the root CA:
$ openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key
$ openssl x509 -req -in CSR.csr -CA /opt/stack/data/CA/root-ca/cacert.pem
-CAkey /opt/stack/data/CA/root-ca/private/cacert.key -days 365 -out john.pem
-CAcreateserial
Create the IdP, mapping and protocol:
$ openstack identity provider create
ad9b5af1ba36ffc36e1fbf7af5e83e25aebf66bbfefc12eed0313a875e6f9663
$ openstack mapping create x509map --rules rules.json
$ openstack federation protocol create x509 --mapping x509map
--identity-provider
ad9b5af1ba36ffc36e1fbf7af5e83e25aebf66bbfefc12eed0313a875e6f9663
Create a local user with role assignments:
$ openstack domain create openstack
$ openstack user
[Yahoo-eng-team] [Bug 1811565] [NEW] Live migrations gives a cryptic message when instance is paused
Public bug reported: root@utu1604template:~# nova list +--+---+++-+---+ | ID | Name | Status | Task State | Power State | Networks | +--+---+++-+---+ | 0a07b744-90ef-4250-adce-cd5adfc2925a | FEDORA-1 | ACTIVE | - | Running | net1=1.0.0.9 | | c1c5dbbc-ba4d-4c43-9fa4-b50284be6b44 | FEDORA-1 | ACTIVE | - | Running | net1=1.0.0.4 | | 05858218-80c3-4f6c-a6c0-e2094faa20a6 | FEDORA-3 | ACTIVE | - | Running | net1=1.0.0.5 | | fddae0ed-fdff-4962-9284-84303ad0cf5c | FEDORA-4 | ACTIVE | - | Running | net1=1.0.0.20 | | d5dcef89-c89d-4e75-b567-ef6ddf0d95e8 | FEDORA-vsan-1 | ACTIVE | - | Running | net1=1.0.0.6 | | c4d9ea4e-7f5d-4ffa-a091-15cf817b6656 | FEDORA-vsan-2 | ACTIVE | - | Running | net1=1.0.0.21 | | 112df993-c9dd-4c6c-b0ae-c20e133bba07 | vm1 | ACTIVE | - | Running | net1=1.0.0.8, 172.24.4.14 | | da3fb215-8665-4761-b197-076c80b7b2bd | vm1 | ACTIVE | - | Running | net1=1.0.0.3, 172.24.4.3 | | dbb473c9-248a-4f5e-ab9b-30c6526f4978 | vm1 | ACTIVE | - | Running | net1=1.0.0.7, 172.24.4.10 | | 3882b496-1433-4cb7-a0c5-151a0c47f8bd | vm2 | ACTIVE | - | Running | net1=1.0.0.12 | +--+---+++-+---+ root@utu1604template:~# nova pause d5dcef89-c89d-4e75-b567-ef6ddf0d95e8 root@utu1604template:~# nova service-list ++--+---+--+-+---++-+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | ++--+---+--+-+---++-+ | 11 | nova-conductor | nova-conductor-f7f8ccbf9-tw48t | internal | enabled | up | 2019-01-08T09:30:36.00 | - | | 14 | nova-consoleauth | nova-consoleauth-6fc4cc74b5-hjlbc | internal | enabled | up | 2019-01-08T09:30:40.00 | - | | 17 | nova-scheduler | nova-scheduler-65d75fb975-8qr25 | internal | enabled | up | 2019-01-08T09:30:39.00 | - | | 20 | nova-compute | nova-compute-04 | az1 | enabled | up | 2019-01-08T09:30:38.00 | - | | 23 | nova-compute | nova-compute-02 | az3 | enabled | up | 2019-01-08T09:30:38.00 | - | | 26 | nova-compute | nova-compute-03 | az2 | enabled | up | 2019-01-08T09:30:40.00 | - | | 29 | nova-compute | nova-compute-01 | az4 | enabled | up | 2019-01-08T09:30:42.00 | - | ++--+---+--+-+---++-+ root@utu1604template:~# nova live-migration d5dcef89-c89d-4e75-b567-ef6ddf0d95e8 nova-compute-02 ERROR (Conflict): Cannot 'os-migrateLive' instance d5dcef89-c89d-4e75-b567-ef6ddf0d95e8 while it is in power_state 7 (HTTP 409) (Request-ID: req-77e0da31-1629-48fb-aaa9-d7a5ff178759) ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1811565 Title: Live migrations gives a cryptic message when instance is paused Status in neutron: New Bug description: root@utu1604template:~# nova list +--+---+++-+---+ | ID | Name | Status | Task State | Power State | Networks | +--+---+++-+---+ | 0a07b744-90ef-4250-adce-cd5adfc2925a | FEDORA-1 | ACTIVE | - | Running | net1=1.0.0.9 | | c1c5dbbc-ba4d-4c43-9fa4-b50284be6b44 | FEDORA-1 | ACTIVE | - | Running | net1=1.0.0.4 | | 05858218-80c3-4f6c-a6c0-e2094faa20a6 | FEDORA-3 | ACTIVE | - | Running | net1=1.0.0.5 | | fddae0ed-fdff-4962-9284-84303ad0cf5c | FEDORA-4 | ACTIVE | - | Running | net1=1.0.0.20 | | d5dcef89-c89d-4e75-b567-ef6ddf0d95e8 | FEDORA-vsan-1 | ACTIVE | - | Running | net1=1.0.0.6 | | c4d9ea4e-7f5d-4ffa-a091-15cf817b6656 | FEDORA-vsan-2 | ACTIVE | - | Running | net1=1.0.0.21 | | 112df993-c9dd-4c6c-b0ae-c20e133bba07 | vm1 | ACTIVE | - | Running | net1=1.0.0.8, 172.24.4.14 | | da3fb215-8665-4761-b197-076c80b7b2bd | vm1 | ACTIVE | - | Running | net1=1.0.0.3, 172.24.4.3 | | dbb473c9-248a-4f5e-ab9b-30c6526f4978 | vm1 | ACTIVE | - | Running | net1=1.0.0.7, 172.24.4.10 | | 3882b496-1433-4cb7-a0c5-151a0c47f8bd | vm2 | ACTIVE | - | Running | net1=1.0.0.12 | +--+---+++-+---+ root@utu1604template:~# nova pause d5dcef89-c89d-4e75-b567-ef6ddf0d95e8 root@utu1604template:~# nova service-list
