Reviewed: https://review.opendev.org/681736 Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=a4be0cb9e842e4eca2023189b19113be76f3b4d0 Submitter: Zuul Branch: master
commit a4be0cb9e842e4eca2023189b19113be76f3b4d0 Author: Ralf Haferkamp <rha...@suse.de> Date: Thu Sep 12 14:44:08 2019 +0200 Fix PostgreSQL specifc issue with credentials encoding Decode the encrypted credential value to a string value before handing it over the database, when running under Python 3.x. Otherwise the underlying database driver (e.g. psycopg2) might treat it as binary data. Change-Id: I87425b54f471e66a9ab3974ab46c4b7f3838b962 Closes-Bug: #1833739 ** Changed in: keystone Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1833739 Title: keystone (stein), python3, and postgresql: hex in database Status in OpenStack Identity (keystone): Fix Released Bug description: When using Keystone (Stein) with Python3 and PostgreSQL, and creating new credentials, we end up with hex in the database for the encrypted_blob in the credential. This causes Keystone to start throwing errors, as it cannot decrypt the encrypted_blob. Adding debugging code to various portions of Keystone narrowed down the location of this problem. Relevant information, and examples will follow. Here is the error we were seeing when we'd create a new credential: 2019-06-21 15:18:10.189 21487 ERROR keystone.credential.providers.fernet.core [req- ff10ee5f-c009-487a-8091-90a65b55eb80 STUFF STUFF - STUFF STUFF] Credential could not be decrypted. Please contact the administrator: cryptography.fernet.InvalidToken After doing a lot of tracing, we ended up mutating: https://github.com/openstack/keystone/blob/stable/stein/keystone/credential/backends/sql.py#L41 @sql.handle_conflicts(conflict_type='credential') def create_credential(self, credential_id, credential): testout = ("cred in sql function: ", credential['encrypted_blob']) LOG.error(testout) testout = ("cred in sql function type: ", type(credential['encrypted_blob'])) LOG.error(testout) with sql.session_for_write() as session: ref = CredentialModel.from_dict(credential) testout2 = ("cred in sql function model: ", ref['encrypted_blob']) LOG.error(testout2) testout3 = ("cred in sql function model type: ", type(ref['encrypted_blob'])) LOG.error(testout3) session.add(ref) return ref.to_dict() Here is what the output looks like when generating a credential: 2019-06-21 15:18:10.090 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function: ', b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=') 2019-06-21 15:18:10.090 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function type: ', <class 'bytes'>) 2019-06-21 15:18:10.091 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function model: ', b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=') 2019-06-21 15:18:10.092 21485 ERROR keystone.credential.backends.sql [req-15159d3b-34ed-4520-81ec-55b9bf55531d STUFF STUFF - STUFF STUFF] ('cred in sql function model type: ', <class 'bytes'>) This is what it looks like in the database: -[ RECORD 6 ]--+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id | IDWASHERE user_id | USERIDWASHERE project_id | PROJECTIDWASHERE type | ec2 extra | {} key_hash | KEYHASHWASHERE encrypted_blob | \x67414141414142644450557943363371417034344c7946777a664c7973385743714649373239715935523153723131506435614c4c36693477584b48765f41735845494a366f5253756a4658656a414d3637595134565a41764b6b67706a79716d77305f6a76464d314e464a754e4a52596d4b4f3933664b5a38316b6e6f7a5364317a4a7858383566515a4b50366354434a77465f7838414c4b334d454a754941794b6364384a30446e3570763865497954662d38534c6b642d6b424c59564d64574f577a78637831537843705f5059523847416930666f4177454c754c6a66786169666a774c4d5947553355696f55676c7a714652733d If you take that encrypted_blob, and decode it: >>> import codecs >>> codecs.decode("67414141414142644450557943363371417034344c7946777a664c7973385743714649373239715935523153723131506435614c4c36693477584b48765f41735845494a366f5253756a4658656a414d3637595134565a41764b6b67706a79716d77305f6a76464d314e464a754e4a52596d4b4f3933664b5a38316b6e6f7a5364317a4a7858383566515a4b50366354434a77465f7838414c4b334d454a754941794b6364384a30446e3570763865497954662d38534c6b642d6b424c59564d64574f577a78637831537843705f5059523847416930666f4177454c754c6a66786169666a774c4d5947553355696f55676c7a714652733d", "hex") b'gAAAAABdDPUyC63qAp44LyFwzfLys8WCqFI729qY5R1Sr11Pd5aLL6i4wXKHv_AsXEIJ6oRSujFXejAM67YQ4VZAvKkgpjyqmw0_jvFM1NFJuNJRYmKO93fKZ81knozSd1zJxX85fQZKP6cTCJwF_x8ALK3MEJuIAyKcd8J0Dn5pv8eIyTf-8SLkd-kBLYVMdWOWzxcx1SxCp_PYR8GAi0foAwELuLjfxaifjwLMYGU3UioUglzqFRs=' >>> Database and database server info: Name | keystone Owner | postgres Encoding | UTF8 Collate | en_US.UTF-8 Ctype | en_US.UTF-8 ii postgresql-9.5 9.5.17-0ubuntu0.16.04.1 amd64 object-relational SQL database, version 9.5 server ii postgresql-client-9.5 9.5.17-0ubuntu0.16.04.1 amd64 front-end programs for PostgreSQL 9.5 ii postgresql-client-common 173ubuntu0.2 all manager for multiple PostgreSQL client versions ii postgresql-common 173ubuntu0.2 all PostgreSQL database-cluster manager ii postgresql-contrib-9.5 9.5.17-0ubuntu0.16.04.1 amd64 additional facilities for PostgreSQL postgres@openstack-db01:~$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS" Keystone and associated oslo.db/sqlalchemy/psycopg information: ii python3-keystone 2:15.0.0-0ubuntu1~cloud0 all OpenStack identity service - Python 3 library ii python3-keystoneauth1 3.13.1-0ubuntu1~cloud0 all authentication library for OpenStack Identity - Python 3.x ii python3-keystoneclient 1:3.19.0-0ubuntu1~cloud0 all client library for the OpenStack Keystone API - Python 3.x ii python3-keystonemiddleware 6.0.0-0ubuntu1~cloud0 all Middleware for OpenStack Identity (Keystone) - Python 3.x ii python3-oslo.db 4.44.0-0ubuntu1~cloud0 all database connectivity to the different backends and helper utils - Python 3.x ii python3-sqlalchemy 1.2.15+ds1-1~cloud0 all SQL toolkit and Object Relational Mapper for Python 3 ii python3-sqlalchemy-ext 1.2.15+ds1-1~cloud0 amd64 SQL toolkit and Object Relational Mapper for Python3 - C extension ii python3-psycopg2 2.7.4-1 amd64 Python 3 module for PostgreSQL root@keystone01:/usr/lib/python3/dist-packages# cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS" Using the exact same database server, running the Rocky version of Keystone with Python2, we do not see this problem. Keystone is being installed using the Ubuntu Cloud Archive: https://wiki.ubuntu.com/OpenStack/CloudArchive To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1833739/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp