** Also affects: ec2-api Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1840647
Title: EC2 API Auth Failure Status in ec2-api: New Status in OpenStack Identity (keystone): New Bug description: When trying to validate user token through Keystone an inconsistent behavior is observed: The very first request for a given user to Keystone is successful, while the second and subsequent requests for the same user result in a Key Error in Keystone _decrypt_credential method and thus auth failure (See the attached traceback). The problem is observed by merely repeating the same aws-cli commands for the same user 2 or more times. After inserting some debug log messages in core.py the problem seem to happen in _get_credential method in keystone.credential.core: In first request to Keystone: Aug 15 20:14:13 ubuntu-vm6 devstack@keystone.service[22678]: #033[00;36mINFO keystone.credential.core [#033[01;36mNone req-14485eaa-4172-4074-ac28-aa22fea7ce32 #033[00;36mNone None#033[00;36m] #033[01;35m#033[00;36mcontents of credential after _get_credential are user_id=u'4ad247ca077743078bf6109ff5c57dc1', key_hash=u'716d67b1efbe8c7b46d13e6f4b5ccfbe095a3f73', encrypted_blob=u'gAAAAABdVZLCpdDXGjdHk3kglftCMUDnxOKWNuLR6_6ZJtVTFt7VvJ-Y7VzyfxO6JwDv88OQtSMlJzbG8hUcfFftqhH6l5jpx6eNXdhTpO6_ZF57xyey1BmFI8JYh-PWdJkYtfDsOWJzCyWACQaUXU9N2J_r83xEYx691ToCwGCrkL-NgfFX-hVy3dYl9_4-5HcCOR4RoK2VXjLdvuAjWdvWCAxiVs-kxARpT0jkoZP1dnw-zs3pgDE=', project_id=u'9be611818d0b4bac82ea0d468f91541b', type=u'ec2', id=u'11755be627c5e6f22f3a31062afbd631bbe0f3176701ea23e5284920ded89e76'#033[00m#033[00m In second request to Keystone: Aug 15 20:14:43 ubuntu-vm6 devstack@keystone.service[22678]: #033[00;36mINFO keystone.credential.core [#033[01;36mNone req-4e4cc471-f9c7-461b-9245-65f8ac8c843b #033[00;36mNone None#033[00;36m] #033[01;35m#033[00;36mcontents of credential after _get_credential are user_id=u'4ad247ca077743078bf6109ff5c57dc1', blob=u'{"access": "09b2bb6d93ff47098198ceaf060baa8a", "secret": "d25c0c2c4ef542a29f84e7924ec89773", "trust_id": null}', project_id=u'9be611818d0b4bac82ea0d468f91541b', type=u'ec2', id=u'11755be627c5e6f22f3a31062afbd631bbe0f3176701ea23e5284920ded89e76'#033[00m#033[00m => Basically _get_credential method returns normal encrypted credential (encrypted_blob) for the first call, and an unencrypted credential (blob) for the second and subsequent calls by the same credential_id after 30 seconds. After that, the already decrypted credential incorrectly returned by _get_credential is passed to _decrypt_credential which fails because there is no 'encrypted_blob' key in the dictionary. The stack trace for the second and subsequent calls led to a package named 'dogpile' and some cache-related methods inside it. The problem seem to happen due to in-place modification of the credential dictionary in _decrypt_credential method. During the first request to Keystone the normal encrypted credential dictionary is returned by _get_credential method and the returned result is cached. However, after that the credential dictionary is modified in-place by _decrypt_credential method. For the second and subsequent calls to the _get_credential for the same user it tries to return the result from cache. But the cached result has been modified by _decrypt_credential. So, the already decrypted during first request credential is returned and passed again to _decrypt_credential breaking it. The solution is to modify a copy of the credential dictionary inside _decrypt_credential method, not the original credential in-place (the same approach as in _encrypt_credential where a copy is modified). This way the encrypted value of the credential is cached and it does not gets modified afterwards. So the normal encrypted credential is returned from cache and passed to _decrypt_credential again and again even for the same user which fixes the problem. To manage notifications about this bug go to: https://bugs.launchpad.net/ec2-api/+bug/1840647/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp