Re: [Openstack-operators] Keystone upgrade issues

2016-08-25 Thread Jonathan Proulx 
On Thu, Aug 25, 2016 at 10:55:51AM -0400, Jonathan Proulx wrote:
:Hi All,
:
:working on testing our Kilo-> Mitaka keystone upgrade, and I've
:clearly missied something I need to do or undo.


D'Oh  why is it that public postings always lead me do discover my own
idiocy soon after (no matter howlong I've been staring at the
problem).

After most recent prodcution DB load into testing I forgot to switch
the endpoints to be in the test cluster.

mysql> update keystone.endpoint set 
url=replace(url,'production-endpoint','test-endpoint');

Obviously getting tokens from test and presenting them to production
endoints won't work :)

/me re-caffeinates

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Keystone upgrade issues

2016-08-25 Thread Matt Fischer 
Jonathan,

Are you using caching for tokens (not the middleware cache but keystone
cache)? There's a bug in the caching so that when it tries to read the
cache and unpack the token its missing some fields. It's been fixed and
backported but may not be in your packages:
https://bugs.launchpad.net/keystone/+bug/1592169

Until that is fixed you can just flush memcache in a loop during the
upgrade.

Also - heads-up that you will have this issue if you use caching in Mitaka
that will lead to intermittent API call failures -
https://bugs.launchpad.net/keystone/+bug/1600394

And finally, this Cinder bug will show up once you're on Keystone Mitaka:
https://bugs.launchpad.net/cinder/+bug/1597045



On Thu, Aug 25, 2016 at 10:55 AM, Jonathan Proulx  wrote:

> Hi All,
>
> working on testing our Kilo-> Mitaka keystone upgrade, and I've
> clearly missied something I need to do or undo.
>
> After DB migration and the edits I belive are required to paste and
> conf files I can get tokens (using password auth) but it won't seem to
> accept them (for example with an admin user I get 'action requires
> authorization' errors when trying to show users )
>
> Current setup is pretty simple and past upgrades of keystone have been
> super easy, so other that reread and recheck not sure where I should
> focus my attention.
>
> using:
> fernet tokens
> mysql local users
> apache/wsgi
> Ubuntu 14.04 cloud archive packages
>
> This is what I can see with --debug the client (both
> python-keystoneclient and python-openstackclient) after getting the
> initial auth token through password exchange:
>
> REQ: curl -g -i -X GET https://controller:35358/v2.0/users -H
> "User-Agent: python-keystoneclient" -H "Accept: application/json" -H
> "X-Auth-Token: {SHA1}"
> "GET /v2.0/users HTTP/1.1" 401 114
> RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Alive: timeout=5
> Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Thu, 25 Aug 2016
> 14:41:26 GMT WWW-Authenticate: Keystone uri="https://nimbus.csail.mit.
> edu:35358" Content-Type: application/json X-Distribution: Ubuntu
> RESP BODY: {"error": {"message": "The request you have made requires
> authentication.", "code": 401, "title": "Unauthorized"}}
>
> (v3 requests are similar modulo API differences)
>
> Keysote.log in debug mode issues a couple deprecation warnings but no
> errors (http://pastebin.com/WriB6u6i).  Not this log is for the same
> event but response is UTC where log is local time (-0400)
>
> Any pointer to where I should focus my investigations woudl be most
> welcome :)
>
> Thanks,
> -Jon
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Keystone upgrade issues

2016-08-25 Thread Jonathan Proulx 
Hi All,

working on testing our Kilo-> Mitaka keystone upgrade, and I've
clearly missied something I need to do or undo.

After DB migration and the edits I belive are required to paste and
conf files I can get tokens (using password auth) but it won't seem to
accept them (for example with an admin user I get 'action requires
authorization' errors when trying to show users )

Current setup is pretty simple and past upgrades of keystone have been
super easy, so other that reread and recheck not sure where I should
focus my attention.

using: 
fernet tokens 
mysql local users
apache/wsgi
Ubuntu 14.04 cloud archive packages 

This is what I can see with --debug the client (both
python-keystoneclient and python-openstackclient) after getting the
initial auth token through password exchange:

REQ: curl -g -i -X GET https://controller:35358/v2.0/users -H "User-Agent: 
python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: 
{SHA1}"
"GET /v2.0/users HTTP/1.1" 401 114
RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Alive: timeout=5 
Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Thu, 25 Aug 2016 
14:41:26 GMT WWW-Authenticate: Keystone 
uri="https://nimbus.csail.mit.edu:35358; Content-Type: application/json 
X-Distribution: Ubuntu 
RESP BODY: {"error": {"message": "The request you have made requires 
authentication.", "code": 401, "title": "Unauthorized"}}

(v3 requests are similar modulo API differences)

Keysote.log in debug mode issues a couple deprecation warnings but no
errors (http://pastebin.com/WriB6u6i).  Not this log is for the same
event but response is UTC where log is local time (-0400)

Any pointer to where I should focus my investigations woudl be most
welcome :)

Thanks,
-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators