[ceph-users] RadosGW not working after upgrade to Hammer

[Gerson Jamal]

Hi everyone,

I upgrade ceph from firefly to hammer and everything looks OK on upgrade
but after that RadosGW not working, I can list all buckets but i cant list
the objects inside the buckets, and I receive the following error:

format=json 400 Bad Request   []{"Code":"InvalidArgument"}

On Radosgw log I got the following error:

2017-06-17 01:37:25.325505 7f0108801700 10 ver=v1 first= req=
2017-06-17 01:37:25.325508 7f0108801700 10 s->object= s->bucket=
2017-06-17 01:37:25.325513 7f0108801700  2 req 21:0.49:swift:GET
/swift/v1/::getting op
2017-06-17 01:37:25.325516 7f0108801700  2 req 21:0.53:swift:GET
/swift/v1/:list_buckets:authorizing
2017-06-17 01:37:25.325529 7f0108801700 10 swift_user=sysmonitor:xx
2017-06-17 01:37:25.325541 7f0108801700 20 build_token
token=15007379736d6f6e69746f723a6c69676874686f7573652c7d7d41da54e3ba35bd45595a0df912
2017-06-17 01:37:25.325572 7f0108801700  2 req 21:0.000108:swift:GET
/swift/v1/:list_buckets:reading permissions
2017-06-17 01:37:25.325580 7f0108801700  2 req 21:0.000116:swift:GET
/swift/v1/:list_buckets:init op
2017-06-17 01:37:25.325582 7f0108801700  2 req 21:0.000119:swift:GET
/swift/v1/:list_buckets:verifying op mask
2017-06-17 01:37:25.325584 7f0108801700 20 required_mask= 1 user.op_mask=7
2017-06-17 01:37:25.325586 7f0108801700  2 req 21:0.000122:swift:GET
/swift/v1/:list_buckets:verifying op permissions
2017-06-17 01:37:25.325588 7f0108801700  2 req 21:0.000125:swift:GET
/swift/v1/:list_buckets:verifying op params
2017-06-17 01:37:25.325590 7f0108801700  2 req 21:0.000127:swift:GET
/swift/v1/:list_buckets:executing
2017-06-17 01:37:25.328258 7f0108801700 20 reading from
.rgw:.bucket.meta.CHECK_CEPH:default.4576.17572
2017-06-17 01:37:25.328284 7f0108801700 20 get_obj_state:
rctx=0x7f01087ff250 obj=.rgw:.bucket.meta.CHECK_CEPH:default.4576.17572
state=0x7f05641389c0 s->prefetch_data=0
2017-06-17 01:37:25.328294 7f0108801700 10 cache get:
name=.rgw+.bucket.meta.CHECK_CEPH:default.4576.17572 : hit
2017-06-17 01:37:25.328304 7f0108801700 20 get_obj_state: s->obj_tag was
set empty
2017-06-17 01:37:25.328308 7f0108801700 10 cache get:
name=.rgw+.bucket.meta.CHECK_CEPH:default.4576.17572 : hit
2017-06-17 01:37:25.330351 7f0108801700  0 ERROR: could not get stats for
buckets
2017-06-17 01:37:25.330378 7f0108801700 10 WARNING: failed on
rgw_get_user_buckets uid=sysmonitor
2017-06-17 01:37:25.330407 7f0108801700  2 req 21:0.004943:swift:GET
/swift/v1/:list_buckets:http status=400
2017-06-17 01:37:25.330412 7f0108801700  1 == req done
req=0x7f053023c0a0 http_status=400 ==
2017-06-17 01:37:25.330418 7f0108801700 20 process_request() returned -22
2017-06-17 01:37:28.470724 7f05837fe700  2
RGWDataChangesLog::ChangesRenewThread: start



Anyone can help me

-- 
Regards,

Gerson Razaque Jamal
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW not working after upgrade to Hammer

[Gerson Jamal]

Hi,

Anyone can found the solution for this issue.
I upgrade from Firefly do Hammer, and i'm facing with this problem.

Thanks in advance

On Mon, Jun 19, 2017 at 10:32 AM, Gerson Jamal 
wrote:

> Hi,
>
> Anyone can found the solution for this issue.
> I upgrade from Firefly do Hammer, and i'm facing with this problem.
>
> Thanks in advance
>
> --
> Regards
>
> Gerson Razaque Jamal
>



-- 
Cumprimentos

Gerson Razaque Jamal
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW not working after upgrade to Hammer

[James Page]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi Arnoud

On 26/05/15 16:53, Arnoud de Jonge wrote:
> Hi,
[...]
> 
> 2015-05-26 17:43:37.352569 7f0fce0ff840  0 ceph version 0.94.1
> (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process radosgw, pid
> 4259 2015-05-26 17:43:37.435921 7f0f8a4f2700  0 ERROR: can't get
> key: ret=-2 2015-05-26 17:43:37.435932 7f0f8a4f2700  0 ERROR:
> sync_all_users() returned ret=-2 2015-05-26 17:43:37.436179
> 7f0fce0ff840  0 framework: fastcgi 2015-05-26 17:43:37.436191
> 7f0fce0ff840  0 framework: civetweb 2015-05-26 17:43:37.436198
> 7f0fce0ff840  0 framework conf key: port, val: 7480 2015-05-26
> 17:43:37.436208 7f0fce0ff840  0 starting handler: civetweb 
> 2015-05-26 17:43:37.453013 7f0fce0ff840  0 starting handler:
> fastcgi 2015-05-26 17:43:41.347403 7f0d487a0700  1 == starting
> new request req=0x7f0d7c011590 = 2015-05-26 17:43:41.498851
> 7f0d487a0700  0 validated token: openstack:admin expires:
> 1432658621 2015-05-26 17:43:41.509877 7f0d487a0700  0 ERROR: could
> not get stats for buckets 2015-05-26 17:43:41.509943 7f0d487a0700
> 1 == req done req=0x7f0d7c011590 http_status=400 == 
> 2015-05-26 17:43:44.140730 7f0d897fa700  1 == starting new
> request req=0x7f0d7c0162e0 = 2015-05-26 17:43:44.285039
> 7f0d897fa700  0 validated token: openstack:admin expires:
> 1432658624 2015-05-26 17:43:44.290175 7f0d897fa700  0 ERROR: could
> not get stats for buckets 2015-05-26 17:43:44.290222 7f0d897fa700
> 1 == req done req=0x7f0d7c0162e0 http_status=400 == 
> 2015-05-26 17:43:46.972310 7f0d77fff700  1 == starting new
> request req=0x7f0d7c016270 = 2015-05-26 17:43:47.121784
> 7f0d77fff700  0 validated token: openstack:admin expires:
> 1432658626 2015-05-26 17:43:47.125191 7f0d77fff700  0 ERROR: could
> not get stats for buckets 2015-05-26 17:43:47.125241 7f0d77fff700
> 1 == req done req=0x7f0d7c016270 http_status=400 ==

I just debugged what I think is the same problem in one of our
clusters; the Gateway was working fine for a bit, but then started
throwing this error.

Turned out the back-end mon and osd daemons had not been restarted so
where still running firefly code - if you're running on Ubuntu, the
packaging won't restart daemons automatically - it has to be done
manually post upgrade (so you can co-ordinate it across your clusters).

Cheers

James

- -- 
James Page
Ubuntu and Debian Developer
james.p...@ubuntu.com
jamesp...@debian.org

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJV8qWSAAoJEL/srsug59jDHasP+weu5D7P5DR4zbRxdvGXw8DR
RWkI2oo4FSB5QYeVBR5NbCOBKQivUoDKs8wcldD7rRLBwNuxxkacrpoBviKHJkZF
XYya8ZGufIX4RTseF9F3qGoJnA5rxCrTPojcB8KGRlXMFutswnv5sgS11J3OlSZn
QG4bB/oW9Cmdw3slxc/Poe3UdxQxgTfyXIoj9eiUWNcCZtbElcggF+EFiN47EneG
mA8kRIZY7ofhl3Lr1AGvXffK/4tjgpeWmPnCPIZlzuigmxbTWvwBjDk2H3gt0C2f
l3vUi1fT2NRslR4v+4MJuI9JtxhKNCaX3QjU5vNVJkRV11nd2RYw+a+YQNBPqEbQ
TstCLDBg2rAnPHpPduqsb+tujx8+p7SHNcgMHPJpxkeZ96XPts0Qxhr8M3i++MlM
blZvRYyRf3KDsWRG7kq65msFGPW9H4eWP4gmCP02Uy3VJ1LtawpypAX0Elq+SC8a
wWRV4GyVEv1tpNyP7gSqBIGhHRthODXQFii6TYJNvN2YpRWiO1XTMUvooNaH/kg5
/P42QUifQ7K2XjVaB9yFQLH4TkQkYjiHY3ro/DxARpVtb41M3yJinCSJaHlgWArc
pDlJm4mz6lRVIo6evTJsqOLoe0cew/+m+atlfuucTM3+SVONSCay9H56wB/Z3/VG
ZTlm19gjHjp6Rpwi401i
=Nfh4
-END PGP SIGNATURE-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RadosGW not working after upgrade to Hammer

[Arnoud de Jonge]

Hi,

After upgrading our OpenStack cluster to Kilo and upgrading Ceph from Giant to 
Hammer, RadosGW stopped working. All other services using Ceph work fine. 
RadosGW is configured to use Keystone for authentication.

# swift list
Account GET failed: http://object.api.openstack.cyso.net/swift/v1?format=json 
400 Bad Request   {Code:InvalidArgument}

Funny thing is that when I do a Curl on that URL and remove the format=json 
part it does work.

root@controller1.eq1:~# curl -i 
'http://object.api.openstack.cyso.net/swift/v1?format=json' -X GET -H 
X-Auth-Token: X
HTTP/1.1 400 Bad Request
Date: Tue, 26 May 2015 15:49:17 GMT
Server: Apache/2.4.7 (Ubuntu)
Accept-Ranges: bytes
Content-Length: 26
Connection: close
Content-Type: application/json; charset=utf-8

{Code:InvalidArgument}
root@controller1.eq1:~# curl -i 'http://object.api.openstack.cyso.net/swift/v1' 
-X GET -H X-Auth-Token: XX
HTTP/1.1 200 OK
Date: Tue, 26 May 2015 15:49:36 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: Accept-Encoding
Connection: close
Transfer-Encoding: chunked
Content-Type: text/plain; charset=utf-8

testbucket


In the RadosGW log I see:

2015-05-26 17:43:37.352569 7f0fce0ff840  0 ceph version 0.94.1 
(e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process radosgw, pid 4259
2015-05-26 17:43:37.435921 7f0f8a4f2700  0 ERROR: can't get key: ret=-2
2015-05-26 17:43:37.435932 7f0f8a4f2700  0 ERROR: sync_all_users() returned 
ret=-2
2015-05-26 17:43:37.436179 7f0fce0ff840  0 framework: fastcgi
2015-05-26 17:43:37.436191 7f0fce0ff840  0 framework: civetweb
2015-05-26 17:43:37.436198 7f0fce0ff840  0 framework conf key: port, val: 7480
2015-05-26 17:43:37.436208 7f0fce0ff840  0 starting handler: civetweb
2015-05-26 17:43:37.453013 7f0fce0ff840  0 starting handler: fastcgi
2015-05-26 17:43:41.347403 7f0d487a0700  1 == starting new request 
req=0x7f0d7c011590 =
2015-05-26 17:43:41.498851 7f0d487a0700  0 validated token: openstack:admin 
expires: 1432658621
2015-05-26 17:43:41.509877 7f0d487a0700  0 ERROR: could not get stats for 
buckets
2015-05-26 17:43:41.509943 7f0d487a0700  1 == req done req=0x7f0d7c011590 
http_status=400 ==
2015-05-26 17:43:44.140730 7f0d897fa700  1 == starting new request 
req=0x7f0d7c0162e0 =
2015-05-26 17:43:44.285039 7f0d897fa700  0 validated token: openstack:admin 
expires: 1432658624
2015-05-26 17:43:44.290175 7f0d897fa700  0 ERROR: could not get stats for 
buckets
2015-05-26 17:43:44.290222 7f0d897fa700  1 == req done req=0x7f0d7c0162e0 
http_status=400 ==
2015-05-26 17:43:46.972310 7f0d77fff700  1 == starting new request 
req=0x7f0d7c016270 =
2015-05-26 17:43:47.121784 7f0d77fff700  0 validated token: openstack:admin 
expires: 1432658626
2015-05-26 17:43:47.125191 7f0d77fff700  0 ERROR: could not get stats for 
buckets
2015-05-26 17:43:47.125241 7f0d77fff700  1 == req done req=0x7f0d7c016270 
http_status=400 ==

Permissions haven't changed from the previous version:

client.radosgw.controller1
key: 
caps: [mon] allow rwx
caps: [osd] allow rwx

My ceph.conf:

[client.radosgw.controller1]
rgw keystone token cache size = 100
rgw keystone revocation interval = 3600
log file = /var/log/radosgw/client.radosgw.controller1.log
rgw keystone admin token = ----
rgw keystone accepted roles = s3, swift, admin, _member_
rgw s3 auth use keystone = true
host = controller1
keyring = /etc/ceph/ceph.client.radosgw.controller1.keyring
rgw keystone url = https://identity-admin.api.openstack.cyso.net:35357
nss db path = /var/lib/ceph/radosgw/nss
rgw print continue = false
rgw socket path = /var/run/ceph/ceph.radosgw.controller1.fastcgi.sock
rgw usage max user shards = 1
rgw usage log tick interval = 60
rgw usage log flush threshold = 1024
rgw usage max shards = 32
rgw enable usage log = true

Apache config:

VirtualHost 10.1.0.11:80
  ServerName object.api.openstack.cyso.net

  DocumentRoot /var/www

  Directory /var/www
Options +ExecCGI
AllowOverride All
Require all granted
SetHandler fastcgi-script
  /Directory

  ErrorLog /var/log/apache2/object_endpoint_error.log
  ServerSignature Off
  CustomLog /var/log/apache2/object_endpoint_access.log combined

  RewriteEngine On
  RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} 
[E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]


  ServerAlias *.object.api.openstack.cyso.net

  FastCgiExternalServer /var/www/s3gw.fcgi -socket 
/var/run/ceph/ceph.radosgw.controller1.fastcgi.sock
  AllowEncodedSlashes on
/VirtualHost

Any thoughts on what could be wrong?



Kind regards,

Arnoud de Jonge
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com