Re: [ceph-users] Missing bucket

[Yehuda Sadeh-Weinraub]

On Fri, Nov 13, 2015 at 12:53 PM, Łukasz Jagiełło
 wrote:
> Hi all,
>
> Recently I've noticed a problem with one of our buckets:
>
> I cannot list or stats on a bucket:
> #v+
> root@ceph-s1:~# radosgw-admin bucket stats --bucket=problematic_bucket
> error getting bucket stats ret=-22

That's EINVAL, not ENOENT. It could mean lot's of things, e.g.,
radosgw-admin version mismatch vs. version that osds are running. Try
to add --debug-rgw=20 --debug-ms=1 --log-to-stderr to maybe get a bit
more info about the source of this error.

> ➜  ~  s3cmd -c /etc/s3cmd/prod.cfg ls
> s3://problematic_bucket/images/e/e0/file.png
> ERROR: S3 error: None
> #v-
>
> ,but direct request for an object is working perfectly fine:
> #v+
> ➜  ~  curl -svo /dev/null
> http://ceph-s1/problematic_bucket/images/e/e0/file.png
> […]
> < HTTP/1.1 200 OK
> < Content-Type: image/png
> < Content-Length: 379906
> […]
> #v-
>
> Any solution how to fix it? We're still running ceph 0.67.11
>

You're really behind.


Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing bucket

[Yehuda Sadeh-Weinraub]

On Fri, Nov 13, 2015 at 1:37 PM, Łukasz Jagiełło
 wrote:
>> >> > Recently I've noticed a problem with one of our buckets:
>> >> >
>> >> > I cannot list or stats on a bucket:
>> >> > #v+
>> >> > root@ceph-s1:~# radosgw-admin bucket stats
>> >> > --bucket=problematic_bucket
>> >> > error getting bucket stats ret=-22
>> >>
>> >> That's EINVAL, not ENOENT. It could mean lot's of things, e.g.,
>> >> radosgw-admin version mismatch vs. version that osds are running. Try
>> >> to add --debug-rgw=20 --debug-ms=1 --log-to-stderr to maybe get a bit
>> >> more info about the source of this error.
>> >
>> >
>> > https://gist.github.com/ljagiello/06a4dd1f34a776e38f77
>> >
>> > Result of more verbose debug.
>> >
>> 2015-11-13 21:10:19.160420 7fd9f91be7c0 1 -- 10.8.68.78:0/1007616 -->
>> 10.8.42.35:6800/26514 -- osd_op(client.44897323.0:30
>> .dir.default.5457.9 [call rgw.bucket_list] 16.2f979b1a e172956) v4 --
>> ?+0 0x15f3740 con 0x15daa60
>> 2015-11-13 21:10:19.161058 7fd9ef8a7700 1 -- 10.8.68.78:0/1007616 <==
>> osd.12 10.8.42.35:6800/26514 6  osd_op_reply(30
>> .dir.default.5457.9 [call] ondisk = -22 (Invalid argument)) v4 
>> 118+0+0 (3885840820 0 0) 0x7fd9c8000d50 con 0x15daa60
>> error getting bucket stats ret=-22
>>
>> You can try taking a look at osd.12 logs. Any chance osd.12 and
>> radosgw-admin aren't running the same major version? (more likely
>> radosgw-admin running a newer version).
>
>
> From last 12h it's just deep-scrub info
> #v+
> 2015-11-13 08:23:00.690076 7fc4c62ee700  0 log [INF] : 15.621 deep-scrub ok
> #v-

This is unrelated.

>
> But yesterday there was a big rebalance and a host with that osd was
> rebuilding from scratch.
>
> We're running the same version (ceph, rados) across entire cluster just
> double check it.
>

what does 'radosgw-admin --version' return?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing bucket

[Łukasz Jagiełło]

>
> >> > Recently I've noticed a problem with one of our buckets:
> >> >
> >> > I cannot list or stats on a bucket:
> >> > #v+
> >> > root@ceph-s1:~# radosgw-admin bucket stats
> --bucket=problematic_bucket
> >> > error getting bucket stats ret=-22
> >>
> >> That's EINVAL, not ENOENT. It could mean lot's of things, e.g.,
> >> radosgw-admin version mismatch vs. version that osds are running. Try
> >> to add --debug-rgw=20 --debug-ms=1 --log-to-stderr to maybe get a bit
> >> more info about the source of this error.
> >
> >
> > https://gist.github.com/ljagiello/06a4dd1f34a776e38f77
> >
> > Result of more verbose debug.
> >
> 2015-11-13 21:10:19.160420 7fd9f91be7c0 1 -- 10.8.68.78:0/1007616 -->
> 10.8.42.35:6800/26514 -- osd_op(client.44897323.0:30
> .dir.default.5457.9 [call rgw.bucket_list] 16.2f979b1a e172956) v4 --
> ?+0 0x15f3740 con 0x15daa60
> 2015-11-13 21:10:19.161058 7fd9ef8a7700 1 -- 10.8.68.78:0/1007616 <==
> osd.12 10.8.42.35:6800/26514 6  osd_op_reply(30
> .dir.default.5457.9 [call] ondisk = -22 (Invalid argument)) v4 
> 118+0+0 (3885840820 0 0) 0x7fd9c8000d50 con 0x15daa60
> error getting bucket stats ret=-22
>
> You can try taking a look at osd.12 logs. Any chance osd.12 and
> radosgw-admin aren't running the same major version? (more likely
> radosgw-admin running a newer version).


>From last 12h it's just deep-scrub info
#v+
2015-11-13 08:23:00.690076 7fc4c62ee700  0 log [INF] : 15.621 deep-scrub ok
#v-

But yesterday there was a big rebalance and a host with that osd was
rebuilding from scratch.

We're running the same version (ceph, rados) across entire cluster just
double check it.

-- 
Łukasz Jagiełło
lukaszjagielloorg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing bucket

[Łukasz Jagiełło]

On Fri, Nov 13, 2015 at 1:07 PM, Yehuda Sadeh-Weinraub 
wrote:

> > Recently I've noticed a problem with one of our buckets:
> >
> > I cannot list or stats on a bucket:
> > #v+
> > root@ceph-s1:~# radosgw-admin bucket stats --bucket=problematic_bucket
> > error getting bucket stats ret=-22
>
> That's EINVAL, not ENOENT. It could mean lot's of things, e.g.,
> radosgw-admin version mismatch vs. version that osds are running. Try
> to add --debug-rgw=20 --debug-ms=1 --log-to-stderr to maybe get a bit
> more info about the source of this error.


https://gist.github.com/ljagiello/06a4dd1f34a776e38f77

Result of more verbose debug.


> You're really behind.
>

I know, we've got scheduled update for 2016 it's a big project to ensure
everything is fine.

-- 
Łukasz Jagiełło
lukaszjagielloorg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing bucket

[Łukasz Jagiełło]

On Fri, Nov 13, 2015 at 1:47 PM, Yehuda Sadeh-Weinraub 
wrote:

> >> >> > Recently I've noticed a problem with one of our buckets:
> >> >> >
> >> >> > I cannot list or stats on a bucket:
> >> >> > #v+
> >> >> > root@ceph-s1:~# radosgw-admin bucket stats
> >> >> > --bucket=problematic_bucket
> >> >> > error getting bucket stats ret=-22
> >> >>
> >> >> That's EINVAL, not ENOENT. It could mean lot's of things, e.g.,
> >> >> radosgw-admin version mismatch vs. version that osds are running. Try
> >> >> to add --debug-rgw=20 --debug-ms=1 --log-to-stderr to maybe get a bit
> >> >> more info about the source of this error.
> >> >
> >> >
> >> > https://gist.github.com/ljagiello/06a4dd1f34a776e38f77
> >> >
> >> > Result of more verbose debug.
> >> >
> >> 2015-11-13 21:10:19.160420 7fd9f91be7c0 1 -- 10.8.68.78:0/1007616 -->
> >> 10.8.42.35:6800/26514 -- osd_op(client.44897323.0:30
> >> .dir.default.5457.9 [call rgw.bucket_list] 16.2f979b1a e172956) v4 --
> >> ?+0 0x15f3740 con 0x15daa60
> >> 2015-11-13 21:10:19.161058 7fd9ef8a7700 1 -- 10.8.68.78:0/1007616 <==
> >> osd.12 10.8.42.35:6800/26514 6  osd_op_reply(30
> >> .dir.default.5457.9 [call] ondisk = -22 (Invalid argument)) v4 
> >> 118+0+0 (3885840820 0 0) 0x7fd9c8000d50 con 0x15daa60
> >> error getting bucket stats ret=-22
> >>
> >> You can try taking a look at osd.12 logs. Any chance osd.12 and
> >> radosgw-admin aren't running the same major version? (more likely
> >> radosgw-admin running a newer version).
> >
> >
> > From last 12h it's just deep-scrub info
> > #v+
> > 2015-11-13 08:23:00.690076 7fc4c62ee700  0 log [INF] : 15.621 deep-scrub
> ok
> > #v-
>
> This is unrelated.
>
> >
> > But yesterday there was a big rebalance and a host with that osd was
> > rebuilding from scratch.
> >
> > We're running the same version (ceph, rados) across entire cluster just
> > double check it.
> >
>
> what does 'radosgw-admin --version' return?
>

Everywhere the same:
ceph version 0.67.11 (bc8b67bef6309a32361be76cd11fb56b057ea9d2)

-- 
Łukasz Jagiełło
lukaszjagielloorg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Missing bucket

[Łukasz Jagiełło]

Hi all,

Recently I've noticed a problem with one of our buckets:

I cannot list or stats on a bucket:
#v+
root@ceph-s1:~# radosgw-admin bucket stats --bucket=problematic_bucket
error getting bucket stats ret=-22
➜  ~  s3cmd -c /etc/s3cmd/prod.cfg ls
s3://problematic_bucket/images/e/e0/file.png
ERROR: S3 error: None
#v-

,but direct request for an object is working perfectly fine:
#v+
➜  ~  curl -svo /dev/null
http://ceph-s1/problematic_bucket/images/e/e0/file.png
[…]
< HTTP/1.1 200 OK
< Content-Type: image/png
< Content-Length: 379906
[…]
#v-

Any solution how to fix it? We're still running ceph 0.67.11

Thanks,
-- 
Łukasz Jagiełło
lukaszjagielloorg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Missing bucket

[Yehuda Sadeh-Weinraub]

On Fri, Nov 13, 2015 at 1:14 PM, Łukasz Jagiełło
 wrote:
> On Fri, Nov 13, 2015 at 1:07 PM, Yehuda Sadeh-Weinraub 
> wrote:
>>
>> > Recently I've noticed a problem with one of our buckets:
>> >
>> > I cannot list or stats on a bucket:
>> > #v+
>> > root@ceph-s1:~# radosgw-admin bucket stats --bucket=problematic_bucket
>> > error getting bucket stats ret=-22
>>
>> That's EINVAL, not ENOENT. It could mean lot's of things, e.g.,
>> radosgw-admin version mismatch vs. version that osds are running. Try
>> to add --debug-rgw=20 --debug-ms=1 --log-to-stderr to maybe get a bit
>> more info about the source of this error.
>
>
> https://gist.github.com/ljagiello/06a4dd1f34a776e38f77
>
> Result of more verbose debug.
>
2015-11-13 21:10:19.160420 7fd9f91be7c0 1 -- 10.8.68.78:0/1007616 -->
10.8.42.35:6800/26514 -- osd_op(client.44897323.0:30
.dir.default.5457.9 [call rgw.bucket_list] 16.2f979b1a e172956) v4 --
?+0 0x15f3740 con 0x15daa60
2015-11-13 21:10:19.161058 7fd9ef8a7700 1 -- 10.8.68.78:0/1007616 <==
osd.12 10.8.42.35:6800/26514 6  osd_op_reply(30
.dir.default.5457.9 [call] ondisk = -22 (Invalid argument)) v4 
118+0+0 (3885840820 0 0) 0x7fd9c8000d50 con 0x15daa60
error getting bucket stats ret=-22

You can try taking a look at osd.12 logs. Any chance osd.12 and
radosgw-admin aren't running the same major version? (more likely
radosgw-admin running a newer version).

>>
>> You're really behind.
>
>
> I know, we've got scheduled update for 2016 it's a big project to ensure
> everything is fine.
>
> --
> Łukasz Jagiełło
> lukaszjagielloorg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com